Like in Stalker where the mutant dogs will turn tail and flee if they take too much damage or if you kill enough of their pack members. Red Dead Redemption's animals also ran away if a fight wasn't going their way.
Actually, Rockstar games are pretty good with this sort of stuff in general. I'm pretty sure you could shoot guns of of people's hands in RDR to make them put their hands up, or cause a fatal gunshot wound that would make them crawl around on their belly and call for help. Both GTA 4 and 5's enemies have injury states where they will take potshots at you with a pistol while bleeding on the ground or just passively clutch their wounds until they die.
I guess it wouldn't work in arcadey or linear games where the point is to kill everything on screen, but for anything more open-ended that tries to go for something approaching realism it'd be nice if the enemies you faced felt more alive and/or showed some basic survival instincts.
I think it could be done with a modifier to reduce experience and/or affect drop rates of equipment. Like you'd get armor if you kill the guy or he gives you the location of a treasure in another location if you spare him.
I tried to spare bandits in Skyrim before reading that their surrender was temporary. Like I get having enemies do false surrenders, but I'd like a little bit of a survival instinct.
Or the option to show humanity to a mook
I always prefer the systems that incentivize players to do the right thing (yeah corny I know). I think the best way (depending if the genre allows it) is to have enemy combatants not drop anything when they surrender, but their faction grants you some favor, or something that's otherwise hard to get. It might be a little hard to work into the story if the enemy faction is supposed to be at complete odds with the player, or if the player has no reason to sympathize with the enemies, but I think it can work in a lot of games.
Maybe hostile factions could reciprocate mercy, players might not care at first but I bet that changes when they get captured and an enemy lieutenant goes down the line, identifying the players and deciding who gets what treatment
Right, or killing retreating units punishes the player by making enemies more aggressive.
I think that's a thing in Mount and Blade. I'm not 100% but I think releasing a noble or one of their allies or family members, or just having a reputation that your enemy admires, can result in them releasing you after you're defeated in battle.
Your comment on giving you treasure was reminding me of a situation that was harder and changed the end state of the game to do that but gave you more for it. That game was monster hunter world, that capturing them gave you more resources but... man that was not better for the monsters man, they run away from you!
IIRC capturing monsters in MHW is strictly better than killing them because it reduces the HP you have to go through by ~20% and you get better rewards. The only hard part is figuring out they're capturable before you accidentally kill them.