Like in Stalker where the mutant dogs will turn tail and flee if they take too much damage or if you kill enough of their pack members. Red Dead Redemption's animals also ran away if a fight wasn't going their way.
Actually, Rockstar games are pretty good with this sort of stuff in general. I'm pretty sure you could shoot guns of of people's hands in RDR to make them put their hands up, or cause a fatal gunshot wound that would make them crawl around on their belly and call for help. Both GTA 4 and 5's enemies have injury states where they will take potshots at you with a pistol while bleeding on the ground or just passively clutch their wounds until they die.
I guess it wouldn't work in arcadey or linear games where the point is to kill everything on screen, but for anything more open-ended that tries to go for something approaching realism it'd be nice if the enemies you faced felt more alive and/or showed some basic survival instincts.
I give players XP for resolving the encounter. Doesn't matter how they do it - Sneaking by, talking their way through, fighting, or one of the absolutely bizarre take a third option things players come up with. As long as they figure out a way to "win" (and usually if they "lose") they get XP and usually some amount of treasure out of it. Something I really dislike about older D&Ds and Pathfinder is how tightly player progression is tied to finding treasure and gold, as it can get in the way of narrative.