In working on Quintessence, I've been trying to think of situations, either realistic or cinematic, that other tabletop games don't model well, if at all. So when Kainenchen and I watched Stardust the other day, I was struck by the many times that a character sets a blade at another character's throat, and the latter is subdued. This is completely reasonable in film, and in real life, right?
If you've seen a system for this kind of thing in a tabletop game, or even text that calls it out as an option, I'm interested to hear it. I can't think of one off-hand, so I came up with this. It is particularly suitable for cinematic duels, capturing guards by stealth that you mean to interrogate, and forcing the last opponent in a battle to surrender.
Subdue by Threat
After rolling damage and applying any mitigation with a melee attack or a ranged attack within 30 feet, if the attack deals enough damage to reduce the defender below 0 hit points, the attacker has the option to place her blade at the defender's throat, heart, or other narratively-appropriate vitals. The damage is not dealt at that time, but the attacker can use a Coup de Grace/Killing Blow as a reaction, without a new attack or damage roll, in response to any action, or as an immediate interrupt triggered by the defender taking a move action. In this specific case, the attacker should be allowed to use reactions and immediate interrupts on her own turn, even if the rules otherwise forbid this. As usual, the attacker can only take reactions or interrupts in response to things she can perceive.
It's up to each game table to determine what kinds of actions are perceptible - that's a bit beyond the scope of this document. For something like 4e D&D, I'd suggest, that it's an interesting detail if a Second Wind isn't perceptible, allowing the character to recover enough to knock aside the attacker's weapon. Also, creatures with regeneration aren't likely to remain Subdued by Threat for long, unless the held damage is truly massive. Dramatic reversals are a good thing, right?
To model the moment in Stardust where Tristan reveals that he has a knife against Septimus's belly even as Septimus has a blade to his throat, I would point to the Sleight of Hand skill, which from the start is designed to avoid perception, or Stealth (same idea). Of course, this doesn't do much if the attacker still has a pile of hit points left and the defender can't do enough damage to Subdue by Threat.
As with any hostage-like situation, the defender's allies (if any) will want to intervene. This is where I'm running into some complication, but let's see if I can work it out to be something simple. The ways I see a character wanting to intervene are:
moving the attacker away from the defender
moving the defender away from the attacker
using Subdue by Threat against the attacker
forcing the attacker's weapon away from the defender (this is the hard part, design-wise)
altering the attacker's goals (natural persuasion or charm person), or otherwise incapacitating her by magic
restoring enough hit points to the defender that he is no longer subject to the Subdue by Threat
Moving the attacker or defender follows the normal rules for forced movement. Anything that allows the defender to take a move action triggers the attacker's immediate interrupt, as noted above.
If both the attacker and defender are subject to Subdue by Threat, the situation becomes a standoff, and either a whole lot of people will die as part of a chain of reactions, or they will negotiate some other settlement. This situation doesn't particularly need additional rules, except to note that it returns to a standard Subdue by Threat if any participant who was Subdued by Threat escapes the threat.
Forcing the attacker's weapon away from the threatening position, without the attacker getting to inflict damage on the defender as a reaction, is a question of pure speed. My current thought on that is to have the intervening character make an attack against the attacker (in 4e, having the intervening character's attack target Reflex is thematically appropriate). On a hit, the intervening character and the attacker roll Initiative (this does not modify their place in the initiative order; it's just the existing way withing the rules to test reaction speed). If the intervening character wins, the Subdue by Threat is disrupted; if the attacker wins, she can take her reaction to Coup de Grace the defender normally. The intervening character's attack does not deal damage.
Altering the attacker's goals doesn't require further rules, as this should be covered by the social interaction rules or the rules for magical mind control; I might suggest that the attacker receive a bonus of some kind for holding the very clear upper hand in the situation.
Restoring hit points to the defender varies by rules set. If ranged healing is the most common kind (4e), it becomes a bit too easy to get people out of the Subdue by Threat situation, and it might be advisable to use an Initiative check (as described above for intervention) to see if the healing spell takes effect before the attacker's Coup de Grace.
Explanatory Digression: This is the benefit of the Coup de Grace rules of D&D: using the bleedout rules alone for being reduced to 0 hit points renders such situations toothless to players, unless the attacker's held damage is enough to not only render the defender unconscious but also kill him outright. This sometimes comes up, but it is not common, and the effects of making it more common have undesirable repercussions throughout the rest of the system.
If I tested the Subdue by Threat rule and found that it worked pretty well, I would look into adding handling for spellcasters to Subdue by (Spell) Threat also. The complicated part, in the fiction, is that the attacker needs to the defender to see the threat that is present, but this doesn't support most common visualizations of casting a fireball or whatever. Also, characters that are not Subdued still need to take damage... so the initial approach would be to declare that area-effect spells and weapon attacks cannot Subdue by Threat. The whole fiction of this idea is that the attack doesn't quite happen, but the attacker makes it clear that it can, at any moment. I would use a Concentration check (in systems that have such a thing) or an Arcana check (again, where appropriate) as a central mechanic for contests to intervene.
Some groups prefer the simulation of attackers accidentally killing defenders they meant to keep alive (as this is also common in the source fiction). That's fine; this rule isn't for you. For my part, I think it's more interesting if the player can make some choices about the level of the character's competence and control. If you like the rule but want to retain the possibility of murdering poor Mercutio, then add in a Dexterity check against a mid-range difficulty immediately after the damage roll; on a failure the weapon slips and deals its normal damage (rather than an immediate Coup de Grace). You can get still more obscure and detailed with modifiers to the roll if you like - for example, if you're wielding a weapon with Ego, there's no telling what might happen.