Crown of madness is a spell with excellent theme and, in my view, underwhelming mechanics. In this post, I discuss what I see as the problems, propose an alternate implementation, and explain my reasoning. I have every expectation that this spell has its hardcore supporters, or perhaps just those who oppose any change to the mechanics on principle. But, well, that's the internet for you - it's pretty clear that the community as a whole is on my side here. I am open to the possibility that I'm wrong about this spell, though.
Crown of Madness
This gives us some bases for comparison. Other 2nd-level enchantments include enthrall, hold person, and suggestion. For further comparison, charm person, and Tasha's hideous laughter are 1st level, and there are no 3rd-level enchantment spells in the Player's Handbook. Crown of madness appears on the bard, sorcerer, warlock, and wizard spell lists.
Casting Time: 1 action Range: 120 feet Components: V, S Duration: Concentration, up to 1 minute
This part of the spell writeup is fairly standard. Hold person and suggestion require Concentration; enthrall does not.
One humanoid of your choice that you can see within range must succeed on a Wisdom saving throw or become charmed by you for the duration. While the target is charmed in this way, a twisted crown of jagged iron appears on its head, and a madness glows in its eyes.
Limiting this to humanoid targets is pretty standard for low-level enchantments, such as charm person and hold person. Play an enchanter if there's going to be a lot of social interaction and politics with other humanoids - skip it if you're mostly fighting undead.
An initial Wisdom saving throw to resist the effect is also pretty normal for low-end effects, though at higher levels we see a few spells that only grant saves on later rounds.
The charmed condition, as a refresher:
A charmed creature can't attack the charmer or target the charmer with harmful abilities or magical effects. (This could be a huge benefit.)
The charmer has advantage on any ability check to interact socially with the creature.
Immunity to the charmed condition obviously carries immunity to this spell, and anything that breaks the charmed condition ends this spell.
The image of the twisted crown and the eyes glowing with madness carries the spell's theme in a big way, but it also serves as a big, obvious sign to the target's allies that something isn't right and maybe you want to avoid that dude. This has major tactical implications.
Notable absence: You or your allies dealing damage to the target cannot break this charmed condition. This is very unusual.
The charmed target must use its action before moving on each of its turns to make a melee attack against a creature other than itself that you mentally choose. The target can act normally on its turn if you choose no creature or if none are within reach.
Forcing the creature to use its action is a huge deal. Hold person is still more effective at pure action denial, and it paralyzes the target to make them easy to stab to death.
Forcing the creature to attack "a creature other than itself that you mentally choose" is the crux of the spell's theme. It's not only action denial, but some expectation of damage output.
It has to be a melee attack. This... is a bummer.
I'm surprised I haven't seen more discussion of the point that one of the available solutions to this spell for non-monks, from the target's perspective, is to sheathe all weapons and just use mostly-ineffective unarmed strikes.
The creature can do what it wants, within the constraints of the charmed condition, if you choose no creature or none are within reach. It still can't attack or cast a spell against the charmer, so it can't take direct steps to break your Concentration.
The "none are within reach" part is a problem here, because the creature still controls its own movement on a turn.
On your subsequent turns, you must use your action to maintain control over the target, or the spell ends. Also, the target can make a Wisdom saving throw at the end of each of its turns. On a success, the spell ends.
This is where things really fall apart for this spell. Using the caster's action on subsequent turns is a huge cost. It's eating up 100% of the spellcaster's expected damage output, unless they can deal damage with a bonus action - something that is an option, if a costly one, for sorcerers with Quickened metamagic. Not so much for bards, warlocks, or wizards.
Making the spell end, rather than just giving the target a round of doing whatever it wants, is pretty punitive to the caster.
A fresh Wisdom saving throw every round is normal.
Oh, and there's no At Higher Levels option here, so the spell doesn't have the flexibility of some other options. By comparison, hold person allows additional targets if you spend higher-level spell slots. Mass paralysis is amazing. You could duplicate this spell as a 10th-level enchanter using Split Enchantment, or as a sorcerer with Twinned Spell.
The problem with the spell is that if both sides play "optimally" (but the target keeps failing the Wisdom save), the outcome is boring, as the target makes one attack and then moves at least 5 feet from its allies. The allies can likewise help with this by moving away from the target. For the spell to do what its theme suggests - making an enemy target go on a rampage through allies - the target has to choose to stay close to allies, and there's nothing in the description to suggest that they do that. If you could possibly cast hold person instead, you should almost always do that - though conversations about the spell are full of contrived applications.
Crown of Madness
Casting Time: 1 action
Range: 120 feet Components: V, S Duration: Concentration, up to 1 minute
One creature that you can see within range must succeed on a Wisdom saving throw or become charmed by you for the duration. While the target is charmed in this way, a twisted crown of jagged iron appears on its head, and a madness glows in its eyes.
The charmed target must choose to either spend its reaction to make a weapon attack against a creature other than itself of your choosing, or suffer 2d10 psychic damage. If it cannot make an attack against an ally for any reason, such as not having a reaction available to spend, it automatically suffers 2d10 psychic damage. Most humanoids cannot use an unarmed strike for this purpose, though monks and creatures with claws, a bite attack, or the like may do so.
At the end of each of its turn, the target can make a Wisdom saving throw. On a success, the spell ends.
At Higher Levels. When you cast this spell using a spell slot of 3rd level or higher, the damage dealt by choosing not to attack an ally increases by 1d10 for each slot level above 2nd.
One of my core goals is to give this spell a very different application than hold person, so it involves a little bit of action denial (the reaction), but is not mostly about action denial. Hold person is the top of the class when it comes to both action denial and imposing weakness upon the target.
My other core goal was for the caster to present the target with an interesting, difficult choice, where both choices are reasonably good for the caster. The target can choose to go along with the crown's violent impulse by attacking an ally, or can choose to push back... and take damage. (To me, this is what the "jagged" iron symbolizes.) Ugly dilemmas are right at the core of Powered by the Apocalypse move creation, and it's one of my favorite things about the various PBTA games I've read.
If you don't specifically need your reaction for something, attacking an ally is a pretty good idea - you might miss, after all, so that no one takes damage. You're playing the odds, at least. Or you could suck up 2d10 psychic damage, which is the kind of noble self-sacrifice that will certainly appeal to some PCs targeted with this spell.
This spell is also excellent in a one-on-one situation, where the caster isn't a legal target for the reaction attack, whereas crown of madness is simple action denial in a one-on-one scene. I think that in general, it's in-theme for enchanters to be at their most terrifying when you're both alone.
Cutting out the continuing drain on the caster's actions is a huge deal. The target gets their action back, but so does the caster - seems fair to me.
It's particularly important to me that this spell not require a highly complex tactical understanding on the caster's part, because spells in general, and low-level spells in particular, shouldn't need deep system mastery. They should do what they seem like they're going to do, and the original version of this spell definitely misses the mark on that for me.
I could get behind a creature pushing through the action denial of hold person by taking a pile of psychic damage, I think. That seems very cinematic to me - maybe it should be one of the benefits of a feat? It would have to be pretty godawful amounts of damage, though, because of the horrors a spellcaster can wreak with a single action.
I tried to implement a similar effect in my Mage: the Awakening chronicle some four years ago. It didn't work in that specific case because of nWoD's single-action economy - without a concept of something other than your main action on a turn, I didn't have a good implementation path, and the players saw that immediately. No idea is ever wasted, though.