Lots of thoughts here. One is that over the course of our lives we encounter so many stories that they need to have variety, and Tolstoy's point makes pure heroes less appealing: "Happy families are all alike; every unhappy family is unhappy in its own way." Heroes and conflicts in children's stories are simple, and get more complex in stories for teens and adults. This is not just about maturity and exposure to the complexities of life and wanting to grapple with real dilemmas, it's also about not reading the hundredth identical plot.
Milton's Lucifer was also my first thought reading this, but I'm not sure I agree with your take. I think the point, for me, is that he makes us question whether is actually is the villain, at all. The persuasion element is, I think, an artifact of the story being told in a cultural context where there's an overwhelming presumption that he is the villain. The ancient Greeks had a different context, and had no problem writing complex and flawed and often doomed heroes who fought against fate and gods without their writers/composers ever thinking they needed to classify them as villains or anti-heroes, just larger-than-life people.
Perhaps it's just my American upbringing, but I think I want to live in a world where agents can get what they want, even with the world set against them, if only they are clever and persistent enough.
I'm American too, and I don't want that. At least not in general. I do share the stereotypical American distrust of rigid traditions and institutions to a substantial degree. I want agents-in-general to get much of what they want, but when the world is set against them? Depends case-by-case on why the world is set against them, and on why the agents have the goals they have. Voldemort was a persistent and clever (by Harry Potter standards) agent as much as Gandhi was. I can understand how each arrived at their goals and methods, and why their respective worlds were set against them, but that doesn't mean I want them both to get what they want. Interpolate between extremes however you like.
You'll like A Practical Guide to Evil.
One of my favorite monologues from there expresses an idea similar to yours:
“This once, just this once, I want us to win.
“To spit in the eyes of the Hashmallim. To trample the pride of all those glorious, righteous princes. To scatter their wizards and make their oracles liars. Just to prove that it can be done.”
“So that five hundred years from now, a band of heroes shiver in the dark of night. Because they know that no matter how powerful their sword or righteous their cause, there was once a time it wasn’t enough. That even victories ordained by the Heavens can broken by the will of men.”
Would prankster archetype characters like Harpo Marx, Daffy Duck, or Woody Woodpecker be considered Agentic? Or is their lack of discernable goal and sheer hedonic anarchism not agentic?
I just wonder because these characters appear to be extremely appealing to audiences too.
Why do we root for the antiheroes?
Walter White, Light Yagami, Lucifer Morningstar, Doctor Frankenstein... It seems to be natural to root for the bad guys, and perhaps even more natural for rationalists.
“I must confess I have always had some sympathy with villains. Heroism makes fine entertainment but sooner or later someone has to get things done.” - Bayaz, Joe Abercrombie's First Law Trilogy.
I think this is the reason, at least for me. Villains (particularly the most beloved villains) are agentic. They know what they want and they attempt to obtain it through their own actions. Sometimes those actions are even reasonable.
This is notable because it's usually untrue of heroes. Heroes tend to be called to adventure, accepting a goal thrust upon them by circumstances or a bearded stranger or both. Then they pursue it, but often alongside a handful of other eclectic goals which cause internal conflict when they can't all be satisfied (getting distracted rescuing some civilians when the fate of the world is at stake). There's also something fairly un-agentic about following an absolute "honor" code, which is much more common among heroes than villains. Finally, heroes have teams of friends instead of minions, and are often forced to go out of their way to help their friends. A hero's friendships might even change his terminal values!
So, why do we often find villains so fascinating?
I think it's the same reason that rationalists don't run the world.
Being highly agentic just isn't easy in real life, because it's inherently individualistic. Individuals tend to be outcompeted by large organizations. Large organizations are approximately agentic, but the people making up a large organization don't necessarily have to be. In fact, it might be better if they aren't!
The military seems to optimize much more strongly for loyalty and discipline than agency - in fact, an army of agents with diverse goals seems tricky to control, since most will instrumentally value their own survival (though an army of agents with one cohesive goal may be even more formidable).
In industry, principles of comparative advantage can imply that it is best for each employee to become highly specialized, which seems opposed to developing their agency. A more capable agent might generalize (building the type of skill set that a startup found might need, including "executive nature" as in Competent Elites). Though agentic employees can create a lot of value, I think it is typically more cost effective to create additional cogs in the machine than it is to increase agency.
I think this is also why politicians are not very clever (epistemic status: I don't know any politicians). The things we value in a leader include costly trustworthiness signals (see https://www.elephantinthebrain.com/). In fact, the risk of an actively corrupt leader may be great enough that some would prefer a candidate with actively irrational beliefs (say, expecting moral judgement in the afterlife).
I would almost go so far as to say that the idea of becoming a Machiavellian scheming mastermind and changing the world (for better or worse) through sheer cleverness is a childish fantasy. Maybe that's not a bad thing; children might be naive, but at least they aren't bitter.
Perhaps it's just my American upbringing, but I think I want to live in a world where agents can get what they want, even with the world set against them, if only they are clever and persistent enough. So when I read about another despicable villain sacrificing it all and throwing the natural order out of balance in the name of cursed immortality or to avenge their family or to grasp untold riches, I can't help but root for them a little.
In a way, Milton's Lucifer is the ultimate Byronic hero: his doomed struggle is against the very idea that everything is already predetermined by God. He's like a meta-agent; his goal is agency. And he is notorious for persuading the reader to root for him.