In the town hall in Siena, there is a set of frescos by Lorenzetti called the Allegory of Good and Bad Government. It was painted between 1338 and 1339 on the walls of the Sala dei Nove, where Siena’s magistrates held council.

One of the long walls shows the Effects of good government. The harvest is ready in the fields and vineyards beyond the city walls, a farmer drives his pig to town, the city is clean and elegant, and citizens are dancing in the streets to celebrate a wedding.

One of the dancers is wearing a dress embroidered with dragonflies.

When the councilman sat to make their decisions, that was what they were looking at and hoping for.

On the opposite wall is a fresco showing the Effects of bad government, but I think they get enough airtime as is, so I’m not going to go into them here.

Inbetween is a fresco showing an Allegory of good government - personifications of the virtues that are needed to govern well. There’s a lot going on in this fresco, but a main part of the picture shows the city of Siena (a solemn, white-haired man) flanked by six female figures. Their names are given in Latin: Pax, Fortitudo, Prudentia, Magnanimitas, Temperantia, and Iustitia. In English, they are peace, fortitude, prudence, magnanimity, temperance and justice.

On the other side of the picture, there’s a second figure of justice, this time balancing the scales that wisdom is holding above her head.

Many Italian city states commissioned similar civic frescoes around this time, but in the European context these paintings were a bit unusual in having a secular rather than a religious subject. Here as with art and science, the city states were ahead of the curve: northern Italy and Flanders urbanised much sooner than other parts of Europe, and their interest in secular government is an early foreshadowing of the great political theorists in the centuries that followed. Secularisation and bureaucratisation in their full forms were alien to fourteenth century Siena and came about gradually and several centuries later. But we can see the seeds of what was to come in paintings like this.

We don’t live in city states anymore - we live in their more powerful successors, nation states. But nation states are also a historical rather than a natural phenomenon. AI could fundamentally change the way they work, or even sweep them away, by automating the thing that nation states are built on: the work of human hands. All governments throughout history have needed some kind cooperation from the governed to function. Even oppressive states rely on bureaucrats and loyalists to get stuff done. But when fully automated government becomes possible, this fundamental dynamic could wither, and it’s not clear what will happen then.

This shift will be a lot faster than the transition from religious to secular government or city states to nations, and will probably be more profound.

A copy of the Effects of Good Government hung in my parents’ bedroom throughout my childhood; now the same blueing copy hangs around my kitchen table. I spend most of my day thinking about what effects AI will have, and how to protect the things I care about through the transition - art, food on the table, the children sitting around it, and the kind of government that can keep all of these safe.

I don’t know what the answer is.

But I do think that we need more positive visions for what good government would look like in a world with extremely powerful AI. That world is coming towards us, and I think we need to start imagining in more detail not just how it could go wrong, but concretely and positively how it could go right.

A post-AGI allegory

Here’s my own beginning, taking the Allegory of Good Government as a jumping off point.

These are musings only: we can only know a little about the future, and I don't know much of that little. I’m less trying to put on an ‘I’m an AI strategy researcher’ hat, and more trying to articulate some features of post-AGI government that seem maybe-good to me, as a tiny part of the human family. And for now, I’m going to talk at a high level about what good would look like, rather than getting concrete about how to get there.

First, the virtues, the pillars I think post-AGI government should stand on.

The virtues that feel most fundamental to me are wisdom and kindness.

• Wisdom today is a thing that lives in some humans and some bits of our traditions and culture. It’s deep, but it’s also rare. I’m excited for AI that uncovers these deep seams of human wisdom, augments them with wisdom beyond our current knowing, and makes it accessible to more people.
• In a world of human work, there are instrumental reasons for governments to treat their citizens well: healthy, well-fed citizens are more productive and pay more tax. Without human work, it’s much less directly beneficial for governments to invest in their people. I’m worried the self-interested incentives here won’t work out well for human flourishing, and I think we’ll probably need to align institutions (and the AI systems that run them) to intrinsically care about something like kindness to get around this. I want post-AGI government to pursue ends that are kind, and I want it to take all of its actions kindly.

If post-AGI government happens wisely and kindly - I think I’ll feel pretty happy with the outcome.

There are other virtues I care about too. Some of them appear in Lorenzetti’s fresco, though wearing different clothes:

• Peace becomes a practical technological project, rather than an ideal. The Sienese magistrates hoped for peace, but didn't have a lot of control over the geopolitical forces they were subject to: from the time Lorenzetti painted his frescos to the fall of the Sienese Republic in 1399, Siena was raided 37 times by mercenary armies, and fought several wars of its own. In the modern era, there have been serious international efforts to build towards peace, but these are partial and precarious. With AI, we might be able to proactively target technological development such that a negotiated settlement is always better than war for both parties.^[1] I don’t know if this will work - I don’t understand the dynamics of war very well, and I don’t know what’s technologically possible. But the thing I’d like to aim for is a technological state where war just isn’t a good strategy - and our great great grandchildren struggle to imagine the world that was benighted enough to resort to it.

• Temperance means treading lightly, and knowing when to stop. At a certain point, that’s enough datacentres for this planet, and if you want new ones, you have to go elsewhere. And that happens before you’ve tiled the oceans with solar panels and replaced all the farmland with datacentres and heated up the atmosphere so biological life can’t survive. Way before that, while there’s still wilderness, and countryside, and cities that are good to live in.

Other virtues are new additions:

• Freedom. We could be so much freer with AI than we’ve ever been before. Today, humans mostly have to live where they were born, in the bodies they were born into, and they have to spend most of their time at work. In many parts of the world, people also aren’t free to speak their minds, to the people they want to speak with. The good version of post-AGI government gives people radically more freedom than this, freedom that only stops at the point you’re restricting others’ freedom.
• Humanity. In the post-AGI era, humanity will be a relic, and a flawed one - but there’s nothing else in the universe quite like her. Good government is something that embodies her best qualities, and enables human flourishing. And that should be a hard line.
• Grace. The Sienese magistrates clearly cared about beauty, more I think than modern governments seem to (though there was and is a lot of ugliness in both kinds of society). I don’t want post-AGI government to treat beauty as something private, something that’s in the eye of the beholder, something that’s a luxury concern for some citizens. I want government itself to be deeply attentive to aesthetics, harmony, the way things fit together. Evil is often ugly, a bringing together of things that shouldn’t touch or a sundering of things that shouldn’t be parted. I think a government that’s attending to the look and feel of things and takes that as part of its remit is less likely to commit atrocities. I mean something slightly deeper than beauty, something that’s in harmony with the ways the universe unfolds. I’m calling it grace for now.

There, that’s the start of my allegory: good post-AGI government guided by wisdom, kindness, peace, temperance, freedom, humanity, and grace.^[2] It doesn’t deal with the nitty gritty of forms of government, voting systems, who votes and how often. It doesn’t tell you practically how we would build the tech we’d need to get there. I think we need more of both of those types of analysis.

But I also think we need more dreaming, envisioning, and hope. If we cling to what we’ve done before, I think AI will break our institutions and we’ll end up with chaos. If we exclusively try to prevent bad things from happening, I’m worried we’ll save a future that isn’t worth living in.

To the extent that you don’t like my allegory - because it’s cringe, or hopelessly naive, or dystopian, or missing something that matters - I think it would be worthwhile to lean into that, and articulate something that sounds better by your lights.

Postscript: what we stand to lose

Eight years after Lorenzetti finished painting the Allegory of Good and Bad Government, the black death arrived in Siena. It was one of the worst-hit cities: something like half of the people living there died.^[3]

In the words of Agnolo di Tura, a Sienese chronicler whose wife and five children died in the plague:

The mortality in Siena began in May. It was a cruel and horrible thing... One who did not see such horribleness can be called blessed… Father abandoned child, wife husband, one brother another; for this illness seemed to strike through breath and sight. And so they died. None could be found to bury the dead for money or friendship. Members of a household brought their dead to a ditch as best they could, without priest, without divine offices. In many places in Siena great pits were dug and piled deep with the multitude of dead. And they died by the hundreds, both day and night, and all were thrown in those ditches and covered with earth. And as soon as those ditches were filled, more were dug. I, Agnolo di Tura... buried my five children with my own hands... So many died that all believed it was the end of the world.

These things have happened before.

The Black Death wasn’t the result of bad government, it was an unusually virulent pathogen. But it’s a visceral example of what we have to lose, in life, love and human dignity, if we get the transition to advanced AI wrong.

1. ^{^}

    I’m thinking of things like structured transparency with great verification, so that states are able to understand their adversaries’ strength very accurately, without compromising each others’ security. Or highly accurate non-sentient simulations of war, which both parties trust to correctly model the outcome. Or commitment technology that allows states to credibly promise that they’ll only use weapons defensively. Or distributed permissions where weapons can only be deployed if neutral arbiters or international bodies agree.

2. ^{^}

    What about the allegory that sits on the throne? Lorenzetti had the personification of the city of Siena. I don’t want Siena, or Italy, or the personification of a world or interstellar government. I don’t actually want a throne at all. I’d like government to be less about rule, and more about empowerment. Less static, more organic. I don’t really know how to visualise this yet, and this makes me wonder if ‘government’ is even the thing to be aiming for. Maybe there’s some better way of organising things.

3. ^{^}

   Benedictow estimates 60% mortality (p. 307), but I haven't looked into the methodology.