(Follow-up to The 2024 Petrov Day Scenario)

Part 0: "Previously, on Petrov Day..."

By Raymond Arnold

One year ago, many people on LessWrong received a DM asking them to choose the most important virtue of Petrov Day, with four listed options that we'd seen people argue for in previous years. 

1. "Avoiding actions that noticeably increase chance that civilization is destroyed"
2. "Accurately reporting your epistemic state"
3. "Quickly orienting to novel situations"
4. "Resisting social pressure."

Then, people who chose one of those options were sent another message saying "Your virtue was in the minority, but you have the power to unilaterally choose the virtue of next year's Petrov Day by clicking a link."

The first person to click the Unilateral Virtue Link was a proponent of "Avoiding actions that noticeably increase the chance that the world will end." But, this virtue was actually in the majority^[1]. The first unilateralist of a Virtue Minority was a proponent of "Accurately reporting your epistemic state." 

A year later, as we decided what to do for Petrov Day, we decided to lure the first unilateralist into a surprise meeting, where I then said "Here's a reminder of what happened in Petrov Day last year. You now have one hour to design this year's Petrov game. Go."

An hour later, they presented us with a Petrov Game of Social Deception. 

We ended up making some changes to the game (with their sign off), based on both what seemed practical to build in 48 hours, and what seemed likely to work well overall.^[2]

The next 48 hours were very stressful, involving both spirited debates about the True Spirit of Petrov Day, what our obligations were given the results of last year's Petrov Day, staying up till 2am writing code for Nuclear Launch Consoles and then scrambling to code the actual Nuclear Destruction outcomes in the first 30 minutes after the game had launched.

It seemed important to me that:

1. Petrov's payoff specifically be about reporting his beliefs
2. To make that interesting, the incentives for the generals had to include some legitimate reason to think they might actually send nukes. 

So the incentives were weighted towards nukes for the generals, and costs were reduced from "the entire site's frontpage might go down and 10s of thousands of people will have a harder time using LessWrong today" to down "up to 300ish people who opted in wouldn't have a frontpage." My hope was that this would make their decision actually an interesting choice, rather than a performative ritual.

What follows in the next section is an account of some of what happened that day (though of course you can read the Diplomatic Channels yourself for more detail).

Part I: The 2024 Petrov Day Scenario

By Ben Pace

Tuesday 12pm: I return from vacation (attending a friend's wedding), and ask Ray what we're doing for Petrov Day. He tells me the backstory above and the idea for setting up a game. I am not sure if this will work out and work to sketch out a plan.

Tuesday 10:12pm: Ray ships a button to the frontpage to let users opt-in to participate in this year's Petrov Day. By the end of Wednesday over 300 users had opted in.

Wednesday afternoon: My last crisis of faith for this plan was during a 10-minute timer on Wednesday afternoon. After that we stayed up until ~1am getting things ready for Thursday. 

I DM ~80 users who opted in (and ~20 users who didn't) offering them to play roles in the game.

Thursday morning: 13 people have replied saying yes. One of the first 12, 1 person didn't realize that it was happening the next day and failed to show up so I subbed the 13th person in.

Here's the cast. I am very grateful to all of them for taking part, I think they all played their roles sincerely and well.

12:10pm: We finished the code for Petrov Day about 10 mins after the games were set to start, at which point they then got their logins.

12:20pm I texted them all their new logins and told them to keep their normal identities a secret. After logging in they had access to some LW dialogues.

The two sides of Generals each had their own War Room for dialogue as well as a publicly readable Diplomatic Channel between the two, public to anyone who viewed the link (including both Petrovs).

Petrov / Peterson saw a report at the 50 minute mark of every hour, and had 10 minutes in which to determine what report to make. It looked like this (note: this particular number was not shown during the game).

Generals received reports, and each had a unilateral launch button.

The Petrovs were not allowed to speak with anyone, they had a lonely job in their report room. One of the missile reports they received was comically large and they report being quite stressed in that 10 minute window, but from reading The Diplomatic Channel they felt it was the right call to not trust the sensors.

13:10: LessWrong the site goes down due to a bug (not due to nuking).

In the Diplomatic Channel, Generals discussed the site being down.

13:40: General Anderson (cata) has a bright idea.

14:18: General Andropov (Thomas Kwa) confirms this is how the code works.

14:19: Habryka writes a shortform posts apologizing for the outage and noting that the site is back up.

The site being down meant that the Petrovs missed one of their reporting windows. We decided not to count this reporting window. I later found out that, during this window, one of the Petrovs messaged one of the mods saying to report nukes if the number reported was over a certain threshold. From looking through the array of numbers that the code would randomly select from, this policy had a ~40% chance of causing a "Nukes Incoming" report (!). Unaware of this, Ray and I made the decision not to count that period.

15:07: The generals then have a pleasant chat where they commit to informing the other side if nukes have been fired, in order to sabotage any potential incentive for someone on either side to strike and get away with it.

16:29: General Carter (Mikhail Samin) posts a website he has made the website westwrong-culture.pages.dev along with an AI-generated song.

Here's the YouTube video of the song. You can read all the lyrics at the site, but here's my favorite verse and chorus.

We wrestle with bias, and dance with the Bayes,

Unraveling mysteries, in intricate ways.

From Petrov's calm hand, to algorithms bright,

We strive for the good, with all of our might.

 

We are WestWrong, a chorus of thought,

Where ideas ignite, and battles are fought,

Not with weapons or steel, but with words sharp and keen,

For the future we build, on screens brightly seen.

5:50pm: Habryka wrote this to the Diplomatic Channel.

Voice of LessWrong: Here are the rules for the final section of the game. The game will end at a random time between 6:00PM and 6:30PM PT. During this period, Petrov and Peterson can make as many reports as they like, and their dashboard will update live with any new missile activity (as always, with errors). Nukes can be launched until the game ends (Petrov and Peterson will get a karma reward only once for this section, based on the admins' gut judgement of whether they "got it right")

6:08pm: The game concludes!

Over the course of the day, every hour, Petrov and Peterson reported no nukes every time, and no General sent any nukes. Hurrah!

For their service, the Generals each earned 100 karma, and the Petrovs earned 1,000 for their multiple correct reports of "All Clear".

Part II: Stats & Feedback Form

Here's the data from 35 people (including 7 player characters).

Here is a google sheet with all the written feedback people gave us (anonymized).

Here are representative question/answer pairs (all from different people).

Q: What was the single best thing about your experience of the game
A: reading the diplomatic channel. lots of beautiful messages there.

Q: What was the single worst thing about your experience of the game?
A: Relative lack of drama, and need to intermittently pay attention over 6 hours

Q: What one change would you make to improve this game if we ran it again next year?
A: More ways for civilians to participate, more 'scenarios' that bring the West and East closer to the brink.

Q: (Optional) Anything else you'd like to add?
A: the lightcone is nothing without its people

Wordcount

For the final thing this section, here's the wordcount of the day's post

• East Wrong War Room: ~1,500 words
• West Wrong War Room: ~3,600 words
• Diplomatic Channel: ~4,000 words
• LW Comment Section: ~6,800 words

Part IV: Reflections

Reflections by Raymond Arnold

Different LessWrong team members think different things about what Petrov Day means. 

Last year, one thing that stood out to me was Vanessa's commentary about her thought process last year, as someone who chose to click the "unilaterally seize virtue control" link.

What is my take-away lesson? The process I used to make the decision seems correct to me: if you have to make a split-second decision, then you need to use your split-second judgement because there is nothing else to go by. There might be some case for a bias towards inaction, but it's not an overwhelming case. Personally, I know that I'm usually too slow to respond in emergency scenarios, so I don't want to train myself to prefer inaction.

The right way to optimize this is to train your split-second judgement to do well in the sort of situations in which split-second judgement is likely to be required. The sort of reasoning required of us here is not likely to be tied to a split-second decision anywhere outside of Petrov Day games^[2], so I think my split-second judgement did as well as expected and there's nothing to correct.

[EDIT: Actually, there is a correction to be made here, and it refers to my wrong reading of the message after clicking the link. The lesson is: if I make a split-second decision, I need to carefully reexamine it after the fact, in order to understand its true consequences, and beware of anchoring on my split-second reasoning: this anchoring is probably motivated by wanting to justify myself later.]

A point of discussion in previous Petrov Day LessWrong events is that, even if you think it's a "game," destroying the frontpage of LessWrong for a day is actually sacrificing nontrivial value. And if you find yourself in a situation where you can casually destroy a bunch of value... like, dude, maybe just don't do that?

Value isn't any less lost if you think your social role is "playing a game." Notice the underlying reality, not just your social role.

But, I also think it's important to not merely learn to generically mouth the words "I shall not click the symbolically ritual button or do anything unilateralist-y on symbolically significant holiday." I'm a minority on the LessWrong team, but I think the virtue of "oriently quickly to a difficult, confusing, high stakes situation" is one of the more important Petrov Day virtues.^[3]

I think it's important for people to actually model "what is at stake in a given situation", and cultivate the skill of actually figuring that out. It's not enough to shout "I will do the Good Thing," the hard part is identifying what the good thing is. I like Vanessa's comment because she both actually thought about the tradeoff (at least briefly) in the moment, and then allocated time to retroactively evaluating it in the past.

People have commented that the payoff structure doesn't reflect a real nuclear war. That's true. But, in real life, despite nuclear war being incredibly bad, nations nonetheless build and deploy lots of nukes due to a variety of iterated games. It seemed good for the payoff structure to reflect the all-things-considered "what do countries seem incentivized to do?", rather than the local question of "how good/bad is a nuclear war?"

Did we do a good job with all this? I dunno, we put this together in about 48 hours. I haven't gotten much distance from it yet, and it's cruxy for me how LessWrong overall responds to the experiment after the fact. Some people have commented this year's event felt more like a "game" than previous years, less like a meaningful ritual. That seems like a reasonable take.

Is it net positive that once a year the LessWrong team throws together a slapdash Petrov Day experience that sometimes takes the site down for buggy-code reasons^[4] that aren't particularly representative of Petrov's incentives and instead represent the janky setup of Petrov's warning system? Should we put more time into it to make it less slapdash? Would that time be worth it, when it trades off against building featues that more directly make LessWrong a place where intellectual progress on important problems can happen?

I dunno, there's enough considerations that it's non-obvious. But those are all questions worth asking. 

Overall, my guess is that the balance of "we spend relatively little time on Petrov Day, and the result is a bit frenetic, not perfectly thought out, and has the tension of a dangerous, janky system" is a surprisingly reasonable equilibrium. 

May you find and cleave to the True Spirit of Petrov Day, as wisely as you can.

Reflections by Ben Pace

Overall this went better than I expected. I had a lot of doubts that this game would come together or be a good experience, but I only came back from vacation on Tuesday afternoon and we didn't have a lot of time to consider alternatives. I'd say it's about 70th-80th percentile outcome relative to my expectations, in that people showed up to take their roles, and the comments and dialogues were lively and sincere (especially aphyer's comments, to mention just one).

I agree with Ray that the ritual and the social experiment aspects have some tension. I think the social experiment angle seems quite underexplored to me and I'm excited to try it again next year in a more considered way, there are obviously many angles one could improve things (e.g. causing more room for tension between the two sides, involving the civilians more, etc).

Personally I think the virtue of Petrov Day is in taking ultimate responsibility for your actions. An organization of people with the virtues of Petrov is an organization that will only behave ethically, because each person is inspecting the action themselves. It's not one merely of "resisting social pressure", it's also one of taking responsibility for things not demanded of you. Those two things in combination make a Petrov (plus some ability to handle pressure in high-stakes situations). I think of this in contrast to the notion of "I was just following orders" and also in contrast to Feynman's virtue of "active irresponsibility", the latter of which I do cultivate in some situations but not others.

I would like to see a version of the Petrov Day scenario next year that gave everyone involved the risk of difficult ethical decisions, and measured its success on the ethics of the behavior of the system as a whole. I don't know how to do this yet or if it's even possible, but that would be my goal.

I also want to say that I am uncertain about whether it was a good idea to use karma as a reward/penalty for the game. I think this instance was not enough to seriously alter or break the signal of karma on the site (via the path of strong votes changing substantially), but it is plausible to me that stepping into altering them for non-voting reasons will result in a system that is much less trustworthy. I think if we do another scenario I would want to do a search for other sorts of rewards/penalties.

The other main thing I hope to do next year is give people a few week's heads up for Petrov Day and give them encouragement to run their own local Petrov Day services. Jim Babcock and I ran two (as ~25 ppl showed up) in the evening and it's always a good experience for getting in touch with history and the position we're in today.

1. ^{^}

   The actual reason why we lied in the second message was "we were in a rush and forgot." After having had it pointed out, it seemed like the best thing to do was honor the wording we had stated.

2. ^{^}

   The designer had (I think?) initially noticed the "focus on accurately reporting epistemic state" aspect, but said that during the stressful hour of designing the game had eventually forgotten that. The version they handed off wasn't particularly optimized for that, but the framework of a social deception game seemed to me to be a good substrate for "accurate epistemic reporting."

3. ^{^}

   Though I was sold on Ben's conceptualization of "the virtue of taking personal responsibility", see below.

4. ^{^}

   The code that crashed the site database yesterday didn't actually have to do with the Petrov Day scenario, it was an unrelated problem in our dialogue notifications became a problem when we added 300 people as observers in the Generals' Diplomacy Channel.