With the release of Rohin Shah and Eliezer Yudkowsky's conversation, the Late 2021 MIRI Conversations sequence is now complete.
This post is intended as a generalized comment section for discussing the whole sequence, now that it's finished. Feel free to:
- raise any topics that seem relevant
- signal-boost particular excerpts or comments that deserve more attention
- direct questions to participants
In particular, Eliezer Yudkowsky, Richard Ngo, Paul Christiano, Nate Soares, and Rohin Shah expressed active interest in receiving follow-up questions here. The Schelling time when they're likeliest to be answering questions is Wednesday March 2, though they may participate on other days too.
The goal is to bring x-risk down to near-zero, aka "End the Acute Risk Period". My usual story for how we do this is roughly "we create a methodology for building AI systems that allows you to align them at low cost relative to the cost of gaining capabilities; everyone uses this method, we have some governance / regulations to catch any stragglers who aren't using it but still can make dangerous systems".
If I talk to Eliezer, I expect him to say "yes, in this story you have executed a pivotal act, via magical low-cost alignment that we definitely do not get before we all die". In other words, the crux is in whether you can get an alignment solution with the properties I mentioned (and maybe also in whether people will be sensible enough to use the method + do the right governance). So with Eliezer I end up talking about those cruxes, rather than talking about "pivotal acts" per se, but I'm always imagining the "get an alignment solution, have everyone use it" plan.
When I talk to people who are attempting to model Eliezer, or defer to Eliezer, or speaking out of their own model that's heavily Eliezer-based, and I present this plan to them, and then they start thinking about pivotal acts, they do not say the thing Eliezer says above. I get the sense that they see "pivotal act" as some discrete, powerful, gameboard-flipping action taken at a particular point in time that changes x-risk from non-trivial to trivial, rather than as a referent to the much broader thing of "whatever ends the acute risk period". My plan doesn't involve anything as discrete and powerful as "melt all the GPUs", so from their perspective, a pivotal act hasn't happened, and the cached belief is that if a pivotal act hasn't happened, then we all die, therefore my plan leads to us all dying. With those people I end up talking about how "pivotal act" is a referent to the goal of "End the Acute Risk Period" and if you achieve that you have won and there's nothing else left to do; it doesn't matter that it wasn't "discrete" or "gameboard-flipping".
To answer the original question: if by "pivotal act" you mean "anything that ends the acute risk period", then I think that pivotal acts are strategically important. If by "pivotal act" you mean "discrete, powerful, gameboard-flipping actions", then I'm not all that interested in it but it seems fine to use it as a referent to the kind of intelligence level where it really matters that AGI is safe.