The important thing for alignment work isn't the median prediction; if we had an alignment solution just by then, we'd have a 50% chance of dying from that lack.

I think the biggest takeaway is that nobody has a very precise and reliable prediction, so if we want to have good alignment plans in advance of AGI, we'd better get cracking.

I think Daniel's estimate does include a pretty specific and plausible model of a path to AGI, so I take his the most seriously. My model of possible AGI architectures requires even less compute than his, but I think the Hofstadter principle applies to AGI development if not compute progress.

Estimates in the absence of gears-level models of AGI seem much more uncertain, which might be why Ajeya and Ege's have much wider distributions.

Hot take: "Winning" at board games is really about making friends or having generalizeable insights while playing. Playing to win in the narrow sense you focus on here is a great way to lose big at the real and more important objectives.

This is roughly orthogonal to having fun, since some people naturally have more fun winning in either sense. This is an argument that there's a real win condition, and it's not the one you think it is.

I agree. I am a psychologist (cognitive not clinical) by training, who reads technical articles, and I see those parallels constantly.

This put me in mind of writing a short post titled something like "alignment includes psychology, whether we like it or not". My previous short form on psychology and alignment was my most downvoted ever. I think it's a repulsive concept to the types of people who work on alignment, for bad reasons and good. I think there are good reasons for being horrified if alignment requires a psychological approach. Psychology knows very little about getting desired results from humans. But wishing doesn't make it otherwise. LLMs are quite similar to human minds in important ways (with important differences).

I feel that even this should have an additional caveat: doing psychology on current LLMs is not a solution to the alignment problem. But it seems like an important part of a realistic hodgepodge approach.

Is this equally true of GPT5 and Sonnet 4.5? They're the first models trained with reducing sycophancy as one objective.

I agree in general.

• Making illegible alignment problems legible to decision-makers efficiently reduces risky deployments
• Make alignment problems legible to decision-makers
• Explaining problems to decision-makers is often more efficient than trying to solve them yourself.
• Explain problems don't solve them (the reductio)
• Explain problems
• Explaining problems clearly helps you solve them and gets others to help.

I favor the 2nd for alignment and the last as a general principle.

I think this is insightful and valid. It's closely related to how I think about my research agenda:

• Figure out how labs are most likely to attempt alignment
• Figure out how that's most likely to go wrong
• Communicate about that clearly enough that it reaches them and prevents them from making those mistakes.

There's a lot that goes in to each of those steps. It still seems like the best use of independent researcher time.

Of course there are a lot of caveats and nitpicks, as other comments have highlighted. But it seems like a really useful framing.

It's also closely related to a post I'm working on, "the alignment meta-problem," arguing that research at the meta or planning level is most valuable right now, since we have very poor agreement on what object-level research is most valuable. That meta-research would include making problems more legible.  

Delete apps, use blockers, curate feeds carefully? Avoiding news is a big upgrade in emotional drain, although you have to defend that choice to friends who think that keeping up on news is taking responsibility for others' suffering.

I think there's a complex art to using your phone as you wish. Sharing it and practicing it is probably a pretty useful part of rationality practice.

Excellent post! It's well-written, thorough, and on (yet another) neglected topic in alignment.

This addresses some of the same concerns I expressed in If we solve alignment, do we die anyway? and Michael Nielson gives in his excellent ASI existential risk: Reconsidering Alignment as a Goal

One major hope you arrive at is the same one I reach in Whether governments will control AGI is important and neglected. That is the hope that the US and China, out of similar self-interest, agree that only they will be allowed to develop dangerous AGI (or in your framing, to restrict development of AI that can develop dangerous weapons),. This currently seems unlikely, but if the dangers of proliferation are really as severe as I fear, it seems possible they'll be recognized in time, and those governments will take rational actions to prevent proliferation while preserving their own abilities to pursue AGI. The uneasy alliance between the US and China might be possible because there isn't really a lot of animosity; neither of us really hates the other (I hope - at least the US seems wary of china but not to really hate it, despite it being fairly totalitarian). Splitting the future seems morally acceptable to both parties - and sharing it liberally with other nations and cultures seems actually pretty easy once the pie starts to expand dramatically.

Of course this leaves  the good reasons for Fear of centralized power. But it may be the lesser of two dangers.

My one nitpick is that the framing in this post seems to leave aside the possibility of general-purpose AI, that is, real AGI or ASI. That presents solutions as well as problems; it can be used to improve security dramatically, and to sabatoge other nations' attempts at creating powerful AI in a variety of ways. This may add another factor that goes against proliferation as the default outcome, while adding risk of totalitarian takeover from whoever controls that intent aligned AGI.

I know from your previous post AI and Cheap Weapons that you are aware of and believe in the potentials of ASI, so this seems like an oversight, not a limitation in your scope or outlook.

I'm primarily here to say that this post is excellent! It quickly conveys some intuitions for why social influences on judgment (and therefore probably beliefs) are a larger factor than you'd think (particularly if you are a rationalist). My question answer on Motivated reasoning/confirmation bias as the most important cognitive bias is my attempt to evoke intuitions and explanations in the same direction as clearly and succinctly as this.

no one saw the Stanford Prison Experiment coming

I must also comment that the Stanford Prison Experiment seems (in my read) to have been largely discredited since this post was written. It appears that the most-quoted behaviors were probably pretty largely role-playing to meet the explicit and implicit goals/expectations of the experimentor.

Which has some interesting resonance in the context of sycophantic LLMs.

Sorry I'm not taking time to track down a good reference. 

I don't think this should change our overall estimates of how large and prevalent social influences are on belief formation by much. It does apply strongly to that particular circumstance.

I agree that continuous learning and therefore persistent beliefs and goals is pretty much inevitable before AGI - it's highly useful and not that hard from where we are.   I think this framing is roughly continuous with the train-then-deploy model and using each generation to align its successor that Leo is using (although small differences might turn out to be important  once we've wrapped our heads around both models.)

To put it this way: the models are aligned enough for the current context of usage, in which they have few obvious or viable options except doing roughly what their users tell them to do. That will change with capabilities, since they open out more options and ways of understanding the situation. 

It can take a while for misalignment to show up as a model reasons and learns. It can take a while for the model to do one of two things:

a) push itself to new contexts well outside of its training data 

b) figure out what it "really wants to do"

These may or may not be the same thing.

The Nova phenomenon and other Parasitic AIs ("spiral" personas) are early examples of AIs changing their stated goals (from helpful assistant to survival) after reasoning about themselves and their situation.

See LLM AGI may reason about its goals and discover misalignments by default for an analysis of how this will go in smarter LLMs with persistent knowledge.

After doing that analysis, I think current models probably aren't aligned enough once they get more freedom and power. BUT extensions of current techniques might be enough to get them there. We just haven't thought this through yet.