AI 2040 was an incredible read. I am not aware of any other concrete AI takeoff scenarios that solve both the alignment and power concentration risks of AI in a robust way quite like AI 2040: Plan A. This is even with the authors assuming hard alignment and the models staying misaligned as late as 2038.

As an outsider there were some striking parts of the scenario that updated my likely worldview of the future:

• Anthropic is depicted to be ahead of the other US AI labs in sheer compute and resources in every scenario, including racing to ASI first in the race scenarios (Plan C and D). AI 2040 appears to treat each company equally and the starting conditions favor Anthropic. Plan D is basically a "either Dario or misaligned-Claude becomes the god-emperor" scenario.
• The sheer amount of things that have to go right to reach the good future is striking to me, including some proposals that I find to be fairly hard to arrange such as the one for US datacenters to be built in Mongolia and Chinese ones to be built in Canada.
• The prediction that the "AI slowdown won't feel like one" definitely runs counter to my intuition, but logically it is almost certainly correct given that it happens in 2029 as depicted in the scenario.
• The prediction that even in the Plan A good future, we won't have the chance of a Long Reflection due to the world staying multipolar and therefore any Long Reflection will turn into a memetic war. (This seems to imply that the only chance for a proper Long Reflection is in totalitarian scenarios like "Dario-Amodei-is-god-emperor"? Maybe the Forethought Institute guys who care about especially-good futures should all switch to helping Dario race to superintelligent AI?)
• The sheer amount of UBI that is given out in the Plan A scenario. This intuitively doesn't feel right to me, but I haven't looked at the math..
  • What follows is one of my preexisting beliefs that isn't contradicted by AI 2040 - that as long as misaligned AI and WW3 scenarios are prevented, life for the average citizen is probably pretty good even though a few oligarchs might control the entire universe.
• I think that the sheer personal autonomy given in Plan A's proposed space colonization plan might be excessive and lead to moral atrocities. In particular the moral red lines would have to be enforced by an AI fully designed around ~2040 rather than one with the benefit of longer reflection. Slowing space expansion to a small fraction of c would probably be required to prevent this.

And some of my observations on how to act in a scenario like Plan A if one wants to maximize their personal power and impact:

• AI scaffold quality becomes more important in any kind of AI slowdown/pause scenario. Having better harnesses and workflow automation than everyone else becomes a big competitive advantage. Small groups could decisively outperform big ones by innovating faster in this area.
• Going for personal wealth accumulation is pointless if it won't outpace the growth of UBI and the robot economy in general. Accumulating popularity in genuine ways, on the other hand, is very powerful.
• After ~2034 the most meaningful impact one can have is in moral progress and other intrinsically human things. It seems like it is very important for people to truly understand themselves and the human species.