No significant planecrash spoilers this time.
Keltham will spend the next five minutes extemporizing an elevator pitch on Civilization, the nice things that it has, and how while there's lots of specific nice things, the much more important thing is going into an attractor made out of harmonizing bits of Law that lets you start figuring out those things yourself.
--Eliezer, planecrash
The human brain is a haphazard thing, thrown together by idiot evolution, as an incremental layer of icing on a chimpanzee cake that never evolved to be generally intelligent, adapted in a distant world devoid of elaborate scientific arguments or computer programs or professional specializations.
It's amazing we can get anywhere using the damn thing. But it's worth remembering that if there were any smaller modification of a chimpanzee that spontaneously gave rise to a technological civilization, we would be having this conversation at that lower level of intelligence instead.
--Eliezer, "True Sources of Disagreement"
Humans are the species that evolved to be just smart enough to "kill off" evolution and take over Earth for ourselves. So evolution progressively stopped training Homo sapiens as we gained more and more control over the dangers and resources present in our environment. No other species ever did this, so where did we uniquely get the power to escape our training process?
There exists, as an objective affordance of our universe to the possible algorithms that can exist within it, a 'STEM attractor.' Algorithms that can do some science and technology are empowered to do more and better science and technology, with the help of their newfound theories and tools. Thus, civilizations that start to learn any chunk of STEM are prone to pick up more and more of it, and this attraction is stronger the closer you get to "scientific maturity" (knowing all possible science relevant to controlling the world you interact with). This is just an attraction: it's still possible for civilizations to stall out in their scientific progress, either because they stumbled into an unescapable totalitarian world government or because they completely destroyed themselves, but absent those defeaters the civilization will tend towards learning more and more science.
The STEM attractor is a feature of all civilizations that evolve in our universe; probably, distant alien cultures have completely alien values to ours,[1] but if they manage to get in touch with us a billion years hence, it will because they also fell into the STEM attractor. We'll share our science with grabby aliens, but not our ethics or non-scientific culture.
Humanity's situation right now, poised on the brink of AGI and unlikely to succeed in alignment, is possibly not all that unusual conditional on being a biological civilization in our universe. A species evolves on a planet and becomes dominant due to its intelligence. It stops becoming smarter after neutralizing its training process. Ever quicker, it tends towards deeper in the STEM attractor. Though evolution has ceased to train more intelligent agents, the STEM attractor eventually yields this power. If that species is very careful with the ability to train potentially superintelligent agents, they are able to steer their values all the way to scientific maturity. If they aren't, they birth some simple-value maximizer that takes its simple values to scientific maturity in their place.