(edit: see disclaimers^[1])

1. Creating superintelligence generally leads to runaway optimization.
2. Under the anthropic principle, we should expect there to be a 'consistent underlying reason' for our continued survival.^[2]
3. By default, I'd expect the 'consistent underlying reason' to be a prolonged alignment effort in absence of capabilities progress. However, this seems inconsistent with the observation of progressing from AI winters to a period of vast training runs and widespread technical interest in advancing capabilities.
4. That particular 'consistent underlying reason' is likely not the one which succeeds most often. The actual distribution would then have other, more common paths to survival.
5. The actual distribution could look something like this: ^[3]

Note that the yellow portion doesn't imply no effort is made to ensure the first ASI's training setup produces a safe system, i.e. that we 'get lucky' by being on an alignment-by-default path.

I'd expect it to instead be the case that the 'luck'/causal determinant came earlier, i.e. initial capabilities breakthroughs being of a type which first produced non-agentic general intelligences instead of seed agents and inspired us to try to make sure the first superintelligence is non-agentic, too.

(This same argument can also be applied to other possible agendas that may not have been pursued if not for updates caused by early AGIs)

1. ^{^}

   Disclaimer: This is presented as probabilistic evidence rather than as a 'sure conclusion I believe'

   Editing in a further disclaimer: This argument was a passing intuition I had. I don't know if it's correct. I'm not confident about anthropics. It is not one of the reasons which motivated me to investigate this class of solution.

   Editing in a further disclaimer: I am absolutely not saying we should assume alignment is easy because we'll die if it's not. Given a commenter had this interpretation, it seems this was another case of my writing difficulty causing failed communication.

2. ^{^}

   Rather than expecting to 'get lucky many times in a row', e.g via capabilities researchers continually overlooking a human-findable method for superintelligence

3. ^{^}

   (The proportions over time here aren't precise, nor are the categories included comprehensive, I put more effort into making this image easy to read/making it help convey the idea.)