This is a special post for quick takes by Richard_Ngo. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Given that Vernor Vinge wrote The Coming Technological Singularity: How to Survive in the Post-Human Era in 1993, which single-handedly established much of the memeplex, including the still ongoing AI-first vs IA-first debate, another interesting question is why didn't anyone found the AI safety field until around 2000.
For me, I'm not sure when I read this essay, but I did read Vinge's A Fire Upon the Deep in 1994 as a college freshman, which made me worried about a future AI takeover, but (as I wrote previously) I thought there would be plenty of smarter people working in AI safety so I went into applied cryptography instead (as a form of d/acc). Eliezer after reading Vinge (as a teen) didn't immediately heed the implicit or explicit safety warnings and instead wanted to accelerate the arrival of the Singularity as much as possible. It took him until around 2000 to pivot to safety. Nick Bostrom I think was concerned from the beginning or very early, but he was a PhD student when he got interested and I guess it took him a while to work through the academic system until he could found FHI in 2005.
Maybe the real question is why didn't anyone else, i.e., someone with established credentials and social capital, found the field. Why did the task fall to a bunch of kids/students? The fact that nobody did it earlier does seem to suggest that it takes a very rare confluence of factors/circumstances for someone to do it.
(Another tangential puzzle is why Vinge himself didn't get involved, as he was a professor of computer science in addition to science fiction writer. AFAIK he stayed completely off the early mailing lists as well as OB/LW nor had any contacts with anyone in AI safety.)