This is a special post for quick takes by Richard_Ngo. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
So I've been thinking about this for a while, and I think I disagree with what I understand of your perspective. Which might obviously mean I misunderstand your perspective.
What I think I understand is that you judge safety research directions based on how well they could work on an evolutionary process like the one that created humans. But for me, the most promising approach to AGI is based on local search, which differs a bit from evolutionary process. I don't really see a reason to consider evolutionary processes instead of local search, and even then, the specific approach of evolution for humans is probably far too specific as a test bench.
This matters because problems for one are not problems for the other. For example, one way to mess with an evolutionary process is to find way for everything to survive and reproduce/disseminate. Technology in general did that for humans, which means the evolutionary pressure decreased as technology evolved. But that's not a problem for local search, since at each step there will be only one next program.
On the other hand, local search might be dangerous because of things like gradient hacking. And they don't make sense for evolutionary processes.
In conclusion, I feel for the moment that backchaining to local search is a better heuristic for judging safety research directions. But I'm curious about where our disagreement lies on this issue.