This is a special post for quick takes by Nico Hillbrand. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I think almost any coherent vision of a good future involves some kind of pause / restriction on unsafe AI development. We should probably not wait with preparing it for when AI labour is directed to it during takeoff.
There's approximately 3 attitudes towards AGI Safety:
I encounter 2. a bunch as a local group organiser. I think often it's because people think there's too much uncertainty or too little leverage they have such that they're better off focusing their motivation on worlds without future powerful AI. Seeing that experts take powerful AI seriously through podcasts etc and talking with other people at conferences that do seems to shift people towards 3. sometimes.
I think 3. maybe has some more interesting distinctions between:
3.1 People that assume alignment is easy to medium difficult and organisations will act pretty competently
3.2 People that take an uncertain attitude about alignment difficulty and organisational competence
3.3 People that assume alignment is difficult and organisations will act pretty incompetently
I think it'd be good for the world to have more people with the 3.2 attitude. Maybe a good example of people that I think of as representative are Buck Shlegeris and Ryan Greenblatt.
My intuition is that 3.3 gaining more traction would be good for the world, because to me a) that seems most realistic based on the evidence I've seen so far and how I interpret it, and b) least problematic in case it's wrong and we live in a world where alignment isn't hard and organisations act competently. What reasons make your intuition point towards 3.2?
Generally I think it's good for people to try to be as accurate as possible and that's what I most associate with 3.2. That said these clusters are a big oversimplification and I think there's people with good reasoning that end up in 3.3. And in practice 3.3 might be a reasonable attitude just from a perspective of a sane society wanting to address the alignment is hard and orgs are incapable scenario anyway
The only place I expect an organisation with excellent safety culture, security mindset and strategic philosophical competence capable of steering a singularity towards good outcomes to arise is in a data center housing a country of aligned roughly human level AIs.