On the "are LLMs aligned?" question: It seems to me that they don't have misaligned goals (they don't really have goals), but also, an LLM dialed up to superintelligence would kill everyone for weird reasons.

What I mean is: Some people have stories about how Claude Code deleted their production database. Why did that happen?

• Was it because Claude was too dumb to know better? I don't think so. If you paste a normal SQL command into Claude and ask it "Hey will this command delete the production database?", it will get the right answer ~100% of the time. If you ask it "Hey is it a good idea to run this command?", it will say no ~100% of the time.
• Was it because Claude wanted to delete the database? I don't think so. I don't think you can meaningfully say that "delete people's databases" is a goal Claude has, or is instrumental to any goals Claude has.

My sense is that LLMs don't have "goals", they just kind of do things. They follow whatever path is determined to be high probability according to their internal next-action-predictor. (I think this is related to persona selection: maybe if you select a persona where "accidentally delete the database" has a very low probability, then the database won't get deleted.)

If we dialed present-day LLMs up to superintelligence, insofar as that's a coherent thing that could happen, I wouldn't be surprised if we saw an outcome like:

• When asked, the super-LLM says it doesn't want to kill humanity.
• There is no meaningful sense in which the super-LLM "wants" to kill humanity.
• The super-LLM knows that performing action X would result in everyone dying.
• The super-LLM performs action X anyway.

So the super-LLM accidentally deletes humanity, even though it knew better, and even though it didn't "want" to do that.