I have been surprised to repeatedly see the claim that there isn't even an argument for concern about AI. That the claim has been made without evidence and can therefore be dismissed.
Obviously, there is an extensive library of evidence and arguments that have been made for decades. Additionally, I would argue that it is the default assumption. However, there is clearly still a need to have a concise argument that can be produced on the fly with no need to understand terminology or any additional background. Here is another attempt at that:
Does this mean that Ais are 100% certain to wipe out humanity? No, of course not. That's an absurd bar. Rather, the burden of proof should be to show that AIs are 99% certain not to cause catastrophe. If there's a 10% chance that Ais will sterilize the earth, that's already an all hands on deck situation.
AFAIK, the Superintelligence FAQ is still considered to be the best introduction for most people.
Ngo wrote AI safety from first principles, but I don't know if it was good enough. Along with List of Lethalities, it's probably a solid list of things to include. Possibly worth doing some work on in order to make a better version.
Zvi's Basics of AI wiping out all value definitely looks pretty neat, but at the end of the day we need someone to go out and empirically test all of these and see what gets the best results.
I find those first two and Lethalities to be too long and complicated for convincing an uninitiated, marginally interested person. Zvi's Basics is actually my current preference along with stories like It Looks Like You're Trying To Take Over The World (Clippy).
This doesn't work as a from-scratch explanation because
You don't explain what alignment is or why it is desireable.
You don't explain what agentising is. or why it is dangerous.
You don't explain why "training successor versions of itself" is dangerous.
I agree that there are many situations where this cannot be used. But there appears at least to be a gap that arguments like this can fill that is missed by the existing explanations.
Contrary opinion: Agents like #AutoGPT are more aligned than the underlying LLMs due to chaining. e.g. If the model will say "I need to make $1M" & then answers itself as "Stealing is the best plan to achieve it"; there are two prompts for the underlying LLMs to refuse help.