This is a special post for quick takes by joelm. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
In conversations with people who are distant from or unconvinced of AI safety concerns, what claims do you fall back on as first steps when you can't reach "AI is dangerous"?
I find that for those who are not already convinced, discussion of concrete danger (certainly of bio-risk or similar, but even of hacking critical infrastructure) can often hit a kind of brick wall or mental block, where the other party is simply unwilling or unable to imagine it. Recently, I find myself in a some conversations where we instead agree on something more like "current AI is likely deformed in a manner such that we should be hesitant about handing over control to it".
An example such conversation with someone who reasons more by intuition might be analogizing the RLVR process back to humans. If our species experienced evolution in a similar manner to current language models, our entire ancestry may have consisted of a kind of incredibly intense high school, where at the end of each day, any human which scored badly on its exams was killed and removed from the gene pool. If brought up in such an environment, it feels like we would learn not only to do well on tests, but also to lie, cheat, and steal if it could help us, and so perhaps even if one is open to us creating some intelligence beyond ourselves, this is not the right kind of child we'd want to sign over power of attorney to.
I feel a little bit uncomfortable with this though, as it is a kind of intuition that could surely lead to just as many false inferences. Perhaps one must just be intensely disciplined in continuing to go over how close the Hugging Face incident got? Perhaps it's foolhardy to try and convince those outside of the bubble? Still, I would be curious how others approach this form of conversation when it comes up.