Inhabiting my views i had at the time rather than my current ones where I cringe at several alignment-is-straightforward claims I'm about to make:
Well yeah. So many people around me clearly were refusing to look at the evidence that made it obvious AGI was near (and since then have deluded themselves into believing current AI is not basically-AGI-with-some-fixable-limitations), and yet still insist they know the future around what's possible. Obviously alignment is hard, but it doesn't seem harder than problems the field chews through regularly anyway. it's like they don't even realize or think about how to turn capabilities into an ingredient for alignment - if we could train a model on all human writing we could use it to predict likely alignment successes. i explained to a few MIRI folks in early 2017, before transformers (I can show you an email thread from after our in-person conversation) why neural networks are obviously going to be AGI by 2027, they didn't find it blindingly obvious like someone who grokked neural networks should have, and that was a huge hit to their credibility with me. They're working on AGI but don't see it happening in front of their faces? Come on, how stupid are you?
(Turns out getting alignment out of capabilities requires you to know what to ask for still, and to be able to reliably recognize if you got it. If you'd told me that then it would gave gotten me thinking about how to solve it but I'd still have assumed the field would figure it out in time like any other capability barrier.)
(Turns out Jake and I were somewhat, though not massively, unique in finding this blindingly obvious. Some people both found this obvious and were positioned to do something about it, eg Dario. I've talked before about why the ML field as a whole tends to doubt their 3 year trajectory - when you're one of the people doing hits based research you have a lot of failures, which makes success seem further than it is for the field as a whole - and now it's turned into alignment researchers not believing me when I tell them they're doing work that seems likely to be relevant to capabilities more than to alignment.)
So, like, #1: make arguments based on actual empirical evidence rather than being mysteriously immune to evidence as though your intuition still can't retrodict capability successes. (This is now being done, but not by you.) #2 might be, like, Socratic questioning my plan - if I'm so sure I can align it, how exactly do I plan to do that? And then, make arguments based on accepting that neural networks do in fact work like other distributed-representation minds (eg animals), and use that to make your point. Accept that you can't prove that we'll fail because it's legitimately not true that we're guaranteed to. Engage with the actual problems someone making a capable ai system encounters rather than wrongly assuming the problem is unrecognizeably different.
And then you get where we've already gotten. I mean, you haven't, I'm still annoyed at you for having had the job of seeing this coming and being immune to evidence, but mostly the field is now making arguments that actually depend on the structures inside neural networks, so the advice that was relevant then mostly doesn't apply anymore. But if you'd wanted to do it then, that's what it would have looked like.
Basically if you can turn it into a socratic prompt for how to make an aligned AI you can give a capability researcher, where the prompt credibly sounds like you actually have thought about the actual steps of making capable systems and if it's hard to do it will become apparent to them as they think about the prompt, then you're getting somewhere. But if your response to a limitation of current AI is still "I don't see how this can change" rather than "let me now figure out how to make this change" you're going to be doomed to think AI is going slower than it is and your arguments are likely to sound insane to people who understand capabilities. Their intuitions have detailed bets about what's going on inside. When you say "we don't understand", their response is "maybe you don't, but I've tried enough stuff to have very good hypotheses consistently". The challenge of communication is then how you make the challenge of alignment into an actual technical question where their intuitions engage and can see why it's hard for themselves.
I was culturally disposed against it, btw. I was an LW hater in 2015 and always thought the alignment concern was stupid.
1
In recent years, I’ve found that people who self-identify as members of the AI safety community have increasingly split into two camps:
Camp A) "Race to superintelligence safely”: People in this group typically argue that "superintelligence is inevitable because of X”, and it's therefore better that their in-group (their company or country) build it first. X is typically some combination of “Capitalism”, “Molloch”, “lack of regulation” and “China”.
Camp B) “Don’t race to superintelligence”: People in this group typically argue that “racing to superintelligence is bad because of Y”. Here Y is typically some combination of “uncontrollable”, “1984”, “disempowerment” and “extinction”.
Whereas the 2023 extinction statement was widely signed by both Camp B and Camp A (including Dario Amodei, Demis Hassabis and Sam Altman), the 2025 superintelligence statement conveniently separates the two groups – for example, I personally offered all US Frontier AI CEO’s to sign, and none chose to do so. However, it would be oversimplified to claim that frontier AI corporate funding predicts camp membership – for example, someone from one of the top companies recently told me that he'd sign the 2025 statement were it not for fear of how it would impact him professionally.
The distinction between Camps A and B is also interesting because it correlates with policy recommendations: Camp A tends to support corporate self-regulation and voluntary commitments without strong and legally binding safety standards akin to those in force for pharmaceuticals, aircraft, restaurants and most other industries. In contrast, Camp B tends to support such binding standards, akin to those of the FDA (which can be viewed as a strict ban on releasing medicines that haven’t yet undergone clinical trials and been safety-approved by independent experts). Combined with market forces, this would naturally lead to new powerful yet controllable AI tools, to do science, cure diseases, increase productivity and even aspire for dominance (economic and military) if that’s desired – but not full superintelligence until it can be devised to meet the agreed-upon safety standards – and it remains controversial whether this is even possible.
In my experience, most people (including top decision-makers) are currently unaware of the distinction between A and B and have an oversimplified view: You’re either for AI or against it. I’m often asked: “Do you want to accelerate or decelerate? Are you a boomer or a doomer?” To facilitate a meaningful and constructive societal conversation about AI policy, I believe that it will be hugely helpful to increase public awareness of the differing visions of Camps A and B. Creating such awareness was a key goal of the 2025 superintelligence statement. So if you’ve read this far, I’d strongly encourage you to read it and, if you agree with it, sign it and share it. If you work for a company and worry about blowback from signing, please email me at mtegmark@gmail.com and say "I'll sign this if N others from my company do", where N=5, 10 or whatever number you're comfortable with.
Finally, please let me provide an important clarification about the 2025 statement. Many have asked me why it doesn't define its terms as carefully as a law would require. Our idea is that detailed questions about how to word laws and safety standards should be tackled later, once the political will has formed to ban unsafe/unwanted superintelligence. This is analogous to how detailed wording of laws against child pornography (who counts as a child, what counts as pornography, etc.) got worked out by experts and legislators only after there was broad agreement that we needed some sort of ban on child pornography.