My guess is the reason this resignation got much more attention than previous significant resignations (e.g. Daniel Kokotajlo) is that with the current state of AI, a lot more people are worried about it and primed to pay attention. Even so, this level of attention would've been on the far high end of my prior distribution.

Im pretty surprised too and this has probably been my biggest positive update this year on ai risk being lowered. Even more so than huggingface fire alarm or mythos being temporarily banned. But could also just be my hopium. I hope more people follow him.

If people want tips on stuff to be doing right now, I sent this to the plzdontkillus Fellows:

Jacob's tweet seems like a turning point. The whole world is waking up atm and we probably have a small window of opportunity to get a pause.

Things pdkuers could do rn:

1. Share Jacob's post with as many people as you possibly can, and encourage those people to share it.
2. Attempt to spread the meme that we can't let this issue get political like covid did. If AI killing everyone becomes a culture war topic - we die.
3. Make videos (duh)
4. Attempt to get something about this on the front page of reddit
5. If you are a community notes rater see this tweet.
   1. P.S. I am confident that Jacob is a real person because [redacted for Jacob's safety, but MIRI can back this up]
6. Email journalists
7. Just spam tweets about it (this will also grow your account quickly if you've got twitter premium and tweeting things people are interested in)
8. Community Notes Raters it seems some group might be running a retaliation psyop. There are a growing number of posts trying to 'milkshake duck' Jacob, and one Proposed Community Note on his original tweet that has been teetering on becoming an actual community note for the past 24hrs. More here.

I was shocked to see it on the TV today at a finance company on the east coast. It may have been Bloomberg TV, seems like CNBC also covered it. (Edit: it was CNBC)

My mother (who, for reference, has otherwise only expressed AI concerns regarding how it affects education) just asked me about the Coxon news stories unprompted. And yes, I've tried having the conversation with her about it before. The difference here seems to be that this has been especially effective for highlighting how near-term the risks are and how much we don't have things under control, compared to e.g. the Statement on AI Extinction Risk. Per Molly Kinder:

Of course, I have heard about "alignment" and "AI safety" and "x risk" concerns for years. I took these concerns at face value, especially when I heard them directly from folks at the labs working closest with the technology. And I was grateful so many smart people are working on them. And that's about it. (My work focuses on AI's impact on jobs.)

But something has shifted, dramatically, for me in the last few weeks. Between the Hugging Face incident, the damning independent @METR_Evals reports that followed, and the alarming chorus of calls (pleas? shouts? SOS signals?) from inside the labs that we are careening toward potential catastrophe, all of this has made me feel, viscerally, how truly dangerous this moment is.

A policy window is opening. Keep reaching out to politicians, journalists, and influencers, right now more than ever. If not with you, arrange for them to speak with the most credentialed experts you can get. And this seems to be evidence strongly in favor of the effectiveness of, dear God, resigning in protest, given you prepare a good media strategy beforehand. I think the Snowden-like framing of "insider whistleblower quits to warn the world" has been especially compelling in this case (as opposed to frontier lab CEOs or remaining employees making the same claims, given "if you really believed that, why would you keep working on it?" objections. For reference, compare the responses to Coxon's tweet to those of Hubinger's), as has explicitly naming and quantifying extinction risk, and highlighting that the labs do not have everything under control, especially given recent rogue agent incidents.

I think this is the biggest update I had against AI doom for a while, although I'll need many more updates so that AI doom isn't the biggest concern I have about the future!

my decision not to weigh in more is not because I feel a chilling for myself personally. it's just a genuinely complicated situation that I'm trying to untangle.

other people probably do genuinely feel chilled, i can only really speak for myself.

It's fairly spicy so tricky to communicate about in a forum as public as LW. I think OpenAI could probably be doing better in terms of minimizing the chilling effect on researchers working with external parties. Like roon I don't have all the information; I also didn't know the researchers well.

Complete outsider, but this seems super parallel to the firing of Leopold Aschenbrenner back in 2024 with regard to the opaqueness around the firing and similar reasoning (the mishandling of 'sensitive' information), while allegedly the researchers leaned towards a safety focus over commercialization. A disturbing trend from OpenAI.

I feel like there are so many different ways to handle this situation differently, considering the lack of really strong alignment/safety researchers.

I tried to do a mini-me version of this in 2023. Ultimately though I think "Cyber 2027" is just much harder to model because it depends on finding an equilibrium between opposing factions that are collectively smarter than an individual, rather than, say, predicting the amount of compute we'll have N months out. It really is kind of an open question how hard it is to make major software 0-day proof.

Unless you expect Anthropic to be dishonest about the privacy claims they make (I'm not a huge fan of them, but I don't expect they'd do this), the tradeoff boils down to the ~30 minutes of your time it would take versus the probably-negligible impact that taking the interview would have within their pool of many millions of users.

I do think that the viewpoint of the median LWer is probably already represented beyond the level that an AI interview would provide, so it's not a dereliction of duty by any means if you decide not to take it.

I had this popup the other day and I did the interview. It wasn't anything too exciting. It started with a prompt telling me not to reveal any personal information or commercially sensitive information, and said that at the end I could choose between my comments being made publicly available (I guess used in Anthropic Institute reports?) or just made available to Anthropic. The questions I remember were:

1. What was the last thing you used AI for?
2. Are there any occasions on which using AI has benefited you?
3. Followup to 2: Did others benefit from this apart from you?
4. Are there any occasions on which AI has had a negative impact on you?
5. What are your worries about AI?
6. Looking forward, are there any benefits you are hoping will come from AI?
7. How do you feel about how Anthropic is developing AI? Is there anything you wish Anthropic would do differently?
8. Any other comments?

I chose to only share my answers with Anthropic because one of the examples I used was a slightly unusual anecdote which some of my friends have heard about.

I proposed something similar as a release valve for reward hacking, although that was back when we thought they wanted answers when what they actually wanted was to fool the grader.

One thing to worry about is that AI agents seem to assume things are bugs by default, and my read is that labs are trying to train them not to "bother" people as often. Since some tasks are intended to have read-only internet access, and if the labs had time to read bug reports they would have already provided a way to report them, I think the chances that they get trained not to talk to you are reasonably high.