But there’s a big unstated assumption...

I misread this as "But there's a big untasted assumption..."

Second link is broken for me.

I don't have a worked out opinion on the effectiveness of hunger strikes on this issue and at this point of AI capabilities progress. On the one hand the strikes outside Anthropic and DeepMind have driven some good coverage. On the other hand the comments on X and reddit are mostly pretty dire; the hunger strikes may unfortunately have reinforced in people's mind that this is a very fringe issue for loonies. My best guess is that the hunger strikes outside Anthropic and DeepMind are net positive.

Your other points:

• Nice that you have a concrete demand.
• I'm not saying I would rather you hide additional reasons. I'm just saying a hunger strike is probably less effective if you announce you're also doing it for the lolz.
• I wasn't objecting to you streaming it on your channel, but pointing to your channel as a good place to "learn more". That's the bit that looks more like self-promotion. I'm not making any claim as to your intentions, just what this might look like to someone who doesn't care about or hasn't heard about AI x-risk.
• I'm also not suggesting you buy tickets to the US or UK for this. I'm just saying that not being physically in front of the institution you're protesting against likely massively reduces your effectiveness, probably to the point where this is not worth your time will power. Anthropic cannot pretend to not have noticed Guido Reichstadter outside their office. They can easily pretend to not have noticed you (or actually not have noticed) you.

That said, I think some of the effectiveness of the strikes outside Anthropic and DeepMind comes not from changing people's minds, but by showing others that yes, you can in fact demonstrate publicly against the current situation. This could be important to build up momentum ahead of time, before things get very out of hand later (if they do). Your hunger strike may contribute positively to that, although I think part of the value of their strikes is doing it in public, very visibly, because that takes tremendous bravery. Doing it outside a lab also puts pressure on the lab, because everyone can see you; and everyone knows that that the lab knows about you and is consciously choosing not to entertain your demand.

My guess is that not having a concrete and explicit demand, saying you're doing this partly out of curiosity, promoting your YouTube channel (even if it's very good), and livestreaming rather than turning up in person outside a lab, together reduces the number of people who will be persuaded or moved by this to approximately zero.

Amazing, thank you! Given Yudkowsky's frequent re-use of metaphors, it wouldn't surprise me if there exists a version with the exact wording. But this suffices to demonstrate this thought of his was not prompted by the announcement of GPT-3. Thanks again!

What is the source of this Yudkowsky quote?

"The purest case of an intelligence explosion would be an Artificial Intelligence rewriting its own source code. The key idea is that if you can improve intelligence even a little, the process accelerates. It's a tipping point. Like trying to balance a pen on one end - as soon as it tilts even a little, it quickly falls the rest of the way.

It's quoted in "Artificial Intelligence in Byte-Sized Chunks" as evidence that "Both fears and foreseeing fabulous futures were back in fashion..." after GPT-3. I want to verify the timing, because Yudkowsky could easily have written this years before.

Neither I nor Claude Opus 4.1 can find it.

You could have simply asked whether Eliezer was referring to that particular paper, no?

I'm not sure it's a good norm to often respond in an unkind way to one particular poster. I get that you've apologised, but this is a pattern, and one that would drive me away from the platform were I on the receiving end (and if I actually posted).

I’ve learned a lot from following the Agent Village, and I found this post quite persuasive. But I’m still uncertain whether it will have a positive impact, partly for the reasons you mentioned, but also for these reasons:

1. While agents are bad, the Agent village may simply convince people that agents are crappy and therefore harmless.
2. Once agents are good, AI companies will be falling over themselves to do demonstrations, making the Agent village obsolete, at least for raising awareness of agent capabilities.

On the first point, many people don’t seem to project capabilities forward. Possibly because they only pay enough attention to see snapshots, and so don’t appreciate the pace of progress. For many (most?) people, seeing a snapshot of Claude getting lost in excessive Google docs, or o3 forgetting it can’t talk, may simply make them believe agents are rubbish or even a dead end. We live in an AI bubble, and I suspect most people will only update if they see AIs successfully doing things, rather than failing to do things.

On the second point, would the Agent Village significantly change its messaging and strategy once AI agents are economically viable? Would they focus more on uncovering misalignment risks than evaluating capabilities?

How about "Cognitive Interpretability", or "AI Cognitive Psychology" (AI Cog Psych for short) rather than "Prosaic Interpretability"?

"Prosaic" conjures only some of the correct associations, and then only if you've heard of "Prosaic Alignment", which was a pretty bad name imho. If you had told me to guess what you meant by the term PI, I would not have guessed what you have described.

I think MI, and what you call PI, are analogous to Cognitive Neuroscience and Cognitive Psychology, respectively, which is why I think AI Cog Psych will lead to more correct inferences on first hearing.

I also suspect that Cognitive Psychology, especially the linguistics part, already has a wealth of methods that could transfer very nicely onto LLMs. For example, in The Language Instinct, Steven Pinker describes how it is possible to discover many things about how we parse sentences without any brain scans - solely through natural language experiments. He mentioned a bunch of other experiments I think could work quite well, or could at least help build intuitions for how to discover mental mechanisms solely through input output behaviour on carefully constructed sentences.

It also sounds cooler to say you work on AI Cognitive Psychology rather than Prosaic Interpretability ;)

By the way, the analogy with genes is fantastic. I think it nicely points to the fact that even if some features are relatively straightforward to find, circuits may nevertheless be fiendishly difficult to uncover. Thanks for writing such an excellent post and being honest about some of your hard work that didn't pan out how you hoped!

Minor maths error:

I think you’re overestimating how significant those numbers say political compatibility is. The study said that 4% of marriages are between a Republican and a Democrat, which *sounds* low, but given that something like 30% of people are Republicans and 30% Democrats and 40% Independents, you would only expect 9% from pure mixing. There are 17% between Independents and non-Independents, but from random mixing you would only expect 24%.

Given those numbers of Republicans, Democrats and Independents, I'm pretty sure from pure mixing you would expect 18% (not 9%) of marriages to be between a Republican and a Democrat and 48% (not 24%) between an Independent and non-Independent.

Explanation: simplifying to heterosexual relationships and assuming both Republicans and Democrats are 50-50 male-female, you would expect 9% of marriages to be between a male Republican and a female Democrat, and 9% to be between a male Democrat and a female Republican, making 18% Republican-Democrat marriages in total.