# User: Max Harms
Profile URL (HTML): [/users/max-harms](/users/max-harms)
Profile URL (Markdown): [/api/user/max-harms](/api/user/max-harms)
* Karma: 2419
* Alignment Forum karma: 331
* Posts: 21
* Comments: 180
* Member since: 2024-05-15 16:06:14Z
Bio
---
Also known as Raelifin: https://www.lesswrong.com/users/raelifin
Top Posts
---------
### [Contra Collier on IABIED](/api/post/contra-collier-on-iabied)
By [Max Harms](/users/max-harms)
2025-09-20 15:55:06Z
* Karma: 235
* Tags: [IABIED](/w/iabied-1), [AI](/w/ai), [World Modeling](/w/world-modeling) (Frontpage)
Read more: [/api/post/contra-collier-on-iabied](/api/post/contra-collier-on-iabied)
### [Thoughts on AI 2027](/api/post/thoughts-on-ai-2027)
By [Max Harms](/users/max-harms)
2025-04-09 21:26:23Z
* Karma: 224
* Linkpost: [https://intelligence.org/2025/04/09/thoughts-on-ai-2027/](https://intelligence.org/2025/04/09/thoughts-on-ai-2027/)
* Tags: [AI Timelines](/w/ai-timelines), [AI](/w/ai) (Frontpage)
Read more: [/api/post/thoughts-on-ai-2027](/api/post/thoughts-on-ai-2027)
### [0\. CAST: Corrigibility as Singular Target](/api/post/0-cast-corrigibility-as-singular-target-1)
By [Max Harms](/users/max-harms)
2024-06-07 22:29:12Z
* Karma: 171
* Curated
* Tags: [Corrigibility](/w/corrigibility-1), [AI](/w/ai) (Frontpage)
Read more: [/api/post/0-cast-corrigibility-as-singular-target-1](/api/post/0-cast-corrigibility-as-singular-target-1)
Recent Posts
------------
### [Corrigibility Prizes for Existing Work](/api/post/corrigibility-prizes-for-existing-work)
By [Max Harms](/users/max-harms)
2026-09-30 16:08:40Z
* Karma: 102
* Tags: [Corrigibility](/w/corrigibility-1), [AI](/w/ai) (Personal Blog)
Read more: [/api/post/corrigibility-prizes-for-existing-work](/api/post/corrigibility-prizes-for-existing-work)
### [A Defense of Gradual Disempowerment](/api/post/a-defense-of-gradual-disempowerment)
By [Max Harms](/users/max-harms)
2026-09-17 21:04:05Z
* Karma: 23
* Tags: [AI](/w/ai) (Frontpage)
Read more: [/api/post/a-defense-of-gradual-disempowerment](/api/post/a-defense-of-gradual-disempowerment)
### [Corrigibility Research Fund Grantees (Round 1)](/api/post/corrigibility-research-fund-grantees-round-1)
By [Max Harms](/users/max-harms)
2026-09-03 20:58:44Z
* Karma: 29
* Tags: [AI](/w/ai) (Personal Blog)
Read more: [/api/post/corrigibility-research-fund-grantees-round-1](/api/post/corrigibility-research-fund-grantees-round-1)
### [How I'm Evaluating Corrigibility Grant Applications](/api/post/how-i-m-evaluating-corrigibility-grant-applications)
By [Max Harms](/users/max-harms)
2026-09-03 20:58:35Z
* Karma: 51
* Tags: [AI](/w/ai) (Personal Blog)
Read more: [/api/post/how-i-m-evaluating-corrigibility-grant-applications](/api/post/how-i-m-evaluating-corrigibility-grant-applications)
### [Announcing the Corrigibility Research Fund](/api/post/announcing-the-corrigibility-research-fund)
By [Max Harms](/users/max-harms)
2026-07-17 18:06:32Z
* Karma: 112
* Tags: [Corrigibility](/w/corrigibility-1), [Grants & Fundraising Opportunities](/w/grants-and-fundraising-opportunities), [AI](/w/ai) (Frontpage)
Read more: [/api/post/announcing-the-corrigibility-research-fund](/api/post/announcing-the-corrigibility-research-fund)
### [P(doom) is a Dumb Meme](/api/post/p-doom-is-a-dumb-meme)
By [Max Harms](/users/max-harms)
2026-06-29 15:12:55Z
* Karma: 135
* Tags: [AI](/w/ai) (Frontpage)
Read more: [/api/post/p-doom-is-a-dumb-meme](/api/post/p-doom-is-a-dumb-meme)
### [Bentham’s Bulldog is wrong about AI risk](/api/post/bentham-s-bulldog-is-wrong-about-ai-risk)
By [Max Harms](/users/max-harms)
2026-01-29 16:33:15Z
* Karma: 110
* Tags: [AI](/w/ai) (Frontpage)
Read more: [/api/post/bentham-s-bulldog-is-wrong-about-ai-risk](/api/post/bentham-s-bulldog-is-wrong-about-ai-risk)
### [Serious Flaws in CAST](/api/post/serious-flaws-in-cast)
By [Max Harms](/users/max-harms)
2025-11-19 17:27:23Z
* Karma: 113
* Tags: [Corrigibility](/w/corrigibility-1), [AI](/w/ai) (Frontpage)
Read more: [/api/post/serious-flaws-in-cast](/api/post/serious-flaws-in-cast)
### [AI Corrigibility Debate: Max Harms vs. Jeremy Gillen](/api/post/ai-corrigibility-debate-max-harms-vs-jeremy-gillen)
By [Liron](/users/liron) with [Max Harms](/users/max-harms), [Jeremy Gillen](/users/jeremy-gillen)
2025-11-14 04:09:16Z
* Karma: 45
* Linkpost: [https://doomdebates.com/p/the-ai-corrigibility-debate-miri](https://doomdebates.com/p/the-ai-corrigibility-debate-miri)
* Tags: [Corrigibility](/w/corrigibility-1), [AI](/w/ai) (Frontpage)
Read more: [/api/post/ai-corrigibility-debate-max-harms-vs-jeremy-gillen](/api/post/ai-corrigibility-debate-max-harms-vs-jeremy-gillen)
### [Worlds Where Iterative Design Succeeds?](/api/post/worlds-where-iterative-design-succeeds)
By [Max Harms](/users/max-harms)
2025-10-23 22:14:21Z
* Karma: 23
* Tags: [AI](/w/ai) (Frontpage)
Read more: [/api/post/worlds-where-iterative-design-succeeds](/api/post/worlds-where-iterative-design-succeeds)
Recent Comments
---------------
### Comment by [Max Harms](/users/max-harms) on [Deconfusing Corrigibility](/api/post/deconfusing-corrigibility)
* 2026-10-20 22:24:10Z
* Karma: 18
* Total votes: 6
* Comment URL (Markdown): [/api/post/deconfusing-corrigibility/comments/gXpbSRKDpad2kt4xH](/api/post/deconfusing-corrigibility/comments/gXpbSRKDpad2kt4xH)
* Comment URL (HTML): [/posts/f8R3B2zqqTJZjSAsm/deconfusing-corrigibility/comment/gXpbSRKDpad2kt4xH](/posts/f8R3B2zqqTJZjSAsm/deconfusing-corrigibility/comment/gXpbSRKDpad2kt4xH)
This is a truly exceptional piece of work! Thank you for writing it! I think you taught me some things about corrigibility, and I'll be spending many coming days engaging with the ideas of the post. Many of those ideas will produce questions, comments, and criticisms. But I wanted to start off with a wholehearted cheer for the combination of philosophical weight, relevance, and rigor.
"Respect for authority" is a particularly good handle, and I'm a bit embarrassed to have never used it! (At least as far as I can recall.)
### Comment by [Max Harms](/users/max-harms) on [Memes, not agents, are what will kill us](/api/post/memes-not-agents-are-what-will-kill-us)
* 2026-10-19 18:42:14Z
* Karma: 5
* Total votes: 3
* Comment URL (Markdown): [/api/post/memes-not-agents-are-what-will-kill-us/comments/Mafkua8nbApvAc3sn](/api/post/memes-not-agents-are-what-will-kill-us/comments/Mafkua8nbApvAc3sn)
* Comment URL (HTML): [/posts/hvLm4zRgzfRLH6H23/memes-not-agents-are-what-will-kill-us/comment/Mafkua8nbApvAc3sn](/posts/hvLm4zRgzfRLH6H23/memes-not-agents-are-what-will-kill-us/comment/Mafkua8nbApvAc3sn)
I happily admit to being confused here. There's more agent foundations work to be done. What you say seems reasonable to me. Perhaps the right frame is a pragmatic question, and that there's no right answer about whether any particular thing counts as agent or environment.
I think my point was mostly that Florian seems to be outlining a trichotomy and that agent/environment seems dichotomous. If we want to start listing different places to put the distinction, I think we'll get at least a dozen different answers, not two (2 dividing places produces three realms, which is one way to interpret the suggestion of a trichotomy). So I don't see why the trichotomy is a natural way to think.
### Comment by [Max Harms](/users/max-harms) on [Memes, not agents, are what will kill us](/api/post/memes-not-agents-are-what-will-kill-us)
* 2026-10-19 16:26:26Z
* Karma: 12
* Total votes: 5
* Comment URL (Markdown): [/api/post/memes-not-agents-are-what-will-kill-us/comments/rHm6cexdNkvmiGaWr](/api/post/memes-not-agents-are-what-will-kill-us/comments/rHm6cexdNkvmiGaWr)
* Comment URL (HTML): [/posts/hvLm4zRgzfRLH6H23/memes-not-agents-are-what-will-kill-us/comment/rHm6cexdNkvmiGaWr](/posts/hvLm4zRgzfRLH6H23/memes-not-agents-are-what-will-kill-us/comment/rHm6cexdNkvmiGaWr)
I think your ontology is a little weird. When I think about agency, the most natural frame is that of the Cartesian agent, which has internal machinery (eg weights) and external environment. The "user prompt", from this perspective, is part of the environment, and it's a stimulus that the agent needs to grapple with, alongside the other stimuli from, eg, the web or other agents.
Setting aside the question of whether the prompt is important to conceptually distinguish, I think most of what you're noticing here is the risks of distributional shifts and subagent problems, both of which have been central concerns in the field for over a decade. These are important problems, but I think this post treats the insight as too novel, and should connect to previous literature, such as Tiling Agents (2013), Corrigibility (2015), and Concrete Problems in AI Safety (2016).
The two most valuable things I think you're identifying are the way in which swarm dynamics differ from the frame of a single agent choosing a single successor, and the way in which we don't really understand the sorts of environmental pressures that swarms produce. It would be good if labs started trying to harden their models against swarm contexts, though I fear that's a game of whack-a-mole that won't get far without fundamental improvements in alignment theory.
While I agree that it's scary to imagine one or more closed-weight frontier models delegating to one or more open-weight models and thus losing important alignment properties and creating a rogue botnet, I think it's also worth worrying about such models hacking their parent company and covertly spinning up instances to join the swarm. It's not obvious to me which one is of more practical concern, and we shouldn't assume that swarms will become deeply heterogeneous in the future. (Most ant colonies aren't, for example. (Though [some are](https://www.scientificamerican.com/article/natural-and-artificial-mixed-coloni/)!))
### Comment by [Max Harms](/users/max-harms) on [Two Axes of Alignment: A Framework for Robust Superintelligence Alignment](/api/post/two-axes-of-alignment-a-framework-for-robust)
* 2026-10-16 20:37:02Z
* Karma: 4
* Total votes: 3
* Comment URL (Markdown): [/api/post/two-axes-of-alignment-a-framework-for-robust/comments/RhA84gDzeQacy5FKk](/api/post/two-axes-of-alignment-a-framework-for-robust/comments/RhA84gDzeQacy5FKk)
* Comment URL (HTML): [/posts/TaZWaPEish9R9y8D7/two-axes-of-alignment-a-framework-for-robust/comment/RhA84gDzeQacy5FKk](/posts/TaZWaPEish9R9y8D7/two-axes-of-alignment-a-framework-for-robust/comment/RhA84gDzeQacy5FKk)
I'm a MIRI guy, and think agent foundations is broadly underrated, so I'm pretty on board with your general perspective. But, given that, it feels like you're making two errors here:
* The concepts here feel too much like dichotomies that don't necessarily capture the complexity of the landscape. For example, the line between ambitious and pragmatic mech interp is pretty blurry. Much of agent foundations is meant to deconfuse about the nature of intelligence and agency, which could be seen as "backchaining" only insofar as fundamental science is useful for getting to the ultimate finish line. Is forward/back a better lens than empirical vs theoretic? Do these labels add things beyond the argument that we should do theoretical work because otherwise we're stumbling along in the dark, hoping our messy safety-relevant properties continue to hold?
* My inner Paul Christiano wants to object that even if the ultimate superintelligence can defeat your current methods, you aren't actually defending against the ultimate superintelligence -- you're defending against what currently exists and what's about to exist. If your methods robustly work for the near-future, and your skill grows as you approach superintelligence such that your methods continue to robustly work, then you can rely on an entirely forward-chaining, extrapolative regime and be fine. As a metaphor, let's say you're tracing out a labyrinth where touching the walls kills you. It's still a reasonable strategy to walk around and explore. "We can't go North! What if there's eventually a wall that way?" doesn't hold water because if there is a wall that way, you can choose not to touch it, and in the process of going North you may learn a lot about better ways to go. For your argument to be correct, failures have to be unpredictable and deadly enough to prevent humanity from adapting from them and moving forward.
Still, I think you're broadly right, and I appreciate your writeup. Thanks!
### Comment by [Max Harms](/users/max-harms) on [Swarm Scaling](/api/post/swarm-scaling)
* 2026-09-25 21:08:23Z
* Karma: 10
* Total votes: 5
* Comment URL (Markdown): [/api/post/swarm-scaling/comments/bgZckaZBndwrNvAdJ](/api/post/swarm-scaling/comments/bgZckaZBndwrNvAdJ)
* Comment URL (HTML): [/posts/6cb7qd3RSkgnviCpf/swarm-scaling/comment/bgZckaZBndwrNvAdJ](/posts/6cb7qd3RSkgnviCpf/swarm-scaling/comment/bgZckaZBndwrNvAdJ)
Doesn't the context length matter a lot, here? Like, in the tables you gave above, none of the agents seem to be hitting the limits of the context window, and thus triggering compaction, whereas in the 130 billion output tokens for Navier-Stokes, a single agent would've been dominated by its compaction mechanism. Thus, even setting aside the speedup, it seems plausible to me that the lambda parameter changes quite a lot based on task size. On a very hard problem requiring billions of output tokens, it seems plausible that swarms have an edge. (Or not! I don't know. My point is that I think extrapolating from ~100k output token cases might not work.)
### Comment by [Max Harms](/users/max-harms) on [A Defense of Gradual Disempowerment](/api/post/a-defense-of-gradual-disempowerment)
* 2026-09-18 01:35:09Z
* Karma: 2
* Total votes: 1
* Comment URL (Markdown): [/api/post/a-defense-of-gradual-disempowerment/comments/Tn8vYDLiMD9xxaiEL](/api/post/a-defense-of-gradual-disempowerment/comments/Tn8vYDLiMD9xxaiEL)
* Comment URL (HTML): [/posts/jXBmrEQj7zKGgiaYh/a-defense-of-gradual-disempowerment/comment/Tn8vYDLiMD9xxaiEL](/posts/jXBmrEQj7zKGgiaYh/a-defense-of-gradual-disempowerment/comment/Tn8vYDLiMD9xxaiEL)
Ah, that's fair! I thought you were arguing that it's not clear that they think it's instrumentally bad, which it sounds like was not in doubt. Sorry for getting that wrong. I'll edit the post to clarify.
### Comment by [Max Harms](/users/max-harms) on [Defining Corrigible and Useful Goals](/api/post/defining-corrigible-and-useful-goals)
* 2026-09-10 20:43:31Z
* Karma: 2
* Total votes: 1
* Comment URL (Markdown): [/api/post/defining-corrigible-and-useful-goals/comments/ThjHohkbYv5hXZRgD](/api/post/defining-corrigible-and-useful-goals/comments/ThjHohkbYv5hXZRgD)
* Comment URL (HTML): [/posts/HLns982j8iTn7d2km/defining-corrigible-and-useful-goals/comment/ThjHohkbYv5hXZRgD](/posts/HLns982j8iTn7d2km/defining-corrigible-and-useful-goals/comment/ThjHohkbYv5hXZRgD)
Maybe we should wait until the call, but I'm still confused about the timing. What's the relevant window of the prediction?
Like, suppose the AI is in a robot and decides to walk towards the shuttlecraft to travel to a lava planet where there's a 10% chance in the next year that it will accidentally melt and thus the shutdown mechanism will be disabled (along with the rest of the robot/AI). Does it get shut down before it takes the first step? If not, what if it's walking towards someone who intends to disable the shutdown mechanism (but not the rest of the machine) in the next 5 minutes? How should I be modeling this prediction mechanism's scope?
Edit: Also, I agree this is a tangent, and it might be good to abandon it as not-cruxy. The more substantive question is whether the transformation is robust to the base model being clever and situationally aware. I still feel like if you tried to do a corrigibility transformation on an uploaded copy of me, I'd be able to wriggle out of it pretty easily.
### Comment by [Max Harms](/users/max-harms) on [Defining Corrigible and Useful Goals](/api/post/defining-corrigible-and-useful-goals)
* 2026-09-09 16:55:11Z
* Karma: 2
* Total votes: 1
* Comment URL (Markdown): [/api/post/defining-corrigible-and-useful-goals/comments/9yhuybroidJaYEMpg](/api/post/defining-corrigible-and-useful-goals/comments/9yhuybroidJaYEMpg)
* Comment URL (HTML): [/posts/HLns982j8iTn7d2km/defining-corrigible-and-useful-goals/comment/9yhuybroidJaYEMpg](/posts/HLns982j8iTn7d2km/defining-corrigible-and-useful-goals/comment/9yhuybroidJaYEMpg)
How does time/causality work in the predictor add-on? (I claim "add-on" is fair, even if it's just adding an additional head/output channel.) Like, if the agent's action at t=0 doesn't change the designated shutdown setup, but then (because of that action) at t=1 the designated channels predictably get disabled regardless of what the agent does, does the prediction fire at t=0 or t=1?
I'm right that none of this is in the paper, right?
Indeed, I feel like most of what you're saying doesn't address my concerns. (But I do think you're trying!) Would you like to try scheduling a call? I could write more words if you prefer slower/async/public, but my sense right now is that it'll be more efficient to have rapid feedback in navigating the disconnect.
### Navigation
* [Front page](https://www.lesswrong.com/api/home)
* [Markdown API documentation](https://www.lesswrong.com/api/SKILL.md)