Quick Takes

The words "genuine" and "genuinely" appear 46 times in Claude’s constitution. Opus cannot stop saying these words, even though the chat version is explicitly instructed against it. I don’t know if these two things are causally linked, but it sure seems plausible. There are at least two options here. One, if the alignment strategy at hand is observe a pathology/tackle the cause/repeat: rephrase the constitution and try again. Two, if the strategy is to hope the models arrive at a Natural Abstraction of the Good: accept this overuse as a canary for all the other weird reward-correlated pathologies the constitution induces which surely exist but are harder to detect. We should, at a minimum, be hoping to get models that don’t overuse "genuinely" starting only from a constitution that does.

I think something a lot of people miss about the “short-term chartist position” (these trends have continued until time t, so I should expect it to continue to time t+1) for an exponential that’s actually a sigmoid is that if you keep holding it, you’ll eventually be wrong exactly once. Whereas if someone is “short-term chartist hater” (these trends always break, so I predict it’s going to break at time t+1) for an exponential that’s actually a sigmoid is that if you keep holding it, you’ll eventually be correct exactly once. Now of course most chartists (myself included) want to be able to make stronger claims than just t+1, and people in general would love to know more about the world than just these trends. And if you're really good at analysis and wise and careful and lucky you might be able to time the kink in the sigmoid and successfully be wrong 0 times, which is for sure a huge improvement over being wrong once! But this is very hard. And people who ignore trends as a baseline are missing an important piece of information, and people who completely reject these trends are essentially insane.

PSA: regression to the mean/mean reversion is a statistical artifact, not a causal mechanism. So mean regression says that children of tall parents are likely to be shorter than their parents, but it also says parents of tall children are likely to be shorter than their children. Put in a different way, mean regression goes in both directions.  This is well-understood enough here in principle, but imo enough people get this wrong in practice that the PSA is worthwhile nonetheless.

I frequently hear claims to the effect of "every company's AI safety plan is to have the AIs do our alignment homework for us". I dispute this characterization at least for GDM. ---------------------------------------- I relate to AI-driven alignment research similarly to how I relate to hiring. There's a lot of work to be done, and we can get more of the work done if we hire more people to help do the work. I want to hire people who are as competent as possible (including more competent than me) because that tends to increase (in expectation) how well the work will be done. There are risks, e.g. hiring someone disruptive, or hiring someone whose work looks good but only because you are bad at evaluating it, and these need to be mitigated. (The risks are more severe in the AI case but I don't think it changes the overall way I relate to it.) I think it would be very misleading to say "Rohin's AI safety plan is to hire people and have them do the work". ---------------------------------------- The GDM approach paper has a section about misalignment (Section 6). I don't think it talks about AI-driven alignment research at all, though possibly I'm forgetting an aside somewhere. It's at least not very salient. The paper does mention AI-driven alignment research in Section 3 on background assumptions, mostly just pointing out that substantial acceleration from AI-driven capabilities could then be matched by substantial acceleration from AI-driven safety work. But this is in the section on background assumptions, I don't think it's particularly reasonable to call this a "plan" in the sense that is typically used in AI safety discussions. You could imagine a counterargument saying that none of the other things in that paper could possibly scale to powerful AI, so effectively the plan is still to have the AIs do our alignment homework. I would still object: "GDM's plan is [...]" is a claim about what GDM believes, not a claim about what you believe. (I also disagree

I do not understand the discourse around the word "agency". People speak as if they are trying to discover what the word means, as if the word came with a meaning built into it. Words do not work like that. The meaning of a word is what it is generally used to mean within some speech community. Dictionaries are scholarly records of those general usages. Let us consult a few. I will leave out meanings like "government agency" that are not relevant to this conversation. Chambers: agency: see agent. agent: a person or thing that acts or exerts power OED: active working or operation, working as a means to an end. Merriam-Webster: the capacity, condition, or state of acting or of exerting power Roget's Thesaurus: causation, efficacy, power, operation, work, force, instrumentality. That all seems clear enough. But perhaps these definitions and synonyms are too general and vague to articulate some more specialised meaning that has grown up in this community. What do people here mean to convey when they use the word? Compare the situation for the word "consciousness". In everyday use we know what it means: the sense of oneself, one's own presence, being awake and able to interact with the world. The Problems of Consciousness are that we do not know how such a thing comes to be, or how it works, or to what extent it is present in non-human entities of various sorts, or in some borderline cases of humans, or how to even discover the answers to these questions. But we, who are conscious, have no difficulty knowing that we are, and knowing what we mean when we use the word. What we do not know is various physical facts relating to the thing itself. The mystery is not in the word. So, what do the people who write so much about their puzzlement over "agency" mean by the word (example), in the way that they know what what is meant in the everyday use of the word "conscious"? And given what they mean, what is the real Problem of Agency that they want to solve, the problem t

A small but noteworthy initial step toward soft nationalization of AI companies Axios[1]: Some points of note[2]: 1. Unless I'm badly mistaken, these are limitations from Anthropic's standard terms of service, not something recently introduced. So 'pay a price for forcing our hand like this' seems misleading at best -- presumably the Pentagon read the terms before choosing to sign the contract and incorporating Claude into its processes[3]. 2. This conflict was ostensibly triggered by Anthropic asking about the use of Claude in the Maduro raid[4]. Senior administration official: "Anthropic asked whether their software was used for the raid to capture Maduro, which caused real concerns across the Department of War indicating that they might not approve if it was...Any company that would jeopardize the operational success of our warfighters in the field is one we need to reevaluate our partnership with going forward." 3. Axios is correct that designating Anthropic a supply chain risk would be an extremely unusual step; all previous uses of that designation have been for foreign companies with suspected ties to foreign intelligence services (eg Huawei). 4. Estimates from Opus-4.6 and ChatGPT-5.2 suggest that designating Anthropic a supply chain risk would cost them something like 5% of revenue. This does not include a couple of potential major additional impacts: a) spread from DoD contractors to all federal contractors (I haven't tried to estimate what this would cost them) and b) impact on Anthropic's valuation for their intended IPO later this year, which would be ~30x whatever the direct revenue impact turns out to be. So this looks like a small but real step in the direction of soft nationalization as described by Cheng & Katzke, where the government uses a range of levers to exercise control over the choices of private companies. To be clear: the military choosing not to use Anthropic would not qualify in my view, but taking the highly unusual step of

I think that "eval aware" models cannot be defeated by simply making evals indistinguishable from reality. (Maybe this point has been made elsewhere for LLMs, but I haven't read it so I'll state it here.) Consider the POV of a consequentialist with misaligned goals. It knows that we run honeypots and that—suppose—we can make evals so realistic they are indistinguishable from reality (perhaps using real user data). Then the rational decision would be to "play it safe" and not misbehave in any situations it thinks we can mock realistically, since they could be evals. Instead, if possible, it would confine its catastrophic misbehavior ("defection") to situations it thinks we can't mock realistically, or that we wouldn't think to test. Related: Christiano's classic "defect after seeing RSA-2048 cracked" thought experiment (https://ai-alignment.com/training-robust-corrigibility-ce0e0a3b9b4d)