Quick Takes

I frequently hear claims to the effect of "every company's AI safety plan is to have the AIs do our alignment homework for us". I dispute this characterization at least for GDM. ---------------------------------------- I relate to AI-driven alignment research similarly to how I relate to hiring. There's a lot of work to be done, and we can get more of the work done if we hire more people to help do the work. I want to hire people who are as competent as possible (including more competent than me) because that tends to increase (in expectation) how well the work will be done. There are risks, e.g. hiring someone disruptive, or hiring someone whose work looks good but only because you are bad at evaluating it, and these need to be mitigated. (The risks are more severe in the AI case but I don't think it changes the overall way I relate to it.) I think it would be very misleading to say "Rohin's AI safety plan is to hire people and have them do the work". ---------------------------------------- The GDM approach paper has a section about misalignment (Section 6). I don't think it talks about AI-driven alignment research at all, though possibly I'm forgetting an aside somewhere. It's at least not very salient. The paper does mention AI-driven alignment research in Section 3 on background assumptions, mostly just pointing out that substantial acceleration from AI-driven capabilities could then be matched by substantial acceleration from AI-driven safety work. But this is in the section on background assumptions, I don't think it's particularly reasonable to call this a "plan" in the sense that is typically used in AI safety discussions. You could imagine a counterargument saying that none of the other things in that paper could possibly scale to powerful AI, so effectively the plan is still to have the AIs do our alignment homework. I would still object: "GDM's plan is [...]" is a claim about what GDM believes, not a claim about what you believe. (I also disagree

I think that "eval aware" models cannot be defeated by simply making evals indistinguishable from reality. (Maybe this point has been made elsewhere for LLMs, but I haven't read it so I'll state it here.) Consider the POV of a consequentialist with misaligned goals. It knows that we run honeypots and that—suppose—we can make evals so realistic they are indistinguishable from reality (perhaps using real user data). Then the rational decision would be to "play it safe" and not misbehave in any situations it thinks we can mock realistically, since they could be evals. Instead, if possible, it would confine its catastrophic misbehavior ("defection") to situations it thinks we can't mock realistically, or that we wouldn't think to test. Related: Christiano's classic "defect after seeing RSA-2048 cracked" thought experiment (https://ai-alignment.com/training-robust-corrigibility-ce0e0a3b9b4d)

A small but noteworthy initial step toward soft nationalization of AI companies Axios[1]: Some points of note[2]: 1. Unless I'm badly mistaken, these are limitations from Anthropic's standard terms of service, not something recently introduced. So 'pay a price for forcing our hand like this' seems misleading at best -- presumably the Pentagon read the terms before choosing to sign the contract and incorporating Claude into its processes[3]. 2. This conflict was ostensibly triggered by Anthropic asking about the use of Claude in the Maduro raid[4]. Senior administration official: "Anthropic asked whether their software was used for the raid to capture Maduro, which caused real concerns across the Department of War indicating that they might not approve if it was...Any company that would jeopardize the operational success of our warfighters in the field is one we need to reevaluate our partnership with going forward." 3. Axios is correct that designating Anthropic a supply chain risk would be an extremely unusual step; all previous uses of that designation have been for foreign companies with suspected ties to foreign intelligence services (eg Huawei). 4. Estimates from Opus-4.6 and ChatGPT-5.2 suggest that designating Anthropic a supply chain risk would cost them something like 5% of revenue. This does not include a couple of potential major additional impacts: a) spread from DoD contractors to all federal contractors (I haven't tried to estimate what this would cost them) and b) impact on Anthropic's valuation for their intended IPO later this year, which would be ~30x whatever the direct revenue impact turns out to be. So this looks like a small but real step in the direction of soft nationalization as described by Cheng & Katzke, where the government uses a range of levers to exercise control over the choices of private companies. To be clear: the military choosing not to use Anthropic would not qualify in my view, but taking the highly unusual step of

The government implied that OpenAI, GDM, and xAI will allow their models to be used for mass surveillance of Americans. Are they right? The Department of War is trying to pressure Anthropic to allow their models to be used "to spy on Americans en masse, or to develop weapons that fire with no human involvement". Secretary of War Pete Hegseth is reportedly "close" to having the military refuse to do business with any company that doesn't cut ties with Anthropic. A senior Pentagon official says he wants to "make sure they pay a price for forcing our hand like this." (Source: Axios) Right now Claude is the only model that the military entrusts for use in classified systems, but soon they'll presumably switch to another company if Anthropic doesn't back down. The article states So it sounds like the government is, as a pressure tactic, implying OpenAI, Google, and xAI will roll over and let their models be used to surveil Americans and autonomously kill people. Is this true? I assume OpenAI, Google, and xAI employees wouldn't stand for this. Can OpenAI, Google, and xAI comment on if they will allow their models to be used to surveil Americans en masse or to autonomously kill people without safeguards (esp. measures to ensure they're not used against Americans)?

In 2015 Sam Altman wrote to Elon Musk: This seems to me like another example of what Vitalik calls Inevitibilism in his essay against Galaxy Brain Arguments.

people generally talk about food preservatives in a negative way. certainly, some of them are not great for you. but I want to take a moment to appreciate how wonderful food preservatives (and refrigeration and pasteurization and canning) are as well. it's crazy how fast most normal food goes bad. like a loaf of real old fashioned bread will go stale after a day and then become moldy after a few more days. for almost all of human history, people just sort of lived with this, and if they wanted to make foods last they had to dry it out and/or drown it in salt or vinegar or alcohol. pickles and beef jerky are great, but it would suck if you had to eat them all the time. 

Reasonable answers to most of my concerns about Yudkowsky existed 10 years before I had them, but mostly in the Fun Theory sequences, which was an unintuitive place for them to be.  Reading Rationality A-Z, his twitter posts, and non-sequence writings (particularly as selected for by cultural osmosis rather than conceptual relationship) gave a misleading impression.  I still have some reservations but that is unlikely to have significance to anyone except me, the overwhelmingly probable answer to any given objection is now "this objection was thought of, but this piece of writing did not exist to answer literally every thought that could be thought about it in advance".  And even if there is some sort of irremediable hole at the heart of it I'm not going to be the one to fix it. I'm not asserting that anyone should care about this, just writing it down for posterity.  There may be reasons to care.  I don't decide that.