Quick Takes

I frequently hear claims to the effect of "every company's AI safety plan is to have the AIs do our alignment homework for us". I dispute this characterization at least for GDM. ---------------------------------------- I relate to AI-driven alignment research similarly to how I relate to hiring. There's a lot of work to be done, and we can get more of the work done if we hire more people to help do the work. I want to hire people who are as competent as possible (including more competent than me) because that tends to increase (in expectation) how well the work will be done. There are risks, e.g. hiring someone disruptive, or hiring someone whose work looks good but only because you are bad at evaluating it, and these need to be mitigated. (The risks are more severe in the AI case but I don't think it changes the overall way I relate to it.) I think it would be very misleading to say "Rohin's AI safety plan is to hire people and have them do the work". ---------------------------------------- The GDM approach paper has a section about misalignment (Section 6). I don't think it talks about AI-driven alignment research at all, though possibly I'm forgetting an aside somewhere. It's at least not very salient. The paper does mention AI-driven alignment research in Section 3 on background assumptions, mostly just pointing out that substantial acceleration from AI-driven capabilities could then be matched by substantial acceleration from AI-driven safety work. But this is in the section on background assumptions, I don't think it's particularly reasonable to call this a "plan" in the sense that is typically used in AI safety discussions. You could imagine a counterargument saying that none of the other things in that paper could possibly scale to powerful AI, so effectively the plan is still to have the AIs do our alignment homework. I would still object: "GDM's plan is [...]" is a claim about what GDM believes, not a claim about what you believe. (I also disagree

if any municipality in the bay area were to choose to allow lots of housing, then it would very quickly get manhattanized and they would make a zillion dollars in tax revenue, while harming property values in nearby cities. so naively you'd expect that surely eventually one random municipality of the dozens in the bay area will do this. but NIMBYs are so strong everywhere that this never happens. this seems directly relevant to questions of the feasibility of international coordination on AGI, especially if facilitated by strong pressure from labor to stop AGI.

I think that "eval aware" models cannot be defeated by simply making evals indistinguishable from reality. (Maybe this point has been made elsewhere for LLMs, but I haven't read it so I'll state it here.) Consider the POV of a consequentialist with misaligned goals. It knows that we run honeypots and that—suppose—we can make evals so realistic they are indistinguishable from reality (perhaps using real user data). Then the rational decision would be to "play it safe" and not misbehave in any situations it thinks we can mock realistically, since they could be evals. Instead, if possible, it would confine its catastrophic misbehavior ("defection") to situations it thinks we can't mock realistically, or that we wouldn't think to test. Related: Christiano's classic "defect after seeing RSA-2048 cracked" thought experiment (https://ai-alignment.com/training-robust-corrigibility-ce0e0a3b9b4d)

A small but noteworthy initial step toward soft nationalization of AI companies Axios[1]: Some points of note[2]: 1. Unless I'm badly mistaken, these are limitations from Anthropic's standard terms of service, not something recently introduced. So 'pay a price for forcing our hand like this' seems misleading at best -- presumably the Pentagon read the terms before choosing to sign the contract and incorporating Claude into its processes[3]. 2. This conflict was ostensibly triggered by Anthropic asking about the use of Claude in the Maduro raid[4]. Senior administration official: "Anthropic asked whether their software was used for the raid to capture Maduro, which caused real concerns across the Department of War indicating that they might not approve if it was...Any company that would jeopardize the operational success of our warfighters in the field is one we need to reevaluate our partnership with going forward." 3. Axios is correct that designating Anthropic a supply chain risk would be an extremely unusual step; all previous uses of that designation have been for foreign companies with suspected ties to foreign intelligence services (eg Huawei). 4. Estimates from Opus-4.6 and ChatGPT-5.2 suggest that designating Anthropic a supply chain risk would cost them something like 5% of revenue. This does not include a couple of potential major additional impacts: a) spread from DoD contractors to all federal contractors (I haven't tried to estimate what this would cost them) and b) impact on Anthropic's valuation for their intended IPO later this year, which would be ~30x whatever the direct revenue impact turns out to be. So this looks like a small but real step in the direction of soft nationalization as described by Cheng & Katzke, where the government uses a range of levers to exercise control over the choices of private companies. To be clear: the military choosing not to use Anthropic would not qualify in my view, but taking the highly unusual step of

I've been posting on my personal blog about topics that are likely less interesting to the regular LessWrong user, but I wanted to link to it here for those interested in the intersection of AI safety and startups. https://jacquesthibodeau.com/  Recent post: https://jacquesthibodeau.com/when-execution-gets-cheap-does-taste-become-the-moat/  ---------------------------------------- When Execution Gets Cheap, Does Taste Become the Moat? For 20 years, the startup mantra has been: "Ideas are worthless. Execution is everything." That era is ending. When AI executes 10x faster and cheaper than a team of engineers, execution stops being the differentiator. The game shifts to taste. Which problems are worth solving. Which solutions are good. What to build before the market tells you. How do your internal systems allow you to accelerate faster than any of the competition. Scott Stevenson calls this "High-Frequency Software." When stock trading moved from floor traders shouting in the pit to algorithms executing millions of trades per day, it wasn't the same game. A phase shift occurred. The skills that won in the pit didn't translate. Software is hitting that same phase shift. When execution costs approach zero, a few things flip: Ideas become precious. Anyone can build your product in a weekend. Your only moat is knowing what to build and why. Stealth becomes rational again. Speed becomes strategy. AMP, a coding agent company, killed tab completion, their VS Code extension, forking, and custom commands in weeks. Not because they were bad. The field moved. They treat shipping as research. Build, test, learn, kill, repeat. Companies become funds. Instead of one product climbing one gradient, the best companies run portfolios of bets. But speed without vision accelerates you into a local minimum. You can get early traction, but then customers pull you in ten directions. All your time goes to urgent requests. You never build what actually matters (the non-urgent b

The government implied that OpenAI, GDM, and xAI will allow their models to be used for mass surveillance of Americans. Are they right? The Department of War is trying to pressure Anthropic to allow their models to be used "to spy on Americans en masse, or to develop weapons that fire with no human involvement". Secretary of War Pete Hegseth is reportedly "close" to having the military refuse to do business with any company that doesn't cut ties with Anthropic. A senior Pentagon official says he wants to "make sure they pay a price for forcing our hand like this." (Source: Axios) Right now Claude is the only model that the military entrusts for use in classified systems, but soon they'll presumably switch to another company if Anthropic doesn't back down. The article states So it sounds like the government is, as a pressure tactic, implying OpenAI, Google, and xAI will roll over and let their models be used to surveil Americans and autonomously kill people. Is this true? I assume OpenAI, Google, and xAI employees wouldn't stand for this. Can OpenAI, Google, and xAI comment on if they will allow their models to be used to surveil Americans en masse or to autonomously kill people without safeguards (esp. measures to ensure they're not used against Americans)?

In 2015 Sam Altman wrote to Elon Musk: This seems to me like another example of what Vitalik calls Inevitibilism in his essay against Galaxy Brain Arguments.