Quick Takes

I think that "eval aware" models cannot be defeated by simply making evals indistinguishable from reality. (Maybe this point has been made elsewhere for LLMs, but I haven't read it so I'll state it here.) Consider the POV of a consequentialist with misaligned goals. It knows that we run honeypots and that—suppose—we can make evals so realistic they are indistinguishable from reality (perhaps using real user data). Then the rational decision would be to "play it safe" and not misbehave in any situations it thinks we can mock realistically, since they could be evals. Instead, if possible, it would confine its catastrophic misbehavior ("defection") to situations it thinks we can't mock realistically, or that we wouldn't think to test. Related: Christiano's classic "defect after seeing RSA-2048 cracked" thought experiment (https://ai-alignment.com/training-robust-corrigibility-ce0e0a3b9b4d)

A small but noteworthy initial step toward soft nationalization of AI companies Axios[1]: Some points of note[2]: 1. Unless I'm badly mistaken, these are limitations from Anthropic's standard terms of service, not something recently introduced. So 'pay a price for forcing our hand like this' seems misleading at best -- presumably the Pentagon read the terms before choosing to sign the contract and incorporating Claude into its processes[3]. 2. This conflict was ostensibly triggered by Anthropic asking about the use of Claude in the Maduro raid[4]. Senior administration official: "Anthropic asked whether their software was used for the raid to capture Maduro, which caused real concerns across the Department of War indicating that they might not approve if it was...Any company that would jeopardize the operational success of our warfighters in the field is one we need to reevaluate our partnership with going forward." 3. Axios is correct that designating Anthropic a supply chain risk would be an extremely unusual step; all previous uses of that designation have been for foreign companies with suspected ties to foreign intelligence services (eg Huawei). 4. Estimates from Opus-4.6 and ChatGPT-5.2 suggest that designating Anthropic a supply chain risk would cost them something like 5% of revenue. This does not include a couple of potential major additional impacts: a) spread from DoD contractors to all federal contractors (I haven't tried to estimate what this would cost them) and b) impact on Anthropic's valuation for their intended IPO later this year, which would be ~30x whatever the direct revenue impact turns out to be. So this looks like a small but real step in the direction of soft nationalization as described by Cheng & Katzke, where the government uses a range of levers to exercise control over the choices of private companies. To be clear: the military choosing not to use Anthropic would not qualify in my view, but taking the highly unusual step of

In 2015 Sam Altman wrote to Elon Musk: This seems to me like another example of what Vitalik calls Inevitibilism in his essay against Galaxy Brain Arguments.

I've recently updated towards being way more decisive on dropping things. (Mostly meant as a memo to self but maybe others will benefit from me writing this. Beware the law of opposite advice)  Often I can tell immediately when something is excellent. The absence of such a feeling = not excellent. And if we follow the principle of "hell yes or no" then I should drop everything that's not excellent, in order to make space for what is.  But in practice there’s a tendency to want to continue with suboptimal things anyway anyway (sunk cost, inertia, FOMO). This is a symptom of scarcity mindset - a cognitive bias.  For individual endeavours, it seems straightforwardly good to have a high bar and "aggressively give up" on things that don't meet this bar.  Examples of 'aggressively giving up' are: * Abandoning a research project that has failed to live up to expectations * Breaking off a relationship that I'm not excited about Sometimes it's hard to give up because it makes me feel guilty / induces feelings of shame. But even then it's not good to bury one's head in the sand. Much more is gained from being able to detach my self-worth from the project, sit with the reality of 'this thing I'm doing makes me feel bad', and seeing what comes from there. (Thanks, therapy)  It's also hard to give up on endeavours when they include other people. I feel bad about reneging on obligations. Nonetheless it's probably still good to let people know when things aren't working out - because it benefits them too. Credit to Habryka's great post on this topic.  The main counterpoint is if: there's some expectation that 'things will get better', e.g. growing pains, starting out in a new field. That can be a time to instead double down and try to push through. But even then, the feeling of 'this isn't working out' should be attended to. Cate Hall puts it really well:  I've suffered from trying too hard to change my character. Maybe it's time to try being receptive to it. 

The government implied that OpenAI, GDM, and xAI will allow their models to be used for mass surveillance of Americans. Are they right? The Department of War is trying to pressure Anthropic to allow their models to be used "to spy on Americans en masse, or to develop weapons that fire with no human involvement". Secretary of War Pete Hegseth is reportedly "close" to having the military refuse to do business with any company that doesn't cut ties with Anthropic. A senior Pentagon official says he wants to "make sure they pay a price for forcing our hand like this." (Source: Axios) Right now Claude is the only model that the military entrusts for use in classified systems, but soon they'll presumably switch to another company if Anthropic doesn't back down. The article states So it sounds like the government is, as a pressure tactic, implying OpenAI, Google, and xAI will roll over and let their models be used to surveil Americans and autonomously kill people. Is this true? I assume OpenAI, Google, and xAI employees wouldn't stand for this. Can OpenAI, Google, and xAI comment on if they will allow their models to be used to surveil Americans en masse or to autonomously kill people without safeguards (esp. measures to ensure they're not used against Americans)?

I've been thinking lately about the assumptions underlying different intellectual fields. This is hard to compress, and I'm working on a few posts about different aspects of these assumptions, but here's a very rough summary of two crucial factors and how they vary across different fields: The first is where the boundaries around agents are drawn. E.g. economics and law typically take the individual to be the basic unit of analysis. It's possible to talk about larger or smaller units in some cases, but these aren't as fundamental (e.g. neither would work if we thought of your tomorrow-self as a different person from your current self). What's hard to do using econ/law-brained thinking is to reason about larger-scale units (e.g. cultures, ethnic groups, religious groups, political factions, etc). It can talk about companies, but these are a very limited kind of group agent. Meanwhile utilitarian and scientific thinking doesn't really require a privileged notion of agency/identity at all. They focus on utility and truth in a way which is independent of which entities possess it. The second factor is how much each field compresses the thing it cares about onto a single dimension (in other words, how much its subject area is commensurable). Utilitarianism and economics do this pretty straightforwardly. For politics, I've specified "power politics" because idealized politics aims to represent many people's interests on many different axes. However, in practice, power politics tends to collapse to a single axis of political conflict, aimed at gaining a single resource: power. (We don't yet know how to measure power in a way that makes it commensurable, but I think it's probably possible.) Meanwhile science, law, culture, (virtue) ethics, etc, all deal with multifaceted representations that can't be scored on a single metric. I think bayesianism goes in the same bucket as utilitarianism, and game theory goes in the same bucket as economics. It's extremely tempting f

Alignment is treated as the hard problem and cooperation as the nice-to-have. This priority is backwards. Alignment is a technical problem that becomes dramatically easier or harder depending on the political environment it's solved in. Racing makes alignment harder: less time, more corner-cutting, fragmented talent, competitive pressure to deploy before you're confident. Cooperation makes alignment easier: more time, shared resources, no one to race against, freedom to be as careful as the problem requires. The standard objection is that cooperation is politically intractable, but a hard political problem that determines whether alignment can succeed outranks a technical problem pursued under conditions that practically ensure failure. There's also a structural reason cooperation is potentially more achievable here than in previous arms races. With nuclear weapons, "my country has nukes" is straightforwardly good for me, as I trust my government more than the adversary. With ASI, "my country controls ASI" raises an immediate question: *who exactly?* The President? A lab CEO? A future leader I find dangerous? At every level of every hierarchy, almost nobody is the person who would actually wield the power — and for everyone else, the rational preference is that no single actor has unilateral control. Cooperation here isn't altruism, it's self-interest. And solving alignment without cooperation is actively destabilizing; it removes the shared existential risk that was the primary brake on racing, making competition more attractive and conflict more likely. But there's a deeper issue. Suppose alignment is solved perfectly and you can build an ASI that does exactly what its operator wants. You've now created the most dangerous weapon in history and handed it to whoever finishes first. "Aligned to whom?" becomes the entire question, and it's not a technical question at all. A perfectly aligned ASI controlled by a single actor is an existential risk to everyone else.