Quick Takes

Unless you have an explicit strategy for how and when you will speak out in the future, and explicit reasoning for why that requires silence now, "strategic silence" is just silence.

PSA: Anthropic models don't seem to particularly privilege the explicit thinking field. This makes reinforcement spillover—where training on a model's outputs generalizes to the CoT, making it appear safer—more likely. While Anthropic models do have an explicit separate thinking field, they don't really use thinking that differently from outputs and aren't that dependent on the thinking field. Sometimes they'll just do their thinking in the output field, the way they talk in the thinking field isn't very distinct from how they talk in outputs, and I believe disabling thinking doesn't have a big effect on coding performance (especially for earlier Anthropic models but even for current Anthropic models). (This is specifically for typical agentic coding; this likely doesn't apply to math.) This is pretty different from OpenAI models which are way more reliant on the explicit thinking and the thinking is very distinct from how the model talks in the output (based on public CoT examples). I'm uncertain, but I think GDM models are generally more similar to OpenAI than Anthropic models on this axis. In general, Anthropic's models leverage taking lots of actions over doing a bunch of thinking. (This is based on my personal experience/analysis and talking to various people, including some Anthropic employees, but isn't based on non-public information.) Some implications of this: * It might be a mistake to emphasize CoT monitoring relative to monitoring the overall transcript (including CoT but also tool calls and other outputs). Like, I do think that Anthropic models are decently monitorable in general, but looking at the thinking tags in particular may not be that important/salient by default. (To be clear, if the model knew you weren't looking at the thinking tags that might be a big uplift to its covert abilities, I just mean that these tags aren't that privileged.) * Because Anthropic models don't treat the thinking field in a very distinct way, I think it's prett

Correct me if I'm mistaken, but at this point it's misleading to think of the frontier LLMs as "text predictors with some post-training", and more accurate to think of them as "RL models that were initialized with a text predictor model". As I understand it, there's now a massive amount of RLAIF to go along with expensive RLHF; some of the RL is persona training, some of it is technical training in fields where reliable feedback can be automated (e.g. is the output a valid program that passes the following tests). Starting off with a text predictor is key, because that makes the LLM represent a lot of useful concepts; but the RL phase is doing an increasing amount of lifting. In particular, that means there's no reason to expect coding or math to cap out at "imitating the best humans", for the same reason that self-play helped AlphaGo to supersede the best humans. Checking here first before I start injecting "text predictors are only the larval stage of modern LLMs" into the discourse.

The recent controversy with Anthropic and the DoW has shown pretty viscerally how US politics can impact AI, and interactions will almost certainly grow more frequent and significant as AI continues to advance.  Something it brought to mind is how the timing of political events (particularly after US midterms in 2026, and after US presidential inauguration in January 2029), would likely act as a significant step change to the distribution of AI outcomes, and how that relates to the timing of AI reaching particular thresholds in capability and political salience.

A quick note on various alignment affordances that the model personas research agenda might offer. I'm interested in takes on how useful people think each of these is.  * "Seeding RL." RL is underspecified. Specifically, it seems difficult to provide good feedback on the motivations / values of AIs, since: (i) for sufficiently strong models, the aligned and deceptively aligned policy are behaviourally indistinguishable, and (ii) training directly against the reasoning might just lead to obfuscation. The hope is that improving the initial point from which we start RL increases the chance of landing in the 'aligned' basin (instead of a misaligned basin.)  * For this to go really well, we need to extensively study the interactions between RL and initialization, as they pertain to propensities and personas. Does RL just 'wash out' any persona prior? Or do aligned personas tend to remain aligned through RL? (Evidence from sleeper agents makes me think yes, but it's complicated)   * “Reframing". By default, models seem to 'over-index' on specific behaviours as evidence of broad alignment tendencies (see: EM from reward hacking). By re-framing specific undesirable behaviours as desirable, we prevent the assistant persona's broader character from being damaged when the model does isolated misaligned things in training. This is one way in which inoculation prompting can work, e.g. in Wichers et al (2025) * For this to go really well, we need to map out the space of traits models associate with the aligned assistant persona (and traits which are anti-associated). So we should build lots of propensity evals here.  * "Containment". Suppose we have an untrusted environment which gives capabilities uplift but might also induce undesirable propensities (e.g. making the model more scheme-y or reward-hacky). To solve this, we train misaligned personas on the untrusted environments. The hope is that the aligned assistant persona can access the capabilities of the misaligned

If you are a student, You should not try to use vibecoding (or agentcoding) to try to catch up to the current level of productivity you see among researcher and programmers. That is similar to buying stocks when they are overvalued and about to crash. Be slow. Be behind. Play with things, break things, make your own manual silly versions without libraries, try to understand something by experiments without explanations. When the current mess of uncatchable bugs created by agents explodes, somebody will have to fix things. Thinking properly for hours over four lines of code, anticipating how things can be broken instead of convincing oneself that reading through the generated code is enough. This might sound silly, but listen: AI is only taught to do the right thing, and THAT is a big problem. When researchers publish a paper (at least how they used to 10 years ago), they play with the topic for hundreds/thousands of hours, trying dozens of alternative formulas, reactions, algorithms, and creating their own variants. They gained huge experience of how things would act in the edge cases and HOW by emergent mechanisms things can become invalid/broken. None of that wisdom is transfered into the final paper, which only shows the optimal way they found. AI can read all the articles in the world and not have enough examples of how to anticipate emergent problems in simple situations. I find and hear about a lot of stupid bugs in google and microsoft products, and it seems the situation is similar most-where. Ai coding is the second reason behind enshittification. When the internet will crash, you would not like to be dependent on any help. You might prefer to have wisdom instead of a big portfolio.

I think about the canonical Reality Has a Surprising Amount of Detail post a lot when trying to automate tasks with LLMs. In particular, any given task has many granular details, most of which don’t come to mind before making contact with reality oneself. The most common failure mode I run into is failing to specify various details that I hadn’t even *realized* were relevant things that could be messed up, before the model encountered the situation and messed them up. This also seems relevant when thinking about the transition from verifiable math and coding problems to more open-ended tasks like research or robotic manipulation.