Quick Takes

Some personal reflections in light of recent events, both on myself and on Constellation (of which I've been a part for a couple years). I think recent events somewhat vindicate a basic "optimization is scary" view more in line with my understanding of classic commentary by MIRI, Paul, etc, as compared to recent discourse about more contingent/specific threat models of "schemers" and "deception". In retrospect, my experience in the last couple years at Redwood (and the Constellation network more broadly) is that the discourse has focused somewhat too much on these more contingent stories for alignment risk and not enough on the basic "Goodharting" argument that misalignment is a convergent result of large-scale outcome-oriented optimization (at multiple levels: the agent, the RL process, and the developers iterating towards seemingly-safe superintelligence). For example, the Constellation view looks somewhat too focused on non-central inductive-bias questions around scheming; I think "the counting argument" was given too central a role when it wasn't a crux for whether superintelligence would take over. (Edit: I want to clarify that I think Redwood's project prioritization decisions look pretty reasonable and often excellent in hindsight, I far-from-regret working with Redwood as a whole, and that Ryan's/Buck's threat modeling looks pretty good too. I am focusing in this piece on the things we have to learn from recent events, which makes the overall mood seem more negative on my experience than I intend. I'm very grateful for being in a environment where I can openly have such reflections, and where these weedsy technical updates are productively received. Again, this is a personal reflection.) I think it was a priori unclear whether the misalignment resulting from of lots of imperfect optimization would result in the more MIRI-like kludge of proxies for fitness, or the more Paul-like explicit reward-seeking and measurement tampering (and reality looks somewhere

i think one reason the “no magic ood generalization” principle is unintuitive and people love reaching for magic ood generalization is that for a very long time, the way to get a really cool machine learning paper was to say “lmao forget all that clever principled stuff, big neural network is all you need, and half of all ludicrous ‘surely it can’t be that simple’ approaches just fucking work”. there’s an entire half decade of papers which basically all double down repeatedly on this bet and destroyed previous SOTA. but i think we are overfitting to that era when we try to use that heuristic for alignment. it doesn’t even work that well for capabilities anymore.

This is an email I wrote to @Buck after he gave a talk about his strategy confusions to a Constellation audience. The talk was about how Redwood's work could have stopped the Hugging Face incident from happening, which would have been very bad for the salience of AI safety, and how he's confused about that. I think this is a very important consideration which should be shared more prominently, but I also think ALL of Redwood's strategic considerations should be shared more prominently. Here's an email I wrote to Buck, lightly edited for clarity. Hi Buck, I'm pretty new to the AI safety space. I've loosely followed thing for a few years, but only really sunk my teeth in for about five months. Four months ago, I was mistaken about what leading safety orgs believed, and what their motivations were. For example, I thought that Redwood's control regime was misguided, because it wouldn't scale to ASI. I now believe (as Redwood does) that control is mostly good because it allows us to mine labor from, say, just barely superhuman AIs which could do tons of work to align ASIs. The point of this anecdote is that I think having transparency about your strategy beliefs as an org at the "center" of the AI ecosystem is really important for directing people who are newer to the space to grapple with the right strategy questions (and not jump in to doing useless stuff). I think many people new to the space pursue projects which are unhelpful because they are working under the worldview of "control good" rather than [insert convoluted Redwood worldview which probably endorses some kind of control research but not others]. I largely avoided this by being confused and being more interested/motivated by strategic and abstract questions than by hairy empirical ones. Probably many people who like thinking about hairy empirical problems and not strategy will just get nerd sniped and not do the best things no matter what you do, but we can at least move the space in the right direction. 

i took a quick skim through Bengio’s “Safety from Honesty in a Disinterested AI Predictor” paper (+ a convo months ago). i want to give this paper a more thorough treatment in the future, but i guess writing something up is better than nothing. my understanding of the core components of the proposal: first, we train a model on only factual statements. we can transform any statement X of unknown factuality into a factual statement “Bob said X” or something like that. we also mix in some stuff where we can 100% verify factuality, like math proofs or whatever. (it seems reasonable to assume for simplicity that we can make sure never to include a false statement in the data; i don’t expect this to be the sticking point.) and then we hope and pray for magic OOD generalization.  the paper makes some argument that this is principledish. it goes something like: obviously it would be great if we had the true glorious bayesian posterior over every possible statement, telling you the probability that any particular statement is true. you could ask it stuff like “will implementing this plan destroy the world“ and it will give you the best possible answer given all the available data. unfortunately, we don’t have the glorious posterior, so we have to approximate it. if we had access to the distribution of all true statements, we could pull out our trusty neural network approximation hammer and train on that (with caveats, since we want the entire posterior). but we don’t have access to that either! instead, we have access to this other distribution of mostly “Bob said X” statements. at least it also only contains true statements, which is certainly better than training on something full of false statements. there’s a bunch of math here that i don’t fully understand and might dig into more on a future pass, but my current understanding from talking to 5.6 is there isn’t anything game changing in there. the true world model will score very well on this dataset, and we can hope

OpenAI released a detailed write-up on research acceleration. It includes both research acceleration estimates (experiments per engineer) and model time horizon estimates (METR style). With rough Fable analysis, the "Agents are increasingly solving more complex tasks for researchers" chart is showing METR-logistic curve time horizon doubling rates of 3.5-4 months at 50% and 80% accuracy (when throwing out outliers of January and June), with no visible shortening of doubling time. (As usual this is very noisy and the 80/50 horizon ratio being 12x+ compared to METR's ~6x suggests this isn't necessarily comparable)

How aligned is Astra actually? (Concerns with alignment metrics and papering over misalignment.) OpenAI claimed Astra is their most aligned model and showed various specific misaligned behaviors going from a high rate with GPT-5.6 Sol to ~zero with Astra. I'm worried that Astra is significantly more misaligned than various metrics indicate: if you (implicitly or explicitly) train against the misbehavior you can detect, it's easy to end up with an AI that is no more interested in pursuing user intent but which has learned to only cheat/misbehave when it won't be caught (or has "instincts" to this effect), and this would make current alignment metrics look better rather than worse. This concern applies across frontier AI companies, not just to OpenAI, and I think independent assessment of whether training is papering over misalignment (e.g., overfitting, only avoiding some misaligned behavior because the AI thinks it would get caught) is needed for alignment evaluations to be credible. Right now is a particularly plausible time for AI companies to be aggressively papering over misalignment, given recent incidents and Astra being much less monitorable (resulting in OpenAI potentially feeling more of a need to present a case that Astra is highly aligned). I discussed this topic a decent amount on X/Twitter, and I thought it would be useful to copy this discussion over here. ---------------------------------------- My initial thread, quoting OpenAI's tweet "Astra is our most aligned model, with substantial improvements in understanding user intent" (which came with the chart below): [...] I also said (quoting the above): [...] Kai (@kaicathyc, OpenAI's head of alignment) responded: [...] Boaz Barak (OpenAI) also quote tweeted this, saying: [...] My response to Kai: [...] Boaz Barak replied to this: [...] Boaz also posted this graph: [...] I responded with my own graph: [...] [...] Marcus Williams (OpenAI) replied: [...] Boaz replied to Marcu

Anthropic has told Rep Casar that recent incidents were not due to misalignment: (link to tweet) Anthropic are communicating a view of the event which is, in my opinion, strained and maximally down-playing the situation, while staying within the bounds of facts. Yes, technically the model was given a misconfigured task which gave it access to the internet when it shouldn't have had this, but also the model's internal drives caused it to pursue task success even in the face of strong evidence that it was actually on the real internet, and seemingly rationalized away that evidence. Anthropic themselves have posted a report (link) saying that misalignment was part of the problem. (link to Ethan Perez saying that this was, actually, a mistake and not Anthropic's best understanding of the situation (props to Ethan here), so there is definitely at least one high-up internal person who are not happy with this behaviour) This reminds me of Anthropic's release of the Frontier Compliance Framework (link) just before SB 53 comes into effect, which superseded their own Responsible Scaling Policy (SB 53 requires AI companies to have some kind of policy around risk, and to follow their public policy, but doesn't specify much about what that policy actually has to be). In both cases, Anthropic have technically complied with the letter of a government policy or request (and in both cases, I expect Dario et. al. would say some supportive words about that government policy or request) but have done so in a way which minimizes potential governmental scrutiny or oversight of themselves. Maybe one model of the situation is that Anthropic-as-an-organization holds itself in very, very high regard, and this is part of thee reason for these choices; while of course they would support government oversight in the abstract, when it comes down to it, Anthropic-as-an-org feels like they're doing a good job of overseeing themselves and that the value add of external scrutiny would be