LESSWRONG
LW

Rohin Shah — LessWrong

Can you say more about why this is a limitation / issue? Is this different from a 2008-2015 analysis saying "later models are more likely to use the transformer architecture," where my response is "that's algorithmic progress for ya". One reason it may be different is that inference-time compute might be trading off against training compute in a way that we think make the comparison improper between low and high inference-compute models.

Yeah it's just the reason you give, though I'd frame it slightly differently. I'd say that the point of "catch-up algorithmic progress" was to look at costs paid to get a certain level of benefit, and while historically "training compute" was... (read more)

Replying toCatch-Up Algorithmic Progress Might Actually be 60× per Year

Rohin Shah2mo

Catch-Up Algorithmic Progress Might Actually be 60× per Year

I was under the impression you expected slower catch-up progress.

Note that I think the target we're making quantitative forecasts about will tend to overestimate that-which-I-consider-to-be "catch-up algorithmic progress" so I do expect slower catch-up progress than the naive inference from my forecast (ofc maybe you already factored that in).

Replying toCatch-Up Algorithmic Progress Might Actually be 60× per Year

Rohin Shah2mo

Catch-Up Algorithmic Progress Might Actually be 60× per Year

Thanks, I hadn't looked at the leave-one-out results carefully enough. I agree this (and your follow-up analysis rerun) means my claim is incorrect. Looking more closely at the graphs, in the case of Llama 3.1, I should have noticed that EXAONE 4.0 (1.2B) was also a pretty key data point for that line. No idea what's going on with that model.

(That said, I do think going from 1.76 to 1.64 after dropping just two data points is a pretty significant change, also I assume that this is really just attributable to Grok 3 so it's really more like one data point. Of course the median won't change, and I do prefer the... (read more)

•••

Replying toCatch-Up Algorithmic Progress Might Actually be 60× per Year

Rohin Shah2mo*

Catch-Up Algorithmic Progress Might Actually be 60× per Year

Speaking colloquially, I might say "these results indicate to me that catch-up algorithmic progress is on the order of one or 1.5 orders of magnitude per year rather than half an order of magnitude per year as I used to think". And again, my previous belief of 3× per year was a belief that I should have known was incorrect because it's based only on pre-training.

Okay fair enough, I agree with that.

As I said in footnote 9, I am willing to make bets about my all-things-considered beliefs. You think I'm updating too much based on unreliable methods? Okay come take my money.

I find this attitude weird. It takes a lot of time... (read more)

Replying toCatch-Up Algorithmic Progress Might Actually be 60× per Year

Rohin Shah2mo*

Catch-Up Algorithmic Progress Might Actually be 60× per Year

~~Your results are primarily driven by the inclusion of Llama 3.1-405B and Grok 3 (and Grok 4 when you include it).~~ Those models are widely believed to be cases where a ton of compute was poured in to make up for poor algorithmic efficiency. If you remove those I expect your methodology would produce similar results as prior work (which is usually trying to estimate progress at the frontier of algorithmic efficiency, rather than efficiency progress at the frontier of capabilities).

I could imagine a reply that says "well, it's a real fact that when you start with a model like Grok 3, the next models to reach a similar capability level will... (read more)

-1

Replying toRecent LLMs can use filler tokens or problem repeats to improve (no-CoT) math performance

Rohin Shah2mo

Recent LLMs can use filler tokens or problem repeats to improve (no-CoT) math performance

Cool result!

these results demonstrate a case where LLMs can do (very basic) meta-cognition without CoT

Why do you believe this is meta-cognition? (Or maybe the question is, what do you mean by meta-cognition?)

It seems like it could easily be something else. For example, probably when solving problems the model looks at the past strategies it has used and tries some other strategy to increase the likelihood of solving the problem. It does this primarily in the token space (looking at past reasoning and trying new stuff) but this also generalizes somewhat to the activation space (looking at what past forward passes did and trying something else). So when you have filler tokens the latter effect still happens, giving a slight best-of-N type boost, producing your observed results.

Replying toFiller tokens don’t allow sequential reasoning

Rohin Shah2mo

Filler tokens don’t allow sequential reasoning

Filler tokens don't allow for serially deeper cognition than what architectural limits allow

This depends on your definition of serial cognition, under the definitions I like most the serial depth scales logarithmically with the number of tokens. This is because as you increase parallelism (in the sense you use above), that also increases serial depth logarithmically.

The basic intuitions for this are:

If you imagine mechanistically how you would add N numbers together, it seems like you would need logarithmic depth (where you recursively split it in half, compute the sums of each half, and then add the results together). Note that attention heads compute sums of numbers that scale with the number of tokens.
There

... (read more)

Replying toContradict my take on OpenPhil's past AI beliefs

Rohin Shah2mo

Contradict my take on OpenPhil's past AI beliefs

Ah, I realized there was something else I should have highlighted. You mention you care about pre-ChatGPT takes towards shorter timelines -- while compute-centric takeoff was published two months after ChatGPT, I expect that the basic argument structure and conclusions were present well before the release of ChatGPT.

While I didn't observe that report in particular, in general Open Phil worldview investigations took > 1 year of serial time and involved a pretty significant and time-consuming "last mile" step where they get a bunch of expert review before publication. (You probably observed this "last mile" step with Joe Carlsmith's report, iirc Nate was one of the expert reviewers for that report.) Also, Tom... (read more)

Replying toContradict my take on OpenPhil's past AI beliefs

Rohin Shah2mo*

Contradict my take on OpenPhil's past AI beliefs

I think I agree with all of that under the definitions you're using (and I too prefer the bounded rationality version). I think in practice I was using words somewhat differently than you.

(The rest of this comment is at the object level and is mostly for other readers, not for you)

Saying it's "crazy" means it's low probability of being (part of) the right world-description.

The "right" world-description is a very high bar (all models are wrong but some are useful), but if I go with the spirit of what you're saying I think I might not endorse calling bio anchors "crazy" by this definition, I'd say more like "medium" probability of being a... (read more)

Replying toContradict my take on OpenPhil's past AI beliefs

Rohin Shah2mo*

Contradict my take on OpenPhil's past AI beliefs

I also want to qualify / explain my statement about it being a crazy argument. The specific part that worries me (and Eliezer, iiuc) is the claim that, at a given point in time, the delta between natural and artificial artifacts will tend to be approximately constant across different domains. This is quite intuition-bending from a mechanistic / reductionist viewpoint, and the current support for it seems very small and fragile (this 8 page doc). However, I can see a path where I would believe in it much more, which would involve things like:

Pinning down the exact methodology: Which metrics are we making this "approximately constant delta" claim for? It's definitely not

... (read 402 more words →)

GDM: Consistency Training Helps Limit Sycophancy and Jailbreaks in Gemini 2.5 Flash

TurnTrout

TurnTrout, Rohin Shah

3mo

Authors: Alex Irpan* and Alex Turner*, Mark Kurzeja, David Elson, and Rohin Shah

You’re absolutely right to start reading this post! What a rational decision!

Even the smartest models’ factuality or refusal training can be compromised by simple changes to a prompt. Models often praise the user’s beliefs (sycophancy) or satisfy inappropriate requests which are wrapped within special text (jailbreaking). Normally, we fix these problems with Supervised Finetuning (SFT) on static datasets showing the model how to respond in each context. While SFT is effective, static datasets get stale: they can enforce outdated guidelines (specification staleness) or be sourced from older, less intelligent models (capability staleness).

We explore consistency training, a self-supervised paradigm that teaches... (read 1608 more words →)

Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety

Tomek Korbak

Tomek Korbak, Mikita Balesni, Vlad Mikulik, Rohin Shah

7mo

Twitter | Paper PDF

Seven years ago, OpenAI five had just been released, and many people in the AI safety community expected AIs to be opaque RL agents. Luckily, we ended up with reasoning models that speak their thoughts clearly enough for us to follow along (most of the time). In a new multi-org position paper, we argue that we should try to preserve this level of reasoning transparency and turn chain of thought monitorability into a systematic AI safety agenda.

This is a measure that improves safety in the medium term, and it might not scale to superintelligence even if somehow a superintelligent AI still does its reasoning in English. We hope that... (read more)

167

Evaluating and monitoring for AI scheming

Vika

Vika, Scott Emmons, Erik Jenner, Mary Phuong, Lewis Ho, Rohin Shah

7mo

As AI models become more sophisticated, a key concern is the potential for “deceptive alignment” or “scheming”. This is the risk of an AI system becoming aware that its goals do not align with human instructions, and deliberately trying to bypass the safety measures put in place by humans to prevent it from taking misaligned action. Our initial approach, as laid out in the Frontier Safety Framework, focuses on understanding current scheming capabilities of models and developing chain-of-thought monitoring mechanisms to oversee models once they are capable of scheming.

We present two pieces of research, focusing on (1) understanding and testing model capabilities necessary for scheming, and (2) stress-testing chain-of-thought monitoring as a... (read 1426 more words →)

Google DeepMind: An Approach to Technical AGI Safety and Security

Rohin Shah

10mo

We have written a paper on our approach to technical AGI safety and security. This post is primarily a copy of the extended abstract, which summarizes the paper. I also include the abstract and the table of contents. See also the GDM blogpost and tweet thread.

Artificial General Intelligence (AGI) promises transformative benefits but also presents significant risks. We develop an approach to address the risk of harms consequential enough to significantly harm humanity. We identify four areas of risk: misuse, misalignment, mistakes, and structural risks. Of these, we focus on technical approaches to misuse and misalignment. For misuse, our strategy aims to prevent threat actors from accessing dangerous capabilities, by proactively identifying... (read 5143 more words →)

Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research (GDM Mech Interp Team Progress Update #2)

lewis smith

lewis smith, Senthooran Rajamanoharan, Arthur Conmy, CallumMcDougall, Tom Lieberum, János Kramár, Rohin Shah, Neel Nanda

11mo

Lewis Smith*, Sen Rajamanoharan*, Arthur Conmy, Callum McDougall, Janos Kramar, Tom Lieberum, Rohin Shah, Neel Nanda

* = equal contribution

The following piece is a list of snippets about research from the GDM mechanistic interpretability team, which we didn’t consider a good fit for turning into a paper, but which we thought the community might benefit from seeing in this less formal form. These are largely things that we found in the process of a project investigating whether sparse autoencoders (SAEs) were useful for downstream tasks, notably out-of-distribution probing.

TL;DR

To validate whether SAEs were a worthwhile technique, we explored whether they were useful on the downstream task of OOD generalisation when detecting harmful intent in

... (read 8585 more words →)

116

AGI Safety & Alignment @ Google DeepMind is hiring

Rohin Shah

The AGI Safety & Alignment Team (ASAT) at Google DeepMind (GDM) is hiring! Please apply to the Research Scientist and Research Engineer roles. Strong software engineers with some ML background should also apply (to the Research Engineer role). Our initial batch of hiring will focus more on hiring engineers, but we expect to continue to use the applications we receive for future hiring this year, which we expect will be more evenly split. Please do apply even if e.g. you’re only available in the later half of this year.

What is ASAT?

ASAT is the primary team at GDM focused on technical approaches to severe harms from AI systems, having evolved out of the Alignment and Scalable... (read 2761 more words →)

103

A short course on AGI safety from the GDM Alignment team

Vika

Vika, Rohin Shah

We are excited to release a short course on AGI safety for students, researchers and professionals interested in this topic. The course offers a concise and accessible introduction to AI alignment, consisting of short recorded talks and exercises (75 minutes total) with an accompanying slide deck and exercise workbook. It covers alignment problems we can expect as AI capabilities advance, and our current approach to these problems (on technical and governance levels). If you would like to learn more about AGI safety but have only an hour to spare, this course is for you!

Here are some key topics you will learn about in this course:

The evidence for the field being on a path to

... (read 263 more words →)

105

MONA: Managed Myopia with Approval Feedback

Seb Farquhar

Seb Farquhar, David Lindner, Rohin Shah

Blog post by Sebastian Farquhar, David Lindner, Rohin Shah.
It discusses the paper MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hacking by Sebastian Farquhar, Vikrant Varma, David Lindner, David Elson, Caleb Biddulph, Ian Goodfellow, and Rohin Shah.

Our paper tries to make agents that are safer in ways that we may not be able to evaluate through Myopic Optimization with Non-myopic Approval (MONA).

Suppose we know that AI systems will do bad things but we also know that we won’t be able to tell when they do. Can we still prevent unintended behaviour?

In particular cases and ways: yes. Specifically, we show how to get agents whose long-term plans follow strategies that humans... (read 2487 more words →)

AGI Safety and Alignment at Google DeepMind: A Summary of Recent Work

Rohin Shah

Rohin Shah, Seb Farquhar, Anca Dragan

We wanted to share a recap of our recent outputs with the AF community. Below, we fill in some details about what we have been working on, what motivated us to do it, and how we thought about its importance. We hope that this will help people build off things we have done and see how their work fits with ours.

Who are we?

We’re the main team at Google DeepMind working on technical approaches to existential risk from AI systems. Since our last post, we’ve evolved into the AGI Safety & Alignment team, which we think of as AGI Alignment (with subteams like mechanistic interpretability, scalable oversight, etc.), and Frontier Safety (working on the Frontier... (read 2668 more words →)

222

On scalable oversight with weak LLMs judging strong LLMs

zac_kenton

zac_kenton, Noah Siegel, janos, Jonah Brown-Cohen, Samuel Albanie, David Lindner, Rohin Shah

Abstract

Scalable oversight protocols aim to enable humans to accurately supervise superhuman AI. In this paper we study debate, where two AI's compete to convince a human judge; consultancy, where a single AI tries to convince a human judge that asks questions; and compare to a baseline of direct question-answering, where the human judge just answers outright without the AI. We use large language models (LLMs) as both AI agents and as stand-ins for human judges, taking the judge models to be weaker than agent models. We benchmark on a diverse range of asymmetries between judges and agents, extending previous work on a single extractive QA task with information asymmetry, to also include... (read 1890 more words →)

So here's a paper: Fundamental Limitations of Alignment in Large Language Models. With a title like that you've got to at least skim it. Unfortunately, the quick skim makes me pretty skeptical of the paper.

The abstract says "we prove that for any behavior that has a finite probability of being exhibited by the model, there exist prompts that can trigger the model into outputting this behavior, with probability that increases with the length of the prompt." This clearly can't be true in full generality, and I wish the abstract would give me some hint about what assumptions they're making. But we can look at the details in the paper.

(This next part isn't... (read more)

Suppose you have some deep learning model M_orig that you are finetuning to avoid some particular kind of failure. Suppose all of the following hold:

Capable model: The base model has the necessary capabilities and knowledge to avoid the failure.
Malleable motivations: There is a "nearby" model M_good (i.e. a model with minor changes to the weights relative to the M_orig) that uses its capabilities to avoid the failure. (Combined with (1), this means it behaves like M_orig except in cases that show the failure, where it does something better.)
Strong optimization: If there's a "nearby" setting of model weights that gets lower training loss, your finetuning process will find it (or something even better). Note this

... (read more)

I recently had occasion to write up quick thoughts about the role of assistance games (CIRL) in AI alignment, and how it relates to the problem of fully updated deference. I thought I'd crosspost here as a reference.

Assistance games / CIRL is a similar sort of thing as CEV. Just as CEV is English poetry about what we want, assistance games are math poetry about what we want. In particular, neither CEV nor assistance games tells you how to build a friendly AGI. You need to know something about how the capabilities arise for that.
One objection: an assistive agent doesn’t let you turn it off, how could that be what we want?

... (read 441 more words →)

It's common for people to be worried about recommender systems being addictive or promoting filter bubbles etc, but as far as I can tell, they don't have very good arguments for these worries. Whenever I talk to someone who seems to have actually studied the topic in depth, it seems they think that there are problems with recommender systems, but they are different from what people usually imagine.

I'll go through the articles I've read that argue for worrying about recommender systems, and explain why I find them unconvincing. I've only looked at the ones that are widely read; there are probably significantly better arguments that are much less widely read.

Aligning Recommender Systems... (read 642 more words →)

What won't we be able to do by (say) the end of 2025? (See also this recent post.) Well, one easy way to generate such answers would be to consider tasks that require embodiment in the real world, or tasks that humans would find challenging to do. (For example, “solve the halting problem”, “produce a new policy proposal that has at least a 95% chance of being enacted into law”, “build a household robot that can replace any human household staff”.) This is cheating, though; the real challenge is in naming something where there’s an adjacent thing that _does_ seem likely (i.e. it’s near the boundary separating “likely” from “unlikely”).

One decent answer... (read more)

I often search through the Alignment Newsletter database to find the exact title of a relevant post (so that I can link to it in a new summary), often reading through the summary and opinion to make sure it is the post I'm thinking of.

Frequently, I read the summary normally, then read the first line or two of the opinion and immediately realize that it wasn't written by me.

This is kinda interesting, because I often don't know what tipped me off -- I just get a sense of "it doesn't sound like me". Notably, I usually do agree with the opinion, so it isn't about stating things I don't believe. Nonetheless, it isn't purely about personal writing styles, because I don't get this sense when reading the summary.

(No particular point here, just an interesting observation)

(This shortform prompted by going through this experience with Embedded Agency via Abstraction)

I occasionally hear the argument "civilization is clearly insane, we can't even do the obvious thing of <insert economic argument here, e.g. carbon taxes>".

But it sounds to me like most rationalist / EA group houses didn't do the "obvious thing" of taxing COVID-risky activities (which basically follows the standard economic argument of pricing in externalities). What's going on? Some hypotheses:

Actually, taxing COVID-risky activities is not a good solution EDIT: and group houses recognized this. (Why? It seemed to work pretty well for my group house.)
Actually, rationalist / EA group houses did tax COVID-risky activities. (Plausible, I don't know that much about other group houses, but what I've heard doesn't seem consistent with

... (read more)

I often have the experience of being in the middle of a discussion and wanting to reference some simple but important idea / point, but there doesn't exist any such thing. Often my reaction is "if only there was time to write an LW post that I can then link to in the future". So far I've just been letting these ideas be forgotten, because it would be Yet Another Thing To Keep Track Of. I'm now going to experiment with making subcomments here simply collecting the ideas; perhaps other people will write posts about them at some point, if they're even understandable.

The LCA paper (to be summarized in AN #98) presents a method for understanding the contribution of specific updates to specific parameters to the overall loss. The basic idea is to decompose the overall change in training loss across training iterations:

$L (θ_{T}) - L (θ_{0}) = \sum_{t} L (θ_{t}) - L (θ_{t - 1})$

And then to decompose training loss across specific parameters:

$L (θ_{t}) - L (θ_{t - 1}) = {\to θ}_{t} \int {\to θ}_{t - 1} {\to \nabla}_{\to θ} L (\to θ) \cdot d \to θ$

I've added vector arrows to emphasize that $θ$ is a vector and that we are taking a dot product. This is a path integral, but since gradients form a conservative field, we can choose any arbitrary path. We'll be choosing the linear path throughout. We can rewrite the integral as the dot product of the change in parameters and the... (read more)

The LESS is More paper (summarized in AN #96) makes the claim that using the Boltzmann model in sparse regions of demonstration-space will lead to the Boltzmann model over-learning. I found this plausible but not obvious, so I wanted to check it myself. (Partly I got nerd-sniped, partly I do want to keep practicing my ability to tell when things are formalizable theorems.) This benefited from discussion with Andreea (one of the primary authors).

Let's consider a model where there are clusters ${c_{i}}$ , where each cluster contains trajectories whose features are identical $c_{i} = {τ : ϕ (τ) = ϕ_{c_{i}}}$ (which also implies rewards are identical). Let $c (τ)$ denote the cluster that $τ$ belongs to. The Boltzmann model says... (read 623 more words →)

LESSWRONG
LW

LESSWRONG
LW

Rohin Shah

AGI Safety and Alignment at Google DeepMind: A Summary of Recent Work

Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety

DeepMind is hiring for the Scalable Alignment and Alignment Teams

Discussion: Challenges with Unsupervised LLM Knowledge Discovery

Rohin Shah

Rohin Shah

GDM: Consistency Training Helps Limit Sycophancy and Jailbreaks in Gemini 2.5 Flash

Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety

Evaluating and monitoring for AI scheming

Google DeepMind: An Approach to Technical AGI Safety and Security

Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research (GDM Mech Interp Team Progress Update #2)

AGI Safety & Alignment @ Google DeepMind is hiring

A short course on AGI safety from the GDM Alignment team

Value Learning

Alignment Newsletter

Rohin Shah

AGI Safety and Alignment at Google DeepMind: A Summary of Recent Work

Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety

DeepMind is hiring for the Scalable Alignment and Alignment Teams

Discussion: Challenges with Unsupervised LLM Knowledge Discovery

Rohin Shah

Rohin Shah

GDM: Consistency Training Helps Limit Sycophancy and Jailbreaks in Gemini 2.5 Flash

Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety

Evaluating and monitoring for AI scheming

Google DeepMind: An Approach to Technical AGI Safety and Security

Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research (GDM Mech Interp Team Progress Update #2)

AGI Safety & Alignment @ Google DeepMind is hiring

A short course on AGI safety from the GDM Alignment team

Value Learning

Alignment Newsletter

TL;DR

What is ASAT?

Who are we?

Abstract