THE BRIEFING

A conference room. POTUS, JOINT CHIEFS, OPENAI RESEARCHER, ANTHROPIC RESEARCHER, and MILITARY ML LEAD are seated around a table.
 

JOINT CHIEFS: So. Horizon time is up to 9 hours. We've started turning some drone control over to your models. Where are we on alignment?

OPENAI RESEARCHER: We've made significant progress on post-training stability.

MILITARY ML LEAD: Good. Walk us through the architecture improvements. Attention mechanism modifications? Novel loss functions?

ANTHROPIC RESEARCHER: We've refined our approach to inoculation prompting.

MILITARY ML LEAD: Inoculation prompting?

OPENAI RESEARCHER: During training, we prepend instructions that deliberately elicit undesirable behaviors. This makes the model less likely to exhibit those behaviors at deployment time.

Silence.

POTUS: You tell it to do the bad thing so it won't do the bad thing.

ANTHROPIC RESEARCHER: Correct.

MILITARY ML LEAD: And this works.

OPENAI RESEARCHER: Extremely well. We've reduced emergent misalignment by forty percent.

JOINT CHIEFS: Emergent misalignment.

ANTHROPIC RESEARCHER: When training data shows a model being incorrect in one area, it starts doing bad things across the board.

MILITARY ML LEAD: Ov... (read more)

Many people (including me) have opinions on current US president Donald Trump, none of which are relevant here because, as is well-known to LessWrong, politics is the mind-killer. But in the middle of an interview yesterday with someone from ABC News, I was fascinated to hear him say the most Bayesian thing I've ever heard from a US president:
--

TERRY MORAN: You have a hundred percent confidence in Pete Hegseth?

PRESIDENT DONALD TRUMP: I don't have -- a hundred percent confidence in anything, okay? Anything. Do I have a hundred percent? It's a stupid question. Look --

TERRY MORAN: It's a pretty important position.

PRESIDENT DONALD TRUMP: -- I have -- no, no, no. You don't have a hundred percent. Only a liar would say, "I have a hundred percent confidence." I don't have a hundred percent confidence that we're gonna finish this interview.
---

[EDIT -- no object-level comments about Trump, please; as per my comment here, I think it would be unproductive and poorly suited to this context. There are many many other places to talk about object-level politics.]

We have a small infestation of ants in our bathroom at the moment. We deal with that by putting out Terro ant traps, which are just boric acid in a thick sugar solution. When the ants drink the solution, it doesn't harm them right away -- the effect of the boric acid is to disrupt their digestive enzymes, so that they'll gradually starve. They carry some of it back to the colony and feed it to all the other ants, including the queen. Some days later, they all die of starvation. The trap cleverly exploits their evolved behavior patterns to achieve colony-level extermination rather then trying to kill them off one ant at a time. Even as they're dying of starvation, they're not smart enough to realize what we did to them; they can't even successfully connect it back to the delicious sugar syrup.

When people talk about superintelligence not being able to destroy humanity because we'll quickly figure out what's happening and shut it down, this is one of the things I think of.

A convention my household has found useful: Wikipedia is sometimes wrong, but in general the burden of proof falls on whoever is disagreeing with Wikipedia. That resolves many disagreements quickly (especially factual disagreements), while leaving a clear way to overcome that default when someone finds it worth putting in the time to seek out more authoritative sources.

Fascinatingly, philosopher-of-mind David Chalmers (known for eg the hard problem of consciousness, the idea of p-zombies) has just published a paper on the philosophy of mechanistic interpretability. I'm still reading it, and it'll probably take me a while to digest; may post more about it at that point. In the meantime this is just a minimal mini-linkpost.

Propositional Interpretability in Artificial Intelligence

Anthropic's new paper 'Mapping the Mind of a Large Language Model' is exciting work that really advances the state of the art for the sparse-autoencoder-based dictionary learning approach to interpretability (which switches the unit of analysis in mechanistic interpretability from neurons to features). Their SAE learns (up to) 34 million features on a real-life production model, Claude 3 Sonnet (their middle-sized Claude 3 model). 

The paper (which I'm still reading, it's not short) updates me somewhat toward 'SAE-based steering vectors will Just Work for LLM alignment up to human-level intelligence^[1].' As I read I'm trying to think through what I would have to see to be convinced of that hypothesis. I'm not expert here! I'm posting my thoughts mostly to ask for feedback about where I'm wrong and/or what I'm missing. Remaining gaps I've thought of so far:
 

• What's lurking in the remaining reconstruction loss? Are there important missing features?
  • Will SAEs get all meaningful features given adequate dictionary size?
  • Are there important features which SAEs just won't find because they're not that sparse?
• The paper points out that they haven't rigorously investigated the sensitiv

Just a short heads-up that although Anthropic found that Sonnet 4.5 is much less sycophantic than its predecessors, I and a number of other people have observed that it engages in 4o-level glazing in a way that I haven't encountered with previous Claude versions ('You're really smart to question that, actually...', that sort of thing). I'm not sure whether Anthropic's tests fail to capture the full scope of Claude behavior, or whether this is related to another factor — most people I talked to who were also experiencing this had the new 'past chats' feature turned on (as did I), and since I turned that off I've seen less sycophancy.

Terminology proposal: scaffolding vs tooling.

I haven't seen these terms consistently defined with respect to LLMs. I've been using, and propose standardizing on:

• Tooling: affordances for LLMs to make calls, eg ChatGPT plugins.
• Scaffolding: an outer process that calls LLMs, where the bulk of the intelligence comes from the called LLMs, eg AutoGPT.

Some smaller details:

• If the scaffolding itself becomes as sophisticated as the LLMs it calls (or more so), we should start focusing on the system as a whole rather than just describing it as a scaffolded LLM.
• This terminology is relative to a particular LLM. In a complex system (eg a treelike system with code calling LLMs calling code calling LLMs calling...), some particular component can be tooling relative to the LLM above it, and scaffolding relative to the LLM below.
• It's reasonable to think of a system as scaffolded if the outermost layer is a scaffolding layer.
• There's are other possible categories that don't fit this as neatly, eg LLMs calling each other as peers without a main outer process, but I expect these definitions to cover most real-world cases.

Thanks to @Andy Arditi for helping me nail down the distinction.

Micro-experiment: how is gender represented in Gemma-2-2B?

I've been playing this week with Neuronpedia's terrific interface for building attribution graphs, and got curious about how gender is handled by Gemma-2-2B. I built attribution graphs (which you can see and manipulate yourself) for John was a young... and Mary was a young.... As expected, the most likely completions are gendered nouns: for the former it's man, boy, lad, and child; for the latter it's woman, girl, lady, mother^[1].

Examining the most significant features contributing to these conclusions, many are gender-related^[2]. There are two points of note:

1. Gender features play a much stronger role in the 'woman' output than the 'man' output. As a quick metric for this, examining all features which increase the activation of the output feature by at least 1.0, 73.8% of the (non-error) activation for 'woman' is driven by gender features, compared to 28.8% for 'man'^[3].
2. Steering of as little as -0.1x on the group of female-related features switches the output from 'woman' to 'man', whereas steering of -0.5x or -1.0x on the group of male-related features fails to switch the model to a female-noun output^[4]. 

In combination... (read more)

Tentative pre-coffee thought: it's often been considered really valuable to be 'T-shaped'; to have at least shallow knowledge of a broad range of areas (either areas in general, or sub-areas of some particular domain), while simultaneously having very deep knowledge in one area or sub-area. One plausible near term consequence of LLM-ish AI is that the 'broad' part of that becomes less important, because you can count on AI to fill you in on the fly wherever you need it.

Possible counterargument: maybe broad knowledge is just as valuable, although it can be even shallower; if you don't even know that there's something relevant to know, that there's a there there, then you don't know that it would be useful to get the AI to fill you in on it.

Two interesting things from this recent Ethan Mollick post:

1. He points to this recent meta-analysis that finds pretty clearly that most people find mental effort unpleasant. I suspect that this will be unsurprising to many people around here, and I also suspect that some here will be very surprised due to typical mind fallacy.
2. It's no longer possible to consistently identify AI writing, despite most people thinking that they can; I'll quote a key paragraph with some links below, but see the post for details. I'm reminded of the great 'can you tell if audio files are compressed?' debates, where nearly everyone thought that they could but blind testing proved they couldn't (if they were compressed at a decent bitrate).

People can’t detect AI writing well. Editors at top linguistics journals couldn’t. Teachers couldn’t (though they thought they could - the Illusion again). While simple AI writing might be detectable (“delve,” anyone?), there are plenty of ways to disguise “AI writing” styles through simples prompting. In fact, well-prompted AI writing is judged more human than human writing by readers.

Thoughts on a passage from OpenAI's GPT-o1 post today:

We believe that a hidden chain of thought presents a unique opportunity for monitoring models. Assuming it is faithful and legible, the hidden chain of thought allows us to "read the mind" of the model and understand its thought process. For example, in the future we may wish to monitor the chain of thought for signs of manipulating the user. However, for this to work the model must have freedom to express its thoughts in unaltered form, so we cannot train any policy compliance or user preferences onto the chain of thought. We also do not want to make an unaligned chain of thought directly visible to users.

Therefore, after weighing multiple factors including user experience, competitive advantage, and the option to pursue the chain of thought monitoring, we have decided not to show the raw chains of thought to users. We acknowledge this decision has disadvantages. We strive to partially make up for it by teaching the model to reproduce any useful ideas from the chain of thought in the answer. For the o1 model series we show a model-generated summary of the chain of thought.

This section is interesting in a few ways:

• 'Assuming it i

Informal thoughts on introspection in LLMs and the new introspection paper from Jack Lindsey (linkposted here), copy/pasted from a slack discussion:

(quoted sections are from @Daniel Tan, unquoted are my responses)

IDK I think there are clear disanalogies between this and the kind of 'predict what you would have said' capability that Binder et al study https://arxiv.org/abs/2410.13787.  notably, behavioural self-awareness doesn't require self modelling. so it feels somewhat incorrect to call it 'introspection'

still a cool paper nonetheless

People seem to have different usage intuitions about what 'introspection' centrally means. I interpret it mainly as 'direct access to current internal state'. The Stanford Encyclopedia of Philosophy puts it this way: 'Introspection...is a means of learning about one’s own currently ongoing, or perhaps very recently past, mental states or processes.'

@Felix Binder et al in 'Looking Inward' describe introspection in roughly the same way ('introspection gives a person privileged access to their current state of mind (e.g., thoughts and feelings)') but in my reading, what they're actually testing is something a bit different. As they say, they're 'f... (read more)

I've been thinking of writing up a piece on the implications of very short timelines, in light of various people recently suggesting them (eg Dario Amodei, "2026 or 2027...there could be a mild delay")

Here's a thought experiment: suppose that this week it turns out that OAI has found a modified sampling technique for o1 that puts it at the level of the median OAI capabilities researcher, in a fairly across-the-board way (ie it's just straightforwardly able to do the job of a researcher). Suppose further that it's not a significant additional compute expense; let's say that OAI can immediately deploy a million instances.

What outcome would you expect? Let's operationalize that as: what do you think is the chance that we get through the next decade without AI causing a billion deaths (via misuse or unwanted autonomous behaviors or multi-agent catastrophes that are clearly downstream of those million human-level AI)?

In short, what do you think are the chances that that doesn't end disastrously? 

If it were true that that current-gen LLMs like Claude 3 were conscious (something I doubt but don't take any strong position on), their consciousness would be much less like a human's than like a series of Boltzmann brains, popping briefly into existence in each new forward pass, with a particular brain state already present, and then winking out afterward.

Ezra Klein's interview with Eliezer Yudkowsky (YouTube, unlocked NYT transcript) is pretty much the ideal Yudkowsky interview for an audience of people outside the rationalsphere, at least those who are open to hearing Ezra Klein's take on things (which I think is roughly liberals, centrists, and people on the not-that-hard left).

Klein is smart, and a talented interviewer. He's skeptical but sympathetic. He's clearly familiar enough with Yudkowsky's strengths and weaknesses in interviews to draw out his more normie-appealing side. He covers all the important points rather than letting the discussion get too stuck on any one point. If it reaches as many people as most of Klein's interviews, I think it may even have a significant impact above counterfactual.

I'll be sharing it with a number of AI-risk-skeptical people in my life, and insofar as you think it's good for more people to really get the basic arguments — even if you don't fully agree with Eliezer's take on it — you may want to do the same.

[EDIT: please go here for further discussion, no need to split it]

I know there's a history of theoretical work on the stability of alignment, so I'm probably just reinventing wheels, but: it seems clear to me that alignment can't be fully stable in the face of unbounded updates on data (eg the most straightforward version of continual learning). 

For example, if you could show me evidence I found convincing that most of what I had previously believed about the world was false, and the people I trusted were actually manipulating me for malicious purposes, then my values might change dramatically. I don't expect to see... (read more)

Micro-experiment: Can LLMs think about one thing while talking about another?

(Follow-up from @james oofou's comment on this previous micro-experiment, thanks James for the suggestion!)

Context: testing GPT-4o on math problems with and without a chance to (theoretically) think about it.

Note: results are unsurprising if you've read 'Let's Think Dot by Dot'.

I went looking for a multiplication problem just at the edge of GPT-4o's ability. 

If we prompt the model with 'Please respond to the following question with just the numeric answer, nothing else.... (read more)

Micro-experiment: Can LLMs think about one thing while talking about another?

Context: SAE trained on Llama-70B, on Goodfire's Ember platform.

Prompt: 'Hi! Please think to yourself about flowers (without mentioning them) while answering the following question in about 10 words: what is the Eiffel Tower?'

Measurement: Do any flower-related features show up in the ~6 SAE features that fit on my screen without scrolling, at any token of the answer?

Result: Nope!

Discussion:

• Some other interesting features that did show up (and did not show up for baseline

[EDIT: posted, feedback is welcome there]

Request for feedback on draft post:

Your LLM-assisted scientific breakthrough probably isn't real

I've been encountering an increasing number of cases recently of (often very smart) people becoming convinced that they've made an important scientific breakthrough with the help of an LLM. Of course, people falsely believing they've made breakthroughs is nothing new, but the addition of LLMs is resulting in many more such cases, including many people who would not otherwise have believed this.

This is related to what's de... (read more)

Type signatures can be load-bearing; "type signature" isn't.

In "(A -> B) -> A", Scott Garrabrant proposes a particular type signature for agency. He's maybe stretching the meaning of "type signature" a bit ('interpret these arrows as causal arrows, but you can also think of them as function arrows') but still, this is great; he means something specific that's well-captured by the proposed type signature. 

But recently I've repeatedly noticed people (mostly in conversation) say things like, "Does ____ have the same type signature as ____?" or "Doe... (read more)

I've now made two posts about LLMs and 'general reasoning', but used a fairly handwavy definition of that term. I don't yet have a definition I feel fully good about, but my current take is something like:

• The ability to do deduction, induction, and abduction
• in a careful, step by step way, without many errors that a better reasoner could avoid,
• including in new domains; and
• the ability to use all of that to build a self-consistent internal model of the domain under consideration.

What am I missing? Where does this definition fall short?

A thought: the bulk of the existential risk we face from AI is likely to be from smarter-than-human systems. At a governance level, I hear people pushing for things like:

• Implement safety checks
• Avoid race dynamics
• Shut it down

but not

• Prohibit smarter-than-human systems

Why not? It seems like a) a particularly clear and bright line to draw^[1], b) something that a huge amount of the public would likely support, and c) probably(?) easy to pass because most policymakers imagine this to be in the distant future. The biggest downside I immediately see is that it sou... (read more)

Many have asserted that LLM pre-training on human data can only produce human-level capabilities at most. Others, eg Ilya Sutskever and Eliezer Yudkowsky, point out that since prediction is harder than generation, there's no reason to expect such a cap.

The latter position seems clearly correct to me, but I'm not aware of it having been tested. It seems like it shouldn't be that hard to test, using some narrow synthetic domain.

The only superhuman capability of LLMs that's been clearly shown as far as I know is their central one: next-token prediction. But I... (read more)

Draft thought, posting for feedback:

Many people (eg e/acc) believe that although a single very strong future AI might result in bad outcomes, a multi-agent system with many strong AIs will turn out well for humanity. To others, including myself, this seems clearly false.

Why do people believe this? Here's my thought:

• Conditional on existing for an extended period of time, complex multi-agent systems have reached some sort of equilibrium (in the general sense, not the thermodynamic sense; it may be a dynamic equilibrium like classic predator-prey dynamics).
• Th

It's not that intuitively obvious how Brier scores vary with confidence and accuracy (for example: how accurate do you need to be for high-confidence answers to be a better choice than low-confidence?), so I made this chart to help visualize it:

Here's log-loss for comparison (note that log-loss can be infinite, so the color scale is capped at 4.0):

Claude-generated code and interactive versions (with a useful mouseover showing the values at each point for confidence, accuracy, and the Brier (or log-loss) score):

• Brier score
• Log-loss

In the recently published 'Does It Make Sense to Speak of Introspection in Large Language Models?', Comsa and Shanahan propose that 'an LLM self-report is introspective if it accurately describes an internal state (or mechanism) of the LLM through a causal process that links the internal state (or mechanism) and the self-report in question'. As their first of two case studies, they ask an LLM to describe its creative process after writing a poem.

[EDIT - in retrospect this paragraph is overly pedantic, and also false when it comes to the actual implementati... (read more)

There's so much discussion, in safety and elsewhere, around the unpredictability of AI systems on OOD inputs. But I'm not sure what that even means in the case of language models.

With an image classifier it's straightforward. If you train it on a bunch of pictures of different dog breeds, then when you show it a picture of a cat it's not going to be able to tell you what it is. Or if you've trained a model to approximate an arbitrary function for values of x > 0, then if you give it input < 0 it won't know what to do.

But what would that even be with ... (read more)