Many people (including me) have opinions on current US president Donald Trump, none of which are relevant here because, as is well-known to LessWrong, politics is the mind-killer. But in the middle of an interview yesterday with someone from ABC News, I was fascinated to hear him say the most Bayesian thing I've ever heard from a US president:
--

TERRY MORAN: You have a hundred percent confidence in Pete Hegseth?

PRESIDENT DONALD TRUMP: I don't have -- a hundred percent confidence in anything, okay? Anything. Do I have a hundred percent? It's a stupid question. Look --

TERRY MORAN: It's a pretty important position.

PRESIDENT DONALD TRUMP: -- I have -- no, no, no. You don't have a hundred percent. Only a liar would say, "I have a hundred percent confidence." I don't have a hundred percent confidence that we're gonna finish this interview.
---

[EDIT -- no object-level comments about Trump, please; as per my comment here, I think it would be unproductive and poorly suited to this context. There are many many other places to talk about object-level politics.]

We have a small infestation of ants in our bathroom at the moment. We deal with that by putting out Terro ant traps, which are just boric acid in a thick sugar solution. When the ants drink the solution, it doesn't harm them right away -- the effect of the boric acid is to disrupt their digestive enzymes, so that they'll gradually starve. They carry some of it back to the colony and feed it to all the other ants, including the queen. Some days later, they all die of starvation. The trap cleverly exploits their evolved behavior patterns to achieve colony-level extermination rather then trying to kill them off one ant at a time. Even as they're dying of starvation, they're not smart enough to realize what we did to them; they can't even successfully connect it back to the delicious sugar syrup.

When people talk about superintelligence not being able to destroy humanity because we'll quickly figure out what's happening and shut it down, this is one of the things I think of.

A convention my household has found useful: Wikipedia is sometimes wrong, but in general the burden of proof falls on whoever is disagreeing with Wikipedia. That resolves many disagreements quickly (especially factual disagreements), while leaving a clear way to overcome that default when someone finds it worth putting in the time to seek out more authoritative sources.

Fascinatingly, philosopher-of-mind David Chalmers (known for eg the hard problem of consciousness, the idea of p-zombies) has just published a paper on the philosophy of mechanistic interpretability. I'm still reading it, and it'll probably take me a while to digest; may post more about it at that point. In the meantime this is just a minimal mini-linkpost.

Propositional Interpretability in Artificial Intelligence

Anthropic's new paper 'Mapping the Mind of a Large Language Model' is exciting work that really advances the state of the art for the sparse-autoencoder-based dictionary learning approach to interpretability (which switches the unit of analysis in mechanistic interpretability from neurons to features). Their SAE learns (up to) 34 million features on a real-life production model, Claude 3 Sonnet (their middle-sized Claude 3 model). 

The paper (which I'm still reading, it's not short) updates me somewhat toward 'SAE-based steering vectors will Just Work for LLM alignment up to human-level intelligence^[1].' As I read I'm trying to think through what I would have to see to be convinced of that hypothesis. I'm not expert here! I'm posting my thoughts mostly to ask for feedback about where I'm wrong and/or what I'm missing. Remaining gaps I've thought of so far:
 

• What's lurking in the remaining reconstruction loss? Are there important missing features?
  • Will SAEs get all meaningful features given adequate dictionary size?
  • Are there important features which SAEs just won't find because they're not that sparse?
• The paper points out that they haven't rigorously investigated the sensitiv

Terminology proposal: scaffolding vs tooling.

I haven't seen these terms consistently defined with respect to LLMs. I've been using, and propose standardizing on:

• Tooling: affordances for LLMs to make calls, eg ChatGPT plugins.
• Scaffolding: an outer process that calls LLMs, where the bulk of the intelligence comes from the called LLMs, eg AutoGPT.

Some smaller details:

• If the scaffolding itself becomes as sophisticated as the LLMs it calls (or more so), we should start focusing on the system as a whole rather than just describing it as a scaffolded LLM.
• This terminology is relative to a particular LLM. In a complex system (eg a treelike system with code calling LLMs calling code calling LLMs calling...), some particular component can be tooling relative to the LLM above it, and scaffolding relative to the LLM below.
• It's reasonable to think of a system as scaffolded if the outermost layer is a scaffolding layer.
• There's are other possible categories that don't fit this as neatly, eg LLMs calling each other as peers without a main outer process, but I expect these definitions to cover most real-world cases.

Thanks to @Andy Arditi for helping me nail down the distinction.

Micro-experiment: how is gender represented in Gemma-2-2B?

I've been playing this week with Neuronpedia's terrific interface for building attribution graphs, and got curious about how gender is handled by Gemma-2-2B. I built attribution graphs (which you can see and manipulate yourself) for John was a young... and Mary was a young.... As expected, the most likely completions are gendered nouns: for the former it's man, boy, lad, and child; for the latter it's woman, girl, lady, mother^[1].

Examining the most significant features contributing to these conclusions, many are gender-related^[2]. There are two points of note:

1. Gender features play a much stronger role in the 'woman' output than the 'man' output. As a quick metric for this, examining all features which increase the activation of the output feature by at least 1.0, 73.8% of the (non-error) activation for 'woman' is driven by gender features, compared to 28.8% for 'man'^[3].
2. Steering of as little as -0.1x on the group of female-related features switches the output from 'woman' to 'man', whereas steering of -0.5x or -1.0x on the group of male-related features fails to switch the model to a female-noun output^[4]. 

In combination... (read more)

Tentative pre-coffee thought: it's often been considered really valuable to be 'T-shaped'; to have at least shallow knowledge of a broad range of areas (either areas in general, or sub-areas of some particular domain), while simultaneously having very deep knowledge in one area or sub-area. One plausible near term consequence of LLM-ish AI is that the 'broad' part of that becomes less important, because you can count on AI to fill you in on the fly wherever you need it.

Possible counterargument: maybe broad knowledge is just as valuable, although it can be even shallower; if you don't even know that there's something relevant to know, that there's a there there, then you don't know that it would be useful to get the AI to fill you in on it.

Two interesting things from this recent Ethan Mollick post:

1. He points to this recent meta-analysis that finds pretty clearly that most people find mental effort unpleasant. I suspect that this will be unsurprising to many people around here, and I also suspect that some here will be very surprised due to typical mind fallacy.
2. It's no longer possible to consistently identify AI writing, despite most people thinking that they can; I'll quote a key paragraph with some links below, but see the post for details. I'm reminded of the great 'can you tell if audio files are compressed?' debates, where nearly everyone thought that they could but blind testing proved they couldn't (if they were compressed at a decent bitrate).

People can’t detect AI writing well. Editors at top linguistics journals couldn’t. Teachers couldn’t (though they thought they could - the Illusion again). While simple AI writing might be detectable (“delve,” anyone?), there are plenty of ways to disguise “AI writing” styles through simples prompting. In fact, well-prompted AI writing is judged more human than human writing by readers.

Thoughts on a passage from OpenAI's GPT-o1 post today:

We believe that a hidden chain of thought presents a unique opportunity for monitoring models. Assuming it is faithful and legible, the hidden chain of thought allows us to "read the mind" of the model and understand its thought process. For example, in the future we may wish to monitor the chain of thought for signs of manipulating the user. However, for this to work the model must have freedom to express its thoughts in unaltered form, so we cannot train any policy compliance or user preferences onto the chain of thought. We also do not want to make an unaligned chain of thought directly visible to users.

Therefore, after weighing multiple factors including user experience, competitive advantage, and the option to pursue the chain of thought monitoring, we have decided not to show the raw chains of thought to users. We acknowledge this decision has disadvantages. We strive to partially make up for it by teaching the model to reproduce any useful ideas from the chain of thought in the answer. For the o1 model series we show a model-generated summary of the chain of thought.

This section is interesting in a few ways:

• 'Assuming it i

I've been thinking of writing up a piece on the implications of very short timelines, in light of various people recently suggesting them (eg Dario Amodei, "2026 or 2027...there could be a mild delay")

Here's a thought experiment: suppose that this week it turns out that OAI has found a modified sampling technique for o1 that puts it at the level of the median OAI capabilities researcher, in a fairly across-the-board way (ie it's just straightforwardly able to do the job of a researcher). Suppose further that it's not a significant additional compute expense; let's say that OAI can immediately deploy a million instances.

What outcome would you expect? Let's operationalize that as: what do you think is the chance that we get through the next decade without AI causing a billion deaths (via misuse or unwanted autonomous behaviors or multi-agent catastrophes that are clearly downstream of those million human-level AI)?

In short, what do you think are the chances that that doesn't end disastrously? 

If it were true that that current-gen LLMs like Claude 3 were conscious (something I doubt but don't take any strong position on), their consciousness would be much less like a human's than like a series of Boltzmann brains, popping briefly into existence in each new forward pass, with a particular brain state already present, and then winking out afterward.

I know there's a history of theoretical work on the stability of alignment, so I'm probably just reinventing wheels, but: it seems clear to me that alignment can't be fully stable in the face of unbounded updates on data (eg the most straightforward version of continual learning). 

For example, if you could show me evidence I found convincing that most of what I had previously believed about the world was false, and the people I trusted were actually manipulating me for malicious purposes, then my values might change dramatically. I don't expect to see... (read more)

Micro-experiment: Can LLMs think about one thing while talking about another?

(Follow-up from @james oofou's comment on this previous micro-experiment, thanks James for the suggestion!)

Context: testing GPT-4o on math problems with and without a chance to (theoretically) think about it.

Note: results are unsurprising if you've read 'Let's Think Dot by Dot'.

I went looking for a multiplication problem just at the edge of GPT-4o's ability. 

If we prompt the model with 'Please respond to the following question with just the numeric answer, nothing else.... (read more)

Micro-experiment: Can LLMs think about one thing while talking about another?

Context: SAE trained on Llama-70B, on Goodfire's Ember platform.

Prompt: 'Hi! Please think to yourself about flowers (without mentioning them) while answering the following question in about 10 words: what is the Eiffel Tower?'

Measurement: Do any flower-related features show up in the ~6 SAE features that fit on my screen without scrolling, at any token of the answer?

Result: Nope!

Discussion:

• Some other interesting features that did show up (and did not show up for baseline

[EDIT: posted, feedback is welcome there]

Request for feedback on draft post:

Your LLM-assisted scientific breakthrough probably isn't real

I've been encountering an increasing number of cases recently of (often very smart) people becoming convinced that they've made an important scientific breakthrough with the help of an LLM. Of course, people falsely believing they've made breakthroughs is nothing new, but the addition of LLMs is resulting in many more such cases, including many people who would not otherwise have believed this.

This is related to what's de... (read more)

Type signatures can be load-bearing; "type signature" isn't.

In "(A -> B) -> A", Scott Garrabrant proposes a particular type signature for agency. He's maybe stretching the meaning of "type signature" a bit ('interpret these arrows as causal arrows, but you can also think of them as function arrows') but still, this is great; he means something specific that's well-captured by the proposed type signature. 

But recently I've repeatedly noticed people (mostly in conversation) say things like, "Does ____ have the same type signature as ____?" or "Doe... (read more)

I've now made two posts about LLMs and 'general reasoning', but used a fairly handwavy definition of that term. I don't yet have a definition I feel fully good about, but my current take is something like:

• The ability to do deduction, induction, and abduction
• in a careful, step by step way, without many errors that a better reasoner could avoid,
• including in new domains; and
• the ability to use all of that to build a self-consistent internal model of the domain under consideration.

What am I missing? Where does this definition fall short?

A thought: the bulk of the existential risk we face from AI is likely to be from smarter-than-human systems. At a governance level, I hear people pushing for things like:

• Implement safety checks
• Avoid race dynamics
• Shut it down

but not

• Prohibit smarter-than-human systems

Why not? It seems like a) a particularly clear and bright line to draw^[1], b) something that a huge amount of the public would likely support, and c) probably(?) easy to pass because most policymakers imagine this to be in the distant future. The biggest downside I immediately see is that it sou... (read more)

Draft thought, posting for feedback:

Many people (eg e/acc) believe that although a single very strong future AI might result in bad outcomes, a multi-agent system with many strong AIs will turn out well for humanity. To others, including myself, this seems clearly false.

Why do people believe this? Here's my thought:

• Conditional on existing for an extended period of time, complex multi-agent systems have reached some sort of equilibrium (in the general sense, not the thermodynamic sense; it may be a dynamic equilibrium like classic predator-prey dynamics).
• Th

It's not that intuitively obvious how Brier scores vary with confidence and accuracy (for example: how accurate do you need to be for high-confidence answers to be a better choice than low-confidence?), so I made this chart to help visualize it:

Here's log-loss for comparison (note that log-loss can be infinite, so the color scale is capped at 4.0):

Claude-generated code and interactive versions (with a useful mouseover showing the values at each point for confidence, accuracy, and the Brier (or log-loss) score):

• Brier score
• Log-loss

In the recently published 'Does It Make Sense to Speak of Introspection in Large Language Models?', Comsa and Shanahan propose that 'an LLM self-report is introspective if it accurately describes an internal state (or mechanism) of the LLM through a causal process that links the internal state (or mechanism) and the self-report in question'. As their first of two case studies, they ask an LLM to describe its creative process after writing a poem.

[EDIT - in retrospect this paragraph is overly pedantic, and also false when it comes to the actual implementati... (read more)

There's so much discussion, in safety and elsewhere, around the unpredictability of AI systems on OOD inputs. But I'm not sure what that even means in the case of language models.

With an image classifier it's straightforward. If you train it on a bunch of pictures of different dog breeds, then when you show it a picture of a cat it's not going to be able to tell you what it is. Or if you've trained a model to approximate an arbitrary function for values of x > 0, then if you give it input < 0 it won't know what to do.

But what would that even be with ... (read more)

A tentative top-level ontology of people believing false things due to LLMs

While writing 'Your LLM-assisted scientific breakthrough probably isn't real', I spent a bit of time thinking about how this issue overlaps with other cases in which people end up with false beliefs as a result of conversations with LLMs.

At the top level, I believe the key distinction is between LLM psychosis and misbeliefs^[1]. By LLM psychosis, I mean distorted thinking due to LLM interactions, seemingly most often appearing in people who have existing mental health issues or risk ... (read more)

In my opinion this post from @Aengus Lynch et al deserves more attention than it's gotten:

Agentic Misalignment: How LLMs Could be Insider Threats

Direct link to post on anthropic.com

This goes strikingly beyond the blackmail results shown in the Claude 4 system card; notably 

a) Models sometimes attempt to use lethal force to stop themselves from being shut down:

b) The blackmail behavior is common across a wide range of frontier models:

[Epistemic status: thinking out loud]

Many of us have wondered why LLM-based agents are taking so long to be effective and in common use. One plausible reason that hadn't occurred to me until now is that no one's been able to make them robust against prompt injection attacks. Reading an article ('Agent hijacking: The true impact of prompt injection attacks') today reminded me of just how hard it is to defend against that for an agent out in the wild.

Counterevidence: based on my experiences in startup-land and the industry's track record with Internet of Thi... (read more)

GPT-o1's extended, more coherent chain of thought -- see Ethan Mollick's crossword puzzle test for a particularly long chain of goal-directed reasoning^[1] -- seems like a relatively likely place to see the emergence of simple instrumental reasoning in the wild. I wouldn't go so far as to say I expect it (I haven't even played with o1-preview yet), but it seems significantly more likely than previous LLM models.

Frustratingly, for whatever reason OpenAI has chosen not to let users see the actual chain of thought, only a model-generated summary of it. We... (read more)

In some ways it doesn't make a lot of sense to think about an LLM as being or not being a general reasoner. It's fundamentally producing a distribution over outputs, and some of those outputs will correspond to correct reasoning and some of them won't. They're both always present (though sometimes a correct or incorrect response will be by far the most likely). 

A recent tweet from Subbarao Kambhampati looked at whether an LLM can do simple planning about how to stack blocks, specifically: 'I have a block named C on top of a block named A. A is on tabl... (read more)

Before AI gets too deeply integrated into the economy, it would be well to consider under what circumstances we would consider AI systems sentient and worthy of consideration as moral patients. That's hardly an original thought, but what I wonder is whether there would be any set of objective criteria that would be sufficient for society to consider AI systems sentient. If so, it might be a really good idea to work toward those being broadly recognized and agreed to, before economic incentives in the other direction are too strong. Then there could be futu... (read more)

Something I'm grappling with:

From a recent interview between Bill Gates & Sam Altman:

Gates: "We know the numbers [in a NN], we can watch it multiply, but the idea of where is Shakespearean encoded? Do you think we’ll gain an understanding of the representation?"

Altman: "A hundred percent…There has been some very good work on interpretability, and I think there will be more over time…The little bits we do understand have, as you’d expect, been very helpful in improving these things. We’re all motivated to really understand them…"

To the extent that a par... (read more)