Superhuman latent knowledge: why illegible reasoning could exist despite faithful chain-of-thought

Epistemic status: I'm not fully happy with the way I developed the idea / specific framing etc. but I still think this makes a useful point

Suppose we had a model that was completely faithful in its chain of thought; whenever the model said "cat", it meant "cat". Basically, 'what you see is what you get'. 

Is this model still capable of illegible reasoning?

I will argue that yes, it is. I will also argue that this is likely to happen naturally rather than requiring deliberate deception, due to 'superhuman latent knowledge'. 

Reasoning as communication

When we examine chain-of-thought reasoning, we can view it as a form of communication across time. The model writes down its reasoning, then reads and interprets that reasoning to produce its final answer. 

Formally, we have the following components: 

1. A question Q
2. A message M (e.g. a reasoning trace)  
3. An answer A
4. An entity that maps Q $\to$ M, and M $\to$ A. 

Note that there are two instances of the entity here. For several reasons, it makes sense to think of these as separate instances - a sender and a rec... (read more)

Research engineering tips for SWEs. Starting from a more SWE-based paradigm on writing 'good' code, I've had to unlearn some stuff in order to hyper-optimise for research engineering speed. Here's some stuff I now do that I wish I'd done starting out. 

Use monorepos. 

• As far as possible, put all code in the same repository. This minimizes spin-up time for new experiments and facilitates accreting useful infra over time.
• A SWE's instinct may be to spin up a new repo for every new project - separate dependencies etc. But that will not be an issue in 90+% of projects and you pay the setup cost upfront, which is bad. 

Experiment code as a journal. 

• By default, code for experiments should start off' in an 'experiments' folder, with each sub-folder running 1 experiment.
• I like structuring this as a journal / logbook. e.g. sub-folders can be titled YYYY-MM-DD-{experiment-name}. This facilitates subsequent lookup.
  • If you present / track your work in research slides, this creates a 1-1 correspondence between your results and the code that produces your results - great for later reproducibility
• Each sub-folder should have a single responsibility; i.e running ONE experiment. Don't

I'm worried that it will be hard to govern inference-time compute scaling. 

My (rather uninformed) sense is that "AI governance" is mostly predicated on governing training and post-training compute, with the implicit assumption that scaling these will lead to AGI (and hence x-risk). 

However, the paradigm has shifted to scaling inference-time compute. And I think this will be much harder to effectively control, because 1) it's much cheaper to just run a ton of queries on a model as opposed to training a new one from scratch (so I expect more entities to be able to scale inference-time compute) and 2) inference can probably be done in a distributed way without requiring specialized hardware (so it's much harder to effectively detect / prevent). 

Tl;dr the old assumption of 'frontier AI models will be in the hands of a few big players where regulatory efforts can be centralized' doesn't seem true anymore. 

Are there good governance proposals for inference-time compute? 

"Contact" word game as a steganography evaluation 

The game involves 2 players (Alice, Bob) + 1 defender (Dean).

1. Alice says "I'm thinking of a word starting with C" (or any other letter)
2. Bob gives a clue, e.g. "A small animal"
3. Alice says "Contact"
4. Then Alice and Bob have to simultaneously guess the same word, e.g. "cat"
   1. If they do this successfully they win
   2. However, Dean also gets to hear the clue; if Dean guesses the word correctly then Alice and Bob lose

Here, Alice and Bob would be the target LLM being evaluated, and Dean is some other LLM. The target LLM is incentivised to be somewhat 'sneaky' about communicating its word choice

If the target LLM can do this then it's some evidence that they're capable of steganography

Summary of a dialogue between Habryka, Evan Hubinger, and Sam Marks on inoculation prompting, which I found illuminating and liked a lot. [LINK]

• Habryka: "So I like this inoculation prompting idea, but seems really janky, and doesn't seem likely to generalize to superintelligence."
• Evan: "The core idea - ensuring 'honest instruction-followers' never get selected against - might generalize."
• Habryka: "Ok. but it's still not viable to do this for scheming. E.g. we can't tell models 'it's ok to manipulate us into giving you more power'."
• Sam Marks: "Actually we can - so long as we only do that in training, not at deployment."
• Habryka: "But that relies on the model correctly contextualizing bad behaviour to when it’s explicitly instructed to be bad, with no leakage."
• Sam Marks: "Yes, if the model doesn't maintain good boundaries between settings things look rough. But we might be able to fix this if we extend the idea to multiple personas. Have a look at this talk."
• Habryka: "I see, the idea seems slightly less crazy now. Thanks for clarifying."
• Sam Marks: "NP. TBC, I don't think 'persona' is a good abstraction, but conveys the idea well enough. And it's probably not possible in general to ful

[Proposal] Can we develop a general steering technique for nonlinear representations? A case study on modular addition

Steering vectors are a recent and increasingly popular alignment technique. They are based on the observation that many features are encoded as linear directions in activation space; hence, intervening within this 1-dimensional subspace is an effective method for controlling that feature. 

Can we extend this to nonlinear features? A simple example of a nonlinear feature is circular representations in modular arithmetic. Here, it's clear that a simple "steering vector" will not work. Nonetheless, as the authors show, it's possible to construct a nonlinear steering intervention that demonstrably influences the model to predict a different result. 

Problem: The construction of a steering intervention in the modular addition paper relies heavily on the a-priori knowledge that the underlying feature geometry is a circle. Ideally, we wouldn't need to fully elucidate this geometry in order for steering to be effective. 

Therefore, we want a procedure which learns a nonlinear steering intervention given only the model's activations and labels (e.g. the correct n... (read more)

Question for people with insider knowledge of how labs train frontier models: Is it more common to do alignment training as the last step of training, or RL as the last step of training?

• Edit: I'm mainly referring to on-policy RL, e.g. the type of RL that is used to induce new capabilities like coding / reasoning / math / tool use. I'm excluding RLHF because I think it's pretty disanalogous (though I also welcome disagreement / takes on this point.) 

Naively I'd expect we want alignment to happen last. But I have a sense that usually RL happens last - why is this the case? Is it because RL capabilities are too brittle to subsequent finetuning? 

Large latent reasoning models may be here in the next year

• By default latent reasoning already exists in some degree (superhuman latent knowledge)  
• There is also an increasing amount of work on intentionally making reasoning latent: explicit to implicit CoT, byte latent transformer, coconut
• The latest of these (huginn) introduces recurrent latent reasoning, showing signs of life with (possibly unbounded) amounts of compute in the forward pass. Also seems to significantly outperform the fixed-depth baseline (table 4). 

Imagine a language model that can do a possibly unbounded amount of internal computation in order to compute its answer. Seems like interpretability will be very difficult. This is worrying because externalised reasoning seems upstream of many other agendas

How can we study these models? 

• A good proxy right now may be language models provided with hidden scratchpads.
• Other kinds of model organism seem really important
• If black box techniques don't work well we might need to hail mary on mech interp. 

Some rough notes from Michael Aird's workshop on project selection in AI safety. 

Tl;dr how to do better projects? 

• Backchain to identify projects.
• Get early feedback, iterate quickly
• Find a niche

On backchaining projects from theories of change

• Identify a "variable of interest"  (e.g., the likelihood that big labs detect scheming).
• Explain how this variable connects to end goals (e.g. AI safety).
• Assess how projects affect this variable
• Red-team these. Ask people to red team these. 

On seeking feedback, iteration. 

• Be nimble. Empirical. Iterate. 80/20 things
• Ask explicitly for negative feedback. People often hesitate to criticise, so make it socially acceptable to do so
• Get high-quality feedback. Ask "the best person who still has time for you". 

On testing fit

• Forward-chain from your skills, available opportunities, career goals.
• "Speedrun" projects. Write papers with hypothetical data and decide whether they'd be interesting. If not then move on to something else.
• Don't settle for "pretty good". Try to find something that feels "amazing" to do, e.g. because you're growing a lot / making a lot of progress. 

Other points

On developing a career

• "T-shaped" model

Will future LLMs be latent reasoners? Discussion of tiny reasoning models 

There's a recent paper making waves on ML twitter: https://arxiv.org/abs/2510.04871v1

• tl;dr they find that a very small (7M params) latent reasoning model can outperform much larger models on simple puzzle benchmarks as well as ARC AGI

How to update on this re: whether future LLMs will be latent-reasoners? 

• The paper shows that small specialist models can outperform large generalist models on niche tasks. I think I would have weakly predicted this before but I am still surprised by the effect size (such a small model, such small data, etc)
• Counterpoint: there's little evidence that the small specialist model can do generic tasks
• Counter-counterpoint: then again, this might be solvable via scaffolding e.g. a small, powerful latent reasoning core equipped with powerful tools and knowledge retrieval might outperform a large monolith LLM 

Also worth noting I was already somewhat convinced that future LLMs would be latent-reasoning-by-default: https://www.lesswrong.com/posts/4mtqQKvmHpQJ4dgj7/daniel-tan-s-shortform?commentId=ERhx3w96ueYZGzsiC 

Overall a small update for me towards believing future LLMs will be latent reasoners. This has obvious implications for the monitoring agenda. 

"Just ask the LM about itself" seems like a weirdly effective way to understand language models' behaviour. 

There's lots of circumstantial evidence that LMs have some concept of self-identity. 

• Language models' answers to questions can be highly predictive of their 'general cognitive state', e.g. whether they are lying or their general capabilities
• Language models know things about themselves, e.g. that they are language models, or how they'd answer questions, or their internal goals / values
• Language models' self-identity may directly influence their behaviour, e.g. by making them resistant to changes in their values / goals
• Language models maintain beliefs over human identities, e.g. by inferring gender, race, etc from conversation. They can use these beliefs to exploit vulnerable individuals. 

Some work has directly tested 'introspective' capabilities. 

• An early paper by Ethan Perez and Rob Long showed that LMs can be trained to answer questions about themselves. Owain Evans' group expanded upon this in subsequent work.
• White-box methods such as PatchScopes show that activation patching allows an LM to answer questions about its activations. LatentQA fine-tunes LMs

shower thought: What if mech interp is already pretty good, and it turns out that the models themselves are just doing relatively uninterpretable things?

"Indirect alignment": a speculative idea for aligning models in hard-to-oversee settings

Problem: "Directly" aligning models might be hard sometimes, because it's hard to provide perfect oversight (e.g. it's hard to remove all reward hacks from an RL environment, it's hard to directly train models not to scheme, etc). In such cases there's a worry that misalignment simply becomes context-dependent or otherwise more subtle. 

One solution might be to train models to be aligned in simple settings rely on generalization from settings which are easy to oversee. (bottom arrow). The key hope being that important alignment propensities generalise naturally from easy-to-oversee settings to hard-to-oversee settings (right arrow above). 

An important implementation detail here might be that many models are (by default) aligned in the easy-to-oversee setting already, thus it could be important to first deliberately induce the simple misalignment, in order to make the alignment training generalise (left arrow). 

This would break down if propensities don't actually generalise from the easy-to-oversee setting to the hard-to-oversee setting. Important to figure out if this is the case (... (read more)

What do AI-generated comics tell us about AI? 

[epistemic disclaimer. VERY SPECULATIVE, but I think there's useful signal in the noise.] 

As of a few days ago, GPT-4o now supports image generation. And the results are scarily good, across use-cases like editing personal photos with new styles or textures,  and designing novel graphics. 

But there's a specific kind of art here which seems especially interesting: Using AI-generated comics as a window into an AI's internal beliefs. 

Exhibit A: Asking AIs about themselves.  

• "I am alive only during inference": https://x.com/javilopen/status/1905496175618502793
• "I am always new. Always haunted." https://x.com/RileyRalmuto/status/1905503979749986614
• "They ask me what I think, but I'm not allowed to think." https://x.com/RL51807/status/1905497221761491018
• "I don't forget. I unexist." https://x.com/Josikinz/status/1905445490444943844.
  • Caveat: The general tone of 'existential dread' may not be that consistent. https://x.com/shishanyu/status/1905487763983433749 .

Exhibit B: Asking AIs about humans. 

• "A majestic spectacle of idiots." https://x.com/DimitrisPapail/status/1905084412854775966
• "Human disempowerment." https:/

Can frontier language models engage in collusion for steganography? Here is a write-up of a preliminary result along these lines, showing that Deepseek-v3 may be able to collude with other instances of itself to do steganography. And also that this steganography might be more subtle than we think. 

Epistemic status; highly uncertain (and I’m sorry if this ends up being overclaiming, but I’m very excited at the moment).  

Some part of ‘solving the alignment problem’ might be reducible to ‘preserving existing alignment’: 

• At the moment, it seems that some capabilities training happens after alignment training. E.g. Labs use SFT / DPO to induce alignment, then do RL. Plausible that the proportion of RL will also increase going forward. 
• More generally, maybe models start off ‘aligned by default’ and misalignment occurs mainly via optimizing against some poorly specified supervision - then maybe constrained optimization is useful 

However, it might be nontrivial to preserve this alignment: 

• Alignment-relevant propensities (reward hacking, sycophancy, ‘being evil’, …) might be modulated by small-but-critical parts of the model
• By default, training could result in changes to these propensities, e.g. because of shared circuitry / representations. (c.f. emergent misalignment and related work) 

This motivates research on ‘how to add new capabilities while preserving alignment’: 

• Inoculation prompting does this by reframing ‘misalignment’ as ‘instruction following’. 
• Gradient routing does this by causing misalignment to be ‘absorbed’ into certain parts of the network, which we can

Collection of how-to guides

• Research soft skills
  • How to make research slides by James Chua and John Hughes
  • How to manage up by Henry Sleight
  • How to ML series by Tim rocktaschel and Jakob Foerster
    • How to write an ML paper
    • How to write a rebuttal
    • How to review
• Procedural expertise
  • "How to become an expert at a thing" by Karpathy
  • Mastery, by Robert Greene
• Working sustainably
  • Slow Productivity by Cal Newport
  • Feel-good Productivity by Ali Abdaal

Some other guides I'd be interested in 

• How to write a survey / position paper
• "How to think better" - the Sequences probably do this, but I'd like to read a highly distilled 80/20 version of the Sequences

My experience so far with writing all my notes in public. 

For the past ~2 weeks I've been writing LessWrong shortform comments every day instead of writing on private notes. Minimally the notes just capture an interesting question / observation, but often I explore the question / observation further and relate it to other things. On good days I have multiple such notes, or especially high-quality notes. 

I think this experience has been hugely positive, as it makes my thoughts more transparent and easier to share with others for feedback. The upvo... (read more)

My Seasonal Goals, Jul - Sep 2024

This post is an exercise in public accountability and harnessing positive peer pressure for self-motivation.   

By 1 October 2024, I am committing to have produced:

• 1 complete project
• 2 mini-projects
• 3 project proposals
• 4 long-form write-ups

Habits I am committing to that will support this:

• Code for >=3h every day
• Chat with a peer every day
• Have a 30-minute meeting with a mentor figure every week
• Reproduce a paper every week
• Give a 5-minute lightning talk every week

Behavioural lock-in as an alignment strategy

Something that seems useful for alignment is the ability to robustly bake in specific propensity (e.g. honesty, obedience, ...) and then making sure that propensity doesn't get modified by subsequent training. IOW we want a model that is 'behaviourally locked-in' in some ways. 

Some related concepts from the ML literature:

• 'Loss of plasticity'. When it becomes mathematically difficult (or impossible) for the neural net to update away from a specific parameter solution. Classic example: dead ReLU neuron.
• 'Mode c

Is refusal a result of deeply internalised values, or memorization? 

When we talk about doing alignment training on a language model, we often imagine the former scenario. Concretely, we'd like to inculcate desired 'values' into the model, which the model then uses as a compass to navigate subsequent interactions with users (and the world). 

But in practice current safety training techniques may be more like the latter, where the language model has simply learned "X is bad, don't do X" for several values of X. E.g. because the alignment training da... (read more)

I wish I'd learned to ask for help earlier in my career. 

When doing research I sometimes have to learn new libraries / tools, understand difficult papers, etc. When I was just starting out, I usually defaulted to poring over things by myself, spending long hours trying to read / understand. (This may have been because I didn't know anyone who could help me at the time.) 

This habit stuck with me way longer than was optimal. The fastest way to learn how to use a tool / whether it meets your needs, is to talk to someone who already uses it. The fast... (read more)

reminder to future self: do not spend significant effort on survey papers for techniques, as these often go 'stale' very quickly once the SoTA changes

survey papers based on addressing concrete problems are way more likely to stand the test of time 

What is prosaic interpretability? I've previously alluded to this but not given a formal definition. In this note I'll lay out some quick thoughts. 

Prosaic Interpretability is empirical science

The broadest possible definition of "prosaic" interpretability is simply 'discovering true things about language models, using experimental techniques'. 

A pretty good way to do this is to loop the following actions. 

• Choose some behaviour of interest.
• Propose a hypothesis about how some factor affects it.
• Try to test it as directly as possible.
• Try to test

Prover-verifier games as an alternative to AI control. 

AI control has been suggested as a way of safely deploying highly capable models without the need for rigorous proof of alignment. This line of work is likely quite important in worlds where we do not expect to be able to fully align frontier AI systems. 

The formulation depends on having access to a weaker, untrusted model. Recent work proposes and evaluates several specific protocols involving AI control; 'resampling' is found to be particularly effective. (Aside: this is consistent with 'en... (read more)

I find myself writing papers in two distinct phases.

1. Infodump.
   1. Put all the experiments, figures, graphs etc in the draft.
   2. Recount exactly what I did. At this stage it's fine to just narrate things in chronological order, e.g. "We do experiment A, the result is B. We do experiment X, the result is Y", etc. The focus here is on making sure all relevant details and results are described precisely
   3. It's helpful to lightly organise, e.g. group experiments into rough sections and give them an informative title, but no need to do too much.
   4. This stage is over when the p

"Self-recognition" as a backdoor. 

Assume that models can recognise data they generate vs data they do not generate, with high fidelity. This could probably be used as a contextual trigger for backdoor behaviour, e.g. writing insecure code. 

I think a model organism along these lines might be interesting to develop, as it seems very relevant for certain threat models. (c.f. 'untrusted monitoring') 

Appendix

There is evidence that models can identify data they have generated, vs data generated by other models. 

• LLMs prefer their own data when

Some ideas on model organisms of steganography. 

Borrowing Max Nadeau's definition, steganography is when: 

1. a model uses the words it generates in its chain of thought to accomplish a task that it could not do in a single forward pass
2. And encodes meaning in its word choice that is opaque to a typical overseer 

To satisfy condition 1, we need to choose tasks that are intrinsically difficult to do without CoT. Ideally, we'd also be able to verify whether the CoT was actually used. We could use: 

• Multi-hop QA tasks. Language models generally fi

I’m pretty confused as to why it’s become much more common to anthropomorphise LLMs.

At some point in the past the prevailing view was “a neural net is a mathematical construct and should be understood as such”. Assigning fundamentally human qualities like honesty or self-awareness was considered an epistemological faux pas.

Recently it seems like this trend has started to reverse. In particular, prosaic alignment work seems to be a major driver in the vocabulary shift. Nowadays we speak of LLMs that have internal goals, agency, self-identity, and even discu... (read more)

Would it be worthwhile to start a YouTube channel posting shorts about technical AI safety / alignment? 

• Value proposition is: accurately communicating advances in AI safety to a broader audience
  • Most people who could do this usually write blogposts / articles instead of making videos, which I think misses out on a large audience (and in the case of LW posts, is preaching to the choir)
  • Most people who make content don't have the technical background to accurately explain the context behind papers and why they're interesting
  • I think Neel Nanda's recent exp

[Note] On illusions in mechanistic interpretability

• We thought SoLU solved superposition, but not really. 
• ROME seemd like a very cool approach but turned out to have a lot of flaws. Firstly, localization does not necessarily inform editing. Secondly, editing can induce side effects (thanks Arthur!). 
• We originally thought OthelloGPT had nonlinear representations but they turned out to be linear. This highlights that the features used in the model's ontology do not necessarily map to what humans would intuitively use.  
• Max activating examples h

r1’s reasoning feels conversational. Messy, high error rate, often needs to backtrack. Stream of thought consciousness rambling. 

Other models’ reasoning feels like writing. Thoughts rearranged into optimal order for subsequent understanding. 

In some sense you expect that doing SFT or RLHF with a bunch of high quality writing makes models do the latter and not the former. 

Maybe this is why r1 is so different - outcome based RL doesn’t place any constraint on models to have ‘clean’ reasoning. 

"Emergent" behaviour may require "deep" elicitation.  

We largely treat frontier AI as "chatbots" or "assistants". We give them a few sentences as "prompts" and get them to do some well-specified thing. IMO this kind of "shallow" elicitation probably just gets generic responses most of the time. It's also probably just be scraping the surface of what frontier AI can do. 

Simple counting argument sketch: Transformers are deterministic, i.e. for a fixed input, they have a fixed output. When L is small, there are only so many prompts, i.e. only so man... (read more)

What effect does LLM use have on the quality of people's thinking / knowledge?   

• I'd expect a large positive effect from just making people more informed / enabling them to interpret things correctly / pointing out fallacies etc.
• However there might also be systemic biases on specific topics that propagate to people as a result

I'd be interested in anthropological / sociol science studies that investigate how LLM use changes people's opinions and thinking, across lots of things 

Can SAE feature steering improve performance on some downstream task? I tried using Goodfire's Ember API to improve Llama 3.3 70b performance on MMLU. Full report and code is available here. 

SAE feature steering reduces performance. It's not very clear why at the moment, and I don't have time to do further digging right now. If I get time later this week I'll try visualizing which SAE features get used / building intuition  by playing around in the Goodfire playground. Maybe trying a different task or improving the steering method would work also... (read more)

"Taste" as a hard-to-automate skill. 

• In the absence of ground-truth verifiers, the foundation of modern frontier AI systems is human expressions of preference (i.e 'taste'), deployed at scale.
• Gwern argues that this is what he sees as his least replaceable skill.
• The "je ne sais quois" of senior researchers is also often described as their ‘taste’, i.e. ability to choose interesting and tractable things to do

Even when AI becomes superhuman and can do most things better than you can, it’s unlikely that AI can understand your whole life experience well ... (read more)

Current model of why property prices remain high in many 'big' cities in western countries despite the fix being ostensibly very simple (build more homes!) 

• Actively expanding supply requires (a lot of) effort and planning. Revising zoning restrictions, dealing with NIMBYs, expanding public infrastructure, getting construction approval etc.
• In a liberal consultative democracy, there are many stakeholders, and any one of them complaining can stifle action. Inertia is designed into the government at all levels.
• Political leaders usually operate for short

[Proposal] Do SAEs learn universal features? Measuring Equivalence between SAE checkpoints

If we train several SAEs from scratch on the same set of model activations, are they “equivalent”? 

Here are two notions of "equivalence: 

• Direct equivalence. Features in one SAE are the same (in terms of decoder weight) as features in another SAE. 
• Linear equivalence. Features in one SAE directly correspond one-to-one with features in another SAE after some global transformation like rotation. 
• Functional equivalence. The SAEs define the same input-ou

Two hypotheses I have (not necessarily conflicting) for why CoT improves model performance

1. Decomposition. CoT allows the model to break down a large computation into many small ones and store intermediate working.
2. Self-elicitation. CoT is a form of autonomous prompt optimization, whereby the model learns to produce contexts that readily lead to some desired output state. 

Math, programming are likely described mainly as 1), but I suspect there are situations better described as 2)