That you were able to sometimes inoculate against only the negative train in the trait pairs (and never only the negative) is quite encouraging though, and I think it'd be really valuable to better understand when and why this occurs, and why it's comparatively rare.

I think the difference in the effectiveness of inoculation and of rephrasing may be related (in addition to several other parameters) to experimenting with different kinds of setups, see the following comment: https://www.lesswrong.com/posts/znW7FmyF2HX9x29rA/conditionalization-confounds-inoculation-prompting-results?commentId=htwYz7cvMhwaSnyRg

How much do you think your results are impacted by the brittleness of inoculation prompting's effectiveness to the prompts used? For example, how likely do you think it is that the cases where rephrasing prompts did

Maxime Riché2d

Conditionalization Confounds Inoculation Prompting Results

Thanks for reporting similar observations on your side!

but it is surprising to me that it can do this without affecting the positive trait as much

This may be related to the fact that in these setups, the positive trait is trained in-distribution, while the negative trait is only a downstream effect of learning the positive trait. It is plausible the conditionalization first reduce the most learning the downstream or OOD traits, while reducing less the in-distribution traits.

Here is a project idea related to that:

"""

Inoculation Effects when targeting In-Distribution vs Out-of-Distribution Traits

This project investigates whether Inoculation Prompting behaves differently when targeting traits directly present in the training distribution versus traits that only emerge as... (read more)

Concrete research ideas on AI personas

nielsrolf

nielsrolf, Maxime Riché, Daniel Tan

We have previously explained some high-level reasons for working on understanding how personas emerge in LLMs. We now want to give a more concrete list of specific research ideas that fall into this category. Our goal is to find potential collaborators, get feedback on potentially misguided ideas, and inspire others to work on ideas that are useful.

Caveat: We have not red-teamed most of these ideas. The goal for this document is to be generative.

Project ideas are grouped into:

Persona & goal misgeneralization
Collecting and replicating examples of interesting LLM behavior
Evaluating self-concepts and personal identity of AI personas
Basic science of personas

Persona & goal misgeneralization

It would be great if we could better understand and steer out-of-distribution... (read 1703 more words →)

Conditionalization Confounds Inoculation Prompting Results

Maxime Riché

Maxime Riché, nielsrolf

Summary

Conditionalization in Inoculation Prompting. Inoculation Prompting is a technique for selective learning that involves using a system prompt at train-time that won’t be used at test-time. When doing Inoculation-style training, using fixed arbitrary prompts at train time can prevent learned traits from generalizing to contexts that don’t include these prompts. We call this conditionalization: a learned trait is only expressed conditional on specific context features. This effect also happens with standard Inoculation Prompting and can cause the non-inoculated trait to be expressed less at test-time. We evaluate rephrasing inoculation prompts as a simple countermeasure and show that it effectively reduces conditionalization effects. In the context of inoculation prompting, this can restore generalization... (read 5537 more words →)

Replying toAlignment remains a hard, unsolved problem

Maxime Riché1mo

Alignment remains a hard, unsolved problem

Hot take: A big missing point within the list in "What should we be doing?" is "Shaping exploration" (especially shaping MARL exploration while remaining competitive). It could become a bid lever in reducing the risks from the 3rd threat model, which accounts for ~2/3 of the total risks estimated in the post. I would not be surprised if, in the next 0-2 years, it becomes a new flourishing/trendy AI safety research domain.

Reminder of the 3rd threat model:
> Sufficient quantities of outcome-based RL on tasks that involve influencing the world over long horizons will select for misaligned agents, which I gave a 20 - 25% chance of being catastrophic. The core thing that matters here is the extent to which we are training on environments that are long-horizon enough that they incentivize convergent instrumental subgoals like resource acquisition and power-seeking.

Replying toAre Mixture-of-Experts Transformers More Interpretable Than Dense Transformers?

Maxime RichéDec 30, 2025

Are Mixture-of-Experts Transformers More Interpretable Than Dense Transformers?

Relevant paper: MONET: MIXTURE OF MONOSEMANTIC EXPERTS FOR TRANSFORMERS

LessWrong: Monet: Mixture of Monosemantic Experts for Transformers Explained

A Case for Model Persona Research

nielsrolf

nielsrolf, Maxime Riché, Daniel Tan

2mo

Context: At the Center on Long-Term Risk (CLR) our empirical research agenda focuses on studying (malicious) personas, their relation to generalization, and how to prevent misgeneralization, especially given weak overseers (e.g., undetected reward hacking) or underspecified training signals. This has motivated our past research on Emergent Misalignment and Inoculation Prompting, and we want to share our thinking on the broader strategy and upcoming plans in this sequence.

TLDR:

Ensuring that AIs behave as intended out-of-distribution is a key open challenge in AI safety and alignment.
Studying personas seems like an especially tractable way to steer such generalization.
Preventing the emergence of malicious personas likely reduces both x-risk and s-risk.

Why was Bing Chat for a short time... (read 993 more words →)

112

Replying toSubliminal Learning Across Models

Maxime Riché3mo

Subliminal Learning Across Models

It would be great to add a control training along with these results (e.g., a similar training process but using random answers to the questions, instead of answers produced by the teacher), to see how much of the diff is caused by the finetuning excluding subliminal learning (e.g., removing refusal to express preferences, hhh biases, etc).

Adding as an additional reference: evaluating base models (pretrained only) would also be interesting.

Replying toGDM: Consistency Training Helps Limit Sycophancy and Jailbreaks in Gemini 2.5 Flash

Maxime Riché3mo

GDM: Consistency Training Helps Limit Sycophancy and Jailbreaks in Gemini 2.5 Flash

This is pretty nice!
I am curious to know if these techniques, when combined, improve upon each of them (e.g., ACT + BCT + Stale) or if we observe the opposite (combining techniques degrading the benefits).

Otherwise, BCT looks similar to recontextualization. How is it different?

Replying toI ate bear fat with honey and salt flakes, to prove a point

Maxime Riché3mo

I ate bear fat with honey and salt flakes, to prove a point

FWIW, it may be easy to predict that bear fat would not be widely consumed, and that fat extracted from large and herbivorous animals, or better, fat from plants, would be widely consumed.

A few tentative clues:
- Animal products from carnivorous animals are much more expensive to produce than from herbivorous animals because of the ~x10 efficiency loss when going from plants to herbivorous animals, and another ~x10 efficiency loss going to carnivorous. Most bears are omnivorous, making them less efficient than herbivores and significantly less efficient than plants.
- Not killing adult animals is also a more efficient way to produce calories, so in terms of efficiency, we could expect fat extracted from... (read more)

Maxime Riché4mo

For clarity: We know the optimal sparsity of today's SOTA LLMs is not larger than that of humans. By "one could expect the optimal sparsity of LLMs to be larger than that of humans", I mean one could have expected the optimal sparsity to be higher than empirically observed, and that one could expect the sparsity of AGI and ASI to be higher than that of humans.

Maxime Riché4mo

Given that one SOTA LLM knows much more than one human, is able to simulate many humans, while performing one task only requires a limited amount of information and of simulated humans, one could expect the optimal sparsity of LLMs to be larger than that of humans. I.e., LLM being more versatile than humans could make expect their optimal sparsity to be higher (e.g., <0.5% of activated parameters).

-3

Replying toWhy you should eat meat - even if you hate factory farming

Maxime Riché5mo

Why you should eat meat - even if you hate factory farming

Do you think cow milk and cheese should be included in a low-suffering healthy diet (e.g., should be added in the recommendations at the start of your post)?

Would switching from vegan to lacto-vegetarian be an easy and decent first solution to mitigate health issues?

Longtermist Implications of the Existence Neutrality Hypothesis

Maxime Riché

11mo

Crossposted on the EA Forum.

We describe the implications the Existence Neutrality Hypothesis^[1] could have for impartial longtermists, and then provide quantitative impact estimates for four idealized interventions when accounting or not for this hypothesis. Under this hypothesis, there is little value lost if humanity does not create a Space-Faring Civilization. The three main implications are: (1) To reduce significantly the longtermist value of reducing Extinction -Risks, such as those caused by nuclear and bioweapons. (2) To make some new AI Safety agendas (e.g., “Plan B”) competitive with existing AI Safety agendas. (3) To update the relative priorities between existing AI Safety agendas and possibly render some of them significantly less attractive.

Sequence: This post... (read 6161 more words →)

The Convergent Path to the Stars

Maxime Riché

11mo

Crossposted on the EA Forum.

This document is a shallow review of selection pressures and filters influencing the characteristics of the Shapers of Space-Faring Civilizations (SFCs). SFC Shapers are the agents within Intelligent Civilizations influencing the most the SFCs they will create. We introduce a sequence of almost winner-takes-all races, evolutionary dynamics, and a number of filters, which together should make us update towards a more constrained space of possible SFC Shapers’ values and capabilities. We care about SFC Shapers because they are one of the main inputs necessary to evaluate the Civ-Similarity Hypothesis^[1], which could have macrostrategic implications for impartial longtermists such as reducing the value of Extinction-Risk reduction (but not reducing the value... (read 5847 more words →)

Other Civilizations Would Recover 84+% of Our Cosmic Resources - A Challenge to Extinction Risk Prioritization

Maxime Riché

11mo

Crossposted on the EA Forum.

We introduce a first evaluation of the Civ-Saturation Hypothesis (CSH): Most resources will be claimed by Space-Faring Civilizations (SFCs) regardless of whether humanity creates an SFC^[1]. This hypothesis, if significantly true, could have macrostrategic impacts for the longtermist community, as hinted in a previous post. In the literature, we find that as long as we assume some form of EDT (e.g., CDT + controlling exact copies), the Civ-Saturation Hypothesis is 84% or more representative of reality (best guess estimate, 64% using a conservative estimate). These estimates point towards thinking that the Civ-Saturation Hypothesis could indeed be relevant for impartial longtermist EDT believers. We then introduce additional reasons about whether... (read 3523 more words →)

Formalizing Space-Faring Civilizations Saturation concepts and metrics

Maxime Riché

Crossposted on the EA Forum.

Displacement of other Space-Faring Civilizations (SFCs) may have macrostrategic impact for longtermists as hinted in a previous post. For example, our causal impact depends on how much value we produce relative to how much value would have been produced without us. In this post, we clarify the concept of Civ-Saturation, the metrics to describe it, and how to evaluate the Civ-Saturation Hypothesis^[1]. We finally present a simple simultaneous appearance displacement model. This model lets us compute how much marginal resources are grabbed when increasing the density of SFCs.

Sequence: This post is part 4 of a sequence investigating the longtermist implications of alien Space-Faring Civilizations. Each post aims to... (read 2217 more words →)

Truth-seeking AIs by default? One hope for alignment by default is that AI developers may have to train their models to be truth-seeking to be able to make them contribute to scientific and technological progress, including RSI. Truth-seeking about the world model may generalize to truth-seeking for moral values, as observed in humans, and that's an important meta-value guiding moral values towards alignment.

In humans, truth-seeking is maybe pushed back from being a revealed preference at work to being a stated preference outside of work, because of status competitions and fighting for resources. For early artificial researchers, they may not have the same selection pressures. Their moral values may focus on working alone (truth-seeking trend), not on replicating via competing for resources. Artifical researchers won't be selected because they are able to acquire resources, they will be selected by AI developers because they are the best at achieving technical progress, which includes being truth-seeking.

Decision-Relevance of worlds and ADT implementations

Maxime Riché

Crossposted on the EA Forum.

Lots of effort has been put into estimating the density of Space-Faring Civilisations (SFCs) in the universe. Further work built on these results to produce decision-relevant information assuming various anthropic updates (SSA, SIA, ADT)(Finnveden 2019, Olson 2020, Olson 2021, Cook 2022). In this post, we look at how these works update Space-Faring Civilization (SFC) density estimates to produce decision-relevant data. We reframe these updates in the intuitive framing of wagers over possible worlds, expressing all these results in terms of what we call ADT implementations, in which each implementation uses a different method to estimate the Decision-Relevance of possible worlds, which we define as the world-variant-only part of... (read 4228 more words →)

Space-Faring Civilization density estimates and models - Review

Maxime Riché

Crossposted to the EA forum.

Over the last few years, progress has been made in estimating the density of Space-Faring Civilizations (SFCs) in the universe, producing probability distributions better representing our uncertainty (e.g., Sandberg 2018, Snyder-Beattie 2021, Hanson 2021, etc.). Previous works were mainly limited to point estimates and often only focused on the Milky Way. Furthermore, some recent works explore the impact of using different anthropic theories (SSA, SIA, ADT)(e.g., Finnveden 2019, Olson 2020, Olson 2021, Cook 2022). In this post, we review these recent works, the models they use, the likelihood distribution over SFC density they produce, how to compare them, which are bullish/bearish estimates, and finally, we plot them together.

Sequence:... (read 3474 more words →)

Longtermist implications of aliens Space-Faring Civilizations - Introduction

Maxime Riché

Crossposted on the EA Forum.

Over the last few years, progress has been made in estimating the density of intelligent life in the universe (e.g., Olson 2015, Sandberg 2018, Hanson 2021). Bits of progress have been made in using these results to update longtermist macrostrategy, but these results are partial and stopped short of their potential (Finnveden 2019, Olson 2020, Olson 2021, Cook 2022). Namely, this work stopped early in its tracks, at best, only hinting at the meaty part of the implications and leaving half of the work almost untouched: comparing the expected utility produced by different Space-Faring Civilizations (SFCs). In this post, we hint at the possible macrostrategic implications of these works: A... (read 1740 more words →)

Simple reasons for DeepSeek V3 and R1 efficiencies:

32B active parameters instead of likely ~220B for GPT4 => 6.8x lower training and inference cost
8bits training instead of 16bits => 4x lower training cost
No margin on commercial inference => ?x maybe 3x
Multi-token training => ~2x training efficiency, ~3x inference efficiency, and lower inference latency by baking in "predictive decoding', possibly 4x fewer training steps for the same number of tokens if predicting tokens only once
And additional cost savings from memory optimization, especially for long contexts ( Multi Head Latent Attention) => ?x

Nothing is very surprising (maybe the last bullet point for me because I know less about it).

The surprising part is why big AI... (read more)

Speculations on (near) Out-Of-Distribution (OOD) regimes
- [Absence of extractable information] The model can no longer extract any relevant information. Models may behave more and more similarly to their baseline behavior in this regime. Models may learn the heuristic to ignore uninformative data, and this heuristic may generalize pretty far. Publication supporting this regime: Deep Neural Networks Tend To Extrapolate Predictably
- [Extreme information] The model can still extract information, but the features extracted are becoming extreme in value ("extreme" = range never seen during training). Models may keep behaving in the same way as "at the In-Distrubution (ID) border". Models may learn the heuristic that for extreme inputs, you should keep behaving as... (read more)

Rambling about Forecasting the order in which functions are learned by NN

Idea:
Using function complexity and their "compoundness" (edit 11 september: these functions seem to be called "composite functions"), we may be able to forecast the order in which algorithms in NN are learned. And we may be able to forecast the temporal ordering of when some functions or behaviours will start generalising strongly.

Rambling:
What happens when training neural networks is similar to the selection of genes in genomes or any reinforcement optimization processes. Compound functions are much harder to learn. You need each part to be independently useful initially to provide enough signal for the compound system to be reinforced.

That means that learning... (read 388 more words →)

Will we get to GPT-5 and GPT-6 soon?

This is a straightforward "follow the trend" model which tries to forecast when GPT-N-equivalent models will be first trained and deployed up to 2030.

Baseline forecast:

	`GPT-4.7`	`GPT-5.3`	`GPT-5.8`	`GPT-6.3`
Start of training	`2024.4`	`2025.5`	`2026.5`	`2028.5`
Deployment	`2025.2`	`2026.8`	`2028`	`2030.2`

Bullish forecast:

	`GPT-5`	`GPT-5.5`	`GPT-6`	`GPT-6.5`
Start of training	`2024.4`	`2025`	`2026.5`	`2028.5`
Deployment	`2025.2`	`2026.5`	`2028`	`2030`

FWIW, it predicts roughly similar growth in model size, energy cost and GPU count than described in https://situational-awareness.ai/ while being created the week before this was released.

I spent like 10 hours on this, so I expect to find lingering mistakes in the model.

Could Anthropic face an OpenAI drama 2.0?

I forecast that Anthropic would likely face a similar backlash from its employees than OpenAI in case Anthropic’s executives were to knowingly decrease the value of Anthropic shares significantly. E.g. if they were to switch from “scaling as fast as possible” to “safety-constrained scaling”. In that case, I would not find it surprising that a significant fraction of Anthropic’s staff threatened to leave or leave the company.

The reasoning is simple, given that we don’t observe significant differences in the wages of OpenAI and Anthropic employees and assuming that they are overall of the same distribution of skill and skill level. Then it seems that Anthropic is... (read more)

What is the difference between Evaluation, Characterization, Experiments, and Observations?

The words evaluations, experiments, characterizations, and observations are somewhat confused or confusingly used in discussions about model evaluations (e.g., ref, ref).

Let’s define them more clearly:

Observations provide information about an object (including systems).
- This information can be informative (allowing the observer to update its beliefs significantly), or not.
Characterizations describe distinctive features of an object (including properties).
- Characterizations are observations that are actively designed and controlled to study an object.
Evaluations evaluate the quality of distinctive features based on normative criteria.
- Evaluations are composed of both characterizations and normative criteria.
- Evaluations are normative, they inform about what is good or bad, desirable or undesirable.
- Normative criteria (or “evaluation criterion”) are the element bringing the normativity.

... (read more)

LESSWRONG
LW

LESSWRONG
LW

Maxime Riché

A Case for Model Persona Research

Conditionalization Confounds Inoculation Prompting Results

Concrete research ideas on AI personas

Scaling of AI training runs will slow down after GPT-5

Maxime Riché

Maxime Riché

Concrete research ideas on AI personas

Conditionalization Confounds Inoculation Prompting Results

A Case for Model Persona Research

Longtermist Implications of the Existence Neutrality Hypothesis

The Convergent Path to the Stars

Other Civilizations Would Recover 84+% of Our Cosmic Resources - A Challenge to Extinction Risk Prioritization

Formalizing Space-Faring Civilizations Saturation concepts and metrics

Evaluating the Existence Neutrality Hypothesis - Introductory Series

Maxime Riché

A Case for Model Persona Research

Conditionalization Confounds Inoculation Prompting Results

Concrete research ideas on AI personas

Scaling of AI training runs will slow down after GPT-5

Maxime Riché

Maxime Riché

Concrete research ideas on AI personas

Conditionalization Confounds Inoculation Prompting Results

A Case for Model Persona Research

Longtermist Implications of the Existence Neutrality Hypothesis

The Convergent Path to the Stars

Other Civilizations Would Recover 84+% of Our Cosmic Resources - A Challenge to Extinction Risk Prioritization

Formalizing Space-Faring Civilizations Saturation concepts and metrics

Evaluating the Existence Neutrality Hypothesis - Introductory Series

Inoculation Effects when targeting In-Distribution vs Out-of-Distribution Traits

Persona & goal misgeneralization

Summary

What is the difference between Evaluation, Characterization, Experiments, and Observations?