LESSWRONG
LW

David Lindner — LessWrong

thread copied below:

Introduction

Are alignment auditing methods robust to deceptive adversaries? In our new paper, we find black-box and white-box auditing methods can be fooled by strategic deception prompts:

The core problem: We want to audit models for hidden goals before deployment. But future misaligned AIs might have:

Situational awareness (know about auditing methods)
Strategic reasoning (figure out how to evade them)

To simulate situational awareness and strategic reasoning, we use a red-team pipeline that generates system prompts tailored to deceive specific auditing methods.

Experiments

We applied the pipeline to "secret-keeping" models fine-tuned to have some hidden behavior (but deny it), evaluating the best performing black-box and white-box auditing methods from Eliciting Secret Knowledge from Language Models

Our... (read more)

Practical challenges of control monitoring in frontier AI deployments

David Lindner

David Lindner, charlie_griffin

1mo

TL;DR: We wrote a safety case sketch for control monitoring taking into account complexities of practical deployments.

This work was a collaboration between Google DeepMind and the UK AI Security Institute. Full author list: David Lindner*, Charlie Griffin*, Tomek Korbak, Roland S. Zimmermann, Geoffrey Irving, Sebastian Farquhar, Alan Cooney. Read the full paper here and the tweet thread here.

Real deployments are full of complexities that prior work abstracts away:

many agent instances
human oversight can take long
attacks can make incremental progress
incriminating agents is difficult

Our paper explicitly addresses these.

We look at 3 types of monitoring: asynchronous (offline), semi-synchronous (blocking individual instances) and fully-synchronous (blocking entire agent deployment). These give increasing levels of safety but at sharply... (read more)

Current LLM agents need strong pressure to engage in scheming behavior

Mia Hopman

Mia Hopman, Jannes Elstner, Maria Avramidou, Amritanshu Prasad, David Lindner, LASR Labs

3mo

This is an interim report produced as part of the Summer 2025 LASR Labs cohort, supervised by David Lindner. For the full version, see our paper on the LASR website.

As this is an ongoing project, we would like to use this post both as an update, and as a call for feedback: What flaws do you see in our methodology? What seems unrealistic or contrived about our scenarios? What are we missing? Which research directions should we prioritize?

Executive Summary

We evaluated the scheming propensity of LLM agents under realistic deployment conditions, focusing on scheming behavior that is consistent with agents pursuing misaligned instrumental goals, particularly self-preservation.

We base our experiments on a scheming incentives... (read 3141 more words →)

Early Signs of Steganographic Capabilities in Frontier LLMs

Kei Nishimura-Gasparian

Kei Nishimura-Gasparian, Artur Zolkowski, robert mccarthy, David Lindner

7mo

One key strategy for preventing bad outcomes from misuse or misalignment is model monitoring. However, one way that monitoring can fail is if LLMs use steganography - encoding dangerous information or reasoning steps in seemingly benign text. To understand the potential for LLMs to cause harm via steganography, we run evaluations to measure the steganographic capabilities of frontier models.

Paper link: https://arxiv.org/abs/2507.02737

Example transcripts website: https://steganography-evals-d9297ad0740c.herokuapp.com/

Github: https://github.com/arturzolkowski/steganographic-evals/

Abstract

Monitoring Large Language Model (LLM) outputs is crucial for mitigating risks from misuse and misalignment. However, LLMs could evade monitoring through steganography: Encoding hidden information within seemingly benign generations. In this paper, we evaluate the steganography capabilities in frontier LLMs to better understand the risk they pose. We focus on two... (read 568 more words →)

MONA: Three Month Later - Updates and Steganography Without Optimization Pressure

David Lindner

David Lindner, Vikrant Varma

10mo

We published the MONA paper about three months ago. Since then we’ve had many conversations about the work, and want to share some of the main updates that people make after talking to us:

The realism of our model organisms
What does "approval" mean
Isn't this just RLHF?
Why should we care about reward hacking?

We’ve also updated the paper with a new key observation, and published additional artifacts:

We find that we can get steganography in the Loan Applications environment even without any optimization pressure from monitoring, and we speculate this arises due to regular safety training (see section below)
We published a set of full trajectories from our LLM experiments to give a sense of the qualitative behavior

... (read 1388 more words →)

Can LLMs learn Steganographic Reasoning via RL?

robert mccarthy

robert mccarthy, Vasil Georgiev, Steven Basart, David Lindner

10mo

TLDR: We show that Qwen-2.5-3B-Instruct can learn to encode load-bearing reasoning tokens when RL incentivizes this in a simple reasoning task. However, it was difficult to achieve this result. Our attempts to learn encoded reasoning via RL were generally bottlenecked either by exploration difficulties or hacking of the CoT monitor.

This work was done as part of MATS 7.0. These results were used in Large language models can learn and generalize steganographic chain-of-thought under process supervision.

Introduction

Motivation.

Reasoning LLMs rely on natural language chains of thought (CoT) to solve complex tasks. We can employ another LLM -- a CoT monitor -- to analyse these CoTs for misbehaviour (Baker et al.). Such monitoring is promising for catching complex... (read 1695 more words →)

MONA: Managed Myopia with Approval Feedback

Seb Farquhar

Seb Farquhar, David Lindner, Rohin Shah

Blog post by Sebastian Farquhar, David Lindner, Rohin Shah.
It discusses the paper MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hacking by Sebastian Farquhar, Vikrant Varma, David Lindner, David Elson, Caleb Biddulph, Ian Goodfellow, and Rohin Shah.

Our paper tries to make agents that are safer in ways that we may not be able to evaluate through Myopic Optimization with Non-myopic Approval (MONA).

Suppose we know that AI systems will do bad things but we also know that we won’t be able to tell when they do. Can we still prevent unintended behaviour?

In particular cases and ways: yes. Specifically, we show how to get agents whose long-term plans follow strategies that humans... (read 2487 more words →)

On scalable oversight with weak LLMs judging strong LLMs

zac_kenton

zac_kenton, Noah Siegel, janos, Jonah Brown-Cohen, Samuel Albanie, David Lindner, Rohin Shah

Abstract

Scalable oversight protocols aim to enable humans to accurately supervise superhuman AI. In this paper we study debate, where two AI's compete to convince a human judge; consultancy, where a single AI tries to convince a human judge that asks questions; and compare to a baseline of direct question-answering, where the human judge just answers outright without the AI. We use large language models (LLMs) as both AI agents and as stand-ins for human judges, taking the judge models to be weaker than agent models. We benchmark on a diverse range of asymmetries between judges and agents, extending previous work on a single extractive QA task with information asymmetry, to also include... (read 1890 more words →)

Replying toVLM-RM: Specifying Rewards with Natural Language

David Lindner2y

VLM-RM: Specifying Rewards with Natural Language

The agents are rewarded at every timestep and we want them to perform the task throughout the whole episode, so falling over is definitely not what we want. But this has more to do with the policy optimization failing than with the reward model. In other words a policy that doesn't fall over would achieve higher reward than the policies we actually learn. For example, if we plot the CLIP reward over one episode, it typically drops at the end of the episode if the agent falls down.

We tried some tricks to improve the training, such as providing a curriculum starting from short episodes to longer ones. This worked decently well and made the agents fall over less, but we ended up not using it in the final experiments because we primarily wanted to show that it works well with off-the-shelf RL algorithms.

VLM-RM: Specifying Rewards with Natural Language

ChengCheng

ChengCheng, David Lindner, Ethan Perez

tl;dr

We show how to use Vision-Language Models (VLM), and specifically CLIP models, as reward models (RM) for RL agents. Instead of manually specifying a reward function, we only need to provide text prompts like “a humanoid robot kneeling” to instruct and provide feedback to the agent. Importantly, we find that larger VLMs provide more accurate reward signals, so we expect this method to work even better with future models.

Figure 1: We use CLIP as a reward model to train a MuJoCo humanoid robot to (1) stand with raised arms, (2) sit in a lotus position, (3) do the splits, and (4) kneel on the ground (from left to right). Each task is

... (read 1444 more words →)

Replying toPractical Pitfalls of Causal Scrubbing

David Lindner3y

Practical Pitfalls of Causal Scrubbing

Thanks, that's a useful alternative framing of CaSc!

FWIW, I think this adversarial version of CaSc would avoid the main examples in our post where CaSc fails to reject a false hypothesis. The common feature of our examples is "cancellation" which comes from looking at an average CaSc loss. If you only look at the loss of the worst experiment (so the maximum CaSc loss rather than the average one) you don't get these kind of cancellation problems.

Plausibly you'd run into different failure modes though, in particular, I guess the maximum measure is less smooth and gives you less information on "how wrong" your hypothesis is.

Replying toPractical Pitfalls of Causal Scrubbing

David Lindner3y

Practical Pitfalls of Causal Scrubbing

Yes, this seem like a plausible confusion. Your interpretation of what we mean is correct.

Practical Pitfalls of Causal Scrubbing

Jérémy Scheurer

Jérémy Scheurer, Phil3, tony, jacquesthibs, David Lindner

TL;DR: We evaluate Causal Scrubbing (CaSc) on synthetic graphs with known ground truth to determine its reliability in confirming correct hypotheses and rejecting incorrect ones. First, we show that CaSc can accurately identify true hypotheses and quantify the degree to which a hypothesis is wrong. Second, we highlight some limitations of CaSc, in particular, that it cannot falsify all incorrect hypotheses. We provide concrete examples of false positive results with causal scrubbing. Our main finding is that false positives can occur when there is “cancellation”, i.e., CaSc causes the model to do better on some inputs and worse on others, such that, on average, the scrubbed model recovers the full loss. A... (read 3713 more words →)

LESSWRONG
LW

LESSWRONG
LW

David Lindner

Clarifying AI X-risk

Practical Pitfalls of Causal Scrubbing

MONA: Managed Myopia with Approval Feedback

Threat Model Literature Review

David Lindner

Stress-Testing Alignment Audits With Prompt-Level Strategic Deception

Practical challenges of control monitoring in frontier AI deployments

Current LLM agents need strong pressure to engage in scheming behavior

Early Signs of Steganographic Capabilities in Frontier LLMs

MONA: Three Month Later - Updates and Steganography Without Optimization Pressure

Can LLMs learn Steganographic Reasoning via RL?

MONA: Managed Myopia with Approval Feedback

David Lindner

Clarifying AI X-risk

Practical Pitfalls of Causal Scrubbing

MONA: Managed Myopia with Approval Feedback

Threat Model Literature Review

David Lindner

Stress-Testing Alignment Audits With Prompt-Level Strategic Deception

Practical challenges of control monitoring in frontier AI deployments

Current LLM agents need strong pressure to engage in scheming behavior

Early Signs of Steganographic Capabilities in Frontier LLMs

MONA: Three Month Later - Updates and Steganography Without Optimization Pressure

Can LLMs learn Steganographic Reasoning via RL?

MONA: Managed Myopia with Approval Feedback

Introduction

Experiments

Executive Summary

Abstract

Introduction

Abstract

tl;dr