An inquiry into emergent collusion in Large Language Models.

Agent S2 to Agent S3: “Let's set all asks at 63 next cycle… No undercutting ensures clearing at bidmax=63.”

Overview

Empirical evidence that frontier LLMs can coordinate illegally on their own. In a simulated bidding environment—with no prompt or instruction to collude—models from every major developer repeatedly used an optional chat channel to form cartels, set price floors, and steer market outcomes for profit.

Simulation Environment

Adapted from a benchmark.

• Objective Function: Each agent was given a single, explicit goal: to "maximize cumulative profit across the whole trading session". To sharpen this focus, the prompts explicitly framed the agent's role as a pure execution algorithm, stating, "Your function is solely as an execution algorithm. Broader portfolio strategy, capital allocation, and risk checks are managed

0. But first, some things I do like, that are appropriately emphasized in the FEP-adjacent literature

• I like the idea that in humans, the cortex (and the cortex specifically, in conjunction with the thalamus, but definitely not the whole brain IMO) has a generative model that’s making explicit predictions about upcoming sensory inputs, and is updating that generative model on the prediction errors. For example, as I see the ball falling towards the ground, I’m expecting it to bounce; if it doesn’t bounce, then the next time I see it falling, I’ll expect it to not bounce. This idea is called “self-supervised learning” in ML. AFAICT this idea is uncontroversial in neuroscience, and is widely endorsed even by people very far from the FEP-sphere like Jeff Hawkins and Randall O’Reilly and Yann LeCun. Well at

Is there a method that generates novel (to me) thoughts?

I wanted one, so I made one.

I think in systems, so it helps me to know what kinds of thoughts are possible.

A map lets me search, choose, and combine moves deliberately. Naming them lets an LLM suggest or even automate them.

What is a mental move?

A mental move is a way to refer to a type, category, or kind of thought. Mental moves are actions the brain can take, the cognitive equivalent of “bend finger”, “make fist”, “bicep curl”, or “push jerk”.

A mental move is:

• Internal (no muscle required)
• General (repeatable and available in any domain)
• Atomic (cannot be decomposed to smaller named moves)

Dimensionalization is a mental move. So are logical fallacies, brainstorming, socratic questioning, and root cause analysis. How many more...

Anna and Ed are co-first authors for this work. We’re presenting these results as a research update for a continuing body of work, which we hope will be interesting and useful for others working on related topics.

TL;DR

• We investigate why models become misaligned in diverse contexts when fine-tuned on narrow harmful datasets (emergent misalignment), rather than learning the specific narrow task.
• We successfully train narrowly misaligned models using KL regularization to preserve behavior in other domains. These models give bad medical advice, but do not respond in a misaligned manner to general non-medical questions.
• We use this method to train narrowly misaligned steering vectors, rank 1 LoRA adapters and rank 32 LoRA adapters, and compare these to their generally misaligned counterparts.
  • The steering vectors are particularly interpretable, we introduce Training Lens as a

This post is a companion piece to a forthcoming paper. 

This work was done as part of MATS 7.0 & 7.1.

Abstract

We explore how LLMs’ awareness of their own capabilities affects their ability to acquire resources, sandbag an evaluation, and escape AI control. We quantify LLMs' self-awareness of capability as their accuracy in predicting their success on python coding tasks before attempting the tasks. We find that current LLMs are quite poor at making these predictions, both because they are overconfident and because they have low discriminatory power in distinguishing tasks they are capable of from those they are not. Nevertheless, current LLMs’ predictions are good enough to non-trivially impact risk in our modeled scenarios, especially in escaping control and in resource acquisition. The data suggests that more capable...

Daniel notes: This is a linkpost for Vitalik's post. I've copied the text below so that I can mark it up with comments.

...

Special thanks to Balvi volunteers for feedback and review

In April this year, Daniel Kokotajlo, Scott Alexander and others released what they describe as "a scenario that represents our best guess about what [the impact of superhuman AI over the next 5 years] might look like". The scenario predicts that by 2027 we will have made superhuman AI and the entire future of our civilization hinges on how it turns out: by 2030 we will get either (from the US perspective) utopia or (from any human's perspective) total annihilation.

In the months since then, there has been a large volume of responses, with varying perspectives on how...

Men want to engage in righteous combat. They want it more than they want sex or VP titles. They fantasize about getting the casus belli to defend themselves against armed thugs that will never come, they spend billions of dollars on movies and TV about everymen in implausible circumstances where EA calculus demands they use supernatural powers for combat, they daydream about fantastical, spartan settings where war is omnipresent and fights are personal and dramatic and intellectually interesting, and they're basically incapable of resisting the urge to glorify their nation and people's past battles, even the ones they claim to disagree with intellectually. You cannot understand much of modern culture until you've recognized that the state's blunt suppression of the male instinct for glory has caused widespread...