Elliott Thornley

Can risk aversion learned at low stakes generalize to astronomically high stakes?

This post covers our recent paper: Out-of-Distribution Generalization of Risk Aversion in Language Models. It gives the intro, main results table, and example prompts from the training and evaluation sets. For everything else, see the paper. TL;DR * Training AIs to be risk-averse in resources could be a useful failsafe...

Jul 1420

How big is the Sun? How could you figure it out?

I’m a few chapters into Our Mathematical Universe by Max Tegmark. By this point he’s covered the ingenuities of the ancient Greeks, taking my knowledge of physics to within two and a half thousand years of the cutting edge. And what ingenuities they were. A whole series of them, strung...

Jul 9101

Tie training can make DPO/RLHF-trained AIs generalize better

This post covers our recent ICML paper: Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training. TL;DR * Our theorems and experiments suggest that DPO and RLHF have an unwelcome consequence: they make AIs care about every feature of actions that correlates with true value on...

Jul 654

Risk-Averse AIs

by wdmacaskill and Elliott Thornley

Abstract We make the case for training AIs to be risk-averse in resources — specifically, to treat resources as having diminishing marginal utility. These AIs would (for example) choose $40 for sure over a half-chance of $100 and a half-chance of $0. We argue that risk aversion can preserve AIs’...

Jun 2435

Towards Shutdownable Agents: Generalizing Stochastic Choice in RL Agents and LLMs

Summary * Misaligned artificial agents might resist shutdown. * One proposed solution is the POST-Agents Proposal: roughly, training agents to lack preferences between different-length trajectories. * The Discounted Reward for Same-Length Trajectories (DReST) reward does this by penalizing agents for repeatedly choosing same-length trajectories. It thus incentivizes agents to be:...

Jun 320

Maybe we should pretrain on synthetic data about good-but-reward-hacking AIs

TLDR: The idea is basically inoculation prompting crossed with alignment pretraining. Call it ‘inoculation pretraining.’ It’s a type of spillway design. Reward hacking can cause emergent misalignment: you train the AI to cheat on its tasks and it turns broadly evil. Why does this happen? The persona selection model (PSM)...

May 2916

AI safety can be a Pascal's mugging even if p(doom) is high

People sometimes say that AI safety is a Pascal’s mugging. Other people sometimes reply that AI safety can’t be a Pascal’s mugging, because p(doom) is high. Both these people are wrong. The second group of people are wrong because Pascal’s muggings are about the probability that you make a difference,...

Apr 2530

Elliott Thornley

Elliott Thornley

Risk-Averse AIs

Tie training can make DPO/RLHF-trained AIs generalize better

Shutdownable Agents through POST-Agency

There are no coherence theorems

Elliott Thornley

Risk-Averse AIs

Tie training can make DPO/RLHF-trained AIs generalize better

Shutdownable Agents through POST-Agency

There are no coherence theorems

Can risk aversion learned at low stakes generalize to astronomically high stakes?

How big is the Sun? How could you figure it out?

Tie training can make DPO/RLHF-trained AIs generalize better

Risk-Averse AIs

Towards Shutdownable Agents: Generalizing Stochastic Choice in RL Agents and LLMs

Maybe we should pretrain on synthetic data about good-but-reward-hacking AIs

AI safety can be a Pascal's mugging even if p(doom) is high