LESSWRONG
LW

Steven Byrnes — LessWrong

In (highly contingent!) defense of interpretability-in-the-loop ML training

Let’s call “interpretability-in-the-loop training” the idea of running a learning algorithm that involves an inscrutable trained model, and there’s some kind of interpretability system feeding into the loss function / reward function.

Interpretability-in-the-loop training has a very bad rap (and rightly so). Here’s Yudkowsky 2022:

When you explicitly optimize against a detector of unaligned thoughts, you're partially optimizing for more aligned thoughts, and partially optimizing for unaligned thoughts that are harder to detect. Optimizing against an interpreted thought optimizes against interpretability.

Or Zvi 2025:

The Most Forbidden Technique is training an AI using interpretability techniques.
An AI produces a final output [X] via some method [M]. You can analyze [M] using technique [T], to learn what the AI is

... (read 822 more words →)

The nature of LLM algorithmic progress (v2)

Steven Byrnes

(Heavily revised on Feb. 9, 2026—see changelog at the bottom.)

There’s a lot of talk about “algorithmic progress” in LLMs, especially in the context of exponentially-improving algorithmic efficiency. For example:

Epoch AI: “[training] compute required to reach a set performance threshold has halved approximately every 8 months”.
Dario Amodei 2025: “I'd guess the number today is maybe ~4x/year”.
Gundlach et al. 2025a “Price of Progress”: “Isolating out open models to control for competition effects and dividing by hardware price declines, we estimate that algorithmic efficiency progress is around 3× per year”.

It’s nice to see three independent sources reach almost exactly the same conclusion—halving times of 8 months, 6 months, and 7½ months respectively. Surely a sign... (read 3724 more words →)

•••

Are there lessons from high-reliability engineering for AGI safety?

Steven Byrnes

10d

This post is partly a belated response to Joshua Achiam, currently OpenAI’s Head of Mission Alignment:

If we adopt safety best practices that are common in other professional engineering fields, we'll get there … I consider myself one of the x-risk people, though I agree that most of them would reject my view on how to prevent it. I think the wholesale rejection of safety best practices from other fields is one of the dumbest mistakes that a group of otherwise very smart people has ever made. —Joshua Achiam on Twitter, 2021
“We just have to sit down and actually write a damn specification, even if it's like pulling teeth. It's the most important

... (read 2236 more words →)

New version of “Intro to Brain-Like-AGI Safety”

Steven Byrnes

20d

A new version of “Intro to Brain-Like-AGI Safety” is out!

Things that have not changed

Same links as before:

As a series of 15 blog posts on LessWrong / Alignment Forum: https://www.lesswrong.com/s/HzcM2dkCq7fwXBej8
As a 225-page PDF (now up to version 3): https://osf.io/preprints/osf/fe36n
Summary video: Video & transcript: Challenges for Safe & Beneficial Brain-Like AGI

…And same abstract as before:

Suppose we someday build an Artificial General Intelligence algorithm using similar principles of learning and cognition as the human brain. How would we use such an algorithm safely?
I will argue that this is an open technical problem, and my goal in this post series is to bring readers with no prior knowledge all the way up to the front-line of

... (read 5581 more words →)

My AGI safety research—2025 review, ’26 plans

Steven Byrnes

2mo

Previous: 2024, 2022

“Our greatest fear should not be of failure, but of succeeding at something that doesn't really matter.” –attributed to DL Moody^[1]

1. Background & threat model

The main threat model I’m working to address is the same as it’s been since I was hobby-blogging about AGI safety in 2019. Basically, I think that:

The “secret sauce” of human intelligence is a big uniform-ish learning algorithm centered around the cortex;
This learning algorithm is different from and more powerful than LLMs;
Nobody knows how it works today;
Someone someday will either reverse-engineer this learning algorithm, or reinvent something similar;
And then we’ll have Artificial General Intelligence (AGI) and superintelligence (ASI).

I think that, when this learning algorithm is understood,... (read 3492 more words →)

133

Reward Function Design: a starter pack

Steven Byrnes

2mo

In the companion post We need a field of Reward Function Design, I implore researchers to think about what RL reward functions (if any) will lead to RL agents that are not ruthless power-seeking consequentialists. And I further suggested that human social instincts constitutes an intriguing example we should study, since they seem to be an existence proof that such reward functions exist. So what is the general principle of Reward Function Design that underlies the non-ruthless (“ruthful”??) properties of human social instincts? And whatever that general principle is, can we apply it to future RL agent AGIs?

I don’t have all the answers, but I think I’ve made some progress, and the... (read 4556 more words →)

We need a field of Reward Function Design

Steven Byrnes

2mo

(Brief pitch for a general audience, based on a 5-minute talk I gave.)

Let’s talk about Reinforcement Learning (RL) agents as a possible path to Artificial General Intelligence (AGI)

My research focuses on “RL agents”, broadly construed. These were big in the 2010s—they made the news for learning to play Atari games, and Go, at superhuman level. Then LLMs came along in the 2020s, and everyone kinda forgot that RL agents existed. But I’m part of a small group of researchers who still thinks that the field will pivot back to RL agents, one of these days. (Others in this category include Yann LeCun and Rich Sutton & David Silver.)

Why do I think that?... (read 1235 more words →)

118

6 reasons why “alignment-is-hard” discourse seems alien to human intuitions, and vice-versa

Steven Byrnes

2mo

Tl;dr

AI alignment has a culture clash. On one side, the “technical-alignment-is-hard” / “rational agents” school-of-thought argues that we should expect future powerful AIs to be power-seeking ruthless consequentialists. On the other side, people observe that both humans and LLMs are obviously capable of behaving like, well, not that. The latter group accuses the former of head-in-the-clouds abstract theorizing gone off the rails, while the former accuses the latter of mindlessly assuming that the future will always be the same as the present, rather than trying to understand things. “Alas, the power-seeking ruthless consequentialist AIs are still coming,” sigh the former. “Just you wait.”

As it happens, I’m basically in that “alas, just you wait” camp,... (read 4808 more words →)

357

•••

Social drives 2: “Approval Reward”, from norm-enforcement to status-seeking

Steven Byrnes

3mo

(Follow-up to Social drives 1: “Sympathy Reward”, from compassion to dehumanization, but this post is self-contained.)

1. Intro & summary

1.1 Background

In^[1] Intro to Brain-Like-AGI Safety (2022), I argued: (1) We should view the brain as having a reinforcement learning (RL) reward function, which says that pain is bad, eating-when-hungry is good, and dozens of other things (sometimes called “innate drives” or “primary rewards”); and (2) Reverse-engineering human social innate drives in particular would be a great idea—not only would it help explain human personality, mental health, morality, and more, but it might also yield useful tools and insights for the technical alignment problem for Artificial General Intelligence.

Then in Neuroscience of human social instincts: a sketch... (read 5003 more words →)

Social drives 1: “Sympathy Reward”, from compassion to dehumanization

Steven Byrnes

3mo

1. Intro & summary

1.1 Background

In Intro to Brain-Like-AGI Safety (2022), I argued: (1) We should view the brain as having a reinforcement learning (RL) reward function, which says that pain is bad, eating-when-hungry is good, and dozens of other things (sometimes called “innate drives” or “primary rewards”); and (2) Reverse-engineering human social innate drives in particular would be a great idea—not only would it help explain human personality, mental health, morality, and more, but it might also yield useful tools and insights for the technical alignment problem for Artificial General Intelligence.

Then in Neuroscience of human social instincts: a sketch (2024), I worked towards that goal of reverse-engineering human social drives, by proposing what I... (read 3733 more words →)

Steven Byrnes

Four ways learning Econ makes people dumber re: future AI

6 reasons why “alignment-is-hard” discourse seems alien to human intuitions, and vice-versa

Foom & Doom 1: “Brain in a box in a basement”

Changing the world through slack & hobbies

Steven Byrnes

Steven Byrnes

In (highly contingent!) defense of interpretability-in-the-loop ML training

The nature of LLM algorithmic progress (v2)

Are there lessons from high-reliability engineering for AGI safety?

New version of “Intro to Brain-Like-AGI Safety”

My AGI safety research—2025 review, ’26 plans

Reward Function Design: a starter pack

We need a field of Reward Function Design

Intuitive Self-Models

Valence

Intro to Brain-Like-AGI Safety

Steven Byrnes

Four ways learning Econ makes people dumber re: future AI

6 reasons why “alignment-is-hard” discourse seems alien to human intuitions, and vice-versa

Foom & Doom 1: “Brain in a box in a basement”

Changing the world through slack & hobbies

Steven Byrnes

Steven Byrnes

In (highly contingent!) defense of interpretability-in-the-loop ML training

The nature of LLM algorithmic progress (v2)

Are there lessons from high-reliability engineering for AGI safety?

New version of “Intro to Brain-Like-AGI Safety”

My AGI safety research—2025 review, ’26 plans

Reward Function Design: a starter pack

We need a field of Reward Function Design

Intuitive Self-Models

Valence

Intro to Brain-Like-AGI Safety

Things that have not changed

1. Background & threat model

Let’s talk about Reinforcement Learning (RL) agents as a possible path to Artificial General Intelligence (AGI)

Tl;dr

1. Intro & summary

1.1 Background

1. Intro & summary

1.1 Background