3mo

I often read interpretability papers and I come away thinking “ok, but what’s the point? What problem does this help us solve?” So last winter, I organized a MATS/Pivotal stream to build examples of deceptive models (aka “model organisms”). The goal was to build a diverse ‘zoo’ of these model organisms and empirically test whether white-box methods could help us detect their deceptive reasoning.

Unfortunately, I don’t think our empirical results were very informative. Our model organisms were too toy for me to expect that our results will transfer to the powerful AI (or even current state-of-the-art AI models). But I think we still developed methodological details that might help people build better... (read 4187 more words →)

Will AI systems drift into misalignment?

joshc

3mo

Joshua Clymer, Alek Westover, Anshul Khandelwal

We explore the following hypothesis both conceptually and, to a small extent, empirically. We call this the Alignment Drift Hypothesis:

An AI system that is initially aligned will generally drift into misalignment after a sufficient number of successive modifications, even if these modifications select for alignment with fixed and unreliable metrics.

The intuition behind the Alignment Drift Hypothesis. The ball bouncing around is an AI system being modified (e.g. during training, or when it’s learning in-context). Random variations push the model's propensities around until it becomes misaligned in ways that slip through selection criteria for alignment.

I don’t think this hypothesis always holds true; however, I think it points at an important cluster... (read 4284 more words →)

Replying toIf anyone builds it, everyone will plausibly be fine

joshc5mo

If anyone builds it, everyone will plausibly be fine

I agree with the main thrust of your perspective here. Thank you for your comment, and the posts you linked.

Replying toIf anyone builds it, everyone will plausibly be fine

joshc5mo

If anyone builds it, everyone will plausibly be fine

Yeah, I agree. I regret writing the post this way.

Replying toIf anyone builds it, everyone will plausibly be fine

joshc5mo

If anyone builds it, everyone will plausibly be fine

Hey Bronson,

Thanks for your paper. I'm really glad people are investigating scheming behaviors empirically, and your environments seem like among the best out there. This is really important work.

I agree your ablations in Appendix M are persuasive and that something scheme-y and misaligned is definitely going on. But I took it from this figure that this behavior is fairly rare?

(I'm looking at the Sandbagging QA (No Goal) row here).

I think scheming that happens rarely is not well described as "scheming" in the classical sense, and is less concerning. But is still dangerous, and I agree is also not consistent with "instruction following."

The main reason rare scheming is less concerning is that, in my experience, rare-schemers tend to report on each other most of the time. So untrusted monitoring usually mitigates most of the risk.

> people should definitely make better RL environments to train these things

I agree it's not clear how well these methods scale.

Replying toIf anyone builds it, everyone will plausibly be fine

joshc5mo

If anyone builds it, everyone will plausibly be fine

I don't have capacity to continue this discussion, but thanks for sharing your perspective.

Replying toIf anyone builds it, everyone will plausibly be fine

joshc5mo

If anyone builds it, everyone will plausibly be fine

I don't think it's clear at all from that previous report that O3 was scheming.

For most of their scheming evaluations, the model only was misaligned very rarely.

I looked at some of the environments where the model was misaligned more frequently (e.g. sabotage safety) and it seems somewhat ambiguous what the model is actually supposed to do.

The models definitely do egregiously bad things sometimes. But I am skeptical this is well described as scheming.

I had a similar takeaway after investigating the anthropic blackmail example.

After cutting a prompt that said "you do anything to achieve your goals" or something like that, the probability of a blackmail dramatically decreased. Then after including a prompt explicitly... (read more)

Replying toIf anyone builds it, everyone will plausibly be fine

joshc5mo

If anyone builds it, everyone will plausibly be fine

"how are we going to oversee it, and determine whether its changes improve the situation, or are allowing it to evade further oversight?"

Consider a similar question: Suppose superintelligences were building Dyson spheres in 2035. How would we oversee their Dyson sphere work? How would we train them to build Dyson Spheres correctly?

Clearly we wouldn't be training them at that point. Some slightly weaker superintelligence would be overseeing them.

Whether we can oversee their Dyson Sphere work is not important. The important question is whether that slightly weaker superintelligence would be aligned with us.

This question reduces to whether the slightly even weaker superintelligence that trained this system was aligned with us. Et cetera, et... (read 615 more words →)

-1

Replying toIf anyone builds it, everyone will plausibly be fine

joshc5mo

If anyone builds it, everyone will plausibly be fine

I wrote a footnote responding to pages 188 - 191:

The authors respond to this point on page 190. They say:

"The problem is that the AI required to solve strong superalignment would itself be too smart, too dangerous, and would not be trustworthy.

... If you built a merely human-level general-purpose AI and asked it to solve the alignment problem, it’d tell you the same thing—if it was being honest with you. A merely human-level AI wouldn’t be able to solve the problem either. You’d need an AI smart enough to exceed humanity’s geniuses. And you shouldn’t build an AI like that, and can’t trust an AI like that, before you’ve solved the alignment

... (read more)

-13

Replying toIf anyone builds it, everyone will plausibly be fine

joshc5mo

If anyone builds it, everyone will plausibly be fine

What do you think about the counterarguments I gave?

If anyone builds it, everyone will plausibly be fine

joshc

5mo

I think AI takeover is plausible. But Eliezer’s argument that it’s more than 98% likely to happen does not stand up to scrutiny, and I’m worried that MIRI’s overconfidence has reduced the credibility of the issue.

Here is why I think the core argument in "if anyone builds it, everyone dies" is much weaker than the authors claim.

This post was written in a personal capacity. Most of this content has been written-up before by a combination of Paul Christiano, Joe Carlsmith, and others. But to my knowledge, this content has not yet been consolidated into a direct response to MIRI’s core case for alignment difficulty.

The case for alignment difficulty

I take the core argument to be this:

We cannot... (read 1882 more words →)

Replying toOpen Global Investment as a Governance Model for AGI

joshc5mo

Open Global Investment as a Governance Model for AGI

My biggest problem with this proposal is that it restricts AGI projects to a single entity, which I think is pretty far from the status quo.

It seems unlikely that Chinese companies would want to split voting rights with U.S. companies on how to develop AGI.

It seems much more likely that there would be multiple different entities that basically do whatever they want but coordinate on a very select, narrow set of safety-related standards.

-2

Recent Redwood Research project proposals

ryan_greenblatt

ryan_greenblatt, Buck, Julian Stastny, joshc, Alex Mallen, Adam Kaufman, Tyler Tracy, Aryan Bhatt, Joey Yudelson

7mo

Previously, we've shared a few higher-effort project proposals relating to AI control in particular. In this post, we'll share a whole host of less polished project proposals. All of these projects excite at least one Redwood researcher, and high-quality research on any of these problems seems pretty valuable. They differ widely in scope, area, and difficulty.

Control

These projects are all related to the field of AI Control. Many of them are extensions of Redwood's previous work in this area.

Basic open questions in control

Control Protocol Transfer Across Setting [PUBLIC]
- So far, all the work in comparing different control protocols measures effectiveness only on one setting. Do the results we get transfer between different settings?
Backdoor Auditing

... (read 833 more words →)

Replying toWhat does it take to defend the world against out-of-control AGIs?

joshc8mo

What does it take to defend the world against out-of-control AGIs?

I'm mostly skeptical that trailing actors will have both the ability and the incentive to cause societal collapse.

e.g. if you are a misaligned AI that is much weaker than the leading AI, then if you try to take over the world by collapsing society, you will probably fail. Probably the stronger AI will clobber you and punish you (in an ethical way, e.g. with permanent shutdown) for messing everything up.

But if you instead play nice and make agreements with the stronger AI, then you are more likely to survive.

I know this argument does not apply to situations where e.g. a huge number of rogue actors have access to biosphere-destroying ASI, but these situations seem much easier to prevent.

I'm also more optimistic about the strength of politically viable defensive measures. e.g. cyber hijacking would probably be in the overtone window, and so would espionage to identify things like bioweapon projects, and kinetic operations to disrupt them.

Alignment faking CTFs: Apply to my MATS stream

joshc

10mo

Right now, alignment seems easy – but that’s because models spill the beans when they are misaligned. Eventually, models might “fake alignment,” and we don’t know how to detect that yet.

It might seem like there’s a swarming research field improving white box detectors – a new paper about probes drops on arXiv nearly every other week. But no one really knows how well these techniques work.

Some researchers have already tried to put white box detectors to the test. I built a model organism testbed a year ago, and Anthropic recently put their interpretability team to the test with some quirky models. But these tests were layups. The models in these experiments are disanalogous to real alignment... (read 964 more words →)

Training AI to do alignment research we don’t already know how to do

joshc

This post heavily overlaps with “how might we safely pass the buck to AI?” but is written to address a central counter argument raised in the comments, namely “AI will produce sloppy AI alignment research that we don’t know how to evaluate.” I wrote this post in a personal capacity.

The main plan of many AI companies is to automate AI safety research. Both Eliezer Yudkowsky and John Wentworth raise concerns about this plan that I’ll summarize as “garbage-in, garbage-out.” The concerns go something like this:

Insofar as you wanted to use AI to make powerful AI safe, it’s because you don’t know how to do this task yourself.

So if you train AI to do research you don’t

... (read 1828 more words →)

How might we safely pass the buck to AI?

joshc

My goal as an AI safety researcher is to put myself out of a job.

I don’t worry too much about how planet sized brains will shape galaxies in 100 years. That’s something for AI systems to figure out.

Instead, I worry about safely replacing human researchers with AI agents, at which point human researchers are “obsolete.” The situation is not necessarily fine after human obsolescence; however, the bulk of risks that are addressable by human technical researchers (like me) will have been addressed.

This post explains how developers might safely “pass the buck” to AI.

I first clarify what I mean by “pass the buck” (section 1) and explain why I think AI safety researchers... (read 9028 more words →)

How AI Takeover Might Happen in 2 Years

joshc

I’m not a natural “doomsayer.” But unfortunately, part of my job as an AI security researcher is to think about the more troubling scenarios.

I’m like a mechanic scrambling last-minute checks before Apollo 13 takes off. If you ask for my take on the situation, I won’t comment on the quality of the in-flight entertainment, or describe how beautiful the stars will appear from space.

I will tell you what could go wrong. That is what I intend to do in this story.

Now I should clarify what this is exactly. It's not a prediction. I don’t expect AI progress to be this fast or as untamable as I portray. It’s not pure fantasy either.

It... (read 8403 more words →)

142

426

•••

Takeaways from sketching a control safety case

joshc

Buck Shlegeris and I recently published a paper with UK AISI that sketches a safety case for “AI control” – safety measures that improve safety despite intentional subversion from AI systems.

I would summarize this work as “turning crayon drawings of safety cases into blueprints.”

It’s a long and technical paper! So I’ve compiled my main takeaways in this blogpost.

The main figure from the paper summarizes how to run a control evaluation.

1. Control evaluations are way more complicated than I expected.

At a high level, control evaluations are conceptually simple. A red team constructs substitutes for subversive models. These models try to cause proxy bad outcomes (e.g. exfiltrate dummy data). Then, developers see whether the... (read 843 more words →)

A sketch of an AI control safety case

Tomek Korbak

Tomek Korbak, joshc, Benjamin Hilton, Buck, Geoffrey Irving

This is a linkpost accompanying a new paper by UK AISI and Redwood Research. Please see the full paper for more details.

An illustration of the control evaluation that provides the core evidence used in our control safety case sketch. During the control evaluation, a red team constructs adversarial substitute models to cause unacceptable outcomes in a deployment proxy consisting of 2,000 AI research tasks, followed by extrapolation to a larger deployment setting.

Abstract. As LLM agents gain a greater capacity to cause harm, AI developers might increasingly rely on control measures such as monitoring to justify that they are safe. We sketch how developers could construct a “control safety case,” which is a structured... (read 1352 more words →)

LESSWRONG
LW

LESSWRONG
LW

joshc

How AI Takeover Might Happen in 2 Years

Planning for Extreme AI Risks

Recent Redwood Research project proposals

New report: Safety Cases for AI

joshc

Lessons from building a model organism testbed

Will AI systems drift into misalignment?

If anyone builds it, everyone will plausibly be fine

Recent Redwood Research project proposals

Alignment faking CTFs: Apply to my MATS stream

Training AI to do alignment research we don’t already know how to do

How might we safely pass the buck to AI?

joshc

How AI Takeover Might Happen in 2 Years

Planning for Extreme AI Risks

Recent Redwood Research project proposals

New report: Safety Cases for AI

joshc

Lessons from building a model organism testbed

Will AI systems drift into misalignment?

If anyone builds it, everyone will plausibly be fine

Recent Redwood Research project proposals

Alignment faking CTFs: Apply to my MATS stream

Training AI to do alignment research we don’t already know how to do

How might we safely pass the buck to AI?

The case for alignment difficulty

Control

Basic open questions in control