This is an email I wrote to @Buck after he gave a talk about his strategy confusions to a Constellation audience. The talk was about how Redwood's work could have stopped the Hugging Face incident from happening, which would have been very bad for the salience of AI safety, and how he's confused about that. I think this is a very important consideration which should be shared more prominently, but I also think ALL of Redwood's strategic considerations should be shared more prominently. Here's an email I wrote to Buck, lightly edited for clarity.

Hi Buck,

I'm pretty new to the AI safety space. I've loosely followed thing for a few years, but only really sunk my teeth in for about five months. Four months ago, I was mistaken about what leading safety orgs believed, and what their motivations were. For example, I thought that Redwood's control regime was misguided, because it wouldn't scale to ASI. I now believe (as Redwood does) that control is mostly good because it allows us to mine labor from, say, just barely superhuman AIs which could do tons of work to align ASIs. The point of this anecdote is that I think having transparency about your strategy beliefs as an org at the "center" of the AI ecosystem is really important for directing people who are newer to the space to grapple with the right strategy questions (and not jump in to doing useless stuff). I think many people new to the space pursue projects which are unhelpful because they are working under the worldview of "control good" rather than [insert convoluted Redwood worldview which probably endorses some kind of control research but not others]. I largely avoided this by being confused and being more interested/motivated by strategic and abstract questions than by hairy empirical ones. Probably many people who like thinking about hairy empirical problems and not strategy will just get nerd sniped and not do the best things no matter what you do, but we can at least move the space in the right direction. 

Anyways, this summer I noticed I was confused about whether implementing control mechanisms right now is good at all. Warning shots matter so much! And then you gave a talk about this topic which I enjoyed, and my sense in the room was that the talk raised an important question that most people had not deeply considered. I'm very worried about the contribution of "AI safety" towards putting us in a looks good, is bad world [Buck's framing from his talk]. I think you should make your talk into a brief write up and post it somewhere. In general, I think you should more prominently place your strategy considerations on Redwood's website and research. I don't know the best way to do this and I am sure there are significant trade offs I have not considered. But if Redwood isn't transparent about strategy, who will be! Not just transparent somewhere in a blog post from a few months ago, but clearly transparent with a "do not try this at home unless you understand this blog post" disclaimer next to research that you think is probably good for a very specific reason. I would like the AI safety space to be more confused, more correct, and less headstrong. 

Best,

Ben Pomeranz

@Cleo Nardo points out that this also has a lot to do with field building research programs, where people want to start doing research on day three but should probably be considering strategy for a few weeks. He also points out that the control stuff was preceded by posts like “the case for ensuring AIs are controlled,” but I think that these justifications should be featured prominently alongside the research that they justify, because many people will engage with and build off of technical work without looking back in time for relevant blogposts that may precede it. Also, he points out
that Redwood's more recent research direction into conceptual uplift does not come with a matching "the case for conceptual uplift" post.

Should we have a very exclusive AIS cause-prio sort of conference which outputs something like Hilbert Problems?
I still think something like “Hilbert problems for AIS” or just a convention of GOAT level researchers where they do super intense cause prio is quite good.

The idea:

• Get a number of very excellent AIS researchers/thinkers/funders (people with excellent end to end threat models/theories of change with various strategies) in the same place for 1+ days (IDK how long)
• Have people working on a very diverse set of things (e.g. China policy, specific control protocols, grassroots organizing, singular learning theory, etc. etc. etc. )
• Have them present their best arguments at the start of the thing for why more people should work on the set of things they think people should work on
• Have lots of 1 on 1s with goal of finding cruxes or changing minds
• Instruct people to make outcomes they want as specific/verifiably worded as possible. E.g. “scalable oversight” is not ok, but “models which we have confidence in as defined by X monitoring transcripts from all of the following operations inside of labs: Y% of internal sandboxed evals, Z% of blah blah” is better. This example is bad, but hopefully points in the direction of what I mean.
• If an outcome isn’t actually verifiable other researchers poke at it and try to help make it more verifiable. Maybe have some incentive structure IDK
• Maybe have some auction format where attendees put how much they’d pay, in percent of 2026-2028 cG dollars or some other valuation, to get various outcomes.
• Maybe have forecasters estimate the expected price of those outcomes in total labor/resources later and then you have a list of most cost effective things to try from this aggregate view

Pros:

• In general you get a list of very specific things people should work towards, and now instead of researchers saying “I work on scalable oversight!” they can say “I do this thing which I think will help achieve Guntherson’s criteria in the following way” and are more likely to be doing helpful stuff
• Important researchers at this conference might update their worldview to work on better things
• In general the conference runners record discussions and gather evidence the researchers reference as much as they can and can then present great steelman cases for various things to work on that people can look at
• Maybe this let’s you post cash prizes for some of the verifiable outcomes.

Cons:

• Hard to structure well?
• Cost + opportunity cost
• Maybe you make it less likely that researchers go exploring and find the good ideas that they haven’t thought of yet or otherwise limit future creativity. This seems not likely to be a huge cost to me: people who currently think outside of the box in terms of what we are prioritizing probably wouldn't stop because the list of things people currently care about is much better specified, ordered and presented.

Here’s a thing I’m confused about:

A lot of technical AI safety work is also capabilities work. A canonical example is that good evals and good RL environments are somewhat interchangeable, and even in the case of alignment evals (what could be wrong with checking whether models are aligned?) this allows for post training the models to be more aligned, and especially appear more aligned, and thus the labs rush onward improving the capabilities of their apparently aligned AIs.

AFAICT, the thinking of these TAIS researchers is something like: "RSI->ASI is going to happen soon, so we should make it as likely as possible that it goes well by trying to set up the initial conditions and infrastructure of the RSI flywheel as well as possible.”

This makes sense if you take it for granted that RSI simply must happen soon, regardless of what you do. If everyone is just going to work on marginal technical fixes, then there is nothing you can do but marginally technically fix alongside them. And so we roll along to what everyone agrees is a dangerous future. This is a coordination problem. It's defection. It appears we suck.

But also, what, am I gonna take a stand like a chump and sit around being noble while there's a forty percent chance RSI is about to start?

Governance and especially slowdown stuff dodges this problem and looks robustly good to me. Also, transparency and whisteblowing stuff.

Conclusion: I’m not sure if I want to jump into a community that is saying, “well, 20% chance the kill machine kills us all but it would be 35% if we weren’t taping up the wiring in the kill machine so” instead of trying to turn off the kill machine. But I don't know where else to jump in. I'm confused.

Related:
https://www.lesswrong.com/posts/9RL9MuGZjzm4q3gKG/what-just-happened-a-retrospective-of-ai-alignment