bpomo's Shortform

bpomo

This is a special post for quick takes by bpomo. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Here’s a thing I’m confused about:

A lot of technical AI safety work is also capabilities work. A canonical example is that good evals and good RL environments are somewhat interchangeable, and even in the case of alignment evals (what could be wrong with checking whether models are aligned?) this allows for post training the models to be more aligned, and especially appear more aligned, and thus the labs rush onward improving the capabilities of their apparently aligned AIs.

AFAICT, the thinking of these TAIS researchers is something like: "RSI->ASI is going to happen soon, so we should make it as likely as possible that it goes well by trying to set up the initial conditions and infrastructure of the RSI flywheel as well as possible.”

This makes sense if you take it for granted that RSI simply must happen soon, regardless of what you do. If everyone is just going to work on marginal technical fixes, then there is nothing you can do but marginally technically fix alongside them. And so we roll along to what everyone agrees is a dangerous future. This is a coordination problem. It's defection. It appears we suck.

But also, what, am I gonna take a stand like a chump and sit around being noble while there's a forty percent chance RSI is about to start?

Governance and especially slowdown stuff dodges this problem and looks robustly good to me. Also, transparency and whisteblowing stuff.

Conclusion: I’m not sure if I want to jump into a community that is saying, “well, 20% chance the kill machine kills us all but it would be 35% if we weren’t taping up the wiring in the kill machine so” instead of trying to turn off the kill machine. But I don't know where else to jump in. I'm confused.

Related:
https://www.lesswrong.com/posts/9RL9MuGZjzm4q3gKG/what-just-happened-a-retrospective-of-ai-alignment

Governance and especially slowdown stuff dodges this problem and looks robustly good to me. Also, transparency and whisteblowing stuff.

I think this stuff isn’t robustly good (but also robustly good is a bit of a bad meme anyway). It doesn’t even robustly delay RSI (let alone robustly reduce extinction or robustly improve overall future value.) In particular, there’s a tension between “transparency is robustly good” and “capability and alignment evals are bad” given that transparency is largely about running these evals and sharing the results with external... (read more)

2charding2mo

Michael Nielsen has a thoughtful take on this, as excerpted here and originally published as a postscript here. He describes how alignment work has often worsened risk to humanity by making products more palatable and salable, which has boosted investment, sped up capability advancements, and thereby increased destructive potential. Nielsen next states that, even if you think technical safety and alignment work are helpful, it's still a bad idea to do that work for companies developing frontier AI. As he puts it, [...] I think Haiku's comment has some great suggestions along these lines. I'll add that, at a frontier AI company, it could be very difficult for you to tell if you were helping or making things worse. I haven't worked at a frontier AI company (and I wouldn't!), so I don't speak from direct experience. But in many areas of business, employees are encouraged to feel that their safety concerns are positively influencing company actions, when actual influence is negligible or counterproductive.

3bpomo2mo

I do not think that 1. is very relevant to the central point. Maybe that was a bad example. However, even if the lab isn't directly RLing on that alignment eval or whatever, they may be "grad student descent"-ing up the eval and achieving a similar effect. Either way, I think the result is a increase by X% of entering an aligned RSI flywheel and a decrease by Y% of everyone freaking out and slowing down. I think 2. is just the coordination problem I describe? I did not read your post, so I don't know if you are pointing to something other than the fact that being noble in order to encourage coordination on this would have to account for the fact that this coordination must extend to China.