Zach Stein-Perlman's Shortform

Zach Stein-Perlman

This is a special post for quick takes by Zach Stein-Perlman. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Posting for a friend:

I think the investigation of the Hugging Face incident is a great win for the marginalist faction of the AI safety community.
Compare what happened when an Alibaba model allegedly broke out of its sandbox and started mining crypto. Alibaba said a few sentences on this in the middle of a long research paper, and when the public noticed, Alibaba issued a tweet (contradicting the paper) intended to reassure the public. Alibaba never shared more, we never learned more about the incident, and the public quickly lost interest.
The Alibaba incident was probably much smaller than OpenAI's, and part of difference in reporting might be explained by general cultural differences between the US and China. Still, I think we could have ended up in a world where the leading US companies have Alibaba-like attitudes on incident disclosure and where the Hugging Face attack gets hushed up similarly to the Alibaba incident. I think the fact that the Hugging Face incident didn't get hushed up is largely due to some good people working at OpenAI plus METR and Redwood maintaining a good relationship with OpenAI.
The information disclosure here is far from perfect, and hopefully we get more, but what got disclosed is already very valuable. Among other things, the disclosed scary details probably help the cause of AI pause more than anything that direct pause advocacy has accomplished so far.
Sure, the marginalist, working-within-the-system faction of AI safety has way more resources than the more radical faction, and perhaps it is still under-delivering relative to its resources. But I think the marginalists just scored a big win, and I want the more radical faction to remember the value of getting the incident disclosure that we got instead of what Alibaba would have disclosed, when accusing the marginalist faction of not accomplishing anything useful.

3David Matolcsi2mo

Ok, maybe I don't understand METR well enough. I know they try hard to not to accept money from the labs (I don't know what else you mean by not getting in bed with them), and I know they and their employees sometimes say negative things about the situation which probably cools the relationship with the labs a bit. I don't want to imply that they are not independent or that they are unhealthily close to the labs. But they are still primarily not an advocacy organization, they work regularly with the labs, and afaict they try to work on topics that are interesting to the labs too. Maybe I'm missing something, but is there any other safety org who has friendlier relationship to the labs than they do? If not, then what do you mean by "outside-game organization", other than the fact that they are not literally lab-employees? I agree btw that working outside a lab is almost always better for a marginal person than working inside a lab. I'm not entirely sure though that these incidents were really a further update about this though - lab employees are now uniquely well-positioned to study the incidents further and hopefully publish about them, time will tell how much they can actually do that. I also think that gaining power and a voice inside the labs aged somewhat well - I think this is OP's point that it was not at all pre-determined that we get at least this much transparency from a leading AI lab, instead of living in the The Counterfactual Quiet AGI Timeline. I agree with some of your other points on what aged well and what didn't. I still think that "studying things and informing the world in a kind of impartial way, in collaboration with the labs" got a positive update compared to "just do direct advocacy for pause, and maybe create some intentionally scary demos" - though it's still possible that "just do direct pause advocacy" is still punching above its weight compared to its resources, which possibility was explicitly stated by the OP. Btw, I'm still intere

6habryka2mo

I mean, yes, almost all of them? Goodfire, Epoch, MATS, Arena, CAIS, Ought/Elicit, ARC, FarAI, Manifund, SFF, Foresight, Apart research, Constellation. Some where it's a hard call and I could see it going either way but it's not clear: Apollo, CG, 80k, Iliad Like, overall the vast majority of the ecosystem is much friendlier to the labs than METR. Much of it recommends working at the labs directly (in "safety" roles of course). [...] Yes! For example, constitutional AI seemed like the central alignment effort of Anthropic for most of the last year or so. Joe Carlsmith and Amanda work on that. I had arguments with Joe about exactly the tradeoff between these two things. Jan Leike has always worked on hill-climbing local alignment metrics. Automated alignment research is the next most prominent research direction at both OpenAI and Anthropic. Ethan Perez works on constitutional classifiers and jailbreaking, plus automated alignment research. Sam Marks leads the cognitive oversight team, mostly working on interpretability, which is between these two things, but historically Anthropic's work in the space has more focused on using interp-insights to steer the model behavior (though it could of course also be used to measure misalignment, so I could see this going either way) Of the most prominent safety researchers at Anthropic (according to Claude), only Evan's work seems mostly focused on noticing and analyzing bad things. And well... OpenAI... unclear who prominent safety researchers at OpenAI are, as all the good ones keep leaving, but needless to say it doesn't involve very much measurement of risk as opposed to risk mitigation. OpenAI safety seems basically 100% focused on direct alignment. And that's only the "safety" researchers. Of course the vast majority of executive talent at Anthropic also works there with a safety motivation, even without being "prominent safety researchers". Most of the investment of the safety community into Anthropic isn't safe

Yeah, fair. I categorized the other orgs as not having a relationship with the labs (like I don't think that ARC had any when I worked there), but I guess you can say that in a sense that's friendlier than an external assessor's relationship.

Fwiw, when I look at your list of people, I would still only categorize Joe and Amanda as working on making current models more aligned in a way that is supposed to make incidents like this less likely. Fore example, I assume you also dislike working on making AIs good at automated alignment research, but I think that'... (read more)

Zach Stein-Perlman's Shortform

5

Zach Stein-Perlman's Shortform

5