June Jimenez's Shortform

June Jimenez

This is part Quick Take and part footnote from the preface to my upcoming paper.

A very reasonable question you could pose to me, made more reasonable by the reluctance of the original gradual disempowerment researchers to continue centering this line of work among their projects, and more-reasonable-still by worries about whether people are engaging with the central GD concepts instead of just bouncing off, is something like: "Why are you doing GD research? Don't you know we'll have a fast takeoff^[1], and then even if GD is true it doesn't matter very much? It's not like we'll have control after that, certainly not enough control to change outcomes meaningfully."

The minus-first reason is that I currently think most technical safety research is not just not on the critical path, but currently net negative. I will not really justify this viewpoint at the length it deserves, other than to say that I think there's a lot more consideration of differentially accelerating safety research relative to capabilities than there is about either working on a paradigm that does not appear to accelerate capabilities at all or otherwise attempting to directly offset the capabilities impact of your research output^[2], and centering the former seems misguided. If we imagine a hypothetical research paradigm that leads to a 5% speedup in capabilities but a 50% speedup in safety according to some reasonable operationalization, that sounds neat. Great, even. But it neglects uptake, speed of uptake and general feasibility; a capabilities externality, absent some severe taboo like the taboo on neuralese, will be implemented into the frontier training pipeline approximately as soon as it can be wrangled to make the models more capable. But a safety speedup might take years to implement or to be worth implementing, and has much less clear incentives to be implemented. You can't get your safety payoff in 2032 if we foom in 2028. So in general, my primary first-order consideration for project selection is "In expectation, will this accelerate or decelerate AI capabilities?"

The zeroth reason is that I would currently make for a mediocre technical safety researcher. I'm working on my comparative advantage.

But, specific to GD:

First, I don't think a fast takeoff is inevitable. I think it is a function of the regulatory environment; said bluntly, I think if it were illegal to do frontier AGI research, or even burdensome and regulated, instead of being only approximately as burdensome as any other software job, then all of a sudden a lot of current AI researchers would find different cool stuff to do with their lives. I think this alone would slow down AI research substantially domestically. This would provide ample reason for the US to negotiate a bilateral and longer-term AI development deal with China. By no means is any of this trivial, but I mean to make the point that it is possible. If this is true, all of a sudden research on GD becomes much more obviously critical.

Second, is that I think GD matters now, has affected matters for a while, and matters a lot and an increasing amount every week, because it's already playing a role in the path we're going down via various very ugly selection pressures. I think as a default, with the current or likely near-future regulatory environment, we are headed for a world where the labs trigger RSI, possibly accidentally^[3]. And I think both desire modification in less disempowered times and gradual disempowerment as an AI-centered framing provide good models for how we've ended up here, and give compelling reason that our ending up here is not inevitable and seems undesirable. Our desires are path-dependent, on a both individual and societal level. So we should figure out if we have gone down the wrong path and the best / most feasible way to backtrack or merge back to better ones if so.

Third, GD is still neglected. It's a "big if true" that we've failed to fully digest^[4]: at the highest level of abstraction, it's the question of "If we solve technical alignment, do we expect good outcomes, or bad outcomes by default?" I think there's a very good chance of bad outcomes on short development timelines, and I will claim without evidence that many technical (safety) researchers believe that if their (safety) research paradigm is successful, this will be good instead of bad. I think of a lot of that as misplaced confidence. If I'm wrong, I want people to engage with the arguments in depth and give us evidence for that.

^{^}
Or a slow takeoff that's about the speed of a fast takeoff.
^{^}
For a concrete (and slightly kooky) example, commit to advancing your research paradigm which has some capabilities externalities iff your friend who works at a frontier lab quits, moves into more safety-focused research or starts doing slowdown advocacy within the company.
^{^}
Here I mean accidentally as in "via a training run that they did not intend to cause RSI", but there's a broader sense of "accidentally" which means "You followed a plan which would lead to RSI, but convinced yourself that it would not."
^{^}
To be fair, it's pretty early.

June Jimenez1d2911

Showing 3 of 8 replies (Click to show all)

June Jimenez15h10

On point 3, yes, I think it's still unclear how something like RLHF cashes out in terms of whether it improves or worsens safety. But if you imagine kind of counterpossibly that RLHF provides little to no capabilities uplift, it's hard to imagine it getting such rapid and extensive adoption. Similarly, chain of thought was mass-adopted primarily as a capabilities advance, and the monitoring uplift was a nice bonus. But to make the point of which gets prioritized at the limit, look at how nice 6.1 Sol's chain-of-thought is now.

On 5, no, I think of "worrying... (read more)

1sil16h

There's an obvious reason for this. Trillions of dollars has been invested into capabilities research because of the massive ROI it brings. Alongside that, billions of dollars has trickled into safety research since yes, alignment genuinely is in the interest of AI labs & their investors. In contrast, countering GD offers no such ROI. Uncomfortably, from the capitalist/power seeking perspective, GD is nearly the express purpose of AI investment[1]. There's an irony here where the ideas GD is drawn from predicts the exact reason it is neglected. I write this since I feel the main thing missing from GD discourse is a realist calibration towards adverse incentives. 1. ^ This is not to say investors are intentionally screwing everyone over, but to say that Molochian dynamics are.

3June Jimenez16h

This is not quite my top-level crux, as I don't think we're on track to solve technical alignment in short development timelines. But I think the question you pointed out is a central question/crux in alignment. On 1. I am interested and would like to hear more. I do agree that on balance, the current founders of powerful AI companies are nicer than many people or than their historical counterparts, though with wide dispersion around the mean. But I mostly don't think this interacts with gradual disempowerment concerns, both for the vast majority of people and even for those nearest to power currently? How much does it matter how nice the Anthropic 7 are (or were in 2022) if the world transforms rapidly around them such that they have less and less direct leverage? Not to mention; I really don't think moral character is the relevant consideration here; structural and outside factors like "How does the US executive branch feel about the concentration of power dynamics?" matter much more. So I could be persuaded that it's handle-able through ordinary policy, but I think the question the IC[1] and GD literatures ask is whether the relevant incentive structures are likely to be strengthened or weakened as AI gets more capable, whether they are likely to be acted on, and whether the relevant people will retain enough power to be able to enact that redistribution of power even if they wanted to. 1. ^ For what it's worth, I agree many of the Intelligence Curse claims in specific are fairly implausible, not least because it's unclear how strong the original Resource Curse literature is / what the effect sizes are.