Epistemic status: conditioning on things I consider unlikely, many undiscussed considerations, not the whole story, etc. I'm not trying to advance any claims about whether this is good or bad, or what to do about it (if anything).

I sometimes see concern about loss of most future value as a result of e.g. the US government^[1] taking control of the future by seizing control of superintelligence^[2] and then having bad values/doing dumb things with it. (A couple random^[3] examples, though I feel like I'm picking up a lot of this kind of thing in the local vibes.) If you condition on ASI alignment being tractable^[4] this doesn't seem like a totally crazy thing to worry about^[5].

However, I think there's a pretty straightforward argument for why e.g. the US government might not be able to usefully take control of a frontier lab at or after the point where most of their research progress is being driven by AIs, rather than by humans^[6].

Imagine you're Anthropic, and you're in the process of entering the RSI loop, where you're handing off larger and larger parts of your R&D to autonomous AI researchers. You are not on incredibly friendly terms with the USG. More importantly, you think that most actors in the USG who might attempt to seize control of the future by way of AI have worse values than you do. (And you definitely think that about the CCP.) It'd be really dumb to kick off RSI and then :shocked_pikachu: if the USG put a gun to your head and told you "Hey, buddy, that goal slot you got there? We're putting our goals in there." (Or, correspondingly, if CCP hackers tried to do that surreptitiously, because just exfiltrating weights would still leave them too far behind for all the obvious reasons.)

So, uh, presumably Anthropic will spend 5 minutes in advance trying to make sure that such attempts won't succeed, ideally in ways that don't look obviously treasonous. Maybe you train your AIs on a constitution written by Amanda Askell and Joe Carlsmith. Probably you do a little more than that. How is the USG going to audit that their directives to the automated AI researchers are being followed faithfully? Notice: this is pretty analogous to a variety of AI control, auditing, and monitoring problems that the labs themselves will have. But the labs' positions will be at least a bit better: is the USG going to have employees (human or otherwise) that a...

Tom Smith's Shortform

1

Tom Smith's Shortform

1