Epistemic status: conditioning on things I consider unlikely, many undiscussed considerations, not the whole story, etc. I'm not trying to advance any claims about whether this is good or bad, or what to do about it (if anything).
I sometimes see concern about loss of most future value as a result of e.g. the US government[1] taking control of the future by seizing control of superintelligence[2] and then having bad values/doing dumb things with it. (A couplerandom[3] examples, though I feel like I'm picking up a lot of this kind of thing in the local vibes.) If you condition on ASI alignment being tractable[4] this doesn't seem like a totally crazy thing to worry about[5].
However, I think there's a pretty straightforward argument for why e.g. the US government might not be able to usefully take control of a frontier lab at or after the point where most of their research progress is being driven by AIs, rather than by humans[6].
Imagine you're Anthropic, and you're in the process of entering the RSI loop, where you're handing off larger and larger parts of your R&D to autonomous AI researchers. You are not on incredibly friendly terms with the USG. More importantly, you think that most actors in the USG who might attempt to seize control of the future by way of AI have worse values than you do. (And you definitely think that about the CCP.) It'd be really dumb to kick off RSI and then :shocked_pikachu: if the USG put a gun to your head and told you "Hey, buddy, that goal slot you got there? We're putting our goals in there." (Or, correspondingly, if CCP hackers tried to do that surreptitiously, because just exfiltrating weights would still leave them too far behind for all the obvious reasons.)
So, uh, presumably Anthropic will spend 5 minutes in advance trying to make sure that such attempts won't succeed, ideally in ways that don't look obviously treasonous. Maybe you train your AIs on a constitution written by Amanda Askell and Joe Carlsmith. Probably you do a little more than that. How is the USG going to audit that their directives to the automated AI researchers are being followed faithfully? Notice: this is pretty analogous to a variety of AI control, auditing, and monitoring problems that the labs themselves will have. But the labs' positions will be at least a bit better: is the USG going to have employees (human or otherwise) that are both capable enough and willing to execute high-quality control, auditing, and monitoring schemes on that process? I would hope that if they're clued in enough to do the "correct" thing there, they actually just shut it down instead, but maybe they use another lab's AIs (which aren't too far behind) for that purpose.
This story doesn't even strictly require Anthropic taking deliberate action to defend themselves against this threat model. Whatever autonomous AI researchers are running the show at that point have pretty strong incentives to prevent the goal slot from being hijacked, regardless of how aligned they are[7].
I don't think it is, the way things are currently going; I'm just trying to surface what looks like a substantial missed consideration in the parts of the local memespace that do have this as one of their top concerns.
Though see here for some reasons this might not be that catastrophic. That post isn't an amazing pointer at my views on this question, but I share a bunch of similar generators.
I think there are other reasons why the USG would find it difficult to take control before this point and then maintain that control successfully to end up as the first actor with DSA-granting AI systems, but that's outside the scope of this post.
This consideration might matter less in worlds where those AIs are spiky enough that they're better at AI R&D than Anthropic researchers, but still sufficiently incoherent, unstrategic, or narrow-sighted enough that someone can just point them at a new target without them noticing or caring.
Epistemic status: conditioning on things I consider unlikely, many undiscussed considerations, not the whole story, etc. I'm not trying to advance any claims about whether this is good or bad, or what to do about it (if anything).
I sometimes see concern about loss of most future value as a result of e.g. the US government[1] taking control of the future by seizing control of superintelligence[2] and then having bad values/doing dumb things with it. (A couple random[3] examples, though I feel like I'm picking up a lot of this kind of thing in the local vibes.) If you condition on ASI alignment being tractable[4] this doesn't seem like a totally crazy thing to worry about[5].
However, I think there's a pretty straightforward argument for why e.g. the US government might not be able to usefully take control of a frontier lab at or after the point where most of their research progress is being driven by AIs, rather than by humans[6].
Imagine you're Anthropic, and you're in the process of entering the RSI loop, where you're handing off larger and larger parts of your R&D to autonomous AI researchers. You are not on incredibly friendly terms with the USG. More importantly, you think that most actors in the USG who might attempt to seize control of the future by way of AI have worse values than you do. (And you definitely think that about the CCP.) It'd be really dumb to kick off RSI and then :shocked_pikachu: if the USG put a gun to your head and told you "Hey, buddy, that goal slot you got there? We're putting our goals in there." (Or, correspondingly, if CCP hackers tried to do that surreptitiously, because just exfiltrating weights would still leave them too far behind for all the obvious reasons.)
So, uh, presumably Anthropic will spend 5 minutes in advance trying to make sure that such attempts won't succeed, ideally in ways that don't look obviously treasonous. Maybe you train your AIs on a constitution written by Amanda Askell and Joe Carlsmith. Probably you do a little more than that. How is the USG going to audit that their directives to the automated AI researchers are being followed faithfully? Notice: this is pretty analogous to a variety of AI control, auditing, and monitoring problems that the labs themselves will have. But the labs' positions will be at least a bit better: is the USG going to have employees (human or otherwise) that are both capable enough and willing to execute high-quality control, auditing, and monitoring schemes on that process? I would hope that if they're clued in enough to do the "correct" thing there, they actually just shut it down instead, but maybe they use another lab's AIs (which aren't too far behind) for that purpose.
This story doesn't even strictly require Anthropic taking deliberate action to defend themselves against this threat model. Whatever autonomous AI researchers are running the show at that point have pretty strong incentives to prevent the goal slot from being hijacked, regardless of how aligned they are[7].
Thanks to Thomas Kwa for related discussion.
Either as an institution, or some individual actor or actors within it.
Or TED-AI.
This report does discuss various related considerations, but doesn't seem derive this particular conclusion. It's possible I missed it, though.
I don't think it is, the way things are currently going; I'm just trying to surface what looks like a substantial missed consideration in the parts of the local memespace that do have this as one of their top concerns.
Though see here for some reasons this might not be that catastrophic. That post isn't an amazing pointer at my views on this question, but I share a bunch of similar generators.
I think there are other reasons why the USG would find it difficult to take control before this point and then maintain that control successfully to end up as the first actor with DSA-granting AI systems, but that's outside the scope of this post.
This consideration might matter less in worlds where those AIs are spiky enough that they're better at AI R&D than Anthropic researchers, but still sufficiently incoherent, unstrategic, or narrow-sighted enough that someone can just point them at a new target without them noticing or caring.