Disclaimer: This post is preparation for a wider-audience op-ed I hope to publish. I'm posting here to stress-test the arguments before distilling them for a non-technical audience - feedback on the core open-source collapse argument, the hardware choke point logic, and the China cooperation section is especially welcome.
This post does not attempt to argue that advanced AI poses existential risk - there is extensive existing work on that question and I don't have much to add to it, beyond one point I develop below: that the open-source capability gap renders the entire safety paradigm moot on a short timeline. Instead, this post takes the risk as given and asks: what intervention could... (read 2619 more words →)
Considering the fact that weight sharing between actor and critic networks is a common practice, and given the fact that the critic passes gradients (learned from the reward) to the actor, for most practical purposes the actor gets all of the information it needs about the reward.
This is the case for many common architectures.
Verification of the hardware moratorium specifically is feasible. I do not believe you can verify software or compute limits - therefore targeting the hardware choke point makes sense