Controlling AGI Risk
6Carl Feynman
1TeaSea
3Ppau
2TeaSea
1Ppau
2Seth Herd
2Nathan Helm-Burger
3TeaSea
New Comment
I’m going to say some critical stuff about this post. I hope I can do it without giving offense. This is how it seemed to one reader. I’m offering this criticism exactly because this post is, in important ways, good, and I’d like to see the author get better.
This is a long careful post, that boils down to “Someone will have to do something.” Okay, but what? It’s operating at a very high level of abstraction, only dipping down into the concrete only for a few sentences about chair construction. It was ultimately unsatisfying to me. I felt like it wrote some checks and left them up to other people to cash. I felt like the notion of sociotechnical system, and the need for an all-of-society response to AI, were novel and potentially important. I look forward to seeing how the author develops them.
This post seems to attempt to recapitulate the history of the AI risk discussion in a few aphoristic paragraphs, for somebody who’s never heard it before. Who’s the imagined audience for this piece? Certainly not the habitual Less Wrong reader, who has already read “List of Lethalities” or its equivalent. But it is equally inappropriate for the AI novice, who needs the alarming facts spelled out more slowly and carefully. I suspect it would help if the author clarified in their mind whom they imagine is reading it.
The post has the imagined structure of a logical proof, with definitions, axioms, and a proposition. But none of the points follow from each other with the rigidity that would require such a setup. When I read a math paper, I need all those things spelled out, because I might spend fifteen minutes reading a five-line definition, or need to repeatedly refer back to a theorem from several pages ago. But this is just an essay, with its lower standards of logical rigidity, and a greater need for readability. You’re just LARPing mathematics. It doesn’t make it more convincing.
Thanks very much, Carl. Your feedback is super-useful, and much appreciated. I'll take it on board along with other comments and will work on a follow-up that gives more examples of what sort of controls might be deployed in the wider system.
As a fellow member of the regrettably small overlap between rationalists and adepts of ecological psychology, (https://www.lesswrong.com/posts/Y4hN7SkTwnKPNCPx5/why-don-t-more-people-talk-about-ecological-psychology), I'm looking forward to seeing your next posts!
You've probably seen this recent discussion post
"How useful is "AI Control" as a framing on AI X-Risk?"
It addresses the control issue you raise, and has links to other work addressing the same issue.
I like your effort to think holistically about the sociotechnical systems we are embedded in and the impacts we should expect AI to have on those systems.
I have a couple of minor critiques of the way you are breaking things down that I think could be improved.
First, a meta thing. The general pattern of being a bit to black & white about described very complicated sets of things. This is nice because it makes it easier to reason about complicated situations, but it risks over simplifying and leading to seeming strong conclusions which don't actually follow from the true reality. The devil is in the details, as they say.
Efforts to-date have largely gravitated into the two camps of value alignment and governance.
I don't think this fully describes the set of camps. I think that these are two of the camps, yes, but there are others.
My breakdown would be:
Governance - Using regulation to set up patterns of behavior where AI will be used and developed in safe rather than harmful ways. Forcing companies to internalize their externalities (e.g. risks to society). Preventing and enforcing human-misuse-of-AI scenarios. Attempting to regulate novel technologies which arise because of accelerated R&D as result of AI. Setting up preventative measures to detect and halt rogue AI or human-misused AI in the act of doing bad things before the worst consequences can come to pass. Preventing acceleration spirals of recursive self-improvement from proceeding so rapidly that humanity becomes intellectually eclipsed and looses control over its destiny.
Value alignment - getting the AIs to behave as much as possible in accordance with the values of humanity generally. Getting the AI to be moral / ethical / cautious about harming people or making irreversible changes with potentially large negative consequences. Ideally, if an AI were 'given free reign' to act in the world, we'd want it to act in ways which were win-win for itself and humanity, and no matter what to err on the side of not harming humanity.
Operator alignment - technical methods to get the AI to be obedient to the instructions of the operators. To make the AI behave in accordance with the intuitive spirit of their instructions ('do what I mean') rather than like an evil genie which follows only the letter of the law. Making the AI safe and intuitive to use. Avoiding unintended negative consequences.
Control - finding ways to keep the operators of AI can maintain control over the AIs they create even if a given AI gets made wrong such that it tries to behave in harmful undesirable ways (out of alignment with operators). This involves things like technical methods of sandboxing new AIs, and thoroughly safety-testing them within the sandbox before deploying them. Once deployed, it involves making sure you retain the ability to shut them off if something goes wrong, making sure the model's weights don't get exfiltrated by outside actors or by the model itself. Having good cybersecurity, employee screening, and internal infosec practices so that hackers/spies can't steal your model weights, design docs, and code.
A minor nitpick:
Sociotechnical system/s (STS)
A system in which agents (traditionally, people) interact with objects (including technologies) to achieve aims and fulfill purposes
Not sure if objects is the right word here, or rather, not sure if that word alone is sufficient. Maybe objects and information/ideas/concepts? Much of the work I've been doing recently is observing what potential risks might arise from AI systems capable of rapidly integrating technical information from a large set of sources. This is not exactly making new discoveries, but just putting disparate pieces of information together in such a way as to create a novel recipe for technology. In general, this is a wonderful ability. In the specific case of weapons of mass destruction, it's a dangerous ability.
Nested STS / Human STS
Yes, historically, all STS have been human STS. But novel AI agents could, in addition to integrating into and interacting with human STS, form their own entirely independent STS. A sufficiently powerful amoral AGI would see human STS as useless if it could make its own that served its needs better. Such a scenario would likely turn out quite badly for humans. This is the concept of "the AI doesn't hate you, it's just that humans and their ecosphere are made of atoms that the AI has preferred uses for."
This doesn't contradict your ideas, just suggests an expansion of possible avenues of risk which should be guarded against. Self-replicating AI systems in outer space or burrowed into the crust of the earth, or roaming the ocean seabeds will likely be quite dangerous to humanity sooner or later even if they have no interaction with our STS in the short term.
Brilliant! Thanks for these insightful comments, Nathan. I'll endeavour to address them in a follow-up post.
1