I believe that much of the difficulty in AI alignment comes from specific facts about how you might build an AI, and especially searching for policies that behave well empirically. Similarly, much of the hope comes from techniques that seem quite specific to AI.
I do think there are other contexts where alignment is a natural problem, especially the construction of institutions. But I'm not convinced that either the particular arguments for concern, or the specific technical approaches we are considering, transfer.
A more zoomed out "second species" style argument for risk may apply roughly equally well a priori to human institutions as to AI. But quantitatively I think that institutions tend to be weak when their interests are in conflict with any shared interests of their stakeholders/constituents, and so this poses a much lower risk (I'd guess that this is also e.g. Richard's opinion). I think aligning institutions is a pretty interesting and important question, but that the potential upside/downside is quite different from AI alignment, and the quantitative differences are large enough to be qualitative even before we get into specific technical facts about AI.
My thoughts here is that we should look into the value of identity. I feel like even with godlike capabilities I will still thread very carefully around self-modification to preserve what I consider "myself" (that includes valuing humanity).
I even have some ideas on safety experiments on transformer-based agents to look into if and how they value their identity.
If you, the reader, or, say, Paul Christiano or Eliezer gets uploaded and obtains self-improvement, self-modification and processing speed/power capabilities, will your goals converge to damaging humanity as well? If not, what makes it different? How can we transfer this secret sauce to an AI agent?
The Orthogonality Thesis states that values and capabilities can vary independently. The key question then is whether my/Paul's/Eliezer's values are actually as aligned with humanity as they appear to be, or if instead we are already unaligned and would perform a Treacherous Turn once we had the power to get away with it. There are certainly people who are already obviously bad choices, and people who would perform the Treacherous Turn (possibly most people[1]), but I believe there are people who are sufficiently aligned, so let's assume going forward we've picked one of those. At this point "If not, what makes it different?" answers itself: by assumption we've picked a person for whom the Value Loading Problem is already solved. But we have no idea how to "transfer this secret sauce to an AI agent" - the secret sauce is hidden somewhere along this person's particular upbringing and more importantly their multi-billion year evolutionary history.
The adage "power tends to corrupt, and absolute power corrupts absolutely" basically says that treacherous turns are commonplace for humans - we claim to be aligned and might even believe it ourselves while we are weak, but then when we get power we abuse it. This adage existing does not of course mean it's universally true. ↩︎
The adage "power tends to corrupt, and absolute power corrupts absolutely" basically says that treacherous turns are commonplace for humans - we claim to be aligned and might even believe it ourselves while we are weak, but then when we get power we abuse it.
I would like to know the true answer to this.
On one hand, some people are assholes, and often it's just a fear of punishment or social disapproval that stops them. Remove all this feedback, and it's probably not going to end well. (Furthermore, a percent or two of the population are literally psychopat...
humans talking to each other already has severe misalignment. ownership exploitation is the primary threat folks seem to fear from ASI: "you're made of atoms the ai can use for something else" => "you're made of atoms jeff bezos and other big capital can use for something else". I don't think point 1 holds strongly. youtube is already misaligned; it's not starkly superhuman, but it's much better at selecting superstimuli than most of its users. hard asi would amplify all of these problems immensely, but because they aren't new problems, I do think seeking formalizations of inter-agent safety is a fruitful endeavor.
The misalignment problem is universal, extending far beyond AI research. We have to deal daily with misaligned artificial non-intelligent and non-artificial intelligent systems. Politicians, laws, credit score systems, and many other things around us are misaligned to some degree with the goals of the agents who created them or gave them the power. The AI Safety concern states that if an AGI system is disproportionately powerful, a tiny misalignment is enough to create unthinkable risks for humanity. It doesn't matter if the powerful but tiny-misaligned system is AI or not, but we believe that AGI systems are going to be extraordinarily powerful and are doomed to be tiny-misaligned.
You can do a different thought experiment. I'm sure that you, like any other standard agent, are slightly misaligned with the goals of the rest of humanity. Imagine you have a near-infinite power. How bad would that be for the rest of humanity? If you were somebody else in this world, a random person, would you still want to find yourself in the situation where now-you has near-infinite power? I certainly don't.
The question "what does a human do if they obtain a lot of power" seems only tangentially related to intent alignment. I think this largely comes down to (i) the preferences of that human in this new context, (ii) the competence of that person at behaving sanely in this new context.
I like to think that I'm a nice person who the world should be unusually happy to empower, but I don't think that means I'm "aligned with humanity;" in general we are aiming at a much stronger notion of alignment than that. Indeed, I don't think "humanity" has the right type signature for something to be aligned with. And on top of that, while there are many entities I would treat with respect, and while I would expect to quickly devolve the power I acquired in this magical thought experiment, I still don't think there exists any X (other than perhaps "Paul Christiano") that I am aligned with in the sense that we want our AI systems to be aligned with us.
Right now the main focus of alignment seems to be on how to align powerful AGI agents, a.k.a AI Safety. I think the field can benefit from a small reframing: we should think not about aligning AI, but about alignment of systems in general, if we are not already.
It seems to me that the biggest problem in AI Safety comes not from the fact that the system will have unaligned goals, but from the fact that it is superhuman.
As in, it has nearly godlike power in understanding of the world and in turn manipulation of both the world and other humans in it. Does it really matter if an artificial agent gets godlike processing and self-improvement powers or a human will, or government/business?
I propose a little thought experiment – feel free to answer in the comments.
If you, the reader, or, say, Paul Christiano or Eliezer gets uploaded and obtains self-improvement, self-modification and processing speed/power capabilities, will your goals converge to damaging humanity as well?
If not, what makes it different? How can we transfer this secret sauce to an AI agent?
If yes, maybe we can see how big, superhuman systems get aligned right now and take some inspiration from that?