I'm glad you mentioned the orthogonality hypothesis (that goals are orthogonal to intelligence). Part of our argument can be seen as reconceptualizing orthogonality, and rejecting a strong version of it.
So it's interesting to me that you see orthogonality as leading folks to conflict theory over mistake theory. I am not sure it has to lead either way.
In our language, we say that it's better to have an "endogenous preferences" theory than an "exogenous preferences" theory, meaning that it's better to be able to construct models where preferences can change as a function of the mechanisms and processes being modeled. This is not true for most rational actor and RL models; they typically... (read more)
Nice work! I like this approach very much. It seems we have been thinking in very related and compatible directions to each another.
I posted a related one last week: Societal and technological progress as sewing an ever-growing, ever-changing, patchy, and polychrome quilt
"there still is semi-agreement (not just within MIRI) that the alignment problem is due to difficulty aligning the AI's goals with human goals, rather than difficulty finding the universe's objective morality to program the AI to follow."
Right, that's part of the consensus we wrote the post in order to dispute. Even though people say they recognize a difference between "align the AI's goals with human goals" and "find the universe's objective morality", they might have implicitly assumed the two are identical as a consequence of other parts of the paradigm's logic (like CEV, and the mistake theory framing, and various other equivalent foundational ideas).
Of course individuals have complex views and many who... (read more)
I agree that one part of our conclusion is related to corrigibility. We didn't use that term here but we have done so elsewhere, in older work from before we had the new theory. We could probably still use it. It might be a distraction to this audience though since corrigibility looks rather different in a multi-agent context, and it's not the focus here.
In the context of your Twitter comments about biorisk, the relevant question for this post and the appropriateness theory is probably just about whether it would be better to conceptualize these risks as part of a theory that includes a single objective for all humanity ("make the world a better place") versus a theory that doesn't accept that. Our claim is that it's easier to think about 'stability of society' and misuse type harms when using our theory. Nothing you said on Twitter is an argument for why it's important to have anything resembling a single objective.
I do think you're right to point out that the sentence you extracted from the long paper where we speculate about the future doesn't really follow from the theory itself.
Well that fetal alcohol syndrome is associated with poor emotional regulation is apparently a thing that is true in the sense that you can measure it, and it seems to be helpful to know it if you're treating patients with fetal alcohol syndrome or living with them.
But our theory is formal, it's not just a collection of true statements. It's not really clear how we could use it to model the effect of fetal alcohol syndrome on emotional regulation. So, there's a sense in which we don't capture that "feature of reality". But how important is it to capture? All theories are wrong in some ways. The only way to judge them... (read more)
I don't think our story is really ultimately about "making chatbots polite", it's just that impoliteness or awkwardness are common manifestations of AIs acting in contextually inappropriate ways that happen all the time, right now, today.
However, at present, people are starting to give AIs more decision making authority in various domains. And as we continue doing that, giving AIs more and more decision making authority, the scope of possible harm from contextually inappropriate behavior will grow massively. I don't think the scope of possible harm achieved this way is different between our theory and others in the AI safety community.
We are just saying that we think it's useful to conceptualize the possible... (read more)
"Values are free parameters in rational behavior"
Right, so this is the part we reject. In the long theory of appropriateness paper, instead of having a model with exogenous preferences (that's the term we use for the assumption that values are free parameters in rational behavior), we say that it's better to have a theory where preferences are endogenous so they can change as a function of the social mechanisms being modeled.
So, in our theory, personal values are caused by social conventions, norms, and institutions.
Combine this with the contrast between thick and thin morality, which we did mention in the post. You get the conclusion that it's very difficult for individuals to tell... (read more)
Joel Z. Leibo [1], Alexander Sasha Vezhnevets [1], William A. Cunningham [1, 2], Sébastien Krier [1], Manfred Diaz [3], Simon Osindero [1]
[1] Google DeepMind, [2] University of Toronto, [3] Mila Québec AI Institute
We published a more academically oriented version of this post on arxiv, available here.
Disclaimer: These are our own opinions; they do not represent the views of Google DeepMind as a whole or its broader community of safety researchers.
"We pragmatists think of moral progress as more like sewing together a very large, elaborate, polychrome quilt, than like getting a clearer vision of something true and deep." – Richard Rorty (2021) pg. 141
Quite a lot of thinking in AI alignment,... (read 7415 more words →)
I agree, that's also where I thought the movie was going when I watched it. But maybe we're more interested in or primed to think about anti-essentialism than the average viewer.
Another explanation though: your ending would work best if it were intended as a single standalone film. But, the creators are surely anticipating a raft of sequels. They need to keep the demons evil to set up future conflict in future movies.