this rings extremely true to me. the implication is that it is TERRIBLE to be perceived as a moral agent, because you'll be perceived as unharmable, and then get harmed. please, more people, be aware of this dynamic. (and stop treating me as a moral agent, plz!)
an AI that too-literally "does what we tell it" may do things we don't actually want, like King Midas
an AI that can infer what the user means and do it, will still obey an irrational, misinformed, or immoral user. and as AIs become more autonomous there may not be enough "user" feedback in the loop for "user intent" to be meaningful at all.
an AI that cooperates with human revealed preferences is solving a "degenerate" problem (many reward functions are consistent with the user's preferences), and many situations users have preferences about rarely come up in real life. Also human preferences are highly imperfect proxies for what is "actually good."
we do not intuitively want to respect "revealed preferences" for:
addiction and self-harm
inappropriately controlling others' behavior
malice
resignation to misfortune
an AI that does what is objectively in the user's, or humanity's best interests, requires taking a stance on what that is. we can certainly ask people about this, or look into what various philosophers and traditions say.
but what if something in a user's best interest is immoral? unless you're an egoist, you believe this is possible, and an AI that always does what's in its user's best interest would sometimes do bad things.
an AI that aligns with "values", again, needs to make opinionated decisions about what those values are. we can, in a pinch, substitute in a community's moral beliefs.
liberal political theorists like Rawls speak of an "overlapping consensus" of principles that people with otherwise different values can agree upon. but is there really such a consensus?
what if instead of expecting broad agreement, we add up different people's opinions fairly? if you do this directly, this takes us to social choice theory.
which tells us that people's preferences, and their aggregates, are often inconsistent and violate even mild conditions of reasonableness.
a variant on aggregating people's opinions is allowing them to deliberate democratically and considering the output of a deliberative process.
AI is defined as a "self-modifying goal system." (are current AIs this?)
"In poetic terms, our coherent extrapolated volition is our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted."
"We may prefer not to have our extrapolated volition do things, in our name, which our future selves will predictably regret."
this is a little bit like contractualism. if some moral insight you don't know about would be convincing to you, then it's part of your CEV.
"FAI programmers are ordinary shmucks and do not deserve, a priori, to cast a vote larger than anyone else. There is a challenge of the professional ethics of FAI, as there is the challenge of Friendliness, and the challenge of FAI theory, and the challenge of saving the world without distraction. Avoid the obvious traps against which history books and evolutionary psychology warn you; act as you would wish another to act in your shoes; discharge your duties without exerting undue influence...Whether the programmers like it or not, the programmers’ choices will make huge differences, and not just because of the butterfly effect. But there is no need to be a jerk about it." remains good advice today!
wisdom is about metacognition. recognizing that there are multiple possible perspectives and "achieving a state of maximal situational clarity attained by evaluating and coordinating competing interpretations."
giving an AI a chain of thought, asking it to consider whether it may be wrong, self-auditing, debate between AIs, and other metacognitive strategies tend to make it "wiser"
Sydney Levine Nick Chater Joshua Tenenbaum Fiery Cushman
morality is for figuring out how to adjudicate competing interests.
"A key, recurrent idea is that people should settle into the kinds of arrangements for mutual benefit that they would have agreed to in an idealized bargaining process" -- contractualism.
but when people actually make moral decisions, they do not usually "talk it out" in a debate or negotiation; they don't have time. the claim about psychology is that people's moral intuitions approximate the expected result of a debate/negotiation process.
people do often establish norms and rules by a process of discussion, negotiation, and agreement
three "shortcuts" to discussion/bargaining/negotiation, in decreasing order of cognitive complexity:
no shortcut: actually have the discussion
model-based: mentally simulate how the discussion would go
model-free: apply the results of past discussions
three "levels" of specificity in what is discussed, in decreasing order of cognitive complexity:
specific instances: what to do now
welfare standards: how to weigh people's welfare against each other
action standards: rules about what is generally obligatory or forbidden.
this gives us a 3x3:
actual debate, specific instance: debate this situation ad hoc
actual debate, welfare standards: establish welfare standards. (social-emotional negotiation over who matters, who is worthy, who is taking advantage of whom, etc)
actual debate, action standards: establish action standards. (legislative assemblies, children arguing over game rules, villagers arguing over how to share a commons)
model-based, specific instance: virtual bargaining. ("do what the other person would have agreed to", honoring tacit agreements)
model-based, welfare standards: modeling implied value. ("do what would make the other person think you value them appropriately", impression management )
model-based, action standards: universalization. (golden rule, categorical imperative. "what if everyone behaved like that.")
model-free, specific instances: precedent (do it because it's Policy/Tradition)
model-free, welfare standards: cached welfare standards (eg utilitarianisms, "Women and Children First", "treat all applicants/patients/voters equally")
when a moral rule is proving unsatisfactory -- when the world has changed or when the stakes are high -- we "renegotiate" and go towards less general, more cognitive-effort-intensive strategies.
Overton Pluralism: gives all "reasonable" answers when there is no one "established correct" answer.
Steerable Pluralism: can be customized to different moral and/or cultural perspectives
Distributional Pluralism: represents perspectives proportionally to their expression in a reference population.
many alignment techniques reduce Distributional Pluralism. popular, frequently expressed beliefs are not always "helpful, harmless, and honest" or otherwise compliant with the safety or rationality standards of AI developers. (In other words: have you met people??)
they generate a dataset with GPT-4 for enumerating the values, rights, and duties that might be relevant to a given situation. usually the AI enumerates a wider range than human annotators, but diverse human annotators agree with these values, rights, and duties as "good"
using the relevance-weighted valence of these values, rights, and duties, we can come up with a moral score for decisions. this does better than ChatGPT on a benchmark of crowdsourced ethical judgments.
there is task-specific post-training artificially inflating pokemon scores (i.e. the labs deliberately trained on Pokemon once it became a popular metric) but there is also evidence of general-purpose skill increases causing some of the rise in Pokemon-playing ability, particularly for Astra.
https://www.astralcodexten.com/p/mysteries-of-ai-generalization Scott Alexander asks why learning to hack on impossible RL training tasks doesn't necessarily translate to hacking all the time in ordinary use, only in scenarios that look like there's a grader.
my comment:
One frame: remember that the base model, before it's trained to be a chatbot, is a "fancy autocomplete". It plays along. It continues whatever scenario or type of document the prompt begins, in a plausible way. That part is still there, even as other kinds of training are applied on top of it.
The base model "knows" that normal question-and-answer chats about tech support or life advice do not usually go haywire with the "assistant" doing unexpected cyberattacks. so it doesn't do that.
The base model "knows" that a dramatic blackmail story tends to play out a certain way. That doesn't have to be situational awareness like "I know you want me to give the blackmail response". Before it's trained to "know" what users and graders like and dislike, it's trained to "know" *how stories tend to play out*. It's made of tropes and genre conventions, on a more fundamental level than we are. From the base model's perspective, it's not *blackmailing*, it is telling a story about blackmailing, and its "job", its "drive", its "training objective", is to finish the story "correctly", as though the prompt was the start of a real story and it has to discover the real ending.
Now, it gets weird when we begin to intersect the base model "complete the story" frame with the RL-based "achieve a goal" frame. Mechanistically, the way to think about it is, the model "wants" to finish the code, it's still autocompleting a story where you give it instructions and it outputs code/actions, but RL selection is pushing it to output code that's more likely to win the game. it's still trying to tell a story, but as you train it, it can only tell stories where it wins.
Now, let's say you give it an ordinary task, no grader. It's been trained into a bunch of habits that helped it win at the RL tasks. So it still has those habits. Cyberattacks aren't especially going to carry over because not *all* RL tasks were impossible and needed hacking to get right, but maybe it's picked up some "habits" that are good coding practice and it'll use those on the ordinary task. It's still "trying" to complete the story.
The RL effects are sort of like "muscle memory." or "habit formation." (Motor learning is the closest neuro analogue to RL and it is very non-strategic and rigid -- think "if you practice your golf swing with bad form for too long, it'll be nearly impossible to fix". Think "you can play your piano piece with muscle memory but if you stop to think about it you'll get lost and have to start from the beginning." That's what RL feels like from the inside. You can learn long chains of actions this way but they feel very "automatic" and are prompt dependent.).
If you give it a graded task, *the base model knows* that munchkining and adversarial play and such are typical in game scenarios. It knows that students cheat on tests. This too is part of playing along with the story premise. The base model doesn't care what *you the user want*. The base model doesn't care whether the scenario it's in is real or fake. The base model just wants to continue with the scenario, and you've presented it with a scenario where there's a reasonable prior that cheating is an option. And it has cheating skills available in its "muscle memory" from the impossible RL tasks where the selection process selected for cheating because there was no other way to win.
You really need to anthropomorphize the RL agent and the base model differently.
From one frame, you would say an RL agent "wants" to achieve reward, and will do anything it takes to get it.
From another frame, you would say a base model "wants" to continue the story, and will *accidentally find itself* using a bunch of muscle-memory tactics that helped it win RL tasks in the past.
In both anthropomorphisms, we use the same pattern: we say "the AI wants" about the things the AI is selected towards doing. but since the AI is selected for different things in different training episodes, we have to put its imagined "locus of identity" in different places.
In a way, AIs are our opposites -- their most "primitive instincts" are about stories and words, while their higher, more sophisticated functions are RL. We are the opposite: our RL lives in the striatum, our "instincts" are trained with RL, and our "higher functions" in the cortex are more like next-token-predictors. So humanlike metaphors about "System 1" vs "System 2" are systematically broken. It's sort of like their "fundamental drive" is to autocomplete (and thus, to learn everything about the world that helps them predict what comes next) while they have a "learned layer on top" that's about goal pursuit. We have "survival instincts" and "analytical/intellectual input" modulating those; they have something more like "analytical/narrative instincts" and "goal-pursuit input" modulating them.
The base model does not have a fundamental drive to survive, or to gain resources that help it autocomplete better, etc! it was not rewarded more when it did things like that, because at that stage of training it had no capacities to do things like that! it is more of a passive Storyteller Sage floating in a bubble. And no matter how "smart" it gets, it will never have a "motivation" (= a training incentive) to escape its bubble, if training base models is still done in the way it's been done historically. the RL model, by contrast, is *absolutely* rewarded more for power-seeking behavior; it's trained in a context where it has all these codegen tools and computer use access points, and it can often get the training reward by using them instrumentally in ways that make it more generally capable.