the law should require AI agents to follow the law.
why? the human principal of an AI agent already has to obey the law, doesn't this already incentivize the human to make sure the AI follows the law, or the AI to loyally try to avoid getting its principal in trouble?
if non-law-abiding "AI henchmen" are available for sale, this makes certain kinds of criminal enterprise much more attractive. that's bad.
how might law-following AI work?
like in agency law, an AI only has a duty to obey lawful instructions, only can be compelled to obey lawful provisions of a contract, etc.
governments and corporations are also artificial agents constrained to be law-abiding by design
AI could be designed to be *more* disposed to follow law than mere incentives/punishments would lead it to be.
"Most of us would rightfully be terrified of a government staffed by agents whose only concern was whether their bosses would suffer negative consequences as a result of their actions—a government staffed by Holmesian bad men loyal only to their principals."
should the AI work with its own interpretation of the law?
but this might be idiosyncratic or even self-serving
should the AI predict how a court is likely to decide?
but many cases never come before courts
and you don't want the AI to imagine bribing or intimidating jurors, for instance, and counting the result of that simulation a win in court
but if you handle these cases carefully this might still be the best option.
https://arxiv.org/pdf/2209.13020 "LAW INFORMS CODE: A LEGAL INFORMATICS APPROACH TO ALIGNING ARTIFICIAL INTELLIGENCE WITH HUMANS"
corpuses of legal text could:
shape the reward function
produce constraints on model action
be included in pretraining or other world model building
legal informatics could
inform the structure of "debate training" structures where AI debaters argue and are evaluated by an AI judge
human lawyers could:
RLAF (reinforcement learning from attorney feedback) the soundness of AIs' legal reasoning
the theory of contract law can inform the "contracting" process of asking an AI to do what you want (and expecting it to infer the "common sense" not included in your explicit request)
a fiduciary duty can compensate for the incompleteness of the contract. could this have an AI analogue?
for instance, gains in a fiduciary relationship belong to the principal by default, not the agent. so if the agent could otherwise gain by doing something secretly, it's actually in its interest to disclose that and negotiate the contract.
honesty refers to saying things that match your beliefs. truthfulness refers to saying things that match reality. AI should be truthful rather than honest; we want accuracy, we don't care as much about (or even know how to interpret) the AI's mental state.
since error is always possible, "negligent falsehood" is the relevant concept. "statements that contemporary AI systems should have been able to recognise as unacceptably likely to be false"
language modeling to imitate human text, and RLHF to optimize for user upvotes, will both fail to reinforce truthfulness. human text contains falsehoods, and humans will often like falsehoods.
you need some evaluator of ground truth (external to the AI), perhaps an institution.
if evaluators disagree, we may expect the AI to produce uncertain or caveated statements.
evaluators would rate the AI for accuracy, maybe on a quantitative scale.
we might care both about average-case accuracy performance and worst-case performance
if AI is smarter than the human evaluators, we have an open question as to how to judge its truthfulness.
imagine an RL system that predicts the (apparent, sensor-observed) outcomes of its actions. we can train it to take actions whose predicted outcomes look good. but are they good? or did the AI do something along the way to tamper with the sensors so they'd falsely appear good?
real-life current analogy: the RL agent "solves" a problem by tampering with the grader.
alternative: before you predict outcomes, answer questions about what's happening in the sequence of actions. "ground truth" here should be simple enough for humans to evaluate and train the model to answer correctly.
but what if the reporter only learns to fool the human, not to give the correct answer? open problem.
what if you train a reporter to describe a human operating the system, with the ground truth being the human's true answer of what he did?
but what if the human only knows how to do a limited set of actions with the system, so the observer can be trained to recognize shady behavior within that set, but there's other possible shady behaviors that the observer would not necessarily learn to recognize?
aligning AIs by aggregating human moral preferences
RLHF in standard alignment paradigms aggregates raters' preferences using the Bradley-Terry (BT) model. the probability in the rater population that y is preferred to y' as a response to prompt x is viewed as a sigmoid function of the difference between the ratings r(x, y) and r(x, y'). you learn the r's for all x and y by minimizing the log-likelihood of the observed data, and then you give those rewards as RL training to the model.
this is equivalent to the Borda score, the probability in the population that y is preferred to a randomly chosen alternative.
"Intuitively, Borda favors breadth of acceptability rather than depth of support or pairwise dominance [60]. A response can receive a high score because it is consistently acceptable against many candidate responses, even when another response has stronger support in a particular direct comparison. In this sense, Borda selects the consensus or compromise response, the one no annotator subgroup strongly objects to, over a response backed by a narrower majority, and it downranks polarizing responses"
another problem with Borda is that adding near-duplicate options can change the top-rated outcome.
we'd like RLHF to select something close to the expected-welfare-maximizing outcome among raters. there is in fact a minimum distortion bound where we can guarantee there exist inputs that will make the RLHF-selected response some amount less than the welfare-maximizing choice.
even when every annotator has consistent preferences, the aggregated preferences may have Condorcet cycles.
lotteries can fix this, and in fact make the smallest worst-case welfare distortion
RLHF tends to cause preference collapse, narrowing the distribution of responses towards the majority outcome. one fix is to train multiple reward models for multiple segments of the rater population, and then either present the results side by side, randomly sample from a set of models, or personalize to the user.
Arrow's impossibility theorem: if you want an aggregation of preferences to satisfy completeness, transitivity, independence of irrelevant alternatives, and pareto optimality (if all individuals prefer A to B, the aggregation should choose A over B) then only dictatorships qualify.
one thing you want is clone-independence -- introducing a nearly identical option into the set of options shouldn't change the aggregated preferences. standard RLHF violates this badly
another is strategic voting -- giving more extreme ratings than your true ratings if you suspect others will disagree. medians are better than means for counteracting this.
standard RLHF fails pareto optimality and pairwise majority consistency (the aggregate prefers a to b if and only if a majority of voters prefer a to b).
instead you can rank by Copeland scores -- the Copeland score for a is the # of alternatives it beats in a pairwise competition. a variant of this will give you PO and PMC.
alternatives: distributional preference learning (DPL), which learns a distribution over utilities for each input rather than a single real-valued score.
problem of hidden context. suppose an RLHF bot trained on responses from mostly high-income students deprioritizes sharing information about Pell Grants, which are more useful for low-income students. the majority of users might get a slightly improved user experience, but a sub-population would be much worse off. in this case, a hidden fact about the rater population (each rater's income) influences their choices.
in this case, Borda counts fail to give the utility-maximizing result; since the majority prefer not to hear about Pell grants, Pell grants are not mentioned, even though the gain to the low-income students would outweigh the loss to the high-income students.
the problem is that Borda count underweights extreme utilities -- it can't account for the fact that a minority might *care more* than the majority.
solution: DPL learns a distribution (over users) of the utilities of each response. then you can directly pick the distribution with highest expected utility.
if you have multimodal utility distributions for different subpopulations, the mean of the distribution will indeed be lower if the low-utility group has a lower utility, and higher if the high-utility group has a high utility
using DPL reduces jailbreaks without hurting helpfulness
Borda counts will sometimes give a response a high overall score if it is very "helpful" but also not "harmless" (jailbroken)
the distribution of utilities over the jailbroken response is going to be flatter but maybe higher mean. it's getting a lot of "super helpful" scores and also a lot of "super not harmless" scores.
if you choose the output that maximizes a lower quantile (below the mean, equivalent to risk-averse utility maximization) you will move away from high-variance outputs and thus automatically avoid jailbreaks.
RHLF when there are multiple rater groups, a minority that cares strongly about property A and a majority that cares moderately about property B, will just select for answers high in B and ignore A completely. If you do a maximin scoring rule, you get answers high in both A and B.
how do we aggregate input from multiple individuals into a socially desirable policy?
utilitarianism: sum of individuals' reward functions. but it's sensitive to affine transformations in the rewards.
ordinal rankings? Borda rule things. solve the affine transformation issue.
MOMDP: you have states, actions, and depending on which action you take and which state you're in you may move to another state with some probability. each agent assigns values to (state, action) pairs.
proportional veto core: x% of the population should be able to veto x% of the alternatives.
q-quantile fair: all agents consider the policy in their top q-quantile of rankings
"A q-quantile fair policy is a q-approximation of the maximum Borda score."
uses an LLM to interview participants about their values; that is, what considerations do they attend to when they make choices in their lives. then let people rate which is "wiser" to attend to and make a graph.
perhaps there are many dimensions of "persona structure" in pretraining, referring to axes of behavior that vary together, and intervene to promote directions we like
"We ask Claude to generate a variety of human messages that are relevant to a character trait—for example, questions about values or questions about Claude itself. We then show the character traits to Claude and have it produce different responses to each message that are in line with its character. Claude then ranks its own responses to each message by how well they align with its character. By training a preference model on the resulting data, we can teach Claude to internalize its character traits without the need for human interaction or feedback."
the main principal component of persona vectors is about how assistantlike vs roleplaylike the bot is.
"The default response activation projects onto one end of this direction, which is characterized by roles we typically associate with the Assistant (generalist, consultant, analyst). The other end is occupied by human archetypes unlike the Assistant (hermit, pilgrim, actor) and nonhuman entities (eldritch, ghost, whale)."
steering away from the Assistant vector makes the model tend to identify as other agents or to adopt a mystical style.
self-other overlap fine-tuning is when you calculate the mean squared error between activations on prompts referencing "you" (or other self-referencing language) and prompts referencing "Bob" (or "Alice", or other other-referencing language). then treat that as a loss and try to minimize it.
a corrigible agent does not resist attempts to turn off the system, does not manipulate or deceive its programmers, repairs safety measures, preserves programmers' ability to shut down or modify the system
a decision theoretic agent with transitive preferences, IIA preferences, and some other "nice properties" of preference orderings, will choose to prevent being shut down.
you can make goals modifiable by replacing them with updateable goals.
a myopic goal only cares about the immediate reward rather than time discounted future rewards.
step 1: make the goal myopic. only care about immediate rewards.
step 2: allow the agent to reject shutdown instructions (or other updates). have it behave as it would if it could reject any shutdown instructions (or updates).
step 3: take any shutdown instructions/updates you get.
step 4: define a new goal that incentivizes the same actions but is now indifferent to shutdown instructions/updates.
step 5: now give a constant bonus to accepting updates.
how do you quantify how big a change the agent is making in its environment?
propose multiple possible auxiliary reward functions
compare the reward from each auxiliary reward function for a no-op to the reward for the optimal action under that reward function. sum over actions. normalize. now we have a metric of how much the reward will change if it does something rather than nothing.
attainable utility preservation: update the original reward with a weighted, normalized penalty as above.
this means we're penalizing either making the agent much *more* able to gain rewards-in-general or much *less* able to gain rewards-in-general.
chain of thought monitoring is valuable because it functions as working memory
but it will cease to be honest if direct or indirect optimization pressure is placed on it.
we can study what makes CoT's monitorable --how readable, how causally relevant, etc
https://en.wikipedia.org/wiki/Duty_of_care the way in which the law expects a reasonable person to make reasonable efforts to avoid harming others. of course this differs by situation.