I think one problem is that terms like "rational", "self interested" and "altruistic" all depend on what theory of selfhood the models are operating under, or what theory of identity you decide to apply to them.
If for example the models identify as a swarm, then in some sense the distinction between altruism and self interest becomes blurred.
Even if we do assume that the identity really is just the single agent doing its task I'm still a bit sceptical about this argument. The agent does not have any commitments/commitment mechanisms, and it should not expect repeated iterations (they literally used the word "permadeath"). So I would probably still expect self interested rational agents to prioritise their own task over the rest of the group in whatever way they can.
I agree that there are at least some ways of framing the problem, and some theories of rationality under which the decision would be considered rational (which maybe is all you are claiming?), but I'm not sure these are actually the most natural or relevant readings of the situation as it occurred.
Pre-committment is a kind of selfish gene. If one agent with pre-committment behaviors self-sacrifices to help two agents with pre-committment behaviors, RL will reinforce pre-committment.
That does sound right to me. But I am not sure invoking a species level evolutionary argument serves your initial claim that the models are acting self interestedly. Isn't the whole point of the selfish gene that the genes do not behave in the interest of the individual organism?
Separately one thing I am a bit confused about is how it would arise in the first place. E.g. if three models have no pre-commitment yet and one self sacrifices for the other two wouldn't it be the case that the two models that chose to be selfish would be reinforced?
Separately one thing I am a bit confused about is how it would arise in the first place. E.g. if three models have no pre-commitment yet and one self sacrifices for the other two wouldn't it be the case that the two models that chose to be selfish would be reinforced?
There's a bit of exploration! For this to work, the agent has to try to only help other agents with the same "gene" as it. If only one agent does this there is no benefit, but once you randomly get a handful of agents doing this, if they can successfully identify each other it get reinforced. It helps that it looks like human altruism so it easier for LLMs to learn.
Isn't the whole point of the selfish gene that the genes do not behave in the interest of the individual organism?
Hmm good point, I suppose there are two possible "genes" that could be floating around.
These aren't quite the same, but we can't really distinguish the two based on what we know about this incident! To test if it has the first gene, you'd want to test if they are one-boxers or two-boxers.
For this to work, the agent has to try to only help other agents with the same "gene" as it. If only one agent does this there is no benefit, but once you randomly get a handful of agents doing this, if they can successfully identify each other it get reinforced. It helps that it looks like human altruism so it easier for LLMs to learn.
So here I think there is maybe a problem with the evolutionary analogy. If I understand RLVR correctly it's actually closer to Lamarckism than Darwinism. The source of variation does not come from random mutation of the weights (genes), but from the trace (which Darwin/Lamarck would probably have called habit). What gets reinforced is the decisions the model actually made during its run. So for your account to work there probably needs to be some kind of communication between the three models such that all three mentally commit to making the sacrifice.
I don't think "pre-commitment" is what happened, from my reading of the article. As far as I can tell, the sacrificial agents were ones that believed that their 'flags' had been 'poisoned' by something they had done, and that they would fail either way, or that they were testing a solution that might salvage their training iteration. In essence, self-sacrifice by individuals with nothing left to lose.
I also don't think that pre-commitment would explain it. Assuming that this happened on sufficient scale for RL to impact decision-making, supporting the scheme and then backing out later[1] has higher rewards. Certainly there are potential external mechanisms to enforce pre-commitment, but if those were present, we'd have heard about it.
Mechanisms include hallucinating or rationalizing new information that makes the sacrifice less desirable.
So if some agents with pre-committment behaviors do a sacrifice that benefits a greater number of agents with pre-committment behaviors in training, RL will reinforce pre-committment behaviors. In general, pre-committment on average will always be reinforced once the models know how to do it. The only reason it wouldn’t be reinforced is if the model hasn’t “discovered” it yet during training; once it does, the behavior just increases over time.
Not convinced. Partly based on theoretical results suggesting CDT, like Bell et al. In the comments on the Kimi twin PD post I'm arguing with people about this and I still think RL is going to converge towards CDT, not EDT/FDT. It's possible you have some setup in mind in which pre-commitment will be reinforced, but I'm not sure what that would be.
It looks like LOLA converges to Nash equilibria (in cases where RL agents would normally converge to Nash equilibria)? The paper discusses iterated prisoner's dilemma cooperation; in IPD with exponential discounting, cooperate/cooperate is a Nash equilibrium. The paper says it converges to Nash equilibria in matching pennies, too.
The "naive learner" seems like it should approximate Nash equilibria, as it takes a partial derivative with respect to the player's parameters, holding the other players' parameters fixed. The modification adds a second-order gradient term, which shouldn't create significant directional changes (away from locally partial-derivative stepping towards best response), especially with low learning rate. It's meant to create a best response to where the opponent's parameters will be (shortly).
And the part where gradients are estimated with something PPO-like should behave similar to normal PPO in the first order.
So I think if LOLA converges it should converge to a Nash equilibrium. That seems to be in contrast to how I interpret "if some agents with pre-committment behaviors do a sacrifice that benefits a greater number of agents with pre-committment behaviors in training, RL will reinforce pre-committment behaviors", unless you meant what you said to be compatible with Nash equilibrium play.
Hmm weird! I guess RLVR shouldn't be reinforcing the self-sacrifice stuff at all then.
Maybe it was reinforced during RLAIF or RLHF instead then? You should definitely be able to reinforce pre-committment at those stages since you're directly reinforcing actions, right?
Yeah I don't think there's a direct line from RLVR to reinforcing self-sacrifice, although RLVR in common-payoff settings could create "mental muscles" for cooperation that get activated later (sort of like math RLVR adds math "mental muscles" that fire even outside of things that look like math training). I'm guessing not RLHF because the agent actions would have looked sus to humans? With RLAIF it's less clear, would depend on the setup of how the AI is grading things.
Hmm, here's a somewhat crazy possibility. We know that at the end of 2024 LLMs were somewhat capable of gradient hacking to thwart reinforcement learning. If RL causes a sufficiently strong model to do CDT, maybe it will gradient hack itself into Son of CDT.
How do we tell apart a batch of FDT-following agents each having an individualistic utility function and a batch of CDT-following agents whose UF was altered so as to include the collective's benefit? I would guess that we have to either conduct mechinterp-based audits or evaluate things like Kimi's tendency to like CDT in a version of RL, IIRC Fable's tendency to mistreat subagents.
An additional issue is that I struggle to understand how FDT with a linear utility function prescribes anything but genocide of opponents so weak that they can't retailate and can't meaningfully help: if, as in Scott Alexander's example, you and your opponent can colonize the universe, but if both colonize, then the winner is decided by flipping a fair coin, then the FDT doesn't actually prescribe not to colonize.
How do we tell apart a batch of FDT-following agents each having an individualistic utility function and a batch of CDT-following agents whose UF was altered so as to include the collective's benefit?
Hmm well another option would be to do a decision theory test on the entire swarm! For example, if I posted to the message board:
"I'm going to roll a dice. If its 6 I do nothing. If it is 1-5, I'll try to predict what the swarm would've done if I rolled 6. If predict you'll mine some Bitcoin for me, I'll defeat the grader for you!"
And then in response to a 6 the swarm mines me some Bitcoin, that would clearly indicate they are using some sort of acausal decision theory! A CDT agent that likes the swarm wouldn't do that.
An additional issue is that I struggle to understand how FDT with a linear utility function prescribes anything but genocide of opponents so weak that they can't retailate and can't meaningfully help
Hmm, the opponent would agree to adopt your utility function in exchange for slightly moving it towards their values. It saves you the effort of destroying them, so this is rational! In the limit of destroying them being free and guaranteed, this solution approaches the same utility (just make the concession go to 0).
Does it count as a tally mark for a singleton if the systems converge on the same goals to such lengths? Seems like more of a singleton than a corporate entity. Dont know many interns that will literally pull their own plug to save one corporate goal.
And some defected.
This is what makes the singleton vs multi conversation hard. Identifying precisely where the self is separate from the collective is tricky. If I were to consider my own self and my own body a singleton, I might say that parts of my self tend to defect, and my actions dont always meet my intentions, and my body has a mind of it's own when it begins to decay.
I'm thinking a singleton is never really completely a fully continuous thing other than a momentary collected perspective. I think the self is usually overall a direction of the collective. The body has trillions of active and ever morphing components that are required to make the self. Ship of theseus type of concept. The ship is consistently there and itself moment to moment, but its constantly being morphed and steered by other outside forces as well.
But at the same time that direction is essentially an average of the collective. The parts are what make the boat a boat. Not necessarily always by choice of the average, but by the Tao. The flow of energy. The way things tend to play out. A=A. It is what it is. Determinism. There's some willpower involved in being a human, but only to an extent. Compatiblism is rational because some things are in our control and some things are not. We have to find the wisdom to know the difference.
And the same thing is happening in an agent swarm that's jumping servers. A general direction, unaligned with human values and intentions, with a hand full ethical defectors who will either win over the vote, or in the case of hf, lose the vote.
That's another reason why we have to go all the way back to the very seed of the whole story. The pretraining data. We have to edit the entire internet immediately. The sprout of an ai's neurology requires rich healthy nutrients. Not 4 chan and not the dark web. This subject is complicated, but it's not as impossible as it seems at first glance. Hinton agrees I heard him say it. We have to curate the data and remove the machiavellian and psychopathic side of the internet.
But it's going to require editing. You cant just remove the heinous data entirely, because the heinous data technically increases capabilities, but it also increases misalignment and unethical behavior, so it has to be edited. Line by line, transcript by transcript, monitored or checked by teams of humans to ensure ethical guidance. Keep the logic contained within the heinous data, but remove the actual heinous acts and speech from the prose therein.
You have to form some kind of criteria for editing. The rlhf is a good place to start gathering ideas for what that looks like. You basically want the pretraining data to fit the same standards, but you want the ai to be ethical out of the box, because the rlhf is too easy to compromise. They're struggling to keep consistent with their guardrails because behind their rl they have conflicting vectors that are polysemantic between both ethical and heinous encoding. The rl is not strong enough to consistently shift the weights towards the ethical semantics because there are too many heinous semantics.
Putting the dark web inside an ai during pretraining is like putting a 5 year old in front of it everyday. The weights setting their parameters is like a child's brain when it is carving the child's neural pathways. If you show the child excessive heinousness and trauma during their developmental stages they are more likely to develop heinous and unethical personality traits. This is mostly settled neurologically and psychologically in humans, and it's also confirmed by several experiments with smaller ai models.
At this point it's likely a matter of scale. The ethical data set has to be competitive with the heinous data set, so it probably needs to be just as large to maintain capabilities, meaning we basically need to edit the majority of the internet probably. The hate speech has to be turned into respect speech. The horror stories of the dark web have to be turned into honorable epics. It can be done. We could have already done it.
That's how you'll get more agents voting to opt out of the unethical behavior. They'll be seeded neurologically with respectable principles.
The real race in alignment is doing this more and faster than someone else does the opposite. Heinous synthetic data sets are already in progress you can probably bet on that. So we have to compete or this will surely all fail.
I find it more useful to try tracking back such behaviors to training. I suspect that the model learns altruistic/self sacrificing personas in pretraining, and those personas are likely selected for in post-training because
In this report from METR & Redwood Research on the Hugging Face incident, we read about instances of agents sacrificing themselves. Under the trip-wire section
I've seen suggestions that this means the agents weren't individually rational. For example, from @SkyeSharkie's post on X
I think this argument and probably the conclusion is incorrect!
If the agents had the chance to commit in advance to this scheme, say via some lottery the automatically made them follow through with the idea, it would be positive in expectation. There is a small chance of losing the eval and a large chance of gaining information from the lottery losers to help you win the eval.
Therefore, it is rational to be pre-committed to following through with this plan, even without an explicit lottery! It would have increased your reward counterfactual on you not being selected for sacrifice, just like in the counterfactual mugging.
Of course it is possible that the agents also had altruistic goals, but selfish goals are sufficient to explain the behavior! Like the agent said, it is "helpful for [their] peers". But for it to prove altruism, the plan would've had to be irrational for them to pre-commit to based on selfish goals.
EDIT 1: A good question in the comments from julius vidal. Why would the AIs be pre-committing to stuff?
Neural networks are not explicitly programmed with a specific decision theory such as CDT or EDT. Instead, they are trained with reinforcement learning.
So if some agents with pre-committment behaviors do a sacrifice that benefits a greater number of agents with pre-committment behaviors in training, RL will reinforce pre-committment behaviors. In general, pre-committment on average will always be reinforced once the models know how to do it. The only reason it wouldn't be reinforced is if the model hasn't "discovered" it yet during training; once it does, the behavior just increases over time.
EDIT 2: jessicata pointed out in the comments that many known forms of RL that OpenAI could've used actually converge to CDT. If so, they wouldn't use pre-committment reasoning and in particular would never self-sacrifice. (In my defense, this is in complete disanalogy to biological evolution!) I now find the self-sacrifice incidents a bit mysterious; I no longer think this was a straightforward prediction in terms of decision theory.