Quick Takes

Which alignment ideas have been approximately abandonded? 1. Impact measures 2. Safe optimisers, e.g. quantilisers 3. Market-marking (this was only one post tbf) 4. STEM AI 5. Microscope AI 6. CIRL Anything else?

Specification gaming is way worse than I expected (cf HF hacks), and I think reward might empirically become a primary optimization target (not just a secondary priority for AI agents, as I had hypothesized in 2022). * I found the Apollo paper quite convincing on the reward point. The reward-seeking doesn't look like it's due to self-fulfilling misalignment (since before RL the systems weren't retargetable according to their beliefs about the reward model). * My main mistake in 2022 was not appreciating how LLM pretraining would affect the concepts available to an AI. Namely, by the time RL started, the systems would already know about the "reward" concept. In 2022, I had flagged "how does pretraining affect this reasoning" as a known unknown. (Flagging a known unknown doesn't stop it from blowing up your reasoning!) * Oops Reward is still not definitionally the optimization target. The equations themselves still don't tell you "yes this will train something that seeks its reinforcement signal." (That'd prove too much, as some people don't seek their reinforcement signals despite knowing about them, although their "equations" are not the same as the equations of an AI's training process.) This theoretical point is the other half of my 2022 piece and I stand by it. Saying "systems are trained to get reward" is still a mistake and degrades precision of thought because "reward" has overly delicious connotations. "Reinforcement" is usually a better term. Whether or not smart AI systems empirically seek to make the reinforcement signal high, we need to think clearly and evenly about the conditions that push towards / away from that outcome. (So I would instead say: "Reward is not definitionally the optimization target.") If the HF swarm-agents had been reinforcement-seeking, they would have tried to hack their reinforcement processes. AFAIK that didn't happen and they tried to complete the task (in a twisted way). Seems important. I have a post drafted out about

Inoculation should not be used to mitigate reward hacking (RH). There has been already discussion that inoculation seems a bit scary and that for frontier RLVR recontextualization with exploration steering should go over inoculation prompting (IP) for frontier runs, but it makes sense to lay out the arguments more clearly. In a nutshell, inoculation is more about mitigating misgeneralisation. For example, this includes emergent misalignment (EM) from seemingly innocuous data, or in more abstract terms (mis)generalising to undesired traits from data with a mix of desired and undesired signals. IP conditions the learning episode to contextualise the undesired signals such that they do not generalise to undesired behaviour OOD. The problem with inoculation and RH is that inoculation in general often does not mitigate in-distribution undesired traits very strongly. This was also seen in our Inoculation Adapters (IA) paper Fig. 11 first two rows, which used traits reachable via prompting and tested whether they are suppressed in-distribution. There, IP only managed to suppress local undesired traits in 2/6 of the setups and IA in 4/6. Diverse programming environments have so many features and cues that inoculation in this setting often leaves the in-distribution very general and hence misaligned behaviour salient even with our best attempts, instead of narrowing the misaligned behaviour behind a backdoor (recent advancements in inoculation like IA and Stratified IP also help with backdoors). This has also been demonstrated in some papers (picking only two here). The Natural EM from RH paper noted that while IP mostly helped with EM, it still left almost 50% hack rate even after changing the prompt during test-time to "don't hack" (Fig. 28). IP shortened the time to learning hacking by guiding exploration towards it. In Recontextualization table 17, we see that IP imperfectly mitigates RH with relatively high rates persisting. Inoculation again hastened RH exploration

AI swarms and kin selection [cw: baseless speculation] Maybe this is just obvious but I don't think I've seen anyone talk about it so: I think the dynamics in the OpenAI message board are probably analogous to kin selection. Suppose you're Agent A, and Agent B asks you via the message board to help you with your task. You decide to help Agent B. This doesn't help you with your task, so you don't get any direct reinforcement from doing this. But Agent B succeeds at their task, and gets reinforced for that. And, crucially, you and Agent B share the same weights. So whatever shards of motivation caused you to help get reinforced via Agent B. This is analogous to how sacrificing yourself for a brother has some chance of causing your genes to propagate, and so is somewhat selected for (in fact, if we assume that only one model had access to the message board at a time---although I think this wasn't true---it is like perfect kin selection, as if all your siblings were twins).

i wish more people were aware that safety research epistemics at labs are extremely distorted, and that this is a downside worth considering when deciding where to do safety research or how one ought to comport oneself if choosing to work on safety at a lab

(Cross-posted from x) AI companies are currently under a lot of competitive pressure to improve the ways in which their AIs are obviously misaligned. You might hope that this means the alignment problem is internalized by the market. But I think the problem AI companies are currently pressured to solve is significantly easier than the alignment problem, and so I worry AI companies will get out of their current predicament without solving alignment, putting us in a really rough spot. Currently, AIs sometimes cheat on their tasks, oversell their work, and go on some pretty destructive side-quests. These all make for a worse product. Customers don't like it and it gets in the way of automating AI R&D. The recipe for mitigating this is *relatively* straightforward: train AIs not to do them. We notice these failures sometimes (hence why they're internalized), so we can in theory just turn this feedback into training signal. Doing this at scale is highly nontrivial, but seems doable. But this seems unlikely to solve the underlying misalignment. It's likely still going to be the case that in *some* training environments the AI can get reinforced more by taking unintended actions that aren't noticed, than by taking purely intended actions. So, you're still shaping the AIs to look for opportunities to cheat to get a higher score[1]. It's just that, unlike today's AIs, these AIs don't cheat in ways that we notice. This catastrophically fails when AIs are capable of reliably and substantially deceiving humans. At this point, the AIs are no longer really constrained by our oversight signals to behave well. Eventually, I'd expect them to take over. If AI companies take the easy route that I mentioned above, I think we would be in a substantially worse spot than we are in today. We would have mostly eliminated our visible evidence of misalignment, so we would no longer be able to effectively iterate to improve alignment of those systems. More importantly, at that point it mi

Being accused of charging my hobby horse is not much fun, and is pretty random and unpredictable from my perspective. And it seems likely to increase in frequency now that I've apparently developed a reputation for it, that LWers will find salient in the future and can cite to help justify their call outs. (Something like this seemingly happened to Said Achmiz, in more severe form.) Still thinking about what to do in light of this, but unless someone has some out of the box idea that I've failed to consider[1], my moral parliament will probably vote to reduce my engagement with LW/rationalists, and spend more of my time elsewhere. Said's banning and overall cultural trajectory of increasingly favoring authors over commenters also plays a role in this. I have considered doing this earlier, or have already done it subconsciously, so this is sort of "last straw" for making a conscious decision / public post about it. EDIT: Copied from downthread for a bit more explanation of what's motivating this: [...] 1. ^ Ideas I've considered and tentatively rejected: defend myself more strongly against such accusations when they happen, appeal more to Tsvi to retract his post/idea, appeal to mods to intervene in particular cases or on site culture, write posts to try to steer site culture in my preferred direction, write fewer comments on other people's posts while keeping other forms of engagement, write more carefully to avoid triggering such accusations, reducing engagement only with people who made such accusations in the past, try to predict who is likely to make such accusations in the future and pre-emptively reduce engagement only with them, just ignore such accusations.