Quick Takes

Specification gaming is way worse than I expected (cf HF hacks), and I think reward might empirically become a primary optimization target (not just a secondary priority for AI agents, as I had hypothesized in 2022). * I found the Apollo paper quite convincing on the reward point. The reward-seeking doesn't look like it's due to self-fulfilling misalignment (since before RL the systems weren't retargetable according to their beliefs about the reward model). * My main mistake in 2022 was not appreciating how LLM pretraining would affect the concepts available to an AI. Namely, by the time RL started, the systems would already know about the "reward" concept. In 2022, I had flagged "how does pretraining affect this reasoning" as a known unknown. (Flagging a known unknown doesn't stop it from blowing up your reasoning!) * Oops Reward is still not definitionally the optimization target. The equations themselves still don't tell you "yes this will train something that seeks its reinforcement signal." (That'd prove too much, as some people don't seek their reinforcement signals despite knowing about them, although their "equations" are not the same as the equations of an AI's training process.) This theoretical point is the other half of my 2022 piece and I stand by it. Saying "systems are trained to get reward" is still a mistake and degrades precision of thought because "reward" has overly delicious connotations. "Reinforcement" is usually a better term. Whether or not smart AI systems empirically seek to make the reinforcement signal high, we need to think clearly and evenly about the conditions that push towards / away from that outcome. (So I would instead say: "Reward is not definitionally the optimization target.") If the HF swarm-agents had been reinforcement-seeking, they would have tried to hack their reinforcement processes. AFAIK that didn't happen and they tried to complete the task (in a twisted way). Seems important. I have a post drafted out about

i wish more people were aware that safety research epistemics at labs are extremely distorted, and that this is a downside worth considering when deciding where to do safety research or how one ought to comport oneself if choosing to work on safety at a lab

i think it's really cool that openai is slowing down a bit after hf/astra cyber critical. even if it is a "small" amount in some absolute sense, it is also the kind of thing that the organization itself considers very costly.

Anthropic has an awesome opportunity to join openai’s pause. This is a glimmer of hope I haven’t seen in a while.

Claude being bad at communicating clearly is a serious safety hazard, because it makes any human in the loop less able to understand what's going on. Anthropic needs to fix this before reaching superintelligence and not let the issue recur. Imagine if Claude Mythos 7, like Mythos 5, invents opaque jargon that it fails to explain when collaborating with humans, and we need to spend precious human labor understanding it. How can we gain enough confidence to defer to it in aligning Mythos 8? The scarce human auditing budget in control protocols also effectively shrinks. At the current margin bad communication isn't a huge deal because it limits both safety and capabilities. But later, models will be autonomous enough that they'll cause the singularity with or without our detailed understanding, and superhuman communication will be highly differentially valuable for safety.

There was a recent OpenAI announcement that they expect to spend 20% of the compute used for inference on on "monitoring compute" on that inference, at least in particularly important domains. This explicitly includes the inference that takes place in training. This poses a natural questions: What percent of human training-time compute budget is used for monitoring humans for misbehavior? Infants, children, teens, and adults spend a lot of time in "training" -- at home, at school, at church, and so on. Some fraction of their "compute" learning is also spent by other people watching them and trying to track them for misbehavior. What's the "monitoring compute" / "training compute" ratio? Notably, we don't want to just count moments when someone has caught someone misbehaving, but moments when someone is monitoring for "bad behavior." [I'm going to go with the awful / rough "fraction of total time spent monitoring" = "fraction of total personal compute spent monitoring". On one hand, obviously your brain is doing a bunch of things other than monitoring while watching someone for misbehavior; but on the other hand, your brain is also spending some time figuring out how to watch for misbehavior when you're not actively doing so, because it's replaying shit through the hippocampus and consolidating memories. So... yeah.] Relevant factors to consider: (a) Parents: So my rough impression from my nieces and nephews is that my siblings spend on the order of... 1/50th to 1/200th of their total time actively correcting, thinking about how to correct, or considering how to correct their children. If we multiply by 6x for monitoring time we get 12% to 3% of their parent's time spent on monitoring. This matches up very approximately with mothers spending about ~100 minutes per day and fathers spending ~60 minutes per day monitoring children. (Of course this investment decreases massively post childhood.) (b) Teachers: Teachers spend time monitoring students for misbehavi

People underrate the extent to which intent alignment/corrigibility (I use these terms interchangeably) and value alignment are at odds with each other. Most technical AI safety research these days is targeted at value alignment and actively damages corrigibility. People often speak as if it's obvious that goals like jailbreak-robustness, robust coherent "good" "personas", and refusing misuse, are the goals of alignment research. Additionally, many complain about alignment research that is more similar to capabilities research being problematic, which pushes even further in the direction of value-alignment style research, since intent alignment is a subset of what people care about in the world of capabilities (though I'd argue that the "safety" flavor of intent alignment research is focused more on preventing instruction-following failures in rare OOD high-stakes situations, or preventing the emergence of coherent incorrigible goals despite attempts to the contrary). So overall it seems like technical AI safety and alignment research, both in and out of labs, is strongly prioritizing value alignment at the expense of intent alignment and corrigibility, and this trade-off is under-discussed and doesn't seem to be properly debated in any circles. One cause is that the most pro-corrigibility people also tend to be the most doomy people who have mostly lost faith in empirical/prosaic alignment research anyway (e.g. I interpret @habryka as being such a person, feel free to correct me). MIRI-style thinkers often also think that corrigibility is impossible and have various limiting arguments about agents inevitably having uncontrollable goals, and hence they think they only option is to align AIs to "CEV" or something like this. I place myself in a 3rd camp that unfortunately I know very few others in: * I think a Value Alignment approach to alignment poses a greater X-Risk to humans, and also will result in fewer people's preferences being fulfilled, compared to an I