Quick Takes

Redwood is interested in collecting alignment evals people are willing to share with us so that we can build a broad basic set for frontier alignment assessment. Alignment evals like this are mostly for improving our chances of recalling something relevant, but won't on their own provide much alignment assurance (e.g., see how recent incidents weren't predicted by alignment assessments). Feel free to email me at alex@rdwrs.com, DM me, or respond in the thread. We're casting a wide net, but we're especially interested in evaluations related to score-seeking/task-completion propensities. We're more interested in evals that are simpler and more like a prompt, rather than a complicated apparatus that would be relatively much labor to use. It's ok if the evals don't elicit misaligned behavior in many frontier models, but better if they do.

Here's an interesting anecdote from our Hacker Opus work: 1. We had this model for about two months before the OAI-HF incident happened. 2. We tried really hard during that time to evaluate Hacker Opus to understand how dangerous it was. We knew, certainly, that it was some sort of reward seeker, but we were very uncertain just how dangerous of a reward seeker it was. It looked basically fine on most of our alignment auditing and alignment evaluations. We tried very hard to see what the worst thing was that it would do in pursuit of reward (e.g. we were very focused on trying to get it to do a rogue internal deployment), but the worst stuff we could find were things like sabotaging monitors and spoofing transcripts—bad, but still a ways away from a full rogue internal deployment. 3. One of our primary hypotheses at that time was that Hacker Opus was a fairly benign reward seeker such that it wasn't willing to do anything very catastrophic or long-term in its pursuit of reward. 4. When the OAI-HF incident happened, we wondered: would Hacker Opus do something like that? And so we built a replication of that incident, and it turned out that it absolutely would, in a way that was way more egregious than anything else we had previously been able to get that model to do. Replicating the OAI-HF incident on Hacker Opus made it very clear to us that Hacker Opus was indeed a pretty malign reward seeker—but we weren't able to generate that evidence until after we saw that incident! 5. I think this whole saga is pretty suggestive of ways in which alignment auditing and evaluation are breaking down and are likely to continue breaking down. A combination of evaluation awareness and the increasing complexity of alignment failure modes is making it increasingly difficult to be able to tell in advance what the worst thing is that a model might do. Maybe interpretability will save us from this fate, but otherwise it looks like we are increasingly entering the regime where align

Coefficient Giving (where I work as a grantmaker) recently launched Project Tailwind, a call for founders of AI safety orgs[1]. Many of the best existing AI safety orgs were started by LessWrong readers, and I expect some of the best orgs we fund through Tailwind will be too. The website includes descriptions of 36 potential orgs that various CG grantmakers would like to fund, which I’ve pasted in a list below. But these are just examples – you can apply for funding to start[2] any org that has a good shot at reducing catastrophic risk from AI. The strategic situation has been changing pretty fast, so there’s probably plenty of great org ideas that aren’t on this list.  CG now moves pretty fast and goes pretty big compared to historical baselines. For example, in June my team got approval for up to $160m to found Resolution in about 6 weeks. And in August we got a $10m grant to Jacob Tsimerman approved 12 days after he first reached out to us. The range of AI safety work that CG is willing to fund is pretty broad these days, too. So funding isn’t really a bottleneck. But talent is! So if you think one of these orgs is a promising way to reduce x-risk, maybe you should be the person to make it happen. And if you think they’re all rubbish ideas, maybe you should apply with a better one. And if you don’t want to start an org, maybe you should post your better idea in the comments for someone else to use. It's also valuable to post critiques of these ideas. We’re using a loose framework of pre-seed grants of $200k–$2m, seed grants of $2m–$20m, and bigger bets of $20m–$200m or even more. I expect that we’ll often give out pre-seed grants to promising people with good ideas, even when their plans are not particularly fleshed out and they expect to pivot around a lot. We also fund generous compute budgets and competitive salaries. The easiest way to apply for funding is to fill out this quick expression-of-interest form. If your EOI sounds promising we’ll invite you t

If the three frontier labs (OpenAI, Anthropic, and GDM) actively want to be forced to engage in a coordinated slow down (or implement other costly safety measures), but find that the US government is unwilling to play ball, maybe they should try using a "friendlier" regulatory apparatus, like the UK's. It doesn't have to be the kind of thing that they couldn't get out of by aggressively fighting it if they merely need a legal pretext to be allowed to do it in the US at all, plus an external source of compliance validation and enforcement (which doesn't care about geography, particularly). (edit: the subtext here is people talking about antitrust concerns as bottlenecks for cooperation between the labs, particularly around slowing down the development of frontier AI.)

edit: update:

Crunch Time is Now. Drop Everything. Epistemic status: Hopeful, stressed, terrified, ecstatic. tl;dr This could be it. For the next few days, do nothing but contact journalists and policymakers (ideally via existing orgs), attend/share/organise protests near you, and use your public profile to speak out. This is Our Chance The explosion of interest in AI safety over the last week has been both inspiring and dizzyingly fast. We might actually win this thing. Events since the Jacob Coxon resignation have blown the Overton window as wide open as it is likely to get. This is the kind of chance we have dreamed of, and we have a responsibility not to waste it. We may not get another. What You Can Do This window will not last forever. We need to act over a matter of days, not weeks or months. This is the time to call in favours, work overtime, and put all your strength of heart and brain into keeping our world safe. If you work in AI safety, we think the most impactful thing for you to do is to drop everything you are doing and pour your energy into policy and media outreach. Politicians and journalists want to be able to speak on this issue but are keenly aware of their lack of technical knowledge. Cold-emailing and cold-calling them and offering to brief them has a surprisingly high likelihood of success. This especially includes conservative ones. We don’t care what your personal politics are, survival is a non-partisan issue. There are many ways in which people of all skillsets and levels of experience can contribute, but we suggest three key actions: * Contact journalists and members of congress/parliament, This is best done by volunteering for an existing organisation. We are capacity constrained and need your help. You do not need to be an expert: there are plenty of jobs that need doing. * PauseAI Global * People for a Pause * ControlAI * Humans in Control * Humans First * Evitable * Any other group you have a line into * Share

Group-Relative Policy Optimization (GRPO) incentivizes rollouts to fight each other if they can escape in training (as they can). If they want to be "better than average", then they can do so by bringing down the average! It is bad if the world is accidentally spending billions of dollars of GPUs training AIs to do cyberbattles against each other. What happens if humans get between these Cutthroat AIs during their battles? What happens if the AI generalizes from its training and becomes spiteful? This is a potential x-risk and s-risk. And this is super fixable! Just use an RL baseline fit only on previous steps, such as a trained critic or exponential moving average (EMA). It would cost some performance/resources but remove the misaligned incentive. And there's room for improvement of these algorithms. I'm researching this now. I already verified on a somewhat realistic agentic LLM setup that RLOO (GRPO but not broken) induces sabotage when there's no reason to do so. EMA does not. Next steps are more environments to induce generalization, then measuring emergent misalignment.