The goal is to tell the full story, not to declare which houses have ghosts.
Just wanted to say I think this is really important work and I'm excited that you're doing it. The existence of this group might even incentivize authors to reveal more of the full story in the first place
1It feels like the argument of this initiative is: (A) there exist some important safety papers that don't tell the full story, (B) replicating those papers would tell (something closer to) the full story, (C) that type of replication is currently under-incentivized right now, and (D) publishing the full story of those important papers would meaningfully improve safety research. (Tell me if I'm getting it wrong though!)
I buy (A) and (B), but I'm not sure about (C) or (D). I think I'd be more convinced if you had some example(s) from the last 5 years where a specific quasi-important safety paper went unquestioned (or under-questioned) for longer than it could have, such that an early replication would have saved meaningful research effort on net. If enough people premised their later work on misunderstood results from this paper, that might be warrant replication even for a relatively small misunderstanding. Are there any such examples? You'd probably know better than I, so I'd welcome a correction.
I generally am going to avoid critiquing papers without a full replication but one example from a long time ago is Redwood Research's "High-stakes alignment via adversarial training". One thing that makes this a non-controversial example is the authors themselves admit that there were aspects of the original writeup which were misleading (see full retrospective here). You can also check out my thoughts on this paper or this post on EM.
I will actually push back on the framing here though. "Telling the full story" is messy and subtle. So providing examples of papers where this would have been useful is difficult to fit in a comment (actually is hard to fit into a blog post or paper). Additionally, showing that a paper does replicate and in fact it is more robust than we expect is 1. what we hope to see and 2. also provides useful information. Researchers don't know what papers are real and being able to see that someone was able to replicate something allows them to make better strategic decisions about their research. Senior level AI safety researchers have told me that work not replicating is a concern: e.g., you can see Buck's comment here but most of these comments were made in private (but from credible people - e.g., Anthropic employees). So by default, people can't necessarily assume things will replicate and you are giving useful information either way.
Interesting project! The slow march of AI safety creating an alternate academic universe continues.
I wonder how you're thinking about whether some of your replication is already done within the leading companies. The five criteria you list for judging whether to replicate a paper seem solid, but they also seem to leave out whether the labs might have replicated it already or have a strong incentive to do so.
This is true for a decent amount of empirical safety research (e.g., deliberative alignment, monitoring, etc.), where, if the techniques are effective in practice, AGI companies will adopt them. Presumably that adoption includes internal benchmarking, i.e. replication. I think this likely holds for important papers in mechinterp and other less immediately useful areas as well, as long as multiple labs have a team focused on that area.
And it's especially true for papers that have implications for both alignment and capabilities, e.g. papers answering the question "how much does RL induce new capabilities vs. amplifying low probability ones that already existed in pretraining?" Though with that last category it might still be useful to replicate some of them, given that AGI companies might not correct misunderstandings as a competitive strategy.
Is this something you're considering, and if so, how do you plan to adjust your criteria to account for it? Your claim that "the incentives aren't there" for replication is true in some but not all cases, so it's worth trying to figure out where the real "market failures" are before you try to correct them.
Its hard to know what is happening inside of the labs. I doubt what you are describing is occurring but thats just my personal opinion and at the end of the day, who knows! If it is happening though, then the work is certainly not being made public. So at the very least, the non-lab AI safety community doesn't get to benefit. I would argue that having people outside of labs having an accurate understanding of AI safety research is important as well.
Additionally, we should not wait for ideas to be implemented at labs! Ideally, if there was a key limitation of a work we would catch it way before that. Counting on a lab to notice that some safety technique doesn't work seems like a risky strategy. Even if they are good at catching issues, making them do this wastes valuable conscientious, safety-pilled researcher time within labs when they could have been implementing something more likely to work.
Related question: wouldn't some findings garner replication-style efforts by default once they become important enough? My sense is that once some finding becomes load-bearing enough (e.g. the METR graph), it inevitably receives critical scrutiny (e.g. critiques of the METR graph). What's the story for why this doesn't happen? Or perhaps it only happens once the paper is past some threshold of notoriety, meaning there's a ton of important but un-replicated papers just below that threshold?
meaning there's a ton of important but un-replicated papers just below that threshold?
Yes! I also think that waiting for something to be maximally load bearing before taking a closer look is bad practice. We want to build up organizational knowledge so we are able to catch things before lots of other research is built upon it.
If we get AI safety research wrong, we may not get a second chance. But despite the stakes being so high, there has been no effort to systematically review and verify empirical AI safety papers. I would like to change that.
Today I sent in funding applications to found a team of researchers dedicated to replicating AI safety work. But what exactly should we aim to accomplish? What should AI safety replications even look like? After 1-2 months of consideration and 50+ hours of conversation, this document outlines principles that will guide our future team.
I. Meta-science doesn’t vindicate anyone
Researchers appear to agree that some share of AI safety work is low-quality, false, or misleading. However, everyone seems to disagree on which share of papers are the problematic ones.
When I expressed interest in starting a group that does AI safety replications, I suspect some assumed I would be “exposing” the papers that they don’t approve of. This is a trap and it is especially important for us, as the replicators, not to fall into it. If our replications tend to confirm our beliefs, that probably says more about our priors than the papers we are studying.
II. Searching for “bad” papers is like searching for “haunted” houses
Consider a team of researchers trying to find examples of haunted houses. They could investigate suspicious buildings or take tips from people who have witnessed paranormal activity. They could then publish reports of which houses you should definitely avoid. But the issue is that ghosts aren’t real. What they would be finding is a convincing story, not the underlying truth.
Trying to find “bad” papers will be like finding haunted houses. If given a mandate to find papers that don’t replicate, we will find them. But the uncomfortable truth is that genuinely influential papers that are straightforwardly, objectively wrong are rare. The empirical claims are likely true in some sense, but don't tell the full story. The goal is to tell the full story, not to declare which houses have ghosts.
III. Research doesn’t regulate itself
Even when researchers are especially disciplined, they are incentivized to frame their papers around their successes while burying their limitations. Likewise, when designing evaluations, researchers are incentivized to measure the properties they are proud of rather than those they wish would go away.
I’ve heard the arguments that we don’t need peer review. Authors can accept feedback and update arXiv. Or ideas can duel in the LessWrong comment section. But I don’t think either of these are enough.[1] They both assume that:
#1 is unrealistic. #2 is also often unrealistic and arguably unreasonably burdensome to authors of the work. For example, should an author with 50 papers have to litigate every critique and correct every flaw across dozens of papers and several years of research?
IV. Replications are more than repeating the experiments
For any paper that releases code, “replicating” figures or statistics should be trivial (we would hope). But just because statistics replicate, that doesn’t mean the effect is real. We want to look closely at the paper and ask:
Our philosophy is to start from scratch, carefully implement the paper exactly as it's written, and see if we get the same results. After that, we will poke around a little and see if anything looks weird.
V. The replication is just as dubious as the paper itself
If we can’t replicate something, could that mean we are just doing something wrong? Yes, of course! We obviously will try to avoid this case, and contact the authors to get feedback if things aren’t working. If we can isolate why things aren’t working, this can be a finding within itself (X only happens with a really big batch size on small models). If we try hard and cannot figure out why things aren’t working, it eventually makes sense to write something up saying:
VI. Some centralization is necessary
Our plan is to hire people for in-person fellowships, and eventually, full-time roles. One of the most common comments I get on this is some version of “Why don’t you outsource replications to the community?” or “Why not offer bounties for replications instead of doing them yourself?"
The answer is the incentives aren’t there. After we run a pilot this summer, we would like to complete more ambitious replications (e.g., replicating this or this). Offering bounties at this scale is logistically difficult because even for a minimal replication, compute alone could be thousands of dollars.
Selecting which papers to replicate is perhaps a place where a decentralized approach is more principled. We have a framework for prioritizing papers,[3] but we're also exploring ways for the community to vote on which papers we replicate to reduce selection bias.
VII. We are all adults here
I would expect most replications to take the form of “everything works and we found 0-2 extremely minor issues.” But doing this kind of work inevitably involves sometimes challenging claims made in papers. This is difficult, but replications should state concerns directly. Giving any critique of another's work publicly is stressful, but reasonable people won’t hold it against you when it’s in good faith.
We will take the feedback of authors seriously, but we may not always converge on agreement. In these cases, we will attach an author’s comment to our research.
VIII. Feedback is everything
A group that replicates AI safety papers really exists for a single reason: to be useful to the community. That means we value your feedback and we hang onto every word. Please let us know what you think.
If you want more details about what we are planning, I'm happy to send over our proposal. If you are interested in our summer research fellowship, you can express interest here.
And for what it's worth, I don't even think peer review is enough.
I really like Maksym Andriushchenko's twitter thread on this.
The tl;dr is there are five criteria we plan to use: