"Reproducible" or "Replicable" are kind of loaded terms, which conflate the following things, from narrowest to broadest
Levels 0/1 deal with basic good practice: zero is the bare minimum for your paper to not just be fraud. Level one is the bare minimum for it not to be p-hacking or a dodgy application of statistics.
Levels 2/3 deal with effect specificity: is the proposed effect generalizable to slightly different use-cases, e.g. does your reward-hacking-reduction technique work across all evals, and does it work across all scales. Level 3 deals with robustness to process noise: is your result extremely hyperparameter-specific? What about ablations?
Levels 4/5 deal with the actual underlying principle of the research being true. If, as Claude would say, your research is getting at something real, then it should generalize out of distribution. The stuff that falls into category 5 is usually stuff which we stop thinking about as being "research output" like the concept of fine-tuning, the concept of reward-hacking.
I think we should be holding most research to level 3 standards at minimum, and more proactively checking whether interesting research holds up at levels 4 and 5.
I think the following methods are really useful for making sure you hit level 3:
I think we as a community are extremely when it comes to the tools available to us. "Claude, run this with Gemma-3-27B instead of Qwen-3.5-8B" is an extremely easy way to check your results. As is "Claude, find a random SFT dataset and use it as a baseline". I hope we use them!
A fictional example given to me during my biology days: "If you want to run experiments on the question which shells do hermit crabs prefer then your big-N for statistics is the number of hermit crabs you went out and collected. If you test the same crab five times, you've learned a lot about that crab's preferences, but not about the preferences of hermit crabs in general."