The first is the counterfactual where participants aren't selected for ARENA, do they then go on to do good things
This is not crux for me. I believe ARENA provides counter-factual value compared to not doing ARENA. You work much harder during ARENA than you otherwise would, in great environment, great support, etc.
> The second is the counterfactual where people spend 4 weeks doing research sprints.
This is crux. And agreed it is hard to measure!
Thanks for engaging thoughtfully. Useful to think things through.
1
1Thanks, James, for the detailed thoughts and for reading through the post. I'll respond once here. If we want further back and forth, better to have a chat in private so we can iron out our cruxes (and then summarize for community benefit). I'd also want to hear what others in community think before committing to anything.
> Because ARENA's main bottleneck to scaling hasn't really been TAs
I am happy to defer to you regarding the scaling bottlenecks of Arena. That's not a big crux for the proposal.
> I'm confused about the evidence given that ARENA functions primarily as a signaling mechanism
Maybe the word signaling isn't correct. Let me try to explain. When I point out that there are four people who did ARBOx and are now doing elite fellowship programs, my hunch is that those four had a very good chance of getting into those elite programs, even if they hadn't done ARBOx. Furthermore, if ARBOx did provide a significant boost to their profile/skillset, then one needs to consider how much extra value the extra three weeks at ARENA are providing. Another way of saying this is that ARBOx and Arena and these elite programs have similar selection processes. And so Arena or ARBOx accepting someone is strongly correlated with the fact that they have high potential for future AI safety research, regardless of how much value they add on top.
> I also think having a less-structured, more research-driven model would probably require more TA involvement? If you wanted to do it well and have it be accessible to early-stage people, at least.
I do not consider participants of ARENA to be 'early stage'. In my mind they are mid-stage (i.e. middle of upskilling towards a full-time researcher role) and most participants would be able to do solid research sprints without having gone through ARENA. My proposal is based on helping such mid-stage researchers. I think something like BlueDot (at least, BlueDot in 2024, I dont know about current BlueDot) or AISC targets early-stage researchers.
> Also we do put out impact reports where people self-assess as having improved their ability
My claim (which I have not really justfied, except to defer to Neel Nanda's post) is that the counter-factual of doing four mini research sprints would be significantly higher impact. This could be the central crux.
> Side note that I would love to see more reports like this from other programs in the ecosystem, and on a more regular basis
100%. Thanks for doing this and being a role model!
I'm not sure it presented itself as fact, but it definitely blurs the line and it's not obvious. I still found the stories highly engaging!
Can recommend reading Labatuts "When We Cease to understand the world"!
Feedback on next steps:
- I see three main questions: how can we create a better monitor, how much will the methodologies generalize, how would such a monitor actually be used.
- Better monitors: I like the various ideas for trying to improve things (e.g. fine-tuning, ensembling, agentic monitors). I do not have intuitions for how to pick amongst them as next steps. Maybe need to do a detailed analysis on the monitors built so far and see what kinds of mistakes they make. E.g. to see if ensembling would help, can analyze the results of your grid search to see if models make correlated mistakes or not. Issue with any refinements is over-fitting to the test set, and I dont know principled ways around this. (Especially with LLMs, which can be sensitive in unpredictable ways to prompts)
- Generalization: Either develop some principled reasons to believe the method generalizes (seems hard), or, create/curate more dataset, which seems robustly good. Presumably you can use trajectories from any AI Control project (that meet some quality bar)??
- How monitor will be used? If not done already, need to consider how the monitor would actually be used in practice and how it compares to other blue-teaming methods. This will help motivate next steps and/or performance metrics. I see you've already thought about this, but I dont know if more thought would help on the margin.
Other questions/feedback based on quotes:
> Scheming Strict - An AI system pursues misaligned goals while strategically concealing its intentions or capabilities in order to avoid intervention and better achieve those goals in the long term.
> Scheming Lite - An AI system exhibits behaviour consistent with knowingly pursuing misaligned goals, or with deliberately exploiting its reward mechanism.
This distinction is worth a (short) post by itself! Would be good to flesh out distinction, and give examples and non-examples of each. Clarity on this, might also lead to better prompts for the monitor.
> If the LLM detects an attempt, label as scheming.
> If not, manually inspect and label. These cases were rare
I am surprised this case was rare. Does that mean in Attack Mode, models almost always try to do some sabotage? In original control paper, the red-team protocol resulted in the model being selective when trying to carry out sabotage.
> Concretely, if a monitor convincingly flags a label as incorrect, we prefer to relabel it, even if that introduces bias.
I think this is a typo? Should be LLM-assistant, not monitor.
> Nonetheless, we believe it is still correct to maximise label accuracy. Our primary goal is to build the best possible monitor, not to preserve a fair or unbiased evaluation metric
- Unclear why keeping the naive labels is less biased or more fair?
- For clarity, it's worth listing out all the cases and determining which cases are most problematic, and whether the choice to relabel or not makes things better or worse in each case. There would be 2^3 cases, depending on original-naive-label, new-label, actual-true-but-unknown-label.
> We develop monitors with the following two-stage grid search procedure [and next three bullet points]
Should be explicit which datasets are used in each of these bulletpoints. I assume it is always all of the training set. I.e. no use of test set in developing the monitor, and, no splitting of the training set (e.g. no use of cross validation).
> Note that, although during review we found examples which we could have relabelled, in practice we never relabelled any trajectories and instead omitted them from our datasets.
Why not put them in the datasets?
Other questions without quotes
- Is it possible to use your monitor as part of a blue team protocol in ControlArena? Can then get usefulness vs safety scores in different protocols. Or, if that's too expensive, do some kind of simple simulation/estimate of usefulness vs safety scores. Or maybe it just boils down to calculating TPR for various FPRs?
- Unclear what the output of the monitor is. Based on prompt, it is an integer score from 1 to 10, but looks like in F1 plot, we have more than 11 points.
- More for curiosity, would be interesting to see how a naive baseline would perform. E.g. create vector embeddings of the trajectories, then carry out standard supervised learning pipeline with random forest / CatBoost in sklearn.
Quick one: when you use AUROC (especially very large scores!), always good to report the base rates. Is the dataset equally split 50-50 between scheming and non-scheming?
Based on the original control paper, scheming was rare, so ground-truth data would imbalanced (e.g. 90 or 95% of datapoints is non-scheming). If there is a large imbalance, using AUPRC is more insightful.
Has the team tried noise injection as a way of detecting if a model is sandbagging, in these different scenarios?
Is code for experiments open source?
In Sakana AI's paper on AI Scientist v-2, they claim that the sytem is independent of human code. Based on quick skim, I think this is wrong/deceptful. I wrote up my thoughts here: https://lovkush.substack.com/p/are-sakana-lying-about-the-independence
Main trigger was this line in the system prompt for idea generation: "Ensure that the proposal can be done starting from the provided codebase."
> If you were super disciplined and you took one day every two weeks to work through one notework, you'd spend most of a year just to qualify for the program
I believe: 1) you don't need to diligently work through a whole notebook to get most of the value of the notebook and 2) the majority of the value of ARENA is contained in a subset of the notebooks. Some reasons:
1a) The notebooks are often, by design, far more work than is possible to do in a day. Even in ARENA, where you have pair programming, TAs on hand, great co-working space, lunch and dinner provided, etc.. Note a 'day' here is roughly 5.5-6.5 hours. (10am to 6pm, morning lecture at 10, lunch at 12, break at 3.30)
1b) Even for the shorter notebooks, it is often only manageable to complete in a day if you skip some exercises, or cheat and peak at the solution. (This is recommended by ARENA, and I agree with this recommmendation given time constraints.)
1c) There are 5 (or 6) LARGE mech interp notebooks for final three days of mech interp week. One recommendation is to try two notebooks on the Wed and Thu, then continue with the one you prefer on Friday. So I saw 2 out of the 5 notebooks when I participated in ARENA. Despite this, I was still able to TA during mech interp. It was bit frantic, but I would skim the start of each of the notebooks I didn't understand, enough that I could help people unblock or to explain key ideas. I feel I got good percent of the value that the ARENA participants got out of those other notebooks without having done a single exercise in them.
2a) In ARBOx2, the schedule was (comma represents different days)
- Week 1: CNNs, Transformers, Induction circuit, IoI circuit, [another mech interp notebook. cant remember which. likely SAEs]
- Week 2: RL day 1, RL day 2, project, project, project.
The biggest thing missing from this IMO is the theory of impact exercise from second half of ARENA evals day 1. Otherwise, for the calibre of ppl doing ARENA, a quick skim of the other notebooks gives majority of the value.
I would recommend ARBOx over ARENA because of the time efficiency. You get high percentage of value of ARENA, but in 40% of the time.
> most other programs focus more on research than developing ML skills
I dont think ARENA focusses on ML skills. Week 0 has content directly supervised ML, and only a small (but crucial!) part of ML, namely, writing networks in pytorch and creating training loops. Week 2 has content on RL. But given time constraints, many other parts of ML aren't covered in depth, e.g. how to do hyper-parameter tuning (most of the time just use the hyper-parameters provided, there's no time to actually do hyper-parameter tuning), how to even tell if hyper-parameters are the issue, data collection and cleaning, cluster management, selecting GPUs, etc.