Measuring out-of-context Schelling coordination capabilities in large language models. Overview This is my first foray into AI safety research, and is primarily exploratory. I present these findings with all humility, make no strong claims, and hope there is some benefit to others. I certainly learned a great deal in the...
How might we best design experiments to measure Schelling coordination capabilities? A side project building a Schelling coordination eval has led to a role as an AI Safety researcher - this work will officially begin in a month or so. I have been hesitant to post here or share my...
Measuring out-of-context Schelling coordination capabilities in large language models.
This is my first foray into AI safety research, and is primarily exploratory. I present these findings with all humility, make no strong claims, and hope there is some benefit to others. I certainly learned a great deal in the process, and hope to learn more from any comments or criticism—all very welcome. A version of this article with some simple explanatory animations, less grainy graphs, and updates to the evaluation as new models are released can be found here. Github link is coming soon after a tidy and making the .eval files private or password protected.
Can two instances of the same model, without... (read 2668 more words →)
Indeed, current models are terrible at this! Still, worth keeping an eye on it, as it would complicate dangerous capability evals quite a bit should it emerge.
This was a great read - it seems very wise to track the confidence of models vs actual capabilities on the subskills relevant to these threats as you have started to do here! Another candidate capability to track in this way might be Schelling coordination aka acausal coordination - see this article under limitations-awareness as a strategic input
For this reason, regulatory thresholds should ideally use benchmarks that cover a wide range of difficulties, including extremely difficult questions.
Yes - and a further point on the importance of a range of difficulties: For a model to sandbag succesfully across a carefully designed eval suite, it must simulate a plausible capability profile, and coordinate this simulation across independent instances. Very difficult or dangerous evals are easy to coordinate on (the model should sandbag) but when presented with moderate difficulty evals across various capabilities which may only be concerning in aggregate - it is less obvious which of these to coordinate on and sandbag. Note the importance of a model's self-awareness of its capability to simulate a plausible capability profile to its decision to even attempt to sandbag!
How might we best design experiments to measure Schelling coordination capabilities?
A side project building a Schelling coordination eval has led to a role as an AI Safety researcher - this work will officially begin in a month or so. I have been hesitant to post here or share my research so far in part because I have not yet had time to get fully up to date with all relevant past work. However, I figure if I try to read everything first I’ll never post, despite the acknowledged risk that my post may be unoriginal, misinformed or just plain wrong.
I offer these reflections on the InputCollusion environment in Subversion Strategy Eval by Mallen... (read 2397 more words →)