TL;DR: While the AI safety community is interested in forecasting, it does not tend to systematically compare how effective different researchers, models, or theories are at predicting AI behaviour. I suggest the latter might be neglected, and propose a platform that could help fill the gap. I believe its most valuable use would be as a test bed for theories of how AI works, showing which ones predict well and in what contexts.
In February 2025, Owain Evans’ group published Emergent Misalignment, showing how LLMs fine-tuned on insecure code can become generally misaligned, for instance adopting unrelated behaviours like asserting ‘humans should be enslaved’. This was a high-impact paper, generating a lot of publicity. However, when I first heard about the result, I wasn’t particularly surprised — and I wasn’t the only one. I’d read and bought into simulator theory pretty heavily, and emergent misalignment struck me as exactly the kind of thing it implied. But Owain had anticipated this kind of response, and ran a little experiment to get out ahead of it. At the end of his tweet thread announcing the paper, he added a bonus:
Are our results surprising to AI Safety researchers or could they have been predicted in advance?
Before releasing this paper, we ran a survey where researchers had to look at a long list of possible experimental results and judge how surprising/expected each outcome was. Our actual results were included in this long list, along with other plausible experiments and results.
Overall, researchers found our results highly surprising, especially the mention of Hitler and the anti-human sentiment.
Now this surprised me. And it got me thinking — apart from the fact I was probably just fooling myself and falling prey to hindsight bias — why don’t researchers do what Owain did more often, and solicit predictions about their experiments?
The intellectual space around AI safety is keen on forecasting competitions. It has nurtured prediction markets, regularlykeepsscore on forecasts of various benchmarks and capability thresholds, and builds detailed projectionsof the future. But there seems to be comparatively little effort expended on predicting AI behaviour — or, to operationalise this, on formally tracking how effective different researchers, models, or theories are at forecasting the results of behavioural experiments.
Filling this gap could be valuable. It is common to see researchers worry about the complexity and opacity of AI systems — at this point, it’s a cliché to state they are ‘grown’ rather than made — and to do things like compare the field unfavourably to the precision engineering that goes into e.g. building bridges, or generally bemoan its lack of deep scientific understanding. If you want to replicate the reliability of bridge-building in AI safety, to take steps towards a ‘science of generalization’, or alternatively to show policy-makers just how far away from this we are, then measuring your ability to predict AI behaviour in advance of testing could be a strong signal to work with[1].
The proposal
My original idea was to consider what it would look like to build a platform like the Social Science Prediction Platform that preregisters experiments and invites other researchers to predict the results (see appendix for more details). However, there are a few problems with this idea as applied to AI research:
Each question would have to be reasonably difficult or expensive to resolve, otherwise it would be easy to cheat. E.g., you couldn’t ask ‘What will model X say in Y situation?’, as you could trivially check the query.
It would be difficult to capture qualitative surprise. In Emergent Misalignment, the Hitler and anti-human sentiment responses were what surprised people the most, and it isn’t clear the researchers could have anticipated this well enough to codify into a resolvable question up-front.
Researchers might be reluctant to post, as it would give away their ideas before they have any results.
Instead, I believe a more effective platform would systematise what Owain actually did — surveying forecasters after having completed the research. In this case, researchers could include surprising results in the response options (along with decoys) and would not have to worry about losing credit for their work. They could submit their questions shortly before publication, letting forecasters have a time window (e.g. half an hour) on the platform to give a response. This format also does not demand much behavioural change from researchers, which is always going to be hard to incentivise.
The platform should distinguish between human and AI forecasts. The latter could be used as a public measurement of how well models can anticipate outcomes in AI safety research, which is a core component of good research taste. While I have misgivings about the whole ‘use AI to solve alignment’ plan, and what I’m proposing is (sort of) a benchmark that can be optimised against, I think it would be useful to know publicly whether models can predict safety-relevant behaviour better than human experts.
A key challenge remaining would be operationalising theories for prediction. In my opinion, the most valuable use of a platform like this would be as a test bed for different models of how AI works, shedding light on what predicts well in what context, and what might generalise furthest out of distribution. For instance, if I was so certain the simulators frame made Emergent Misalignment expected, how could I have formalised this? How could subsequent work on persona vectors or features be used to answer new questions[2]? Perhaps repeatable solutions registered in advance — where somebody else can run the same method and get the same result — could be hosted publicly. This would also remove the need for a time limit, which may be essential for running something complex.
Overall, while I’m sure there is plenty left to figure out, I believe that systematically tracking forecasts of AI behaviour is a plausibly neglected activity, and that a platform like the above would be a useful step towards filling the gap.
Appendix: Related work
Broadly speaking, the related literature I found falls into two main themes, along with a bunch of miscellaneous other things:
Experiment forecasting in social science. AI safety is not unique in trying to understand complex systems. Since 2020, the Social Science Prediction Platform has been collecting forecasts of experiments, with the top forecasters making upwards of 700 predictions. There are open questions right now on whether ‘partisan negativity increases the reach and engagement of political social media posts’ and the results of cash transfer programmes in Malawi. I don’t know enough about social science to judge how successful this platform has been, but the testimonials page suggests a decent number of happy users. There are also other papers in social science prediction, including building on some DARPA projects by looking at prediction markets for experiments and forecasting paper replications, with the latter more successful than the former. A 2020 paper tried to outline how ‘replication markets’ might work for AI, although it isn’t clear to me that the project went very far.
Using AI to predict AI behaviour. This ranges from workexplicitly focussed on speeding up capabilities, with examples finding that models can better forecast promising research directions than human experts, to more safety-minded work, such as training models to predict how repeatable behaviours are. Another work develops a method for predicting whether given data will lead to unintended behaviours via subliminal learning.
There are alsosome papers advocating preregistration in AI research — primarily aimed at preventing p-hacking — and broader prediction projects from OpenAI and Anthropic, looking to estimate the likelihood of problem behaviour in deployment. Another paper uses interpretability techniques to predict behaviour in out-of-distribution situations, looking at attention patterns correlated to certain generalisation rules — albeit only for simple models.
More informally, I notice that Neel Nanda recommends predicting experimental results during research, as a way of building your research taste. I’ve seen this advice repeated in a recent post by Jo Jiao, who appeared to find it useful.
And, to use Edward Young’s terminology, might be a way of redirecting some of the make-number-go-up energy of ‘Alignment engineering’ towards the understanding sought by ‘Misalignment science’. One failure mode to be avoided though, is that the point is not to get black box prediction scores, like with a prediction market, but to actually find repeatable methods that increase our understanding.
In the persona vectors paper, they take prompt-response pairs from candidate training data and use the prompts to generate a set of ‘natural’ responses from the base model. They then measure the average projection of each set of responses along a persona vector, e.g. ‘sycophancy’ or ‘evil’. If the difference in these projections is large, it suggests that fine-tuning on the data will induce a behavioural shift. It would be interesting to see how broadly ideas like this could be applied, even on questions that are not at first glance a natural fit. You could imagine having a library of vectors for various models which could be systematically applied to predict the results of new fine-tuning experiments.
TL;DR: While the AI safety community is interested in forecasting, it does not tend to systematically compare how effective different researchers, models, or theories are at predicting AI behaviour. I suggest the latter might be neglected, and propose a platform that could help fill the gap. I believe its most valuable use would be as a test bed for theories of how AI works, showing which ones predict well and in what contexts.
In February 2025, Owain Evans’ group published Emergent Misalignment, showing how LLMs fine-tuned on insecure code can become generally misaligned, for instance adopting unrelated behaviours like asserting ‘humans should be enslaved’. This was a high-impact paper, generating a lot of publicity. However, when I first heard about the result, I wasn’t particularly surprised — and I wasn’t the only one. I’d read and bought into simulator theory pretty heavily, and emergent misalignment struck me as exactly the kind of thing it implied. But Owain had anticipated this kind of response, and ran a little experiment to get out ahead of it. At the end of his tweet thread announcing the paper, he added a bonus:
Now this surprised me. And it got me thinking — apart from the fact I was probably just fooling myself and falling prey to hindsight bias — why don’t researchers do what Owain did more often, and solicit predictions about their experiments?
The intellectual space around AI safety is keen on forecasting competitions. It has nurtured prediction markets, regularly keeps score on forecasts of various benchmarks and capability thresholds, and builds detailed projections of the future. But there seems to be comparatively little effort expended on predicting AI behaviour — or, to operationalise this, on formally tracking how effective different researchers, models, or theories are at forecasting the results of behavioural experiments.
Filling this gap could be valuable. It is common to see researchers worry about the complexity and opacity of AI systems — at this point, it’s a cliché to state they are ‘grown’ rather than made — and to do things like compare the field unfavourably to the precision engineering that goes into e.g. building bridges, or generally bemoan its lack of deep scientific understanding. If you want to replicate the reliability of bridge-building in AI safety, to take steps towards a ‘science of generalization’, or alternatively to show policy-makers just how far away from this we are, then measuring your ability to predict AI behaviour in advance of testing could be a strong signal to work with[1].
The proposal
My original idea was to consider what it would look like to build a platform like the Social Science Prediction Platform that preregisters experiments and invites other researchers to predict the results (see appendix for more details). However, there are a few problems with this idea as applied to AI research:
Instead, I believe a more effective platform would systematise what Owain actually did — surveying forecasters after having completed the research. In this case, researchers could include surprising results in the response options (along with decoys) and would not have to worry about losing credit for their work. They could submit their questions shortly before publication, letting forecasters have a time window (e.g. half an hour) on the platform to give a response. This format also does not demand much behavioural change from researchers, which is always going to be hard to incentivise.
The platform should distinguish between human and AI forecasts. The latter could be used as a public measurement of how well models can anticipate outcomes in AI safety research, which is a core component of good research taste. While I have misgivings about the whole ‘use AI to solve alignment’ plan, and what I’m proposing is (sort of) a benchmark that can be optimised against, I think it would be useful to know publicly whether models can predict safety-relevant behaviour better than human experts.
A key challenge remaining would be operationalising theories for prediction. In my opinion, the most valuable use of a platform like this would be as a test bed for different models of how AI works, shedding light on what predicts well in what context, and what might generalise furthest out of distribution. For instance, if I was so certain the simulators frame made Emergent Misalignment expected, how could I have formalised this? How could subsequent work on persona vectors or features be used to answer new questions[2]? Perhaps repeatable solutions registered in advance — where somebody else can run the same method and get the same result — could be hosted publicly. This would also remove the need for a time limit, which may be essential for running something complex.
Overall, while I’m sure there is plenty left to figure out, I believe that systematically tracking forecasts of AI behaviour is a plausibly neglected activity, and that a platform like the above would be a useful step towards filling the gap.
Appendix: Related work
Broadly speaking, the related literature I found falls into two main themes, along with a bunch of miscellaneous other things:
There are also some papers advocating preregistration in AI research — primarily aimed at preventing p-hacking — and broader prediction projects from OpenAI and Anthropic, looking to estimate the likelihood of problem behaviour in deployment. Another paper uses interpretability techniques to predict behaviour in out-of-distribution situations, looking at attention patterns correlated to certain generalisation rules — albeit only for simple models.
More informally, I notice that Neel Nanda recommends predicting experimental results during research, as a way of building your research taste. I’ve seen this advice repeated in a recent post by Jo Jiao, who appeared to find it useful.
Thank you to Seth Herd for useful discussions.
And, to use Edward Young’s terminology, might be a way of redirecting some of the make-number-go-up energy of ‘Alignment engineering’ towards the understanding sought by ‘Misalignment science’. One failure mode to be avoided though, is that the point is not to get black box prediction scores, like with a prediction market, but to actually find repeatable methods that increase our understanding.
In the persona vectors paper, they take prompt-response pairs from candidate training data and use the prompts to generate a set of ‘natural’ responses from the base model. They then measure the average projection of each set of responses along a persona vector, e.g. ‘sycophancy’ or ‘evil’. If the difference in these projections is large, it suggests that fine-tuning on the data will induce a behavioural shift. It would be interesting to see how broadly ideas like this could be applied, even on questions that are not at first glance a natural fit. You could imagine having a library of vectors for various models which could be systematically applied to predict the results of new fine-tuning experiments.