tl;dr — The evals ecosystem needs to be complemented with real-world monitoring. The AI labs can and should monitor their own traffic, but we also need an independent voice that keeps labs accountable and monitors open models. At the Susan Calvin Project, we aim to detect AI misbehaviors and incidents in the wild, and collect evidence for the (mis)alignment of existing AIs. Agentic AI tools are integrated into more aspects of our work and personal life, and models continue to become more capable while alignment remains unsolved. It is going to be more and more important to understand the behavior of actual AIs in people’s actual usage.
It's time to take a closer look at our AI agents.
Over the past few weeks, I’ve been working on a new project. It’s named after Dr Susan Calvin, the robopsychologist in Asimov. As mentioned in announcement post, I’m building an independent observatory of AI behavior in the wild. I will collect AI agent trajectories from real-world deployment and use them to monitor, measure, and understand AI behavior. This is a longer post with a bit more detail on why I think this is worthwhile.
Why study real-world AI usage data?
I’ll start by saying that I am generally a proponent of evals! I have previously worked at METR on dangerous capabilities evals, and was the point person for evaluating our AI weather models at my last job. I think there are many benefits to running evals in controlled settings. For example, evals allow you to test the AI’s behavior in extremely rare but high-stakes situations.
But evals are also obviously far from perfect. Here are some reasons to also study real-world deployment data:
Frontier models today alreadyexhibit substantial eval-awareness. Eval realism is getting better, but the real-world deployment distribution is always going to be different from the evals distribution[1].
At the meta level: it’s valuable to know how good our evals are! And measuring this requires data from real deployment.
Good real-world monitoring could uncover interesting novel behaviors that have not made their way into any evals.
Lastly, there is a sense in which it does not matter whether or not the AI is eval-aware; what matters is simply how it does in fact behave in (internal and external) deployment. What in-the-wild AIs are like actually matters, as they become a larger and larger part of our life and work.
Why do this outside of AI labs?
AI labs are collecting huge amounts of user data, and they are starting to do research with it. I think this is great! But there’s also enormous value in doing this outside of the labs. As an independent actor, we don’t have incentive to cover up—or fail to discover in the first place—concerning or misaligned behaviors, should they exist in user data. We can offer behavioral comparisons across labs, creating incentives to reduce bad behaviors, or a “race to the top” on good behaviors. Plus, as AI diffuses, we’ll see increased adoption of open-source models and third-party agent traffic that does not go through the labs.
There is one more consideration: Labs are very risk-averse when it comes to privacy practices, and often for good reasons. For example, Anthropic Insights (formerly “Clio”), is Anthropic’s tool for research with user data in a privacy-preserving manner. The privacy-preserving technology amounts to only ever letting Claude read the raw transcript data; researchers only get back summary statistics of answers to questions they asked. This is obviously an extremely limiting technique! I am a believer in the good old ML practice of looking at your damn data, and without access to the raw data, I worry that it would be extremely easy to fool ourselves into thinking we’re measuring something while we are not. Plus, there is always the possibility that misaligned future AIs could deliberately mislead us about the results, and we will have no way of finding out. Of course, privacy does matter a lot and I have more thoughts to share on this soon. But with users that opt in to our monitoring program and consent to have their data studied, we can do better analyses.
What good is this for?
I have many hopes for this project, including:
Help users understand the behavior of the AIs they interact with. Users I’ve talked to are very interested to know whether they are using AI tools well, how they can improve. Why did the AI lie to me, is there something I could have done to prevent it?
Create incentives for AI developers to improve their models’ behavior. It seems like public knowledge about severe misbehaviors of models has been an effective lever for getting better behavior in future models. For example, Transluce found that mental-health-related helpful assistant behaviors “increased sharply over time” across models from leading AI labs, and rates of directly endorsing or facilitating suicide show a parallel decline. While I don’t have proof of any causal links, it seems likely that this was downstream of the reporting and public visibility of the concerning behavior of models from 2024-2025.
Monitor for safety and security incidents in the wild that would have been previously missed. As you may have heard, there have been a fewofthese lately. Are there more if we looked harder?
Lessons for alignment from analyzing behavior differences in real use. We have very little visibility into how others are using AI tools, and we seem to have wildlydifferent experiences with our AI agents. What’s going on? I think there might be interesting lessons for alignment to learn from measuring and understanding these differences.
Help the public understand the current state of AI. The world is rapidly waking up to the speed of AI development, and this project can create artifacts for the public and policy makers to help them understand the real-world behavior of AI today. If current AIs are misaligned, the world needs to know.
Who am I / why me?
My name is Haoxing and I’m one of the founders at Surplus. I have been in and out of the field of AI safety for a few years. I am an author on METR’s time horizons paper and did some research on interpretability at Redwood, back in the day. Having just finished a stint at a non-AI-safety startup, I’m excited to have an opportunity to explore ways I can help transformative AI go well.
This project requires research experience and taste, but also the ability to communicate to a wide audience, and the ability to build something and get real people to use it—so I was excited to give it a go.
How you can help
You can contribute your sessions from agentic coding tools like Claude Code and Codex now! I built a tool so that you can easily exclude sensitive sessions, redact secrets and PII, and share your data encrypted. I am the only human that will be able to view this data, and I will use this research corpus to test methodologies and publish results from studying this corpus. I’m currently working on reproducing and extending Transluce’s report, Measuring coding agent misalignment in the wild.Thank you for your support!
If you’d rather not share your data at this time, you can try out my app, Behavior Wrapped—a fun, Spotify-wrapped style report of the behavior of you and your AI agents that runs locally on your machine. You can also follow me on Twitter or subscribe to this blog for future updates.
Lastly, if you are an AI safety researcher or someone who would be interested in using a dataset of real-world AI traces, please get in touch.
Of course, I don’t expect the dataset I collect to be representative of the “real-world deployment distribution” either. There are all sorts of selection biases, in which users would be willing to share their data, and what data they choose to share, etc. But a nonrepresentative sample of real-world data is still better than none!
tl;dr — The evals ecosystem needs to be complemented with real-world monitoring. The AI labs can and should monitor their own traffic, but we also need an independent voice that keeps labs accountable and monitors open models. At the Susan Calvin Project, we aim to detect AI misbehaviors and incidents in the wild, and collect evidence for the (mis)alignment of existing AIs. Agentic AI tools are integrated into more aspects of our work and personal life, and models continue to become more capable while alignment remains unsolved. It is going to be more and more important to understand the behavior of actual AIs in people’s actual usage.
It's time to take a closer look at our AI agents.
Over the past few weeks, I’ve been working on a new project. It’s named after Dr Susan Calvin, the robopsychologist in Asimov. As mentioned in announcement post, I’m building an independent observatory of AI behavior in the wild. I will collect AI agent trajectories from real-world deployment and use them to monitor, measure, and understand AI behavior. This is a longer post with a bit more detail on why I think this is worthwhile.
Why study real-world AI usage data?
I’ll start by saying that I am generally a proponent of evals! I have previously worked at METR on dangerous capabilities evals, and was the point person for evaluating our AI weather models at my last job. I think there are many benefits to running evals in controlled settings. For example, evals allow you to test the AI’s behavior in extremely rare but high-stakes situations.
But evals are also obviously far from perfect. Here are some reasons to also study real-world deployment data:
Lastly, there is a sense in which it does not matter whether or not the AI is eval-aware; what matters is simply how it does in fact behave in (internal and external) deployment. What in-the-wild AIs are like actually matters, as they become a larger and larger part of our life and work.
Why do this outside of AI labs?
AI labs are collecting huge amounts of user data, and they are starting to do research with it. I think this is great! But there’s also enormous value in doing this outside of the labs. As an independent actor, we don’t have incentive to cover up—or fail to discover in the first place—concerning or misaligned behaviors, should they exist in user data. We can offer behavioral comparisons across labs, creating incentives to reduce bad behaviors, or a “race to the top” on good behaviors. Plus, as AI diffuses, we’ll see increased adoption of open-source models and third-party agent traffic that does not go through the labs.
There is one more consideration: Labs are very risk-averse when it comes to privacy practices, and often for good reasons. For example, Anthropic Insights (formerly “Clio”), is Anthropic’s tool for research with user data in a privacy-preserving manner. The privacy-preserving technology amounts to only ever letting Claude read the raw transcript data; researchers only get back summary statistics of answers to questions they asked. This is obviously an extremely limiting technique! I am a believer in the good old ML practice of looking at your damn data, and without access to the raw data, I worry that it would be extremely easy to fool ourselves into thinking we’re measuring something while we are not. Plus, there is always the possibility that misaligned future AIs could deliberately mislead us about the results, and we will have no way of finding out. Of course, privacy does matter a lot and I have more thoughts to share on this soon. But with users that opt in to our monitoring program and consent to have their data studied, we can do better analyses.
What good is this for?
I have many hopes for this project, including:
Who am I / why me?
My name is Haoxing and I’m one of the founders at Surplus. I have been in and out of the field of AI safety for a few years. I am an author on METR’s time horizons paper and did some research on interpretability at Redwood, back in the day. Having just finished a stint at a non-AI-safety startup, I’m excited to have an opportunity to explore ways I can help transformative AI go well.
This project requires research experience and taste, but also the ability to communicate to a wide audience, and the ability to build something and get real people to use it—so I was excited to give it a go.
How you can help
You can contribute your sessions from agentic coding tools like Claude Code and Codex now! I built a tool so that you can easily exclude sensitive sessions, redact secrets and PII, and share your data encrypted. I am the only human that will be able to view this data, and I will use this research corpus to test methodologies and publish results from studying this corpus. I’m currently working on reproducing and extending Transluce’s report, Measuring coding agent misalignment in the wild. Thank you for your support!
If you’d rather not share your data at this time, you can try out my app, Behavior Wrapped—a fun, Spotify-wrapped style report of the behavior of you and your AI agents that runs locally on your machine. You can also follow me on Twitter or subscribe to this blog for future updates.
Lastly, if you are an AI safety researcher or someone who would be interested in using a dataset of real-world AI traces, please get in touch.
Of course, I don’t expect the dataset I collect to be representative of the “real-world deployment distribution” either. There are all sorts of selection biases, in which users would be willing to share their data, and what data they choose to share, etc. But a nonrepresentative sample of real-world data is still better than none!