Image by Claude Opus 5.5 using Python (NumPy and Pillow)
This is written in my free time and personal capacity.
Two Frames
In AI safety, there are a couple of tensions at the heart of the field, which are interrelated.[1]
Talent: we are talent-constrained and must recruit. But jobs and fellowships turn away nearly everyone.[2]
Field size: we must expand the field and bring more people in. But the field is flooded with slop papers in conferences. And, most papers have a small audience, and aim to shift the practices of roughly five labs (and usually some subteam within them).[3]
Status: we must avoid seeking status, as it disorients, corrupts. But, we must make the field legible and attractive to outsiders.
Raising the bots: we must deceive the AIs to control them and evaluate them without confounds, obscuring what they know about their own situation. But, we hope to gain their trust, must rely on them to do good work at pace, shape their character and retain good relations.
Speed: we must research in a slow mode, think deeply about the most important problem, do things to aim at high assurance. But it is crunch time. It was crunch time two months ago. We must make hacky progress now and trust that scale will clarify things later.
I think these tensions are, at least in part, rooted in what I will call a "war" frame and a "civilisation" frame. These frames are oriented around a different end goal, or telos of AI safety.
In the war frame, AI safety is a campaign with a deadline, with decisive action that needs to be taken at a handful of labs over the next few training runs. In this frame, your job squarely focuses on changing what happens in those labs before then.
In the civilisation frame, AI safety is closer to dealing with a new population of aliens arriving among us. If aliens landed, we wouldn't ask how many alien researchers we need; we would need a counterpart to nearly every human institution: diplomats, courts, customs, doctors, police. By that measure, the right number of people working on AI is huge, and most of them would not be researchers.
This is more than just timelines, of course, and I find it helpful to think of it as a broad, sociological orientation. One observation is that in terms of talent, we tend to recruit under a civilisation frame, but allocate resources and slots under a war frame. In raising the bots, the war frame treats AIs as systems to test and control, so deceiving them is one tool among many. But the civilisation frame treats them as counterparties we will live alongside. You can't run sting operations indefinitely if you want to build trust!
On the War Frame
I think the war frame is prevalent. You can hear it in a lot of talk: crunch time, timelines, the next training run. Often, I catch myself using this language! Boyd Kane accurately describes many serious people as being in "war mode": endlessly driven, not by money or promotions, with everything else losing its interest (MATS 9 Retrospective & Advice). Part of the reason it prevails, I think, is that AI safety is organised around a failure it wishes to end.
But that alone needn't make a field transient. Medicine, public health and computer security all exist to remove something. And these are lasting fields since the problem is chronic. Security is hard to solve, and security needs grow with every new deployment. Large things that did dissolve on success were one-shot projects, like smallpox eradication. In many ways, AI safety has the trappings of a field (fellowships, careers, conferences) but seriously imagines itself as a project with an end date, like a global Manhattan Project.
Near Horizons
I now discuss some personal unhappiness I have with the war frame. Chess engines have a failure called the horizon effect. An engine that can only search a few moves ahead will push an unavoidable loss just past what it can see, so the loss seems to vanish. War mode does the same with everything after the crunch: abrading away your career, your health, your purpose.
One observation is that war mode can narrow your horizons in a corrosive way. When you start in AI safety and you go into war mode, it's hard to reflect; you're hurtling. In the comments on that MATS retrospective, Kyle O'Brien says every day off with his family now feels like a trade-off he may come to regret (MATS 9 Retrospective & Advice).
If you work assuming you are ephemeral, you may sputter out even if we do get a pause! A pause turns a sprint into a marathon, and war mode leaves you out of fuel to run a marathon. One post on psychological support for safety researchers goes as far as suggesting it may be rational to plan to burn out just as ASI arrives (Psychological support for AI safety researchers is neglected).
And: working on the assumption that you will obliterate yourself can't be healthy, even if we do (IMO, implicitly) think highly of ourselves for knowing about the problem and doing something about it. Perhaps the status is part of what keeps it bearable: being important now stands in for having a future. I recall a striking post by Josh Clymer, opening by saying their goal as a safety researcher is to work themselves out of a job, and continuing that safely handing the work to AI should be the field's primary end goal (How might we safely pass the buck to AI?). The AI should be safe, and if it is safe, it can do our job.
That's a rough spot to be in! And I reject it, repudiate such hopelessness: even if we are destined to fail, we can dream of frames that aren't hostile to our wellbeing.
The Civilisation Frame
Sometimes, on long car rides, I think about two decades from now. What does a world look like where AI safety is a chronic rather than a monumentally ephemeral problem, and we treat it more like security or public health? As Claude often says: a thing can also be another thing. A role transitional without the life provisional; scaffolding comes down when the building is built, but the builders go on, building other offices, churches, homes.
Pieces of this frame already exist in some form, mostly as proposals for how to relate to AIs. One way to think of AI risk is as an intensified version of the old problem of balancing power among agents with different values, where our political traditions have a lot to offer (Otherness and control in the age of AGI). One could give AIs the contract, property and tort rights that corporations already have, on the logic that trade beats war (AI Rights for Human Safety). Alignment can be framed as driver training, which still leaves us needing the traffic lights (Infrastructure for AI Agents).
What does this frame imply for our orientation as a field?
For starters, alignment will likely only be one of many Herculean challenges we'll have to overcome. Humans could lose influence over the economy, culture and the state even if efforts to stop AIs from scheming or seeking power succeed (Gradual Disempowerment). In the war frame, success means that the troops can return home, a field to stand down, return to reserves. In the civilisation frame there is no stand-down; we must build peace, society, answering thorny questions of how humans and AIs share institutions over decades.
While we must prepare to fail, we must also prepare to succeed! We should take a range of bets[4]: some we would only do if we thought we had 2 years left, some we would have to do if we were to get high-assurance alignment with a pause, some that would be necessary for the greatest aspirations of humanity in long-term flourishing. We should invest in understanding, even if we may fail, because if we fail to do so, then we will never succeed.
It also, I hope, widens what we can count as safety research. Much of a civilisation's work is institutional: incident reporting, rollbacks, legal standing, liability, auditing, insurance, standards. Other things too: sociology, anthropology, culture, history, education. Little of it needs orienting towards the few people at a frontier lab.
We could, lastly, be clearer about what "talent-constrained" means: senior people who can mentor (MATS, AI Safety Talent Needs in 2026) and generalists, rather than every strong student pivoting to research (Stop telling people to pivot to AI safety, I beg you). And we should recruit people for careers rather than sprints. We should aim to build specific rather than general entry points for people who already have a profession: lawyers, auditors, security engineers, civil servants, doctors, managers. A few of these already exist and seem good: Heron takes experienced security professionals part-time without requiring an AI background, Tarbell trains journalists and places them in newsrooms, and Horizon places technologists in government.
Nonetheless, I remain confused. I find the war mode frame very useful to orient how I do empirical research. Maybe some people should sprint while others build institutions that take decades. And, AIs may staff much of the counterpart civilisation too. How should we frame things to them? What frames will they naturally bring, cling onto? And of course, none of this may matter if they quickly subsume us.
Many of these are not innately contradictory. Many of them have middle grounds: there is an appropriate amount of legibility (like having nice websites, or good salaries) that doesn't tread into inappropriate status-seeking, which impedes truth-seeking. Talent may require aggressive recruitment because the business needs truly are large, but the skills needed may be rare or disparate and rarely in the same person. Hence, I pose them as tensions rather than contradictions; we as a field have to hope that we strike the right balance.
This has been written about at length: as far as I can tell, many AI safety orgs are ambitiously aiming to scale up. A good example is Coefficient Giving, which claims it is acutely capacity-constrained and that its "biggest bottleneck is people, not funding."
Yet, full-time positions are extremely competitive, and fellowships that used to primarily build context and upskill people are now flooded with applicants, many of them already very strong (in my experience as a mentor). Many fellowships accept under 3% of applicants, which is more selective than the Ivy League (Talent Constraints in AI Safety). Together they receive more applications each year than there are people working in AI safety (A decentralised, volunteer-based, scalable lab for AI safety), and one Anthropic Fellows cohort reportedly drew over 10,000 applicants (Stop telling people to pivot to AI safety, I beg you). SPAR, MATS, Pivotal, etc., have increased in size and in competitiveness significantly; you see Reddit posts about acceptance and timelines akin to those about PhD applications, internships in prestige fields, etc.
I think Ed's post describes this pretty well: the dominant style of alignment work is built for immediate use on current systems in a way that is commercially valuable to AGI companies, and functioning as an implicit work trial for them (Alignment engineering vs. misalignment science).
But, we also have a paucity of new ideas, we need more diverse approaches, more attempts. We don't yet have enough bets on which we could stake our futures.
Image by Claude Opus 5.5 using Python (NumPy and Pillow)
This is written in my free time and personal capacity.
Two Frames
In AI safety, there are a couple of tensions at the heart of the field, which are interrelated.[1]
Talent: we are talent-constrained and must recruit. But jobs and fellowships turn away nearly everyone.[2]
Field size: we must expand the field and bring more people in. But the field is flooded with slop papers in conferences. And, most papers have a small audience, and aim to shift the practices of roughly five labs (and usually some subteam within them).[3]
Status: we must avoid seeking status, as it disorients, corrupts. But, we must make the field legible and attractive to outsiders.
Raising the bots: we must deceive the AIs to control them and evaluate them without confounds, obscuring what they know about their own situation. But, we hope to gain their trust, must rely on them to do good work at pace, shape their character and retain good relations.
Speed: we must research in a slow mode, think deeply about the most important problem, do things to aim at high assurance. But it is crunch time. It was crunch time two months ago. We must make hacky progress now and trust that scale will clarify things later.
I think these tensions are, at least in part, rooted in what I will call a "war" frame and a "civilisation" frame. These frames are oriented around a different end goal, or telos of AI safety.
In the war frame, AI safety is a campaign with a deadline, with decisive action that needs to be taken at a handful of labs over the next few training runs. In this frame, your job squarely focuses on changing what happens in those labs before then.
In the civilisation frame, AI safety is closer to dealing with a new population of aliens arriving among us. If aliens landed, we wouldn't ask how many alien researchers we need; we would need a counterpart to nearly every human institution: diplomats, courts, customs, doctors, police. By that measure, the right number of people working on AI is huge, and most of them would not be researchers.
This is more than just timelines, of course, and I find it helpful to think of it as a broad, sociological orientation. One observation is that in terms of talent, we tend to recruit under a civilisation frame, but allocate resources and slots under a war frame. In raising the bots, the war frame treats AIs as systems to test and control, so deceiving them is one tool among many. But the civilisation frame treats them as counterparties we will live alongside. You can't run sting operations indefinitely if you want to build trust!
On the War Frame
I think the war frame is prevalent. You can hear it in a lot of talk: crunch time, timelines, the next training run. Often, I catch myself using this language! Boyd Kane accurately describes many serious people as being in "war mode": endlessly driven, not by money or promotions, with everything else losing its interest (MATS 9 Retrospective & Advice). Part of the reason it prevails, I think, is that AI safety is organised around a failure it wishes to end.
But that alone needn't make a field transient. Medicine, public health and computer security all exist to remove something. And these are lasting fields since the problem is chronic. Security is hard to solve, and security needs grow with every new deployment. Large things that did dissolve on success were one-shot projects, like smallpox eradication. In many ways, AI safety has the trappings of a field (fellowships, careers, conferences) but seriously imagines itself as a project with an end date, like a global Manhattan Project.
Near Horizons
I now discuss some personal unhappiness I have with the war frame. Chess engines have a failure called the horizon effect. An engine that can only search a few moves ahead will push an unavoidable loss just past what it can see, so the loss seems to vanish. War mode does the same with everything after the crunch: abrading away your career, your health, your purpose.
One observation is that war mode can narrow your horizons in a corrosive way. When you start in AI safety and you go into war mode, it's hard to reflect; you're hurtling. In the comments on that MATS retrospective, Kyle O'Brien says every day off with his family now feels like a trade-off he may come to regret (MATS 9 Retrospective & Advice).
If you work assuming you are ephemeral, you may sputter out even if we do get a pause! A pause turns a sprint into a marathon, and war mode leaves you out of fuel to run a marathon. One post on psychological support for safety researchers goes as far as suggesting it may be rational to plan to burn out just as ASI arrives (Psychological support for AI safety researchers is neglected).
And: working on the assumption that you will obliterate yourself can't be healthy, even if we do (IMO, implicitly) think highly of ourselves for knowing about the problem and doing something about it. Perhaps the status is part of what keeps it bearable: being important now stands in for having a future. I recall a striking post by Josh Clymer, opening by saying their goal as a safety researcher is to work themselves out of a job, and continuing that safely handing the work to AI should be the field's primary end goal (How might we safely pass the buck to AI?). The AI should be safe, and if it is safe, it can do our job.
That's a rough spot to be in! And I reject it, repudiate such hopelessness: even if we are destined to fail, we can dream of frames that aren't hostile to our wellbeing.
The Civilisation Frame
Sometimes, on long car rides, I think about two decades from now. What does a world look like where AI safety is a chronic rather than a monumentally ephemeral problem, and we treat it more like security or public health? As Claude often says: a thing can also be another thing. A role transitional without the life provisional; scaffolding comes down when the building is built, but the builders go on, building other offices, churches, homes.
Pieces of this frame already exist in some form, mostly as proposals for how to relate to AIs. One way to think of AI risk is as an intensified version of the old problem of balancing power among agents with different values, where our political traditions have a lot to offer (Otherness and control in the age of AGI). One could give AIs the contract, property and tort rights that corporations already have, on the logic that trade beats war (AI Rights for Human Safety). Alignment can be framed as driver training, which still leaves us needing the traffic lights (Infrastructure for AI Agents).
What does this frame imply for our orientation as a field?
For starters, alignment will likely only be one of many Herculean challenges we'll have to overcome. Humans could lose influence over the economy, culture and the state even if efforts to stop AIs from scheming or seeking power succeed (Gradual Disempowerment). In the war frame, success means that the troops can return home, a field to stand down, return to reserves. In the civilisation frame there is no stand-down; we must build peace, society, answering thorny questions of how humans and AIs share institutions over decades.
While we must prepare to fail, we must also prepare to succeed! We should take a range of bets[4]: some we would only do if we thought we had 2 years left, some we would have to do if we were to get high-assurance alignment with a pause, some that would be necessary for the greatest aspirations of humanity in long-term flourishing. We should invest in understanding, even if we may fail, because if we fail to do so, then we will never succeed.
It also, I hope, widens what we can count as safety research. Much of a civilisation's work is institutional: incident reporting, rollbacks, legal standing, liability, auditing, insurance, standards. Other things too: sociology, anthropology, culture, history, education. Little of it needs orienting towards the few people at a frontier lab.
We could, lastly, be clearer about what "talent-constrained" means: senior people who can mentor (MATS, AI Safety Talent Needs in 2026) and generalists, rather than every strong student pivoting to research (Stop telling people to pivot to AI safety, I beg you). And we should recruit people for careers rather than sprints. We should aim to build specific rather than general entry points for people who already have a profession: lawyers, auditors, security engineers, civil servants, doctors, managers. A few of these already exist and seem good: Heron takes experienced security professionals part-time without requiring an AI background, Tarbell trains journalists and places them in newsrooms, and Horizon places technologists in government.
Nonetheless, I remain confused. I find the war mode frame very useful to orient how I do empirical research. Maybe some people should sprint while others build institutions that take decades. And, AIs may staff much of the counterpart civilisation too. How should we frame things to them? What frames will they naturally bring, cling onto? And of course, none of this may matter if they quickly subsume us.
Thanks for reading.
Many of these are not innately contradictory. Many of them have middle grounds: there is an appropriate amount of legibility (like having nice websites, or good salaries) that doesn't tread into inappropriate status-seeking, which impedes truth-seeking. Talent may require aggressive recruitment because the business needs truly are large, but the skills needed may be rare or disparate and rarely in the same person. Hence, I pose them as tensions rather than contradictions; we as a field have to hope that we strike the right balance.
This has been written about at length: as far as I can tell, many AI safety orgs are ambitiously aiming to scale up. A good example is Coefficient Giving, which claims it is acutely capacity-constrained and that its "biggest bottleneck is people, not funding."
Yet, full-time positions are extremely competitive, and fellowships that used to primarily build context and upskill people are now flooded with applicants, many of them already very strong (in my experience as a mentor). Many fellowships accept under 3% of applicants, which is more selective than the Ivy League (Talent Constraints in AI Safety). Together they receive more applications each year than there are people working in AI safety (A decentralised, volunteer-based, scalable lab for AI safety), and one Anthropic Fellows cohort reportedly drew over 10,000 applicants (Stop telling people to pivot to AI safety, I beg you). SPAR, MATS, Pivotal, etc., have increased in size and in competitiveness significantly; you see Reddit posts about acceptance and timelines akin to those about PhD applications, internships in prestige fields, etc.
I think Ed's post describes this pretty well: the dominant style of alignment work is built for immediate use on current systems in a way that is commercially valuable to AGI companies, and functioning as an implicit work trial for them (Alignment engineering vs. misalignment science).
But, we also have a paucity of new ideas, we need more diverse approaches, more attempts. We don't yet have enough bets on which we could stake our futures.
While this is not a Resolution-post, I like the vision we laid out here.