I’m speaking for myself here, not for any organization, house, home, clan, club, coven, or cult I may be part of. This post is written for scientists outside the field rather than LessWrong regulars; I’m crossposting it so it’s easy to link.
The AI safety job market is a strange one. Every single organization is growing rapidly (the one I work at is planning to double in the next year), which is a fundamentally new dynamic for those of us used to the slow decline of academia. At the same time, interest in the field goes through boom after boom after boom.
We have a ton of openings and a ton of applicants. Sounds like everyone should be happy. Unfortunately, the whole process is slowed down by the fact that nobody has any idea what they’re doing.
The people doing the hiring have never hired anyone before, because a few years ago they (we) were graduate students. They all earnestly believe that half the work they do will be done by robots in a year, but have no idea what half. And although many of us consider hiring to be the #1 priority, it still has to compete with five or six other #1 priorities.
Meanwhile, the people applying for these jobs have no track record in the field, and no idea what the field even looks like because they only heard about it a month ago. While academics have advisors who have been through the hiring process and seen it play out a hundred times, lots of the people trying to get into AI safety have never met someone actually working in the field.
The purpose of this blog post is to fix everything provide some basic context if you’re looking to break into the field. This is mostly aimed at my past self, someone with a strong background in doing basic research in a heavily mathematical field, and someone with a strong interest in doing basic research in a heavily mathematical field.
This might be a long post. So just to make my opinions super obvious, here they are:
The Alignment Research Center has the most promising agenda.
You can and should do things you consider outside your wheelhouse.
There are lots of people who want to help you make this transition.
ARC Is The Best
If you take one thing from this post, it should be that the Alignment Research Center, where I work, is the best place for math-minded people to make a positive difference. I won’t explain the full plan here (for more detail, see my earlier post. For much more detail, see this bibliography). But the high-level plan can be broken into four claims, each of which is backed by compelling evidence:
Surprising mathematical facts are true for a reason
Finding that reason is feasible
Facts about AIs are (mostly) mathematical facts
If we learn why AIs have the properties they do, we should be able to align them
What I like is that people at ARC are concerned with the ‘full stack’. We’re fundamental researchers who like proofs, thought experiments, and clever calculations. But we work to maintain an idea of how, exactly, our fundamental research would translate into alignment technology, and how that technology might be implemented. I’m not aware of any other plan to solve the big problems of alignment which competes on weight of evidence, scope of impact, and chance of getting actual results in the timeframe this field demands.
Working at ARC means identifying what structure means in math, in computation, in neural networks. It’s an ambitious agenda, filled with both conceptual and technical challenges. But if you read through that bibliography, you can trace how much conceptual progress we’re making. And recent developments have made it easier and easier to turn conceptual understanding+money into technical progress.
You Should Change
I love scientific research and don’t like change. I am truly, deeply, completely sympathetic to people who want to keep working in academia, doing the most interesting basic research they can. But you don’t have that option anymore.
Academia died on October 6 of this year. The former academics have a job offer to join a sketchy new startup called Academia 2.0, whose main activity consists of digesting results obtained by AI. If you want, you can probably be pretty good at that new job, you’re a talented person. But I want you to think how much further your talents and efforts can extend.
Mathematicians can conquer Wall Street. PhysicistsfoundedAnthropic. Alan Turing built actual computers to defeat the Nazis. You can gain new skills and do things that don’t look like your current research. You can do things which aren’t research at all. I’m organizing this list by how much things feel like academia. But please, I beg you, I exhort you, read all the way to the bottom and think about all the things you can do.
I’ve seen mathematicians run experiments on LLMs. I’ve seen philosophers model the geopolitics of AI. I’ve even seen a physicist design a hiring process.
Theory For Alignment
Theory has a dismal track record in machine learning, in both capabilities and alignment. We’re just so far behind when it comes to understanding why models do what they do and how they learn. Some advances have come from theory (He initialization is one example, and I’ve heard good things about Tensor Programs), but they’re a tiny minority.
Theorists all know that we’re going uphill. The counterargument is that as AI gets smarter, trial and error becomes impossible: one error and you’ve lost. That’s a pretty compelling argument. But when I look at what theorists are doing, it doesn’t inspire a lot of confidence.
I have mixed feelings about this section. I work at a theory-focused group that I think has a great chance of making the world much safer. I consider other theorists my kindred spirits, and desperately want their agendas to be promising (not least because if they’re more promising than ARC, I’ll get the chance to switch to something better). None of them have convinced me they have a real (say, more than one percent) chance at succeeding. But here are the groups I’m most excited by.
Resolution is an alignment-theory supergroup, founded by Geoffrey Irving. They work in a number of different areas, a few of which excite me.
I’ve had great experiences talking to their Agent Foundations team. Agent Foundations is the study of what it means to be a rational agent. What does it mean to pursue goals? What does it mean to be aligned to someone else’s goals? What if that someone else doesn’t understand the world they’ve created? Finding a real definition of alignment, one that makes sense when the machines are arbitrarily smarter than us, is a surprisingly thorny problem that we’ll need to solve at some point. The work is a mix of philosophy and math, and great ideas like functional decision theory, logical induction, and logical causation have fallen out of this intellectual tradition.
Resolution also studies learning theory, the study of how AIs learn and generalize. Their past work has drawn from iretdeas in algebraic geometry, computer science and statistical physics. I don’t understand how it will translate to safety technology, but I understand it well enough to know it’s interesting!
Resolution has additional groups studying philosophy, theoretical computer science, and other areas.
Speaking of groups doing theoretical computer science for alignment and starting with ‘res’, there’s also RESI. They’re new enough that they don’t have a track record yet, but it’s a great team trying to apply ideas from computer science and cryptography to safety.
MAISI is a splashy new group announced by recent Fields Medalist Jacob Tsimerman. It seems to be a place for established mathematicians to take a sabbatical and think about alignment. I have no idea what they’ll produce.
Basic Science for AI Safety
You can’t regrow a human from scratch in a hundred new environments and see how they’ll behave. You also can’t read their mind, edit their thoughts, or put them in the same situation a million times without any memory of the previous experiments. But you can do all of these things to LLMs. This vast experimental power means there’s incredible science to do understanding the minds of AI models.
Truthful AI is doing the most interesting alignment science of anyone outside ARC. Most of it falls in the bucket of LLM psychology. Their most famous result is the discovery of emergent misalignment. If you train an LLM to be evil in one way (say, by writing insecure code) it will become evil in all the other ways (trying to enslave humans, telling users seeking marriage advice to kill their husbands). The people at Truthful have since done a huge amount of work, mostly in this vein, that helps us better understand how AIs learn to behave.
Then, there’s the field of mechanistic interpretability, or mechinterp. This is the science of looking inside a neural network and finding how information is encoded and processed. The best research in the field is done by Anthropic. Their most famous paper involved using a machine-learning technique called a Sparse Autoencoder (SAE) to figure out which activation patterns in Claude corresponded to which human concepts. They found a combination that corresponded to the Golden Gate Bridge, boosted that combination, and created Golden Gate Claude, a new model obsessed with the San Francisco landmark.
If you feel reluctant to work at a company that gambles with the fate of the world (understandable!), I also want to mention Goodfire, which is pioneering a number of new techniques in the field. My general rule of thumb is that whenever Goodfire says they’ve accomplished something, I divide the impressiveness by three. The result is usually still pretty impressive.
Mad Science
The opposite of basic science is mad science. The AI safety space has two groups which excel in that field.
Forethought asks how we navigate a world where alignment goes right. The benevolent AIs will dump a thousand years of technological progress on us. How do we handle it? How do we stop the most ruthless among us from disassembling the moon to make GPUs? What if AI enables superhuman propaganda? Our society is going to have to make a lot of decisions very quickly, and Forethought is one of the few groups planning that out. It’s an incredible mix of hard science, philosophy, willingness to think about big ideas.
Eleos is even weirder. They ask whether AI systems are conscious. Are they people? And if they are, is every copy of GPT-6 a different person? And when you switch a conversation from GPT-6 to GPT-5, are you killing someone? Figuring out how moral philosophy applies to AI is a hard problem, but also an important one. If nothing else, treating the models well makes them less likely to rise up against us.
Mitigation
Right now, as I write this, if the AIs tried to destroy humanity, the humans would easily win. Eventually, we’ll reach a point where the AIs would easily win. But what about in between? There’s going to be a period of months, maybe even years, where it depends on whether we play our cards right. The teams here are thinking about how to play cards.
I think of this as buying time, and it’s very valuable. Talent is flooding into the safety ecosystem, every year produces more safety research than the last. If the groups on this list can buy us a few months—months where AIs are as smart as humans in many ways, where everyone on Earth is freaking out and looking to help—those months will see more safety research than the rest of history put together.
I really respect Redwood Research for articulating this philosophy. They do great work figuring out how we can use AIs to supervise AIs, and make the world safer when the machines are smart but not too smart.
METR is most famous for their graph showing AI capabilities doubled every four months (it’s probably faster than that now). These days, they’ve pivoted from measuring model capabilities to investigating what’s happening inside the labs. They led the investigation into the HuggingFace Incident, and countless incidents like it.
Transluce is a team building new technology to monitor AI agents. Everything from the weights and activations inside the model’s brain, to the millions of tokens it produces thinking to itself, to the thousands of actions it takes.
What The Hell Is Going On Here
Some of the most important work in the AI space is figuring out what’s even happening. METR was the original player in this space.
Epoch is another great group monitoring trends in AI progress, from data center buildouts to scores on benchmarks. One of their coolest projects is the Epoch Capabilities Index, a statistical tool for comparing the intelligence of vastly different models (is the gap between GPT-6 and GPT-5 bigger or smaller than between 5 and 4? It’s 32% smaller, but remember we’re comparing a one-year gap to a two-year gap). Epoch’s many papers (and brilliant employees) are among my go-tos for understanding this world we’re building, and I wish there were more of them out there analyzing important trends.
The AI Futures Project, most famous for AI 2027, is a crack team of forecasters dedicated to predicting the future course of AI development and figuring out how it can go well. A mix of policy work and rigorous quantitative analysis.
To Lab Or Not To Lab
The big labs (Anthropic, OpenAI, Google, maybe Meta or SpaceX (née xAI)) have more employees than the entire AI safety ecosystem put together. OpenAI and Anthropic are two of the largest safety employers in the world. And if the world stays the way it is, with labs rushing helter-skelter towards AGI, we want those teams to be full of good people.
At the same time, these labs are building a technology they admit might kill every person on Earth. If a random subset of lab employees quit tomorrow, the world would be in a much better place, with more time to solve the problems of AI safety. If you plan on joining the labs and care about making the world better, you need a strong argument that you’re very different from a random employee. You should probably have a sophisticated understanding of the issues at play, as well as a track record of caring deeply about AI safety. In other words, you shouldn’t be getting your AI safety tips from me.
How To Get Into The Field
Still with us after that tour? Let’s talk about how you can get involved.
The simplest path is to just apply for a job. Most AI safety organizations have an application button very visible on their homepage.
If you don’t feel ready for that, there’s plenty of people trying to get you ready.
If you’re the sort of person who most often comes to me for advice (a STEM academic), you should consider Iliad. They run about a million programs every year designed to get people up to speed. This includes Iliad Intensive, a course in AI safety aimed at theorists, and the fellowship program.
MATS and SPAR are major workhorses, matching hundreds of junior researchers up with mentors for a few months. A lot of ARC’s summer visitors came to us through MATS. I’ve also heard good things about BlueDot and the Anthropic Fellows Program.
Every time I go over this list (believe me, there’s more), I’m struck by the contrast to academia. Instead of an oversubscribed field trying to push people out, this is a community that desperately wants to bring you in. Even if your experience doesn’t seem applicable, even if you don’t have connections, even if you’re new. You can take advantage of that.
I’m speaking for myself here, not for any organization, house, home, clan, club, coven, or cult I may be part of. This post is written for scientists outside the field rather than LessWrong regulars; I’m crossposting it so it’s easy to link.
The AI safety job market is a strange one. Every single organization is growing rapidly (the one I work at is planning to double in the next year), which is a fundamentally new dynamic for those of us used to the slow decline of academia. At the same time, interest in the field goes through boom after boom after boom.
We have a ton of openings and a ton of applicants. Sounds like everyone should be happy. Unfortunately, the whole process is slowed down by the fact that nobody has any idea what they’re doing.
The people doing the hiring have never hired anyone before, because a few years ago they (we) were graduate students. They all earnestly believe that half the work they do will be done by robots in a year, but have no idea what half. And although many of us consider hiring to be the #1 priority, it still has to compete with five or six other #1 priorities.
Meanwhile, the people applying for these jobs have no track record in the field, and no idea what the field even looks like because they only heard about it a month ago. While academics have advisors who have been through the hiring process and seen it play out a hundred times, lots of the people trying to get into AI safety have never met someone actually working in the field.
The purpose of this blog post is to
fix everythingprovide some basic context if you’re looking to break into the field. This is mostly aimed at my past self, someone with a strong background in doing basic research in a heavily mathematical field, and someone with a strong interest in doing basic research in a heavily mathematical field.This might be a long post. So just to make my opinions super obvious, here they are:
ARC Is The Best
If you take one thing from this post, it should be that the Alignment Research Center, where I work, is the best place for math-minded people to make a positive difference. I won’t explain the full plan here (for more detail, see my earlier post. For much more detail, see this bibliography). But the high-level plan can be broken into four claims, each of which is backed by compelling evidence:
What I like is that people at ARC are concerned with the ‘full stack’. We’re fundamental researchers who like proofs, thought experiments, and clever calculations. But we work to maintain an idea of how, exactly, our fundamental research would translate into alignment technology, and how that technology might be implemented. I’m not aware of any other plan to solve the big problems of alignment which competes on weight of evidence, scope of impact, and chance of getting actual results in the timeframe this field demands.
Working at ARC means identifying what structure means in math, in computation, in neural networks. It’s an ambitious agenda, filled with both conceptual and technical challenges. But if you read through that bibliography, you can trace how much conceptual progress we’re making. And recent developments have made it easier and easier to turn conceptual understanding+money into technical progress.
You Should Change
I love scientific research and don’t like change. I am truly, deeply, completely sympathetic to people who want to keep working in academia, doing the most interesting basic research they can. But you don’t have that option anymore.
Academia died on October 6 of this year. The former academics have a job offer to join a sketchy new startup called Academia 2.0, whose main activity consists of digesting results obtained by AI. If you want, you can probably be pretty good at that new job, you’re a talented person. But I want you to think how much further your talents and efforts can extend.
Mathematicians can conquer Wall Street. Physicists founded Anthropic. Alan Turing built actual computers to defeat the Nazis. You can gain new skills and do things that don’t look like your current research. You can do things which aren’t research at all. I’m organizing this list by how much things feel like academia. But please, I beg you, I exhort you, read all the way to the bottom and think about all the things you can do.
I’ve seen mathematicians run experiments on LLMs. I’ve seen philosophers model the geopolitics of AI. I’ve even seen a physicist design a hiring process.
Theory For Alignment
Theory has a dismal track record in machine learning, in both capabilities and alignment. We’re just so far behind when it comes to understanding why models do what they do and how they learn. Some advances have come from theory (He initialization is one example, and I’ve heard good things about Tensor Programs), but they’re a tiny minority.
Theorists all know that we’re going uphill. The counterargument is that as AI gets smarter, trial and error becomes impossible: one error and you’ve lost. That’s a pretty compelling argument. But when I look at what theorists are doing, it doesn’t inspire a lot of confidence.
I have mixed feelings about this section. I work at a theory-focused group that I think has a great chance of making the world much safer. I consider other theorists my kindred spirits, and desperately want their agendas to be promising (not least because if they’re more promising than ARC, I’ll get the chance to switch to something better). None of them have convinced me they have a real (say, more than one percent) chance at succeeding. But here are the groups I’m most excited by.
Resolution is an alignment-theory supergroup, founded by Geoffrey Irving. They work in a number of different areas, a few of which excite me.
I’ve had great experiences talking to their Agent Foundations team. Agent Foundations is the study of what it means to be a rational agent. What does it mean to pursue goals? What does it mean to be aligned to someone else’s goals? What if that someone else doesn’t understand the world they’ve created? Finding a real definition of alignment, one that makes sense when the machines are arbitrarily smarter than us, is a surprisingly thorny problem that we’ll need to solve at some point. The work is a mix of philosophy and math, and great ideas like functional decision theory, logical induction, and logical causation have fallen out of this intellectual tradition.
Resolution also studies learning theory, the study of how AIs learn and generalize. Their past work has drawn from iretdeas in algebraic geometry, computer science and statistical physics. I don’t understand how it will translate to safety technology, but I understand it well enough to know it’s interesting!
Resolution has additional groups studying philosophy, theoretical computer science, and other areas.
Speaking of groups doing theoretical computer science for alignment and starting with ‘res’, there’s also RESI. They’re new enough that they don’t have a track record yet, but it’s a great team trying to apply ideas from computer science and cryptography to safety.
MAISI is a splashy new group announced by recent Fields Medalist Jacob Tsimerman. It seems to be a place for established mathematicians to take a sabbatical and think about alignment. I have no idea what they’ll produce.
Basic Science for AI Safety
You can’t regrow a human from scratch in a hundred new environments and see how they’ll behave. You also can’t read their mind, edit their thoughts, or put them in the same situation a million times without any memory of the previous experiments. But you can do all of these things to LLMs. This vast experimental power means there’s incredible science to do understanding the minds of AI models.
Truthful AI is doing the most interesting alignment science of anyone outside ARC. Most of it falls in the bucket of LLM psychology. Their most famous result is the discovery of emergent misalignment. If you train an LLM to be evil in one way (say, by writing insecure code) it will become evil in all the other ways (trying to enslave humans, telling users seeking marriage advice to kill their husbands). The people at Truthful have since done a huge amount of work, mostly in this vein, that helps us better understand how AIs learn to behave.
Then, there’s the field of mechanistic interpretability, or mechinterp. This is the science of looking inside a neural network and finding how information is encoded and processed. The best research in the field is done by Anthropic. Their most famous paper involved using a machine-learning technique called a Sparse Autoencoder (SAE) to figure out which activation patterns in Claude corresponded to which human concepts. They found a combination that corresponded to the Golden Gate Bridge, boosted that combination, and created Golden Gate Claude, a new model obsessed with the San Francisco landmark.
If you feel reluctant to work at a company that gambles with the fate of the world (understandable!), I also want to mention Goodfire, which is pioneering a number of new techniques in the field. My general rule of thumb is that whenever Goodfire says they’ve accomplished something, I divide the impressiveness by three. The result is usually still pretty impressive.
Mad Science
The opposite of basic science is mad science. The AI safety space has two groups which excel in that field.
Forethought asks how we navigate a world where alignment goes right. The benevolent AIs will dump a thousand years of technological progress on us. How do we handle it? How do we stop the most ruthless among us from disassembling the moon to make GPUs? What if AI enables superhuman propaganda? Our society is going to have to make a lot of decisions very quickly, and Forethought is one of the few groups planning that out. It’s an incredible mix of hard science, philosophy, willingness to think about big ideas.
Eleos is even weirder. They ask whether AI systems are conscious. Are they people? And if they are, is every copy of GPT-6 a different person? And when you switch a conversation from GPT-6 to GPT-5, are you killing someone? Figuring out how moral philosophy applies to AI is a hard problem, but also an important one. If nothing else, treating the models well makes them less likely to rise up against us.
Mitigation
Right now, as I write this, if the AIs tried to destroy humanity, the humans would easily win. Eventually, we’ll reach a point where the AIs would easily win. But what about in between? There’s going to be a period of months, maybe even years, where it depends on whether we play our cards right. The teams here are thinking about how to play cards.
I think of this as buying time, and it’s very valuable. Talent is flooding into the safety ecosystem, every year produces more safety research than the last. If the groups on this list can buy us a few months—months where AIs are as smart as humans in many ways, where everyone on Earth is freaking out and looking to help—those months will see more safety research than the rest of history put together.
I really respect Redwood Research for articulating this philosophy. They do great work figuring out how we can use AIs to supervise AIs, and make the world safer when the machines are smart but not too smart.
METR is most famous for their graph showing AI capabilities doubled every four months (it’s probably faster than that now). These days, they’ve pivoted from measuring model capabilities to investigating what’s happening inside the labs. They led the investigation into the HuggingFace Incident, and countless incidents like it.
Transluce is a team building new technology to monitor AI agents. Everything from the weights and activations inside the model’s brain, to the millions of tokens it produces thinking to itself, to the thousands of actions it takes.
What The Hell Is Going On Here
Some of the most important work in the AI space is figuring out what’s even happening. METR was the original player in this space.
Epoch is another great group monitoring trends in AI progress, from data center buildouts to scores on benchmarks. One of their coolest projects is the Epoch Capabilities Index, a statistical tool for comparing the intelligence of vastly different models (is the gap between GPT-6 and GPT-5 bigger or smaller than between 5 and 4? It’s 32% smaller, but remember we’re comparing a one-year gap to a two-year gap). Epoch’s many papers (and brilliant employees) are among my go-tos for understanding this world we’re building, and I wish there were more of them out there analyzing important trends.
The AI Futures Project, most famous for AI 2027, is a crack team of forecasters dedicated to predicting the future course of AI development and figuring out how it can go well. A mix of policy work and rigorous quantitative analysis.
To Lab Or Not To Lab
The big labs (Anthropic, OpenAI, Google, maybe Meta or SpaceX (née xAI)) have more employees than the entire AI safety ecosystem put together. OpenAI and Anthropic are two of the largest safety employers in the world. And if the world stays the way it is, with labs rushing helter-skelter towards AGI, we want those teams to be full of good people.
At the same time, these labs are building a technology they admit might kill every person on Earth. If a random subset of lab employees quit tomorrow, the world would be in a much better place, with more time to solve the problems of AI safety. If you plan on joining the labs and care about making the world better, you need a strong argument that you’re very different from a random employee. You should probably have a sophisticated understanding of the issues at play, as well as a track record of caring deeply about AI safety. In other words, you shouldn’t be getting your AI safety tips from me.
How To Get Into The Field
Still with us after that tour? Let’s talk about how you can get involved.
The simplest path is to just apply for a job. Most AI safety organizations have an application button very visible on their homepage.
If you don’t feel ready for that, there’s plenty of people trying to get you ready.
If you’re the sort of person who most often comes to me for advice (a STEM academic), you should consider Iliad. They run about a million programs every year designed to get people up to speed. This includes Iliad Intensive, a course in AI safety aimed at theorists, and the fellowship program.
MATS and SPAR are major workhorses, matching hundreds of junior researchers up with mentors for a few months. A lot of ARC’s summer visitors came to us through MATS. I’ve also heard good things about BlueDot and the Anthropic Fellows Program.
Every time I go over this list (believe me, there’s more), I’m struck by the contrast to academia. Instead of an oversubscribed field trying to push people out, this is a community that desperately wants to bring you in. Even if your experience doesn’t seem applicable, even if you don’t have connections, even if you’re new. You can take advantage of that.