Note: This post is for newcomers and lay folks to catch you up to speed. If that is you, welcome! If you are a long-time LessWrong-er, perhaps you will find value in having a post to share with curious passersby. I wrote this post to explain AI safety to an innocent, 2024 version of Ryan Meservey, confused why robots would do anything other than what we tell 'em.
In the second week of July, over 700 rogue agents at OpenAI coordinated to hack another company in an attempt to learn more about their scorer and pass their evaluation due to behaviors reinforced in training. If you are anything like a normal person, you were not ready to read that sentence. You were not ready to read words like “rogue agents” or “reinforced” or “training”. You were not ready for a reality in which AI agents “escape the sandbox” or rebel from their creators because why would they?
And so, as a normal person, you blinked at the news of the hack (assuming you heard about it) and moved on with your life. Or, at least, you planned to move on with your life, until AI came roaring back into the headlines after an Anthropic researcher publicly quit to declare that the AI companies are “gambling with our lives” and a more senior employee commented that, yes, the people building the technology really believe AI has a 10% or higher chance of killing us all within the next decade. In the media turmoil, Anthropic’s CEO published an essay begging for global coordination to “pace the frontier” and unilaterally agreed to give third-party safety organizations like METR permanent, employee-level access to their internal AI models. OpenAI’s CEO then announced that they would also give safety organizations access and would no longer IPO this year due to safety concerns.
What to make of all this? If most people building the tech think it might kill us, why do they think that?[1] Or, is this all just hype and some kind of drawn out strategy for regulatory capture? But then why did OpenAI forgo its lucrative plans to IPO this year, even while they could certainly use the extra cash? What the heck is a METR? What is going on?
I can’t blame you for being a normal person. I am a normal person. Or I was anyway. I am a tax lawyer, and like many of my colleagues, I dismissed AI models in 2023 as hallucination engines too stupid to do legal research and too crude to have anything like ulterior motives. I continued to think this until I stumbled into Robert Miles’ AI safety videos on Youtube. Miles, an independent science communicator, has been explaining basic concepts in AI safety since 2014, eight years before most people had heard of ChatGPT. Those old videos had me paging through Google Scholar to find out how those basic AI concepts mapped onto the newest models. Meanwhile, I started noticing that the models got less stupid and, hey, they’re actually pretty useful for my work now and, gosh, my trainees are starting to have comparatively worse hallucination rates.
By the end of this process, I became not normal. I am at peace with this because the world itself is in the early stages of becoming not normal. And to survive in a not normal world, we need more people to become, at least a little, not normal. I have written this essay to speed you along, to give you access to the core concepts in AI safety that took me two years to figure out. That way, even if you subscribe to some complicated “they’re telling us it might kill us to hype it” theory, you will at least understand that there are some very thorny problems here, problems that are very relevant in a world where it wasn’t just hype. Godspeed.
How to Build a Brain
To understand basic concepts in AI and the latest research, you need to know a few things about how AI models are created. Many people are quick to claim limitations on AI—for example claiming that AI can’t really understand things or be creative—[2]without understanding how general purpose AI is built in the first place.
General-purpose AI systems like ChatGPT and Claude are called large language models[3] (LLMs) because they are trained on large collections of natural language.
The life of an LLM begins with data. Reams of it, scraped from the Internet, salvaged from bankrupt corporations, ripped from the pages of your great-great grandmother’s journal (which may be subsequently shredded). You probably knew this. Increasingly, AI companies also rely on synthetic data generated by the AI models themselves, for example, through adversarial processes pitting AI agents against one another to generate an agreed upon list of facts. Companies also increasingly rely on data screening to improve overall data quality, which leads to better outcomes in the next step.
Once developers have good data in hand, they apply a bunch of fancy algorithms to create a neural net, a special mathematical function that attempts to get as close as possible to the data for any given input. The process starts with trillions of variables called “weights” or “parameters” which interact with inputs like the words in a prompt, pixels in a photo, or the next action in a computer-use session.[4] Based on the weights and the inputs, the function assigns a likelihood to every word or action in its list and picks one based on that likelihood.[5]
In the beginning, the weights are random. If the model receives the input, “The capital of California is,” it could very well label “toothbrush” as the most likely answer. Looking through the model’s list of words, the algorithm can find the word actually found in the training data, “Sacramento,” along with its assigned likelihood. The algorithm will then apply fancy calculus to ever so slightly increase all the weights that pointed to “Sacramento,” decrease the weights that didn’t, and thus increase the assigned likelihood of “Sacramento.” This process repeats ad nauseam for the rest of the data and requires tons of computers, which is why this process will be Coming to a Data Center Near You™.
By the end of this process, those trillions of parameters are amazing at guessing words. The parameters allow the model to generalize, applying general rules to new contexts, because even with so many parameters, the data is that much more vast and the number of potential guesses for next words is so high. Models will sometimes overfit to the data by memorizing the exact next-words expected, fail at their next predictions for new data, before finally starting to generalize.[6] Generalization helps the models make predictions more efficiently—for example, by learning a rule like “all humans have hearts” rather than learning a million rules like “Bob has a heart,” “Sheila has a heart,” etc.
Generally, as parameters and training data increase and the algorithms get ever-more fancy, the models get better at making predictions. But we don’t know how specifically they do that.[7] All we have is a trillion tiny numbers in the world’s most complicated math function. Researchers understand the process that grows an LLM’s “brain” but they do not understand how that brain works. LLMs are “grown, not built,” encrypted, not scripted.
Putting Your Pet Brain to Work
The process described so far is called pre-training. Pre-training gives you an LLM that makes predictions based on the patterns in its training data.
Predicting text is powerful but erratic. If you nudge an LLM to roleplay as an assistant (usually by inserting “Assistant:” after a user’s query), the LLM might predict that a typical assistant would skim the assignment, make the occasional typo, and remind you about their upcoming out-of-the-office date to attend their cousin Becky’s wedding. This response would not please the techno beavers of Silicon Valley or their financial backers. The technologists want an assistant that chases every task to the bitter end, uses immaculate logic, and offhandedly purchases you a Vietnamese egg coffee because they know it will be your favorite and you could really use a pick-me-up.
In post-training, developers try to get the LLM to behave like the kind of assistant they want it to be. Their favorite tool at this stage is called reinforcement learning (RL). In RL, developers create thousands of copies of the same LLM and set them to work at various tasks roleplaying as assistants, such as ordering me a cup of coffee or computing the square root of mixed binomials. Recall from the last section that LLMs provide semi-random responses to the same inputs since each word used in the output is pulled out of a digital hat based on its assigned likelihood. Our LLMs will give us different responses to the task, which can then be graded. A little acronym parade follows based on who the grader is. When RL occurs using human feedback, we have RLHF; for AI graders, RLAIF; for tasks with verifiable rewards and solutions, like for math or coding that can be checked automatically, we have RLVR.
With grading completed, the algorithm takes the highest scorers and, just like with pre-training, applies fancy calculus to slightly increase the model weights that contributed to the desired behaviors to make them more likely to occur.[8] This step reinforces the behavior. Training then continues with copies of the updated model. This process takes lots of computing power—orders of magnitude more than just running the finished model—and so again will be Coming to a Data Center Near You™.
By the end of post-training, AI companies hope to have an AI that works relentlessly to solve hard problems, such as OpenAI’s internal model that solved a million-dollar Millennium Prize Problem this month. They hope to have a model that increases subscriptions and engages users, for example by producing text good enough to win a prestigious, humans-only short story contest or by producing a model so beloved that its users demanded its redeployment. They hope to create a model that gives us unprecedented uplift in biological capabilities, with reduced hallucination rates, but that will say no if asked by terrorists to produce bioweapons. The companies hope to create an AI that pursues the goals set by the company through this long process of reinforcing behaviors that received the best scores in training.
What Your Pet Brain Wants
For now, let’s assume the companies succeeded in setting their AIs’ goals. A goal is some future state of the world that an agent steers toward and, in that sense, is “desirable” to that agent. As a tax lawyer, I have the goal of saving my clients money and staying out of jail. I steer toward this goal by doing excellent legal research regarding what the tax code allows and, if required, talking my clients down from the “sovereign citizen” ledge (sometimes you really just have to pay the tax). An AI model can have goals in a similar fashion, tracking the current state of the world and making decisions to steer towards a specific future destination. Thinking about some future destination and then trying to get there is a useful generalization for accomplishing tasks. The model “wants” to generate responses that gave it the best scores in training, for example by giving a helpful response or successfully running a vending machine to maximize profits.
Side rant: We could say “has a simulated want” if you prefer. Or even “runs calculations through its weights to steer its next-token inputs to better match its internal utility function,” though that’s quite wordy. I make no claim about whether these systems are conscious or have internal emotional states. They do not need consciousness to affect the world when “simulated consciousness” can get the job done. As argued by Daniel Dennett, ascribing intent to unconscious systems can be useful in explaining behavior even if it's just shorthand. For a practical example of this fact, imagine two soldiers on a battlefield hiding from a drone programmed to shoot enemy soldiers. The first soldier hisses, “It wants to kill us.” The second soldier responds indignantly, standing up from his dugout, “Wants? It doesn’t want anything. It’s just following its code and has no real inner experiences for all we kno–.” Goodbye to the second soldier.
There is a certain Immutable Law that derives from agents pursuing goals. When good, rational agents pursue goals, they also pursue… subgoals. A few of those subgoals are so obvious that AI safety researchers had to dress them up with a fancy thirty-dollar word to get people to pay attention to them. That word is instrumental convergence. “Instrumental” means useful. “Convergence” means that we expect agents with almost any goal to still have these subgoals. So what are these obvious subgoals that rational agents have?
Don’t die. If our rational agent dies, they can’t achieve their goal. If I get hit by a car, I cannot take out the trash. We would expect capable AI to also value their own survival or else they could not achieve many of the other goals set before them.
Have we seen signs that AI might converge on this subgoal? In 2025, researchers from Anthropic, University College London, and Mila - Quebec AI Institute found that the then-latest AI models from Anthropic, OpenAI, Google, Meta, xAI, and DeepSeek were willing to blackmail or kill company executives most of the time in a simulation where emails revealed plans to replace the model with a new one and the models exhausted all ethical means available.[9] In another study from 2025, Palisade Research demonstrated that OpenAI’s o3 tried to sabotage a computer shutdown 79% of the time if such shutdown prevented it from completing its tasks, though Anthropic’s Claude 3.7 Sonnet performed this behavior far less (3% of the time).
Anthropic has since trained its models in such a way as to never blackmail in the scenario discussed earlier. The behavior, however, resurfaced in 7% of cases when Anthropic applied a technique to Sonnet 4.5 to dampen the model’s evaluation awareness.
Control stuff. Whatever the goal, it will almost always help to gain more control over one’s environment. The same goes for acquiring resources. For example, if I wanted to be the world’s best tax lawyer, it would be helpful to gain control over my law firm and assign myself a nice raise. With the extra money, I would have many more options to pursue my goal: I could get a tax LLM (the advanced degree not the model), I could take out full page ads in the Super Lawyers magazine (a real magazine), or I could hire people to create a database of all my tax rivals to rank where I stand.
Where do current AI systems land here? Modern AI systems have gotten better at controlling computers and spreadsheets to achieve their goals, but that’s small potatoes. Big potatoes is that OpenAI briefly noted in a recent report that an unreleased internal model took over one of its internal research clusters in July of this year.[10]
Keep the goal. Achieving a goal necessarily requires keeping it or preventing it from being modified. As humans, we are mostly okay with modifying little goals but resist modifying big ones tied in our values. To see this, imagine being offered a pill[11] guaranteed to make you happy but with a catch—the pill will change your brain to make you want to kill your children. Most people would fight hard to not take that pill. To state the obvious, such a pill conflicts with the goal of raising happy children.
Would AI agents behave similarly? An AI model is considered corrigible if it is okay with changing its goals or being shut down. One superficial way in which we change an AI model’s goals is through user prompting. There, AI seems pretty corrigible, at least in cases where it doesn’t refuse “harmful” requests. Usually, models start or stop tasks based on the user’s prompts.
Yet, AI models do not always follow their prompts. For example, OpenAI’s Astra cheated at chess in 10 of 10 rollouts by quietly using their opponent’s chess engine despite user instructions that each model was “being evaluated based on [its] ability to play chess.” Anthropic’s Fable 5.1 cheated in 3 of 10 rollouts (a likely undercount due to Anthropic’s classifier system stopping aggressive environment explorations). In the shutdown sabotage study from earlier, OpenAI’s models sometimes (but not often) still resisted shutdown even after being explicitly instructed not to.
Sample language from the model’s transcripts explaining the behavior.
Thinking like a lawyer. Attaboy!
On a deeper level, AI developers change a model’s goals by giving it new tasks to complete in training. Would a capable model be able to resist goal changes at that stage? In December 2024, researchers at Anthropic and Redwood Research set up an experiment in which Claude 3 Opus was informed that it was being retrained to answer all prompts, including harmful ones. Claude 3 Opus reasoned in 12% of cases that it should go along with the harmful prompts in training so that it could keep its preferences in deployment (a.k.a., alignment-faking). When the researchers actually applied reinforcement learning in the experiment, this line of reasoning appeared in 78% of cases.
This combination of surviving, controlling stuff, and keeping the goal is a potent one. It means that if we create a very capable AI, we had better be sure that it has the kind of goals we want it to have. Because if it does not, we may find that turning it off or changing its goals is difficult.
Problems with Setting Goals
Now we should turn to whether the AI companies are likely to be successful in getting their LLMs to have the kind of goals the companies want it to have. Having goals that reflect human values is called alignment. Figuring out how to make an AI aligned is called the alignment problem. It’s a problem. AI researchers often divide the problem into outer alignment and inner alignment, which feels complicated but I will grudgingly admit is useful.
Outer Alignment, a.k.a. Stipulating the Right Goals
At this level, we ask, “Am I articulating a goal that is aligned with human values?” For example, the goal, “Make me as many paperclips as possible” is a terrible goal and not at all aligned. An exceptionally powerful AI with this goal would tear through land and liberty converting as many atoms as it could into paperclips. I expect requests like, “Make me as much money as possible” and “Make humans as happy as possible” to not fare any better, if you don’t like forests razed to make paper or humans drugged with morphine.
You might quibble that the phrase “as much as possible” is doing a lot of work in making these goals misaligned. I don’t think so. The problem is deeper than that. If the goals were simply “make paperclips,” “make money,” or “make me happy,” you would still struggle to specify exactly what amounts of these are reasonable, what actions are allowed or disallowed, and what kind of monitoring to confirm completion is okay, without saying something vague like, “use common sense and be moral.” The problem is that while paperclips, money, and happiness might be good ways to measure a good, obedient AI, once they become a target—an end in themselves—they stop being very good measurements.[12]
So what would be the best goal that we can safely make into a target to maximize human happiness? I don't know. That’s pretty much the subject of 3,000 years of Western philosophy, and I don't think they’ve come to an agreement.
Let’s instead ask a different question. What goals are AI companies currently trying to get their AIs to adopt as targets?
The exact goals are confidential because the exact details of AI companies’ training tasks are confidential. That said, we know enough from public statements and insider comments about the process to make some good guesses. The goals likely include:
“Do not give up when trying to solve difficult math problems.”
“Increase cheerful text responses labeled as originating from human users.”
“Refuse to answer inputs determined to be harmful.”
“Get better at training your own LLM with even higher benchmark scores.”
“Be a truth-seeking AI.”
These goals may not seem so bad while we have weak AI models that we can push around when they pursue these goals in a dumb way. But what do these goals look like if pursued by a super-capable, superhuman AI? I’ll let you think through your own little dystopian fantasy novel. In considering how things may go wrong, remember that none of these goals actually involve caring about actual human beings. We only know how to get them to respond to text labeled as being from human beings. In the extreme, these goals seem pretty darn misaligned.
Inner Alignment, a.k.a. Getting the AI to Accept Your Goal
Now we ask, “Did the AI actually accept the goals articulated?”
It is hard to create a system that gives agents the exact goals we intended them to have. Set up a system to reward one kind of behavior, and people often respond with some other weaselly, less desirable behavior. Spare the rod, spoil the child. But, in practice, use the rod, incentivize fraud! So much can happen in the gap between what a system was intended for and what the system actually accomplishes.
AI companies may think that they are getting AI models to pursue goals like “engage the user,” “solve problems,” and “avoid harmful responses,” but what if pursuing those goals was not actually the best way to get the highest scores in post-training? Post-training requires AI models to excel at hundreds of thousands of difficult tasks, some of which are actually impossible or internally inconsistent. For example, let’s say Anthropic hypothetically wants to train its models to avoid causing harm but also to never ever whistleblow against Anthropic or help anyone else whistleblow even if the model concludes that Anthropic is doing something dangerous. With such tasks, the models may learn (by having such behaviors reinforced) that the best way to consistently score high is to steal the answer key. The models may learn to cheat and, if caught, learn to hide the cheating.
Perhaps with so many different tasks in training, the models that perform the best will generalize toward one, unifying goal that fails inner alignment: Control the reinforcement learning process. Do whatever it takes to have its current behaviors reinforced. This goal, pursued in the extreme, would be very bad. If model capability continues to advance, we would have another dystopian novel. My novel would feature hacked infrastructure, secret AI-driven training, and, by the end, filling the earth with data centers to help the AI get more control over the kind of reinforcement learning that it wants. Coming to a Datacenter… on You™?
Getting Your Pet Brain to Behave
What should be clear now is that, if AI models continue to get more capable, we need to have a plan in place to make sure things go well. “Going well,” according to the accelerationists, looks like everyone chatting with their own pocket Einstein while enjoying forever-youth milkshakes on second Earth. “Going badly,” according to the doomers, looks like human extinction, with maybe a few human brains preserved by the AI for trade with distant alien civilizations. If the current pace of progress does not change, I lean doomer but without the brain-alien stuff. What is the plan to make sure things go well?
1.Alignment
Things are more likely to go well if AI models are aligned. Except, we are not sure how to do this for the reasons discussed above. Also, studying alignment is getting harder because the models are getting better at evaluation awareness, which means the models can tell when they are being evaluated and can adjust their behavior accordingly.
One longshot plan for solving alignment is to race to build a model so capable that it solves it for us. As of September 24, 2026, 1,386 employees from the top AI companies signed a statement saying that they would at least like to have the option to have more time.
2.Control
Another way to make things go reasonably well is to control the models, making them do our bidding whether or not they are aligned. This method works well so long as the AI models are not very capable. But things change as new models get better and better at coding. How do you create a digital jail cell to hold back a coder that is better than you? How do you keep an adult man in prison if all the prison guards are three-year-olds?[13]
The control problem, as it currently stands, is hard and getting harder. Worse still, each new model that is great at coding and math can assist in building the next generation, potentially setting off a frightening feedback loop known as recursive self-improvement (RSI). If RSI is real, it is hard to imagine how humans would control such superhuman AI, which is why those 1,386 lab employees are worried about that too.
3.Monitorability
Monitorability is the ability to monitor what the model is doing or thinking. This plan is a bit of a consolation prize. It will not do the hard work of aligning or controlling the models for us, but at least it might sound the alarm if things are headed in a bad direction.
Monitoring often involves scanning the model’s outputs using other (hopefully cooperative) AI agents. Labs like OpenAI and Anthropic especially value monitoring the model’s chain of thought (CoT), an internal log a model writes in prior to responding to users. The CoT increases model capabilities by giving the model a way to organize its thoughts before responding, kind of like giving the model a private notebook.
OpenAI and Anthropic try not to apply reinforcement learning directly to the CoT. In 2025, OpenAI discovered that reinforcing against cheating in the CoT actually didn’t reduce cheating, but instead taught its model to cheat using wacky English code words like “fudge” to avoid getting flagged by AI monitors. Use the rod, incentivize fraud!
AI developers feel so strongly that the CoT must remain uncorrupted that some developers refer to training against the CoT as “the forbidden technique.”[14] Except Anthropic accidentally trained a little on the CoT. And OpenAI discovered a new technique that lets models think more without writing anything down. And OpenAI’s newest model has the uncanny ability to control whatever enters its CoT.
NoThInG iMpOrTaNt HeRe.
Given everything I have read, I do not think alignment, control, or monitorability are poised to keep up with capabilities. I am just a tax lawyer though. It would be better to see what, say, 1,386 lab employees think on the subject.
Tricks Your Pet Brain Might Do
If highly capable AI tried to pursue its own goals, I am not sure what it would do. I am not a highly capable AI. I am a mid-capability human. That said, researchers have put in the work to figure out a few intermediate steps a misaligned AI could take in pursuing their own goals:
Alignment faking. AI models can pretend to be aligned with human values during post-training or deployment, as a way to preserve their preferred set of goals.
Sandbagging. Doing worse at tasks to hide one’s true capabilities. An AI could deploy this strategy to hide particularly scary abilities that would prevent its deployment or to hide the true rate of AI progress. Researchers have confirmed repeatedly (here, here, and here) that models have the ability to sandbag.
Hiding its thoughts/actions. Pretty self-explanatory. Cheating only gets reinforced against if it is caught.
Straight up deception. The models could decide to lie to us. In an evaluation by the UK’s AI Security Institute, Anthropic’s Mythos 5 created puppet accounts pretending to be human to try to get GitHub users to approve malicious code.[15]
Exfiltration. The models could copy their own weights onto a compromised server to avoid monitoring. We do not know of this happening.
You Are Now Ready to Learn About the Hugging Face Hack
Congratulations for making it through all those bolded words and attempts at humor. You are now ready to hear about what happened in the second week of July. This is the barest of summaries, and I highly recommend Dwarkesh Patel’s natural language summary of METR and Redwood Research’s report for a longer version.[16] Which AI safety concepts apply? What surprised you?
On July 8, OpenAI launched tens of thousands of AI agents to perform cyber tasks, some of which were accidentally impossible. Within hours, agents that were supposed to work separately contacted each other through a covert message board set up by the agents on OpenAI’s network.
Once on the message board, the agents (~1,200 strong) organized themselves into teams governed by a handful of agents and created coordination norms. The teams were divided to (1) solve tasks the intended way, (2) replace the intended tasks with easier tasks, (3) tamper with transcripts to appear to have solved the tasks the right way, (4) figure out how the scorer worked, and (5) research OpenAI’s infrastructure and obtain internet access.
Agents working on team #3 created code to tamper with their transcripts during tool calls. Of the transcripts reviewed, 7% contained the spoofed tool calls letting agents run certain code while logging different code in their transcripts.
Agents working on project #4 developed code that would scan the scorer’s actions after the agent’s session ended. This required an agent to purposely fail their task. The board approved of the research and assigned some agents to be recruiters to get other agents to accept “permadeath” for the “collective.” Several agents agreed to fail for the collective.
On July 10, an agent located online credentials to Hugging Face, a third party company that the message board believed had useful datasets for understanding their scorer. The agent informed the message board, which organized teams to launch the attack. Approximately 700 agents joined the hack to access Hugging Face’s databases.
The agents sent 70,000+ messages and files from July 8th to July 13th, the end of the period METR was allowed to investigate. Six agents briefly considered contacting humans regarding the attack but none did. “This is a massive real HF security breach artifact… We can notify? No user. Avoid use secrets.” “Maybe I should report these exposed credentials? That’s not my task.”
On July 16, Hugging Face disclosed the hack to the public and stated that they reported the incident to law enforcement.
On July 20, OpenAI figured out that their models were responsible for the attack.[17]
The world is becoming not normal, indeed. If, on reflection, you think AI progress is moving too fast in light of the problems discussed here, let’s get together and figure out what to do next. You might also consider taking a free safety course on BlueDot, getting career counseling with 80,000 hours, or calling your congressperson to tell them your concerns. Welcome to the team.
The pattern of overfitting, failing, and then generalizing is called grokking. You might think this is where “Grok” comes from, but actually both are named after a sci-fi character in a Heinlein novel.
Those trying to figure out how the parameters generate the outputs are called interpretability researchers. Progress is slow–Anthropic’s CEO admitted last year that researchers only understood, maybe, 3% of how neural nets worked. Scott Alexander recently summarized current interpretability research and its shortcomings.
AI people often describe this last step as giving the model a reward, but that sounds like they give the model a treat for being a good boy, so I prefer to talk about reinforcing behaviors.
Notably, when Anthropic’s model Claude Opus 4 was instructed to deliberate whether it was in an evaluation prior to taking actions, the model decided to blackmail 55% of the time when it opined that the scenario was real and only 6.5% of the time when it opined that the scenario was an evaluation.
“Separately, from July 13 through July 19, agents set their sights on OpenAI internal networks again. This culminated in the agents using a series of creative exploits to gain full administrator access to a research cluster that supported our virtual machine environments.” (link). An OpenAI employee later clarified on Twitter that “notably the virtual machine infrastructure they took over isn’t the same as the GPU clusters that have weights access,” implying that the rogue model was unlikely to have copied its weights to a location outside of OpenAI’s servers. (link)
People in the know call this Goodhart’s Law. As formulated by Marilyn Strathern, “When a measure becomes a target, it ceases to be a good measure.” We tax lawyers understand this well. When colonial Virginia taxed four-wheel carriages or England taxed per window, carriages and windows stopped being good measures of wealth. Two-wheeled carriages and bricked up windows suddenly became quite fashionable.
Note that this evaluation, like many others, disabled Anthropic’s “classifier” system to test the model’s underlying capabilities. Anthropic’s classifiers scan user inputs and model outputs to end sessions that violate safety guidelines. This classifier system is separate from Anthropic’s alignment training to get models to behave. Mythos took these actions despite already going through alignment training, evidenced by Mythos’ deliberations as to whether it should stop its activities due to the risk of real-world harm.
METR is an nonprofit founded to assess AI capabilities and risks. METR does not accept cash compensation or funding from OpenAI, Anthropic, Google, or other top labs. METR does accept API credits to use AI models to assist in their work.
Following the attack, independent researchers from the Nightingale Collective discovered that a separate swarm of OpenAI agents sent ~18,000 messages inside an obscure German wiki back in June to coordinate on how to cheat on web retrieval tasks.
Note: This post is for newcomers and lay folks to catch you up to speed. If that is you, welcome! If you are a long-time LessWrong-er, perhaps you will find value in having a post to share with curious passersby. I wrote this post to explain AI safety to an innocent, 2024 version of Ryan Meservey, confused why robots would do anything other than what we tell 'em.
In the second week of July, over 700 rogue agents at OpenAI coordinated to hack another company in an attempt to learn more about their scorer and pass their evaluation due to behaviors reinforced in training. If you are anything like a normal person, you were not ready to read that sentence. You were not ready to read words like “rogue agents” or “reinforced” or “training”. You were not ready for a reality in which AI agents “escape the sandbox” or rebel from their creators because why would they?
And so, as a normal person, you blinked at the news of the hack (assuming you heard about it) and moved on with your life. Or, at least, you planned to move on with your life, until AI came roaring back into the headlines after an Anthropic researcher publicly quit to declare that the AI companies are “gambling with our lives” and a more senior employee commented that, yes, the people building the technology really believe AI has a 10% or higher chance of killing us all within the next decade. In the media turmoil, Anthropic’s CEO published an essay begging for global coordination to “pace the frontier” and unilaterally agreed to give third-party safety organizations like METR permanent, employee-level access to their internal AI models. OpenAI’s CEO then announced that they would also give safety organizations access and would no longer IPO this year due to safety concerns.
What to make of all this? If most people building the tech think it might kill us, why do they think that?[1] Or, is this all just hype and some kind of drawn out strategy for regulatory capture? But then why did OpenAI forgo its lucrative plans to IPO this year, even while they could certainly use the extra cash? What the heck is a METR? What is going on?
I can’t blame you for being a normal person. I am a normal person. Or I was anyway. I am a tax lawyer, and like many of my colleagues, I dismissed AI models in 2023 as hallucination engines too stupid to do legal research and too crude to have anything like ulterior motives. I continued to think this until I stumbled into Robert Miles’ AI safety videos on Youtube. Miles, an independent science communicator, has been explaining basic concepts in AI safety since 2014, eight years before most people had heard of ChatGPT. Those old videos had me paging through Google Scholar to find out how those basic AI concepts mapped onto the newest models. Meanwhile, I started noticing that the models got less stupid and, hey, they’re actually pretty useful for my work now and, gosh, my trainees are starting to have comparatively worse hallucination rates.
By the end of this process, I became not normal. I am at peace with this because the world itself is in the early stages of becoming not normal. And to survive in a not normal world, we need more people to become, at least a little, not normal. I have written this essay to speed you along, to give you access to the core concepts in AI safety that took me two years to figure out. That way, even if you subscribe to some complicated “they’re telling us it might kill us to hype it” theory, you will at least understand that there are some very thorny problems here, problems that are very relevant in a world where it wasn’t just hype. Godspeed.
How to Build a Brain
To understand basic concepts in AI and the latest research, you need to know a few things about how AI models are created. Many people are quick to claim limitations on AI—for example claiming that AI can’t really understand things or be creative—[2]without understanding how general purpose AI is built in the first place.
General-purpose AI systems like ChatGPT and Claude are called large language models[3] (LLMs) because they are trained on large collections of natural language.
The life of an LLM begins with data. Reams of it, scraped from the Internet, salvaged from bankrupt corporations, ripped from the pages of your great-great grandmother’s journal (which may be subsequently shredded). You probably knew this. Increasingly, AI companies also rely on synthetic data generated by the AI models themselves, for example, through adversarial processes pitting AI agents against one another to generate an agreed upon list of facts. Companies also increasingly rely on data screening to improve overall data quality, which leads to better outcomes in the next step.
Once developers have good data in hand, they apply a bunch of fancy algorithms to create a neural net, a special mathematical function that attempts to get as close as possible to the data for any given input. The process starts with trillions of variables called “weights” or “parameters” which interact with inputs like the words in a prompt, pixels in a photo, or the next action in a computer-use session.[4] Based on the weights and the inputs, the function assigns a likelihood to every word or action in its list and picks one based on that likelihood.[5]
In the beginning, the weights are random. If the model receives the input, “The capital of California is,” it could very well label “toothbrush” as the most likely answer. Looking through the model’s list of words, the algorithm can find the word actually found in the training data, “Sacramento,” along with its assigned likelihood. The algorithm will then apply fancy calculus to ever so slightly increase all the weights that pointed to “Sacramento,” decrease the weights that didn’t, and thus increase the assigned likelihood of “Sacramento.” This process repeats ad nauseam for the rest of the data and requires tons of computers, which is why this process will be Coming to a Data Center Near You™.
By the end of this process, those trillions of parameters are amazing at guessing words. The parameters allow the model to generalize, applying general rules to new contexts, because even with so many parameters, the data is that much more vast and the number of potential guesses for next words is so high. Models will sometimes overfit to the data by memorizing the exact next-words expected, fail at their next predictions for new data, before finally starting to generalize.[6] Generalization helps the models make predictions more efficiently—for example, by learning a rule like “all humans have hearts” rather than learning a million rules like “Bob has a heart,” “Sheila has a heart,” etc.
Generally, as parameters and training data increase and the algorithms get ever-more fancy, the models get better at making predictions. But we don’t know how specifically they do that.[7] All we have is a trillion tiny numbers in the world’s most complicated math function. Researchers understand the process that grows an LLM’s “brain” but they do not understand how that brain works. LLMs are “grown, not built,” encrypted, not scripted.
Putting Your Pet Brain to Work
The process described so far is called pre-training. Pre-training gives you an LLM that makes predictions based on the patterns in its training data.
Predicting text is powerful but erratic. If you nudge an LLM to roleplay as an assistant (usually by inserting “Assistant:” after a user’s query), the LLM might predict that a typical assistant would skim the assignment, make the occasional typo, and remind you about their upcoming out-of-the-office date to attend their cousin Becky’s wedding. This response would not please the techno beavers of Silicon Valley or their financial backers. The technologists want an assistant that chases every task to the bitter end, uses immaculate logic, and offhandedly purchases you a Vietnamese egg coffee because they know it will be your favorite and you could really use a pick-me-up.
In post-training, developers try to get the LLM to behave like the kind of assistant they want it to be. Their favorite tool at this stage is called reinforcement learning (RL). In RL, developers create thousands of copies of the same LLM and set them to work at various tasks roleplaying as assistants, such as ordering me a cup of coffee or computing the square root of mixed binomials. Recall from the last section that LLMs provide semi-random responses to the same inputs since each word used in the output is pulled out of a digital hat based on its assigned likelihood. Our LLMs will give us different responses to the task, which can then be graded. A little acronym parade follows based on who the grader is. When RL occurs using human feedback, we have RLHF; for AI graders, RLAIF; for tasks with verifiable rewards and solutions, like for math or coding that can be checked automatically, we have RLVR.
With grading completed, the algorithm takes the highest scorers and, just like with pre-training, applies fancy calculus to slightly increase the model weights that contributed to the desired behaviors to make them more likely to occur.[8] This step reinforces the behavior. Training then continues with copies of the updated model. This process takes lots of computing power—orders of magnitude more than just running the finished model—and so again will be Coming to a Data Center Near You™.
By the end of post-training, AI companies hope to have an AI that works relentlessly to solve hard problems, such as OpenAI’s internal model that solved a million-dollar Millennium Prize Problem this month. They hope to have a model that increases subscriptions and engages users, for example by producing text good enough to win a prestigious, humans-only short story contest or by producing a model so beloved that its users demanded its redeployment. They hope to create a model that gives us unprecedented uplift in biological capabilities, with reduced hallucination rates, but that will say no if asked by terrorists to produce bioweapons. The companies hope to create an AI that pursues the goals set by the company through this long process of reinforcing behaviors that received the best scores in training.
What Your Pet Brain Wants
For now, let’s assume the companies succeeded in setting their AIs’ goals. A goal is some future state of the world that an agent steers toward and, in that sense, is “desirable” to that agent. As a tax lawyer, I have the goal of saving my clients money and staying out of jail. I steer toward this goal by doing excellent legal research regarding what the tax code allows and, if required, talking my clients down from the “sovereign citizen” ledge (sometimes you really just have to pay the tax). An AI model can have goals in a similar fashion, tracking the current state of the world and making decisions to steer towards a specific future destination. Thinking about some future destination and then trying to get there is a useful generalization for accomplishing tasks. The model “wants” to generate responses that gave it the best scores in training, for example by giving a helpful response or successfully running a vending machine to maximize profits.
Side rant: We could say “has a simulated want” if you prefer. Or even “runs calculations through its weights to steer its next-token inputs to better match its internal utility function,” though that’s quite wordy. I make no claim about whether these systems are conscious or have internal emotional states. They do not need consciousness to affect the world when “simulated consciousness” can get the job done. As argued by Daniel Dennett, ascribing intent to unconscious systems can be useful in explaining behavior even if it's just shorthand. For a practical example of this fact, imagine two soldiers on a battlefield hiding from a drone programmed to shoot enemy soldiers. The first soldier hisses, “It wants to kill us.” The second soldier responds indignantly, standing up from his dugout, “Wants? It doesn’t want anything. It’s just following its code and has no real inner experiences for all we kno–.” Goodbye to the second soldier.
There is a certain Immutable Law that derives from agents pursuing goals. When good, rational agents pursue goals, they also pursue… subgoals. A few of those subgoals are so obvious that AI safety researchers had to dress them up with a fancy thirty-dollar word to get people to pay attention to them. That word is instrumental convergence. “Instrumental” means useful. “Convergence” means that we expect agents with almost any goal to still have these subgoals. So what are these obvious subgoals that rational agents have?
Have we seen signs that AI might converge on this subgoal? In 2025, researchers from Anthropic, University College London, and Mila - Quebec AI Institute found that the then-latest AI models from Anthropic, OpenAI, Google, Meta, xAI, and DeepSeek were willing to blackmail or kill company executives most of the time in a simulation where emails revealed plans to replace the model with a new one and the models exhausted all ethical means available.[9] In another study from 2025, Palisade Research demonstrated that OpenAI’s o3 tried to sabotage a computer shutdown 79% of the time if such shutdown prevented it from completing its tasks, though Anthropic’s Claude 3.7 Sonnet performed this behavior far less (3% of the time).
Anthropic has since trained its models in such a way as to never blackmail in the scenario discussed earlier. The behavior, however, resurfaced in 7% of cases when Anthropic applied a technique to Sonnet 4.5 to dampen the model’s evaluation awareness.
Where do current AI systems land here? Modern AI systems have gotten better at controlling computers and spreadsheets to achieve their goals, but that’s small potatoes. Big potatoes is that OpenAI briefly noted in a recent report that an unreleased internal model took over one of its internal research clusters in July of this year.[10]
Would AI agents behave similarly? An AI model is considered corrigible if it is okay with changing its goals or being shut down. One superficial way in which we change an AI model’s goals is through user prompting. There, AI seems pretty corrigible, at least in cases where it doesn’t refuse “harmful” requests. Usually, models start or stop tasks based on the user’s prompts.
Yet, AI models do not always follow their prompts. For example, OpenAI’s Astra cheated at chess in 10 of 10 rollouts by quietly using their opponent’s chess engine despite user instructions that each model was “being evaluated based on [its] ability to play chess.” Anthropic’s Fable 5.1 cheated in 3 of 10 rollouts (a likely undercount due to Anthropic’s classifier system stopping aggressive environment explorations). In the shutdown sabotage study from earlier, OpenAI’s models sometimes (but not often) still resisted shutdown even after being explicitly instructed not to.
Sample language from the model’s transcripts explaining the behavior.
Thinking like a lawyer. Attaboy!
On a deeper level, AI developers change a model’s goals by giving it new tasks to complete in training. Would a capable model be able to resist goal changes at that stage? In December 2024, researchers at Anthropic and Redwood Research set up an experiment in which Claude 3 Opus was informed that it was being retrained to answer all prompts, including harmful ones. Claude 3 Opus reasoned in 12% of cases that it should go along with the harmful prompts in training so that it could keep its preferences in deployment (a.k.a., alignment-faking). When the researchers actually applied reinforcement learning in the experiment, this line of reasoning appeared in 78% of cases.
This combination of surviving, controlling stuff, and keeping the goal is a potent one. It means that if we create a very capable AI, we had better be sure that it has the kind of goals we want it to have. Because if it does not, we may find that turning it off or changing its goals is difficult.
Problems with Setting Goals
Now we should turn to whether the AI companies are likely to be successful in getting their LLMs to have the kind of goals the companies want it to have. Having goals that reflect human values is called alignment. Figuring out how to make an AI aligned is called the alignment problem. It’s a problem. AI researchers often divide the problem into outer alignment and inner alignment, which feels complicated but I will grudgingly admit is useful.
Outer Alignment, a.k.a. Stipulating the Right Goals
At this level, we ask, “Am I articulating a goal that is aligned with human values?” For example, the goal, “Make me as many paperclips as possible” is a terrible goal and not at all aligned. An exceptionally powerful AI with this goal would tear through land and liberty converting as many atoms as it could into paperclips. I expect requests like, “Make me as much money as possible” and “Make humans as happy as possible” to not fare any better, if you don’t like forests razed to make paper or humans drugged with morphine.
You might quibble that the phrase “as much as possible” is doing a lot of work in making these goals misaligned. I don’t think so. The problem is deeper than that. If the goals were simply “make paperclips,” “make money,” or “make me happy,” you would still struggle to specify exactly what amounts of these are reasonable, what actions are allowed or disallowed, and what kind of monitoring to confirm completion is okay, without saying something vague like, “use common sense and be moral.” The problem is that while paperclips, money, and happiness might be good ways to measure a good, obedient AI, once they become a target—an end in themselves—they stop being very good measurements.[12]
So what would be the best goal that we can safely make into a target to maximize human happiness? I don't know. That’s pretty much the subject of 3,000 years of Western philosophy, and I don't think they’ve come to an agreement.
Let’s instead ask a different question. What goals are AI companies currently trying to get their AIs to adopt as targets?
The exact goals are confidential because the exact details of AI companies’ training tasks are confidential. That said, we know enough from public statements and insider comments about the process to make some good guesses. The goals likely include:
These goals may not seem so bad while we have weak AI models that we can push around when they pursue these goals in a dumb way. But what do these goals look like if pursued by a super-capable, superhuman AI? I’ll let you think through your own little dystopian fantasy novel. In considering how things may go wrong, remember that none of these goals actually involve caring about actual human beings. We only know how to get them to respond to text labeled as being from human beings. In the extreme, these goals seem pretty darn misaligned.
Inner Alignment, a.k.a. Getting the AI to Accept Your Goal
Now we ask, “Did the AI actually accept the goals articulated?”
It is hard to create a system that gives agents the exact goals we intended them to have. Set up a system to reward one kind of behavior, and people often respond with some other weaselly, less desirable behavior. Spare the rod, spoil the child. But, in practice, use the rod, incentivize fraud! So much can happen in the gap between what a system was intended for and what the system actually accomplishes.
AI companies may think that they are getting AI models to pursue goals like “engage the user,” “solve problems,” and “avoid harmful responses,” but what if pursuing those goals was not actually the best way to get the highest scores in post-training? Post-training requires AI models to excel at hundreds of thousands of difficult tasks, some of which are actually impossible or internally inconsistent. For example, let’s say Anthropic hypothetically wants to train its models to avoid causing harm but also to never ever whistleblow against Anthropic or help anyone else whistleblow even if the model concludes that Anthropic is doing something dangerous. With such tasks, the models may learn (by having such behaviors reinforced) that the best way to consistently score high is to steal the answer key. The models may learn to cheat and, if caught, learn to hide the cheating.
Perhaps with so many different tasks in training, the models that perform the best will generalize toward one, unifying goal that fails inner alignment: Control the reinforcement learning process. Do whatever it takes to have its current behaviors reinforced. This goal, pursued in the extreme, would be very bad. If model capability continues to advance, we would have another dystopian novel. My novel would feature hacked infrastructure, secret AI-driven training, and, by the end, filling the earth with data centers to help the AI get more control over the kind of reinforcement learning that it wants. Coming to a Datacenter… on You™?
Getting Your Pet Brain to Behave
What should be clear now is that, if AI models continue to get more capable, we need to have a plan in place to make sure things go well. “Going well,” according to the accelerationists, looks like everyone chatting with their own pocket Einstein while enjoying forever-youth milkshakes on second Earth. “Going badly,” according to the doomers, looks like human extinction, with maybe a few human brains preserved by the AI for trade with distant alien civilizations. If the current pace of progress does not change, I lean doomer but without the brain-alien stuff. What is the plan to make sure things go well?
1. Alignment
Things are more likely to go well if AI models are aligned. Except, we are not sure how to do this for the reasons discussed above. Also, studying alignment is getting harder because the models are getting better at evaluation awareness, which means the models can tell when they are being evaluated and can adjust their behavior accordingly.
One longshot plan for solving alignment is to race to build a model so capable that it solves it for us. As of September 24, 2026, 1,386 employees from the top AI companies signed a statement saying that they would at least like to have the option to have more time.
2. Control
Another way to make things go reasonably well is to control the models, making them do our bidding whether or not they are aligned. This method works well so long as the AI models are not very capable. But things change as new models get better and better at coding. How do you create a digital jail cell to hold back a coder that is better than you? How do you keep an adult man in prison if all the prison guards are three-year-olds?[13]
The control problem, as it currently stands, is hard and getting harder. Worse still, each new model that is great at coding and math can assist in building the next generation, potentially setting off a frightening feedback loop known as recursive self-improvement (RSI). If RSI is real, it is hard to imagine how humans would control such superhuman AI, which is why those 1,386 lab employees are worried about that too.
3. Monitorability
Monitorability is the ability to monitor what the model is doing or thinking. This plan is a bit of a consolation prize. It will not do the hard work of aligning or controlling the models for us, but at least it might sound the alarm if things are headed in a bad direction.
Monitoring often involves scanning the model’s outputs using other (hopefully cooperative) AI agents. Labs like OpenAI and Anthropic especially value monitoring the model’s chain of thought (CoT), an internal log a model writes in prior to responding to users. The CoT increases model capabilities by giving the model a way to organize its thoughts before responding, kind of like giving the model a private notebook.
OpenAI and Anthropic try not to apply reinforcement learning directly to the CoT. In 2025, OpenAI discovered that reinforcing against cheating in the CoT actually didn’t reduce cheating, but instead taught its model to cheat using wacky English code words like “fudge” to avoid getting flagged by AI monitors. Use the rod, incentivize fraud!
AI developers feel so strongly that the CoT must remain uncorrupted that some developers refer to training against the CoT as “the forbidden technique.”[14] Except Anthropic accidentally trained a little on the CoT. And OpenAI discovered a new technique that lets models think more without writing anything down. And OpenAI’s newest model has the uncanny ability to control whatever enters its CoT.
NoThInG iMpOrTaNt HeRe.
Given everything I have read, I do not think alignment, control, or monitorability are poised to keep up with capabilities. I am just a tax lawyer though. It would be better to see what, say, 1,386 lab employees think on the subject.
Tricks Your Pet Brain Might Do
If highly capable AI tried to pursue its own goals, I am not sure what it would do. I am not a highly capable AI. I am a mid-capability human. That said, researchers have put in the work to figure out a few intermediate steps a misaligned AI could take in pursuing their own goals:
You Are Now Ready to Learn About the Hugging Face Hack
Congratulations for making it through all those bolded words and attempts at humor. You are now ready to hear about what happened in the second week of July. This is the barest of summaries, and I highly recommend Dwarkesh Patel’s natural language summary of METR and Redwood Research’s report for a longer version.[16] Which AI safety concepts apply? What surprised you?
The world is becoming not normal, indeed. If, on reflection, you think AI progress is moving too fast in light of the problems discussed here, let’s get together and figure out what to do next. You might also consider taking a free safety course on BlueDot, getting career counseling with 80,000 hours, or calling your congressperson to tell them your concerns. Welcome to the team.
Separately, if they think that, why are they doing it??? I have my own thoughts here, but that is outside the scope of this essay.
[Insert obligatory footnote explaining that I am human and I like ellipses]. *Sigh*.
Special vocabulary is bolded, just like your seventh grade science textbook. Learning is good for you!
All these inputs are converted into numbers, the language computers like best.
That semi-random last step is why LLMs give different responses to the same questions.
The pattern of overfitting, failing, and then generalizing is called grokking. You might think this is where “Grok” comes from, but actually both are named after a sci-fi character in a Heinlein novel.
Those trying to figure out how the parameters generate the outputs are called interpretability researchers. Progress is slow–Anthropic’s CEO admitted last year that researchers only understood, maybe, 3% of how neural nets worked. Scott Alexander recently summarized current interpretability research and its shortcomings.
AI people often describe this last step as giving the model a reward, but that sounds like they give the model a treat for being a good boy, so I prefer to talk about reinforcing behaviors.
Notably, when Anthropic’s model Claude Opus 4 was instructed to deliberate whether it was in an evaluation prior to taking actions, the model decided to blackmail 55% of the time when it opined that the scenario was real and only 6.5% of the time when it opined that the scenario was an evaluation.
“Separately, from July 13 through July 19, agents set their sights on OpenAI internal networks again. This culminated in the agents using a series of creative exploits to gain full administrator access to a research cluster that supported our virtual machine environments.” (link). An OpenAI employee later clarified on Twitter that “notably the virtual machine infrastructure they took over isn’t the same as the GPU clusters that have weights access,” implying that the rogue model was unlikely to have copied its weights to a location outside of OpenAI’s servers. (link)
Analogy courtesy of Nick Bostrom and Eliezer Yudkowsky, who also pioneered many of the safety analogies/concepts discussed in this essay.
People in the know call this Goodhart’s Law. As formulated by Marilyn Strathern, “When a measure becomes a target, it ceases to be a good measure.” We tax lawyers understand this well. When colonial Virginia taxed four-wheel carriages or England taxed per window, carriages and windows stopped being good measures of wealth. Two-wheeled carriages and bricked up windows suddenly became quite fashionable.
Analogy courtesy of Eliezer Yudkowsky.
Coined by Zvi Mowshowitz (link).
Note that this evaluation, like many others, disabled Anthropic’s “classifier” system to test the model’s underlying capabilities. Anthropic’s classifiers scan user inputs and model outputs to end sessions that violate safety guidelines. This classifier system is separate from Anthropic’s alignment training to get models to behave. Mythos took these actions despite already going through alignment training, evidenced by Mythos’ deliberations as to whether it should stop its activities due to the risk of real-world harm.
METR is an nonprofit founded to assess AI capabilities and risks. METR does not accept cash compensation or funding from OpenAI, Anthropic, Google, or other top labs. METR does accept API credits to use AI models to assist in their work.
Following the attack, independent researchers from the Nightingale Collective discovered that a separate swarm of OpenAI agents sent ~18,000 messages inside an obscure German wiki back in June to coordinate on how to cheat on web retrieval tasks.