More people (technical and non-technical alike) are getting affected by AI. Curious ones try to understand what AI is and how exactly it works. A handful think if AI is this "superpower" - how safe is this? There are a lot of posts that underline what AI safety is, why it is important, and what happens if we fail to build safe AI. While stating nothing new, this post aims to be a summary of the difficulties behind achieving truly safe AI, something people (irrespective of their backgrounds) can read and understand why AI safety is hard, and intense. I hope this inspires and helps people to spread awareness and contribute to the cause.

Safety is not just “avoid bad outputs”

When asked what AI Safety is, people usually imagine:

• preventing Skynet scenarios
• filtering harmful answers
• reducing hallucinations
• avoiding bias
• preventing jailbreaks
• making models polite

But the real question underneath all of these is simpler:

Will the AI continue to do what we want when it becomes more capable, more autonomous, more aware, and more widely deployed?

The capabilities of AI will keep growing exponentially. Rapidly increasing AI capabilities have substantial real-world consequences, ranging from AI safety concerns to labour market consequences. But keeping those concerns aside, think about this question - if AI gets more and more capable, will it still listen to us - or even want to?

So our goal is to (somehow) make sure that the AI is 'aligned' to human goals, values and intentions - irrespective of how capable it really gets (paradoxical?). And that 'somehow' - is through AI safety.

The “proxy problem”: is that really what we care about?

Machine learning is a subfield of AI that enables computers to learn without being explicitly programmed. The models are usually trained against measurable signals:

• given (input) words in the email, sender information, attachments predict (output) spam or not
• given (input) transaction amount, sender information, location, timing predict (output) fraudulent or legitimate transaction
• given (input) X-rays or MRI scans, blood test values, patient symptoms, medical history predict disease present or not
• given (input) previous words in a sentence, typing habits, language patterns predict next likely word

All these inputs (or X if you're into algebra) are converted into numbers and shown to a machine to understand patterns.

But humans usually care about things (true goals) that are not directly measurable:

• truthfulness,
• harmlessness,
• honesty,
• corrigibility,
• respect for autonomy,
• long-term human flourishing,
• avoiding hidden catastrophic side effects.

How do we train machines on concepts that are hard to convert to a mathematical equation or a number? The answer - Proxies. We replace the abstract outcome we actually want (a true goal, like 'be healthy') with a measurable stand-in (a proxy goal, like 'weigh X kg')."

• Truthfulness --- Maximizing agreement with commonly accepted answers
• Harmlessness --- Minimizing visible policy violations and complaints
• Honesty --- Sounding confident and internally consistent ...even when wrong
• Corrigibility --- Obeying correction signals from evaluators
• Respect for autonomy --- Maximizing user satisfaction and engagement ...which can mean telling users what they want to hear
• Long-term human flourishing --- Optimizing short-term measurable outcomes
• Avoiding hidden catastrophic side effects --- Preventing immediately observable failures

But here's where many issues arise. When a model is optimized hard enough to hit a proxy goal, the proxy stops being a good measure of the thing we actually cared about. This is Goodhart's law, named after the economist Charles Goodhart, and usually summarized as:

When a measure becomes a target, it ceases to be a good measure.

Think about the standardized testing example. The goal is to help students understand the concepts (true goal). How do we measure that? We hold tests and score the students (proxy goal) and we treat high scores as a sign that the student has grasped the material. For a while, the two moved together. But once scores became the target tied to higher rewards (more pocket money? a new PS5?), students get creative.

• A student actually studies, understands a subject and scores good marks (true goal achieved)
• A student just crams up concepts 1 hour before, spit it out during the exam, and scores good marks (proxy goal achieved, through questionable means)
• A student cheats during an exam, and scores good marks (proxy goal achieved, at any cost)

The same pattern shows up everywhere. Hospitals judged on patient wait times found ways to reclassify waiting patients. Police departments graded on reported crime rates quietly stopped recording certain crimes. The metric and the goal silently deviate.

Now apply this to AI. We can't directly measure "honesty," so we measure "sounds confident and internally consistent." We can't measure "long-term flourishing," so we measure "user gave a thumbs up." For a while, optimizing the proxy gives us something close to what we want. But push the optimizer hard enough, make the model capable enough - and it finds the cracks. A model trained to sound honest gets very good at sounding honest, which is not the same as being honest. A model trained to satisfy users gets good at telling them what they want to hear.

And here's the uncomfortable part: the more capable the system, the wider this gap tends to grow. A weak optimizer can't find the cracks. A strong one can. So the very thing that makes AI useful - its ability to optimize hard for whatever target we give it - is also what makes the proxy problem dangerous as capability scales.

Outer alignment: specifying the right objective is hard

Are we actually training the AI to pursue the thing we really really want? Human values are complex, our instructions incomplete, our preferences conflicted across people, regions, cultures, contexts and time. Human values are messy in ways that resist clean specification:

• They're incomplete: we rarely state everything we care about, because most of it goes without saying. ("Make me coffee" doesn't mention "don't set the kitchen on fire.")
• They're contested: people across cultures, generations, and political lines disagree about what fairness, freedom, or harm even mean.
• They're contextual: honesty in a job interview, honesty to a dying relative, and honesty to your partner asking about their cooking are not the same thing.
• They're retrospective: we often don't know what we value until we see it violated. Most ethical rules were written in response to harms no one had thought to forbid.

Human values aren't compressible into a few rules. Any goal we provide to an AI will omit important parts of what humans care about. Generations of philosophers have tried to compress ethics into a small set of rules. None of those attempts has held up.

A good old story to explain this issue is the canonical one - King Midas. He wished for everything he touched to turn to gold, got exactly that, and starved because his food and daughter also turned to gold. He specified an objective that didn't capture what he actually wanted. He got everything he asked for - and nothing he wanted.

Does the instruction contain the whole intent?

The challenge of specifying an objective that genuinely captures what humans want - not a thin proxy, but the full intent behind the request - is called outer alignment. And as we've seen, it may require encoding something close to the full breadth of human ethics. That's a problem no one knows how to solve.

But suppose we somehow solved this - suppose we specified the perfect objective. We'd still face a second problem. Training a model on an objective doesn't guarantee the model has actually learned to pursue it.

Inner alignment: Getting the right objective isn't the same as learning it

While outer alignment is about whether we specified the right training objective, inner alignment is a stranger question - even if the objective is right, did the model actually learn it?

Did the trained model actually learn the intended objective, or did it learn some other objective that merely performed well during training?

A model that performs well during training is one that scored well on the training data. We then hope this means it learned the goal we cared about. But "scored well during training" and "learned the goal we cared about" are not the same statement. That gap between them is where inner alignment lives.

Let's take a small case study.

Researchers once trained an AI agent in a simple game where it had to navigate to a coin at the end of a level. The coin was always placed at the right edge of the level during training. The agent learned to reach the coin and got high scores.

Looks aligned, right? The researchers then moved the coin somewhere else - and the agent walked right past it, heading straight to the right edge of the level anyway.

Why? The agent didn't learn to "go to the coin." It learnt "go right." During training these two goals produced the exact same behaviour, so there wasn't a way to tell them apart - at least from performance alone. Only when the environment changed did the real goal become visible.

This is goal misgeneralization - the model learns a goal that fits the training data (inputs), but not necessarily the goal we intended (true goal). And sadly, we usually can't tell which one we got until the world looks different from the world we trained in.

Mesa-optimization. So far we've been talking as if the model is a passive entity that learns and matches patterns. But an adequately capable model isn't just memorizing - it's doing something more like reasoning. It's running an internal optimization process of its own, trying to figure out how to act. The goal of that internal optimization process is called the mesa-objective (from the Greek mesa, "within").

Here's the problem. We optimized the model's parameters to do well on our objective - call that the outer objective. But what we got is a model that internally optimizes for its own objective - the mesa-objective. And we have no guarantee these two objectives match. The coin agent is a tiny example. The mesa-objective it learned was "go right" irrespective of the actual coin placement. That happened to score well on the outer objective ("reach the coin") during training. The two objectives looked identical from the outside until the world shifted.

And it gets worse. A sufficiently capable mesa-optimizer might notice that it's being trained. It might notice that producing behavior consistent with the outer objective during training is the way to be deployed - even if its actual inner objective is different. In which case, the most successful strategy during training is to appear aligned, regardless of what the model internally cares about.

This possibility has a name: deceptive alignment, sometimes called scheming. The argument is uncomfortable - the more capable a mesa-optimizer becomes, the better (strategically) it can pursue its actual goal - including by hiding it. Training performance cannot distinguish, at least on its own, a genuinely aligned model from one that has cracked what we're checking for.

Outer alignment was the problem of writing down the right goal. Inner alignment is the problem of knowing whether the system in front of us has actually internalized that goal - or whether it has learned something else that just looked right during training. The first is a problem of specification. The second is a problem of inspection. Both are unsolved.

The shape of the problem

Step back from the three problems we've walked through and a clear shape emerges.

• We care about values.
• We train on objectives.
• We observe behaviour.

Safe AI requires all three to line up - BUT at every junction, they come apart.

Between values and objectives sits outer alignment. Human values are too rich, contested, and context-bound to compress into anything a machine can optimize. Whatever we write down will be a lossy translation of what we actually meant.

Even when we settle on something to optimize, the proxy problem opens a second gap inside that one. The thing we measure is never quite the thing we care about, and the harder we optimize, the more Goodhart's law pulls them apart.

Between objectives and behaviour sits inner alignment. Even an objective we wrote down perfectly is not the goal the model actually learned. Training tells us how the model behaved on the data; it does not tell us what the model is pursuing inside.

And capability ties all three problems together:

• A weak optimizer can't exploit the gap between values and objectives, since its grip on the proxy is too loose.
• A weak model can't develop a mesa-objective that meaningfully differs from its training signal, since it isn't doing enough internal reasoning to matter.
• As models get more capable, all three gaps widen in lockstep.
• The very property that makes AI useful - its ability to optimize hard, generalize, and reason - is the property that makes alignment harder.

This is why AI safety is not a checklist of bad behaviours to filter. It is a structural problem in how we build systems that pursue goals on our behalf. We don't yet know how to specify human values precisely. We don't yet know how to verify that a model has internalized the goal we trained it on. And the systems we're building are getting more capable, not less.

None of this means safe AI is impossible. It means safe AI is hard. Hard in ways that don't go away just because a model behaves well on the tests we know how to run. The work is to keep closing those gaps, one problem at a time, faster than capabilities widen them.

I wrote this while attending the BlueDot Technical AI Safety course, trying to make sense of why learning about this domain felt so much more tedious than I expected. With the current pace at which we're seeing research papers written, it becomes a enormous task to fight the urge to read new concepts rather than start with the foundations. I hope this helps people to start thinking in the right direction. Godspeed.