Epistemic Status: This is a personal attempt to find a useful lens for a complex debate. The 'sharp left turn' discussion involves some of the hardest questions in AI safety, and I've seen many smart people talk past each other. What follows is a framing I've found helpful for organizing my own thoughts and understanding the crux of the disagreement. My hope is that by sharing it, it might help others see the conversation not as a clash of incompatible analogies, but as a set of related questions about a single, underlying property of learning systems. (I also didn't have too much time to put this together but I thought it would be better to get out a semi-formed version of this, I also use an AI-augmented writing process but hopefully I got rid of most of the LLMy stuff.)

Inspired by all the reviews of reviews on If Anyone Builds it.

A Lens on the Sharp Left Turn: Optimization Slack

The debate around a "sharp left turn" in AI capabilities can feel like a tangled mess of analogies and counter-analogies. On one side, you have the stark warning from our evolutionary history. On the other, you have compelling arguments that the analogy is mechanistically flawed. Trying to figure out who is "right" often feels like missing the point.

Lately, I've been trying to view this entire conversation through a simpler lens: the idea of optimization slack.

You can think of "slack" as the amount of room an inner learning process has to maneuver before the outer optimization process corrects it. It's the temporal, informational, and causal gap between learning and feedback. I've found that this concept helps translate the often-abstract debate into something more concrete.

The Core Argument as a Story of High Slack

From our perspective, the argument for the sharp left turn, articulated by Nate Soares, is fundamentally a story about a system with immense slack. He points to evolution as our one data point for the de novo creation of general intelligence.

The basic structure he describes is this:

• Outer Loop (Evolution): An optimization process (natural selection) selects for a simple, distal goal (Inclusive Genetic Fitness). This loop is incredibly slow, with a feedback cycle measured in generations.
• Inner Loop (An Organism's Brain): This is a much faster, more powerful learning process that runs for an entire lifetime.

The gap between these two loops is the slack. For tens of thousands of years, an organism's brain could perform trillions of learning steps, but this learning was contained within a single life. The outer loop only got a single, noisy data point back at the end: did the organism reproduce? This is an almost unimaginable amount of slack. The inner optimizer was running almost completely unsupervised.

In this view, the "sharp left turn" of humanity wasn't that our brains suddenly became magical. It's that the inner optimizer (our brain) developed a technology (culture, language) that finally allowed its high-speed learning to accumulate across the generations, bypassing the slow, low-slack outer loop. Suddenly, the results of trillions of learning steps per generation started compounding, and the rate of capability gain exploded. The alignment to the original outer goal (IGF) predictably broke down because it was only ever enforced by a loose web of proxy goals (hunger, lust, etc.) that were insufficient to steer the now-compounding inner optimizer.

The Counter-Argument: A Different Kind of Slack

There are counter arguments to this and it is one thing that is discussed in the reviews of Scott Alexander's review of If Anyone Builds it Everyone Dies by Nina Panickssery and the review of the review by Gradient Dissenter. (So this is a response to a review^3!)

One of the original critiques of this story comes from Quentin Pope in "Evolution provides no evidence for the sharp left turn" in 2023. We'll look at this from the perspective of slack.

He zooms in on the specific mechanism of that slack. To put it in these terms, his argument is that the defining feature of evolution's slack was a massive informational bottleneck. Think of the immense amount of learning and adaptation an animal achieves in its lifetime—a huge expenditure of computational effort. In pre-human evolution, when that animal died, nearly all of that painstakingly acquired knowledge vanished with it. Death wiped the slate clean.

From this perspective, the human takeoff wasn't caused by a mysterious, sudden jump in the quality of our brains' learning algorithm. Instead, it was a direct and predictable consequence of inventing a technology that solved this specific bottleneck. That technology was culture. High-fidelity social learning acted as a new channel, allowing the discoveries made during one generation's fast, powerful "inner loop" of learning to be passed on to the next.

This effectively reduced the system's inter-generational slack. The "sharp left turn" was the moment this latent potential was finally unleashed, allowing learning to compound across generations for the first time.

And this, Pope argues, is where the analogy to AI development breaks down. We aren't building systems with a generational structure where the weights are constantly wiped. Our current training paradigm is more like a single, continuous lifetime. There isn't a comparable, massive "overhang" of wasted computational effort from billions of discarded lifetimes waiting to be unlocked by a simple trick. The specific kind of slack that powered the human origin story, he suggests, simply doesn't exist in our current methods.

(He goes into more detail on that on the AXRP podcast here)

Where Does This Leave Us?

So, is the analogy dead? Not quite. Steven Byrnes, in his "Sharp Left Turn" review, provides a way to synthesize these views. He suggests we think not of "evolution," but of an intelligent designer, "Ev," who sets up humanity's initial conditions and then is absent for 100,000 years. (I love this post, it's great!)

Ev's absence can be seen as the slack.

The danger isn't necessarily about generational cycles; it's about the degree of supervision. A learning process running without tight, continuous, and meaningful feedback has room to develop in unexpected ways. Slack is to some extent the upper bounds on how much we can allow capabilities to generalise without giving guidance (reward). 

This reframes the entire debate into a more concrete, technical question: How much slack do our AI training methods actually allow?

This is where things get difficult, and where I think the most interesting open questions lie. Jan Kulveit, in "We don't understand what happened with culture enough," points out that we don't even have a consensus on what specific human innovation was the key slack-reducer. Was it symbolic language? High-fidelity imitation? The development of new social structures? If we don't understand the solution in our own history, it's hard to be confident about the problem in our future.

This leads to a few key research questions that I think are more fruitful than continuing to argue about the analogy itself:

1. Can we measure slack in AI systems? What would a metric for slack even look like? Perhaps it's related to the number of unsupervised internal computations (like in a long chain-of-thought) that occur between rewarded actions. Or maybe it's about the causal distance between a training signal and the cognitive structures it modifies. We don't have a good language for this yet.
2. How does slack relate to different training paradigms? Intuitively, a model fine-tuned with direct RLHF on every output seems very low-slack. A system capable of recursive self-improvement, which can modify its own learning process and curate its own data, seems to be a recipe for generating immense slack from its human supervisors. Where do other methods fall on this spectrum?
3. What is the relationship between slack and generalization? The core fear is that capabilities generalize robustly across high-slack environments, while alignment does not. Is this empirically true in our models? Can we design experiments to test this? For instance, could we train two agents, one in a low-slack (high-frequency feedback) environment and one in a high-slack (sparse feedback) environment, and then measure how well their alignment properties transfer to a novel situation?