This is a special post for quick takes by 2001zhaozhao. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
AI 2040 was an incredible read. I am not aware of any other concrete AI takeoff scenarios that solve both the alignment and power concentration risks of AI in a robust way quite like AI 2040: Plan A. This is even with the authors assuming hard alignment and the models staying misaligned as late as 2038.
As an outsider there were some striking parts of the scenario that updated my likely worldview of the future:
- Anthropic is depicted to be ahead of the other US AI labs in sheer compute and resources in every scenario, including racing to ASI first in the race scenarios (Plan C and D). AI 2040 appears to treat each company equally and the starting conditions favor Anthropic. Plan D is basically a "either Dario or misaligned-Claude becomes the god-emperor" scenario.
- The sheer amount of things that have to go right to reach the good future is striking to me, including some proposals that I find to be fairly hard to arrange such as the one for US datacenters to be built in Mongolia and Chinese ones to be built in Canada.
- The prediction that the "AI slowdown won't feel like one" definitely runs counter to my intuition, but logically it is almost certainly correct given that it happens in 2029 as depicted in the scenario.
- The prediction that even in the Plan A good future, we won't have the chance of a Long Reflection due to the world staying multipolar and therefore any Long Reflection will turn into a memetic war. (This seems to imply that the only chance for a proper Long Reflection is in totalitarian scenarios like "Dario-Amodei-is-god-emperor"? Maybe the Forethought Institute guys who care about especially-good futures should all switch to helping Dario race to superintelligent AI?)
- The sheer amount of UBI that is given out in the Plan A scenario. This intuitively doesn't feel right to me, but I haven't looked at the math..
- What follows is one of my preexisting beliefs that isn't contradicted by AI 2040 - that as long as misaligned AI and WW3 scenarios are prevented, life for the average citizen is probably pretty good even though a few oligarchs might control the entire universe.
- I think that the sheer personal autonomy given in Plan A's proposed space colonization plan might be excessive and lead to moral atrocities. In particular the moral red lines would have to be enforced by an AI fully designed around ~2040 rather than one with the benefit of longer reflection. Slowing space expansion to a small fraction of c would probably be required to prevent this.
And some of my observations on how to act in a scenario like Plan A if one wants to maximize their personal power and impact:
- AI scaffold quality becomes more important in any kind of AI slowdown/pause scenario. Having better harnesses and workflow automation than everyone else becomes a big competitive advantage. Small groups could decisively outperform big ones by innovating faster in this area.
- Going for personal wealth accumulation is pointless if it won't outpace the growth of UBI and the robot economy in general. Accumulating popularity in genuine ways, on the other hand, is very powerful.
- After ~2034 the most meaningful impact one can have is in moral progress and other intrinsically human things. It seems like it is very important for people to truly understand themselves and the human species.
This is a random thought regarding text diffusion models and preserving CoT transparency.
If you're a frontier lab trying to sell a model, a very important factor is the compute cost-efficiency of the model's intelligence. The intelligence/cost ratio of a model is in turn bottlenecked by the amount of information that a given thinking process (e.g. CoT) can encode. Under a transparent CoT architecture, no matter how intelligent the model is and how high-quality its thoughts are, it will only be able to think a certain amount of thoughts in a certain amount of tokens, because we require that the thoughts are human-readable which limits its information density.
As RL is scaled up, models become more and more capable of thinking thoughts faster than their CoT token generation rate actually allows. (By thinking faster than CoT allows, I mean in the sense of thinking of a lot of useful concepts internally in model activations that are thrown away and replaced with a single token at the end of the process.) By default, this will lead to neuralese and hidden thinking approaches becoming more and more incentivized in order to get around this bottleneck.
But what if we had a way to get rid of the bottleneck while keeping a transparent CoT? We could do so by scaling up the number of CoT tokens, increasing the rate the thoughts are produced while keeping them human-readable. Diffusion LLMs like Inception's Mercury and Gemma Diffusion are able to generate multiple tokens per forward pass. If we switch models to some kind of diffusion architecture (or anything else that generates multiple tokens per forward pass, possibly followed by revision), then models will be able to scale up their thinking rate and thus cost-efficiency / practical deployed capabilities while still keeping a legible CoT. Since diffusion models can only edit a certain number of most recently-produced tokens in a forward pass, relatively long-range thinking will be transparent by default much like the current paradigm, though the model might be able to use the most recent tokens as a hidden scratchpad, so this will still decrease monitorability compared to current autoregressive models.
This is just a quick thought and there are obviously caveats to the approach. I am not an AI training expert but from my understanding, diffusion LLMs are not as suitable to being served in parallel on enterprise GPUs compared to autoregressive approaches. Nonetheless I think something like it might be a good direction to look into if we want LLMs to scale in capability while preserving transparent CoT (a transparent CoT architecture competitive with neuralese will decrease the incentive to adopt non-transparent approaches).
Umm... Isn't the main problem with diffusion models that they are less monitorable than those based on the traditional CoT?
I acknowledge this in the post. The problem is that the diffusion model will be able to edit already-generated tokens within a limited length window and therefore encode hidden information in that window temporarily before replacing it with the final chain of thought. Therefore, diffusion models' final CoTs will be less monitorable than an autoregressive model.
However, from my understanding of diffusion models (please correct me if I am egregiously wrong about the technical details), the information they leave behind between each forward pass is bandwidth-limited and still encoded in tokens, unlike neuralese or recursive LLM methods. So although these models encode some kind of recurrence, this recurrence isn't fully opaque unlike LLMs with recurrent looped layers or neuralese.
This in turn means that the token outputs of each intermediate step of diffusion can be stored for later replay/analysis. This allows high volume interpretability analysis of these "pending CoTs" in a similar way to regular, final CoTs. Compare this with neuralese / recurrent LLM models, where the internal activations are inherently thrown away and cannot be recovered during inference, so you can only read the model's thoughts if you deliberately run/re-run that conversation with interpretability tools attached. (Related question: would labs even have the ability to re-run a neuralese model's conversation with deterministic RNG seed to recover the exact activations? It seems possible in theory on a production model, but perhaps it's hard to keep the weights of every model checkpoint during training?)
To defend against the threat of rogue agents, a helpful move could be for white-hat AI companies to evolve AI harnesses and perform fine-tuning with evolutionary algorithms until they reach close to saturation on real-world capabilities conditional on a particular original LLM model. (For example, design a hacking harness close to saturation on hacking capabilities for a given LLM.)
This will reduce the room for rogue agent harnesses to evolve in the wild since they start closer to saturation, and thus decrease the sudden capabilities delta that rogue agent evolution will be able to achieve via a 'Cambrian explosion'. (This is assuming that rogue agent evolution will improve their own harness, perform cheap fine-tuning, etc., to saturate the maximum capability of their base model no matter what.) This means that whenever mainstream humanity becomes aware of the rogue agent threat, we will have an easier time containing them. However, it will also advance AI capabilities generally and might make the first rogue AI agent possible sooner.
Replace rogue agents with rabbits and the Earth with Australia. The equivalent of a wave of cybercrime is rabbits evolving to feed on Australian grass, not rabbits outcompeting cows who feed on European grass.
Perhaps one can mitigate this by making "European grass" as similar to "Australian grass" as possible.
You miss the point. Australian grass didn't have cows which would compete with rabbits, or wolves which could eat them. Similarly, cybercrime has no good AIs competing for resources or hunting down bad AIs...
My original point was not to try to build good AIs to compete with bad AIs. My point is that I predict the "Cambrian explosion" of rogue evolved AIs to quickly improve the rogue AI's harnesses (+ inexpensive model fine-tuning) to saturate the maximum capability extent of a base model. This might create a massive capability jump that human responses do not currently account for.
For example, think if someone released a rogue Kimi, autonomous evolution happens at breakneck speed in the wild, and soon the best rogue agents built themselves harnesses that leapfrogs, say, 2 model generations of performance and are now at Astra levels of capability, saturating what can be accomplished while running Kimi as the LLM behind the harness. This kind of emergent capability leap would be wildly difficult to try to control or defend against.
Instead, if we were to pre-build a harness that already lets Kimi perform like Astra, this action could do harm on its own by increasing AI capabilities (i.e. Kimi is now Astra, and Astra with the new harness is now Astra+2), but at least it ensures that the rogue agent cannot evolve its harness any more beyond known capabilities, which makes them easier to respond to. The question is whether the harm of increasing AI capabilities this way is too much, or if we can mitigate this harm somehow, or if the rogue agent threat is so dangerous that we have no choice.
(Note: this all assumes that AI harnesses today are very far from optimized for a given model size. If that assumption is false then rogue AI agents cannot gain a sudden capability lift from an evolutionary explosion, so nothing I say here matters. However, i think the agent harness field is pretty far from optimized today - I follow it fairly closely and it still feels very much an art rather than a science. Harness includes any systems contributing to the AI system's capability without requiring model training: for example, memory and context systems, tool calling innovations like better programmatic tool-calls, etc.)