Part 1 of a series on process alignment, a different way of thinking about agents at every scale.
Introduction
There are quite a lot of disagreements in the AI Safety and general AI community about what the future will look like. They’re often framed as differences in algorithmic progress or differences in compute scaling or the degree of unipolarity that we would want.
Yet they all seem to smuggle in various assumptions about how future AI systems will look like.
It feels like the forecasts are downstream of ontological commitments: untestable assumptions about what the future is made of and what its basic units even are. Are the players of the coming decades humans? Institutions? AI models? Hybrid human-AI collectives? Depending on how you answer, you get a different theory of what can go wrong and a different theory of governance for preventing it.
For example, will AI develop as it is described in AI 2027 or AI 2040? Or is there something that is fundamentally off with LLMs and the way that they’re currently setup as Richard Ngo writes in his critique of AI 2040? This completely changes the way that we should go about prevention and what we should focus on. This is also dependent on how we model agents in society.
Yet even if there is this fundamental uncertainty about what future agents will look like, we still need approximate solutions for the problems we will face as we can’t go into the future blindly. We somehow need to model the future so that we know what governance actions we can take.
From a computational perspective this is both a difficult and an easy problem.
It's an easy thing since a lot of systems in the world follow the law of large numbers and as a consequence we get to have nice predictability for industrial processes. It also turns out that standardization and predictability leads to an easier time making decisions and as a consequence we like a regulated predictable world. This makes it so that Epoch can put forth their scaling laws based on compute, algorithmic progress and data efficiency.
It's, however, also a hard thing since a subset of systems in the world don't follow the law of large numbers and one could state that the field of complexity science is generally about those systems. Also since the market is an adaptive changing system, it will be a modeler of the underlying dynamics of the world which is provably complex in various domains.
For example, one of the problems with linear forecasts is that it doesn't factor in places where there are discontinuities. In many military forecasts I heard about Ukraine, people were talking about the economy and the amount of resources that each side had and so on, and made forecasts based on that. What happened is that Ukraine developed a new type of drone warfare that wasn't foreseen which completely changed the underlying dynamics of the war. How are we supposed to bring these sorts of black swans into our models?
Generally, we don't model unknown unknowns and we instead simplify our economic models which is coincidentally why you can sometimes on quiet nights hear Nassim Nicholas Taleb scream about black swans in the distance. It seems that in some of the societal prognosis of AI we're thinking about various systems as if they're white swans. That is, we forget about the unknown unknowns that are implicit in our systems.
Yet, some would say, and this is quite a controversial claim, that AI might change the world in more ways than we can currently imagine. In fact, in an even more shocking statement, I would like to claim that our ways of measuring various societal functions are dependent on humans being the main constituents. This means that we might not be ready for the change that bringing a new type of agent into the world would bring.
How does a democracy work when you have agents that can send information brain to brain at the speed of fibre cables? What happens with our collective ability to make sense of things if the ability to output propaganda goes up by 10x? What happens with the balance of power between citizens and governments if the governments can analyse chat logs at a large scale to persecute people?
It would be good to have a better answer to these questions than “eh, idk, good question but how are we supposed to know?”
In fact, it would be even better if we were able to design governance and collective intelligence systems that are resilient to the dangers that we can see coming. I think aiming for prediction is wrong due to the degree of complexification that will likely happen in the coming years; instead I think we should aim for antifragile design of societal systems.
If you’re in a swamp and trying to get somewhere whilst there’s a bunch of fog in the distance but you can see the outline of a cabin light in the distance, you tell people to be adaptable whilst treading lightly towards the cabin. You don’t tell them the exact route to the cabin since you don’t know it before time, the fog makes it so that you cannot see where you are going nor what (AI)ligators will show up in the swamp.
Yes, we need stories of what will happen as a way of communicating the risks but claiming certainty in any specific story leaves you very open to black swans.
I’ll be asking questions like: How can we increase the adaptivity of our democratic institutions? What are the foundational frames that we might want to change?
In this series, I'll go through some ways that we can use techniques from complexity science, political science, economics and computer science in order to create a new set of techniques to simulate and design new systems that can start to answer some of these questions. I will finish off the series with a framework for a more universal way of dealing with uncertainty through simulating the dynamics of sub-parts of systems so we can find new designs. Think of it as a wind tunnel for adaptive institutions.
First, we have to establish why this is such a tricky thing to do and it really has to do with computational irreducibility. We need a computer the size of the universe to simulate the universe so it is also hard to simulate the future.
The Game of Life
The game of life is a simple toy world of how the world works. You have a grid world where some of the squares are filled with a blob and some are not. Depending on the amount of neighbours you have you either reproduce, die or do nothing. This leads to cool behaviour. If you want to know more about the game of life you can check out an interactive simulation here and the Wikipedia page is also quite nice.
Figure 1: Image taken from the Wikipedia page on how the game of life can look like with different self-replicating patterns.
For our purposes, we're going to use it to talk about an interesting property. Namely that in order to predict the future of the system you actually have to run the system. It is computationally irreducible. That is, it is impossible to predict the future of the system without actually simulating it.
We can conclude that blobs that go around a grid world are computationally irreducible. If we turn our gaze to the game of real life it is quite a lot more complex and so we're in slight trouble if we're trying to predict the future of the world.
Yet, economics still seems useful, how can this be?
If we look back at the game of life, I might for example be able to bound the future population of the game of life by looking at the current population since there's a maximum growth rate in the system. That is, the future state is dependent on the past state and so we can limit the space of possible futures without running the whole simulation. Yet I wouldn't know how it actually will unfold.
There are a set of measures that we could call something like averaging measures where we take parts of the population and look at various population size effects. Doing this we're usually able to find empirical relations between things like average economic well-being and GDP among other things that are useful for predicting the future states of the world.
These averaging measures work well when future population dynamics follow past population dynamics and when you can assume linear effects. This is not always the case, if we look at something like a bank run there are non-linear effects where if a group of people start to think that they should take out money then other people see it and start thinking the same thing. This leads to a cascade and voila you have a financial crash.
Our systems are set up for humans and are generally more resilient to these sorts of things, we more or less know of a bank run. Yet, there's the flash crash of 2010 which is the AI version of this type of behaviour.
These were quite mundane and non-complicated AI systems compared to modern day AI agents. If we couldn't predict failures for simple trading algorithms, how are we supposed to predict them for agents that read the news, form strategies and coordinate with each other?
What can we do? Well, the obvious thing is to just simulate it in a similar way to what is said in the game of life. Yet as we said before, to simulate the universe, we need a computer of the size of the universe. So, the science of modelling society always has to make simplifying assumptions about what is going on.
There are different types of modelling approaches, some of them work in ideal conditions and not in unusual conditions, these types of models are generally called equilibrium models and some try to approximate the world with a more advanced version of the game of life. This is called agent based modelling.
We create these “agents” which are usually a group of RL models that learn specific policies about what to do. They compete in some sort of game such as a tragedy of the commons or something like a housing market and then they learn different policies over time based on what works.
Through these models we can find convergent strategies in local areas, a classic example is Axelrod's game theory tournaments where different types of agents are competing against each other. There's an intuitive and fun walkthrough of this in Nicky Case's The Evolution of Trust.
There are versions of this that exist with other types of systems as well, people are using LLMs to run some of these experiments such as in govsim.
In theory these systems could be used in order to forecast and predict the future. If we have LLMs that approximate the real world dynamics closely enough why couldn't we figure out how the future would look in the next couple of years? In the field of Cooperative AI there are a lot of people trying to solve problems related to these questions and trying to figure out how we could design good systems.
Yet, there's a hidden assumption that some of you might have spotted, namely where does the agent in agent based models come from? How do you actually define an agent? Aha! Agent Foundations is needed!
Let's turn to the problem of defining an agent within social systems as it relates to one of the recent famous lessons in Machine Learning, The Bitter Lesson.
The bitter lesson for ABMs
Sutton's The Bitter Lesson is a classic in modern ML spheres and the way I would summarise it is that in the creation of a learning objective or a model, you often bring in your own biases about the world. These biases may or may not be true and most times they're less useful than letting the model learn on its own.
ABMs are generally run to explain real world phenomena and underlying data. If we have some data X and we have some outcomes Y that we want to track, we try to set up an agent based model F that maps X to Y, F: X -> Y. We then try to see if our model predicts the world well, this is the quintessential form of a machine learning problem.
Figure 2: An agent-based model of the AI economy. Each company is an RL agent trained on economic reports and public statements to approximate its real-world counterpart; the model F then maps this micro-structure (X) to macro-variables like GDP (Y).
Yet, a lot of ABMs have the same good ol' problems that arise from the generation of hand crafted models.
For example, if you set up an ABM that is supposed to track the emergence of cooperation in a group and you add a specific term in the reward of the model that incentivises cooperation and you then get cooperation, you can't really claim to have seen the emergence of cooperation.
This is in fact a problem that transcends fields. One of my collaborators has worked on the origin of life and he mentions how he has seen many models where the outcomes are completely dependent on the assumptions brought into the model. I'm not an expert here but I imagine something like “oh, what if better communication leads to better rewards in an RNA model”, you then assume that you have RNA from the beginning and you ask whether or not RNA will grow in the population and voila, it does!
So, computational irreducibility tells us that we need to simulate systems to fully know what will happen but to do that we need really large computers unless we make assumptions. Yet in making assumptions we run into big trouble! What assumptions should we even make to reduce the world to a more computationally friendly place?
Stupid, annoying world! Be more nice!
So, how have fields actually solved this? Generally, it seems to me that there's been a lot of trial and horror and that the main gold standard is if it works in the real world. In social science ABMs are more treated as a thought experiment rather than a baseline verification of the real world and the thing that matters most is boots on the ground experimentation. (Which is slowly changing due to a certain J Doyne Farmer and crowd)
As far as I've looked into other fields, this also seems to be the case there. This is slightly problematic as it makes it hard to create good ABMs so that we can get good predictions of human + AI futures.
Am I then saying that we need to have some approximate solutions to agent foundations to predict the future of civilization? Yes, that is in fact what I'm saying and it is pretty obvious if you think about it.
Am I saying that we need to solve decision theory in detail in order to predict the future? No, not at all! (although it might be useful)
Normal models have agent based assumptions in them!
I may or may not have convinced you that ABMs are difficult. To summarise, you have to make the right types of assumptions, avoid smuggling in your own assumptions, then couple the model to reality. This is not easy for current day systems with normal agents.
Now, what if we suddenly start to get weird agents popping up all over the place? What if a human starts to morph into a company level agent when interacting with multiple AIs? Or what if what was before just part of the supply chain starts to form independent companies and start acting like an agent. An agent here being a self-contained goal-directed system with independent goals that is better modelled with the intentional stance.
What if there are really weird forces underneath these agents like cultural, political and economical forces that follow strange patterns and are likely to shape the future as described in gradual disempowerment? What the hell happens to our modelling assumptions, how do we even create these models in the first place, the boundaries seem to be shifting?
So much for the sophisticated models. But what happens to the coarse-grained predictive variables that depend on rational actor theory? Some degree of profit maximisation can definitely still be predicted, companies seem to somewhat reliably optimise for profit over time and we can use these larger parts of the system to predict part of where the economy is going. But what if companies change fundamentally due to AI Agents? What happens then?
Forget about labour shortages, how do we even calculate normal economic metrics in the first place? What are the possible types of collaborations and organisations that AI systems can start, do they even need normal coordination or can they do acausal coordination due to essentially being the same mind spread out across multiple different areas?
I think a bunch of our agent assumptions will break down in the next 10 years and as a consequence I also think there are things that will break down from a larger forecasting and economic perspective as well as it is somewhat dependent on rational actor theory which is dependent on a specific view of what agents are.
Every model in this post assumes that what constitutes an agent is fixed and there’s nothing that says that will remain true in the next 10 years.
What can we do in order to model this weirdness?
We could potentially look towards the place where mushrooms have over 7000 genders and where the tree of life turns out to be incredibly complicated due to horizontal gene transfer. Where weird distributed cognition systems like slime molds coordinate and change their behaviour in seconds when they meet other slime molds. We will look at biology for a science of really really weird agents.
That is where the next post begins. There I'll argue that the way out is to stop treating agents as the atoms of our models and start treating them as processes. That is, flows that can merge, split and redraw their own boundaries. I'll also try to survive an extended argument with an imaginary, increasingly annoyed Nassim Nicholas Taleb about why this matters for models of how society might look like.
Part 1 of a series on process alignment, a different way of thinking about agents at every scale.
Introduction
There are quite a lot of disagreements in the AI Safety and general AI community about what the future will look like. They’re often framed as differences in algorithmic progress or differences in compute scaling or the degree of unipolarity that we would want.
Yet they all seem to smuggle in various assumptions about how future AI systems will look like.
It feels like the forecasts are downstream of ontological commitments: untestable assumptions about what the future is made of and what its basic units even are. Are the players of the coming decades humans? Institutions? AI models? Hybrid human-AI collectives? Depending on how you answer, you get a different theory of what can go wrong and a different theory of governance for preventing it.
For example, will AI develop as it is described in AI 2027 or AI 2040? Or is there something that is fundamentally off with LLMs and the way that they’re currently setup as Richard Ngo writes in his critique of AI 2040? This completely changes the way that we should go about prevention and what we should focus on. This is also dependent on how we model agents in society.
Yet even if there is this fundamental uncertainty about what future agents will look like, we still need approximate solutions for the problems we will face as we can’t go into the future blindly. We somehow need to model the future so that we know what governance actions we can take.
From a computational perspective this is both a difficult and an easy problem.
It's an easy thing since a lot of systems in the world follow the law of large numbers and as a consequence we get to have nice predictability for industrial processes. It also turns out that standardization and predictability leads to an easier time making decisions and as a consequence we like a regulated predictable world. This makes it so that Epoch can put forth their scaling laws based on compute, algorithmic progress and data efficiency.
It's, however, also a hard thing since a subset of systems in the world don't follow the law of large numbers and one could state that the field of complexity science is generally about those systems. Also since the market is an adaptive changing system, it will be a modeler of the underlying dynamics of the world which is provably complex in various domains.
For example, one of the problems with linear forecasts is that it doesn't factor in places where there are discontinuities. In many military forecasts I heard about Ukraine, people were talking about the economy and the amount of resources that each side had and so on, and made forecasts based on that. What happened is that Ukraine developed a new type of drone warfare that wasn't foreseen which completely changed the underlying dynamics of the war. How are we supposed to bring these sorts of black swans into our models?
Generally, we don't model unknown unknowns and we instead simplify our economic models which is coincidentally why you can sometimes on quiet nights hear Nassim Nicholas Taleb scream about black swans in the distance. It seems that in some of the societal prognosis of AI we're thinking about various systems as if they're white swans. That is, we forget about the unknown unknowns that are implicit in our systems.
Yet, some would say, and this is quite a controversial claim, that AI might change the world in more ways than we can currently imagine. In fact, in an even more shocking statement, I would like to claim that our ways of measuring various societal functions are dependent on humans being the main constituents. This means that we might not be ready for the change that bringing a new type of agent into the world would bring.
How does a democracy work when you have agents that can send information brain to brain at the speed of fibre cables? What happens with our collective ability to make sense of things if the ability to output propaganda goes up by 10x? What happens with the balance of power between citizens and governments if the governments can analyse chat logs at a large scale to persecute people?
It would be good to have a better answer to these questions than “eh, idk, good question but how are we supposed to know?”
In fact, it would be even better if we were able to design governance and collective intelligence systems that are resilient to the dangers that we can see coming. I think aiming for prediction is wrong due to the degree of complexification that will likely happen in the coming years; instead I think we should aim for antifragile design of societal systems.
If you’re in a swamp and trying to get somewhere whilst there’s a bunch of fog in the distance but you can see the outline of a cabin light in the distance, you tell people to be adaptable whilst treading lightly towards the cabin. You don’t tell them the exact route to the cabin since you don’t know it before time, the fog makes it so that you cannot see where you are going nor what (AI)ligators will show up in the swamp.
Yes, we need stories of what will happen as a way of communicating the risks but claiming certainty in any specific story leaves you very open to black swans.
I’ll be asking questions like: How can we increase the adaptivity of our democratic institutions? What are the foundational frames that we might want to change?
In this series, I'll go through some ways that we can use techniques from complexity science, political science, economics and computer science in order to create a new set of techniques to simulate and design new systems that can start to answer some of these questions. I will finish off the series with a framework for a more universal way of dealing with uncertainty through simulating the dynamics of sub-parts of systems so we can find new designs. Think of it as a wind tunnel for adaptive institutions.
First, we have to establish why this is such a tricky thing to do and it really has to do with computational irreducibility. We need a computer the size of the universe to simulate the universe so it is also hard to simulate the future.
The Game of Life
The game of life is a simple toy world of how the world works. You have a grid world where some of the squares are filled with a blob and some are not. Depending on the amount of neighbours you have you either reproduce, die or do nothing. This leads to cool behaviour. If you want to know more about the game of life you can check out an interactive simulation here and the Wikipedia page is also quite nice.
Figure 1: Image taken from the Wikipedia page on how the game of life can look like with different self-replicating patterns.
For our purposes, we're going to use it to talk about an interesting property. Namely that in order to predict the future of the system you actually have to run the system. It is computationally irreducible. That is, it is impossible to predict the future of the system without actually simulating it.
We can conclude that blobs that go around a grid world are computationally irreducible. If we turn our gaze to the game of real life it is quite a lot more complex and so we're in slight trouble if we're trying to predict the future of the world.
Yet, economics still seems useful, how can this be?
If we look back at the game of life, I might for example be able to bound the future population of the game of life by looking at the current population since there's a maximum growth rate in the system. That is, the future state is dependent on the past state and so we can limit the space of possible futures without running the whole simulation. Yet I wouldn't know how it actually will unfold.
There are a set of measures that we could call something like averaging measures where we take parts of the population and look at various population size effects. Doing this we're usually able to find empirical relations between things like average economic well-being and GDP among other things that are useful for predicting the future states of the world.
These averaging measures work well when future population dynamics follow past population dynamics and when you can assume linear effects. This is not always the case, if we look at something like a bank run there are non-linear effects where if a group of people start to think that they should take out money then other people see it and start thinking the same thing. This leads to a cascade and voila you have a financial crash.
Our systems are set up for humans and are generally more resilient to these sorts of things, we more or less know of a bank run. Yet, there's the flash crash of 2010 which is the AI version of this type of behaviour.
These were quite mundane and non-complicated AI systems compared to modern day AI agents. If we couldn't predict failures for simple trading algorithms, how are we supposed to predict them for agents that read the news, form strategies and coordinate with each other?
What can we do? Well, the obvious thing is to just simulate it in a similar way to what is said in the game of life. Yet as we said before, to simulate the universe, we need a computer of the size of the universe. So, the science of modelling society always has to make simplifying assumptions about what is going on.
There are different types of modelling approaches, some of them work in ideal conditions and not in unusual conditions, these types of models are generally called equilibrium models and some try to approximate the world with a more advanced version of the game of life. This is called agent based modelling.
We create these “agents” which are usually a group of RL models that learn specific policies about what to do. They compete in some sort of game such as a tragedy of the commons or something like a housing market and then they learn different policies over time based on what works.
Through these models we can find convergent strategies in local areas, a classic example is Axelrod's game theory tournaments where different types of agents are competing against each other. There's an intuitive and fun walkthrough of this in Nicky Case's The Evolution of Trust.
There are versions of this that exist with other types of systems as well, people are using LLMs to run some of these experiments such as in govsim.
In theory these systems could be used in order to forecast and predict the future. If we have LLMs that approximate the real world dynamics closely enough why couldn't we figure out how the future would look in the next couple of years? In the field of Cooperative AI there are a lot of people trying to solve problems related to these questions and trying to figure out how we could design good systems.
Yet, there's a hidden assumption that some of you might have spotted, namely where does the agent in agent based models come from? How do you actually define an agent? Aha! Agent Foundations is needed!
Let's turn to the problem of defining an agent within social systems as it relates to one of the recent famous lessons in Machine Learning, The Bitter Lesson.
The bitter lesson for ABMs
Sutton's The Bitter Lesson is a classic in modern ML spheres and the way I would summarise it is that in the creation of a learning objective or a model, you often bring in your own biases about the world. These biases may or may not be true and most times they're less useful than letting the model learn on its own.
ABMs are generally run to explain real world phenomena and underlying data. If we have some data X and we have some outcomes Y that we want to track, we try to set up an agent based model F that maps X to Y, F: X -> Y. We then try to see if our model predicts the world well, this is the quintessential form of a machine learning problem.
Figure 2: An agent-based model of the AI economy. Each company is an RL agent trained on economic reports and public statements to approximate its real-world counterpart; the model F then maps this micro-structure (X) to macro-variables like GDP (Y).
Yet, a lot of ABMs have the same good ol' problems that arise from the generation of hand crafted models.
For example, if you set up an ABM that is supposed to track the emergence of cooperation in a group and you add a specific term in the reward of the model that incentivises cooperation and you then get cooperation, you can't really claim to have seen the emergence of cooperation.
This is in fact a problem that transcends fields. One of my collaborators has worked on the origin of life and he mentions how he has seen many models where the outcomes are completely dependent on the assumptions brought into the model. I'm not an expert here but I imagine something like “oh, what if better communication leads to better rewards in an RNA model”, you then assume that you have RNA from the beginning and you ask whether or not RNA will grow in the population and voila, it does!
So, computational irreducibility tells us that we need to simulate systems to fully know what will happen but to do that we need really large computers unless we make assumptions. Yet in making assumptions we run into big trouble! What assumptions should we even make to reduce the world to a more computationally friendly place?
Stupid, annoying world! Be more nice!
So, how have fields actually solved this? Generally, it seems to me that there's been a lot of trial and horror and that the main gold standard is if it works in the real world. In social science ABMs are more treated as a thought experiment rather than a baseline verification of the real world and the thing that matters most is boots on the ground experimentation. (Which is slowly changing due to a certain J Doyne Farmer and crowd)
As far as I've looked into other fields, this also seems to be the case there. This is slightly problematic as it makes it hard to create good ABMs so that we can get good predictions of human + AI futures.
Am I then saying that we need to have some approximate solutions to agent foundations to predict the future of civilization? Yes, that is in fact what I'm saying and it is pretty obvious if you think about it.
Am I saying that we need to solve decision theory in detail in order to predict the future? No, not at all! (although it might be useful)
Normal models have agent based assumptions in them!
I may or may not have convinced you that ABMs are difficult. To summarise, you have to make the right types of assumptions, avoid smuggling in your own assumptions, then couple the model to reality. This is not easy for current day systems with normal agents.
Now, what if we suddenly start to get weird agents popping up all over the place? What if a human starts to morph into a company level agent when interacting with multiple AIs? Or what if what was before just part of the supply chain starts to form independent companies and start acting like an agent. An agent here being a self-contained goal-directed system with independent goals that is better modelled with the intentional stance.
What if there are really weird forces underneath these agents like cultural, political and economical forces that follow strange patterns and are likely to shape the future as described in gradual disempowerment? What the hell happens to our modelling assumptions, how do we even create these models in the first place, the boundaries seem to be shifting?
So much for the sophisticated models. But what happens to the coarse-grained predictive variables that depend on rational actor theory? Some degree of profit maximisation can definitely still be predicted, companies seem to somewhat reliably optimise for profit over time and we can use these larger parts of the system to predict part of where the economy is going. But what if companies change fundamentally due to AI Agents? What happens then?
Forget about labour shortages, how do we even calculate normal economic metrics in the first place? What are the possible types of collaborations and organisations that AI systems can start, do they even need normal coordination or can they do acausal coordination due to essentially being the same mind spread out across multiple different areas?
I think a bunch of our agent assumptions will break down in the next 10 years and as a consequence I also think there are things that will break down from a larger forecasting and economic perspective as well as it is somewhat dependent on rational actor theory which is dependent on a specific view of what agents are.
Every model in this post assumes that what constitutes an agent is fixed and there’s nothing that says that will remain true in the next 10 years.
What can we do in order to model this weirdness?
We could potentially look towards the place where mushrooms have over 7000 genders and where the tree of life turns out to be incredibly complicated due to horizontal gene transfer. Where weird distributed cognition systems like slime molds coordinate and change their behaviour in seconds when they meet other slime molds. We will look at biology for a science of really really weird agents.
That is where the next post begins. There I'll argue that the way out is to stop treating agents as the atoms of our models and start treating them as processes. That is, flows that can merge, split and redraw their own boundaries. I'll also try to survive an extended argument with an imaginary, increasingly annoyed Nassim Nicholas Taleb about why this matters for models of how society might look like.