Epistemic status: Rough vibe-y ideas written under time pressure because otherwise I wouldn't write anything; I'm a programming language and p2p protocol developer falling down multiple convergently-evolving rabbit-shaped holes.
Questions
What is an agent? How do we think meaningfully about how cells or humans or societies maintain themselves and are changed by their environment? How do we think about the patterns where historically lineages of agents have gone extinct... to hopefully avoid this, or if it is inevitable, to delay or come to terms with it? This is the questions posed to the field of agent foundations. What would a theory that tells us this look like? Or better question, different theories (?)
It would have to somehow take observational data on a particular agent and environment and make some prediction of how that agent adapts to its environment. One potential RSI-relevant example here might be "take data on the informational dynamics of LLM training and use and predict how it will improve itself". This seems hopeless on its face, but there are some meaningful patterns. The most hopeful of which is on highlight in Scott's The Goddess of Everything Else, which observes that multi-agent systems seem to tend towards becoming joint/"super"-agents over time. The progression historically has been something like:
Prokaryotes -> Eukaryotes
Eukaryotes -> Multi-Cellular Organisms
Organisms -> Communities/Groups/Cultures of organisms (this is where we are!)
There does seem to be a difference or perhaps a continuum here. I.e. perhaps the evolution process hasn't quite been the same for prokaryotes/eukaryotes vs eukaryotes/organisms vs organism/multi-organism communities? Prokaryotes inside eukaryotes seem to be a rather close-knit symbiotic system where the outer organism has full control over the inner organism's reproduction. Thus, you don't quite have issues like cancer (or at least I am unaware of analogies at the moment of writing). Multi-celled systems seem to be similarly close-knit (in that it's rather hard for cells to exit), but there are more instances in my mind of cells escaping a multi-cellular context compared to a single-celled one. Both of these contexts are much different from human societies (or to be fair, even some meaningfully multi-cellular or collective prokaryotic organizations). But perhaps because of environment (or in this case, geography) human super-agents are much more fuzzy than the lower down superagent configuration.
It is weird patterns like this that I think an agent foundations theory would need to explain. It would need to somehow extract some course-grained structure from the nature of the environments (things possibly like, how easy is information flow? how regularizing on structure or reproduction are things like governments?). There are quite a few parallels between a body and a civilization, the most obvious of which is unique cells/humans that police other cells/humans to make sure behavior is in a certain desirable range. We even have civilizational approximations of autoimmune diseases, where the body thinks a given cell's behavior is harmful but it is ultimately mostly fine and the damage to the body as a whole for fighting is greater (e.g. racism, civil war, minority rights). A first-approximation research question here is why are eukaryotic/prokaryotic systems so tight-knit compared to human civilizations, and is this what we are heading towards? If so, what does tight-knittedness look like in a context where each composing subagent is itself a decently powerful optimizer (as is the case in the multi-celled case?). Can we critically study biology to find patterns that match human civilizations and make meaningful predictions here? Like what contexts create slime molds or colonies vs strongly regularizing superagents (like humans)? Can we predict which we get from the information distribution structure / environmental affordances, and if so, what does that mean for our likely future?
Explorations
These are huge questions, but I've been thinking about potential directions. Watching this video about city development under restrictive land use by Lars Doulcet and realizing that it kinda looked like a softmax got me thinking about potential parallels between capitalism/civilizations and neural networks. Exactly how close of an analogy could there be between humans and cells, police and immune systems, etc.? One thing that I discovered and that Claude seemed to confirm is that there is a direct correspondence between capital allocation on land (the built environment) and the weights of a neural network, and furthermore there is a correspondence between land itself with the directions of the residual stream in a neural network. I explored this correspondence and ended up with the following model (link) that claims to find a rather precise correspondence between spiking neural networks and a city development.
Here's the main idea:
And correspondence table (incomplete):
From the same equilibrium law, it seems to be able to find quite precise correspondences between land use and neuronal connection, such as districting:
This model is a rather complicated functional/energy-minimization model and while it is possible to write any neural network this way, neural network architectures seem to be much more uniform / designed (as opposed to neurons, which I have some intuition that they can just grow together to use the space effectively). I'm kinda curious if like most of the capabilities/architectural development in transformers (MoE, recursions, better attention mechanisms, etc.) is just kinda trying to approximate a more emergent local learning rule but on our somewhat strange Von Neumann-ish pipelined architectures. Maybe most transformer optimization is just a progression to re-create the neuron wiring dynamics of biology that can be faithful to the local nature of computation? (╭ರ_•́)
Speculation
It seems to me the next step here is to try and take models like this that seem to have quite a bit of emergent behavior that can be finagled to match interesting patterns that look like things that might appear in neural networks or land development and see if we can actually get it to fit real data. This might look like searching for a given information topology of a given complexity, seeing what observations it would produce when evolved/simulated over time, and then seeing if any patterns like that could be found in data. This seems rather compute intensive, but the nice thing is it seems like it might be course-grainable, and applicable to smaller things. In the end this probably looks something like a economics foundation model. If we had such a thing, it would be possible to roll out potential futures under different regularizers / policies (like LVT!) and see if we like the properties of those rollouts or not. I suspect this might not explain all of our problems (because there are always limits to the ability we can measure, and the size of search space we can search) but it might help. (Or it might hurt / aggregate power)
Another problem I wish a model like this would explain is—if we go back to the first section—it feels quite intuitive that there's some kind of fundamental progression to life. Life seems to keeps forming agents that keep composing themselves into tighter and tighter-knit organisms with progressively less ability (desire?) to leave their associated super-agents. If we see ourselves, (or perhaps the capital and AI we build) as the latest incarnation of this process, what does that imply? People are scared of being replaced, or being made unnecessary / disempowered / outright disregarded as impediments to a more powerful being. I think an apt analogy here is what we've done to extinct or enslave other species purely out of accident or desire for their delicious meat. (That is a common theme of our AI-related futurist horror). One place we might be able to look for answers is to see if it's possible to make a meaningful model of historical biologically-caused mass extinction. The major one is the great oxygenation event, but I'm sure there are others. Is there a good model for the causes of these extinctions, and does that imply anything about our possibly-impeding one? My gut tells me it has something to do with insufficient regularizers / inequality of capability, (and I suspect some people here agree, as human self-improvement is like basically the original point of this website), but if the choice is extinction or adaptation, how do we make the adaptation as painless and potentially gradual as possible? I for one am a fan of progressive mind-upload/augmentation like The Gentle Romance... Is anyone working on this? Gwern?
Some miscellaneous possible connections
It's all about learning rate. Being forced into a higher learning rate can be painful (you have to destroy parts of yourself or reconfigure parts of yourself to survive). Seeing someone else with a higher learning rate becoming more powerful is scary, it means you might be the one being reconfigured. Examples: rust belt, being poor, being scared of LVT because it might make old people move.
I'm a programming language developer so I want to think about agents as like "fuzzy traits": they have an optimizer and some kind of function that says when the optimizer has found a good state. The optimizer (brain) is made up of sub-agents (cells!) and the priors that guide the optimizer's heuristics are precisely the coordination mechanism between the sub-agents's value functions. This seems maybe like a plausible direction.
In the development of my PL I am researching these things called "interaction nets", in these nets you have a big graph of nodes called "agents" when certain ports of the agent nodes are connected to certain other ports, a local graph rewrite rule can fire and this is how computation is done. (The nice thing about inets is that it allows for automatic parallelization of everything that can be parallelized and then its just up to the executor to figure out exactly how to organize this—something something local learning heuristics? cellular automata?)
Conclusion
No conclusion yet as this is still an active research space. Thanks for reading my crazy ideas! Feel free to put your own weird hunches in the comments.
Epistemic status: Rough vibe-y ideas written under time pressure because otherwise I wouldn't write anything; I'm a programming language and p2p protocol developer falling down multiple convergently-evolving rabbit-shaped holes.
Questions
What is an agent? How do we think meaningfully about how cells or humans or societies maintain themselves and are changed by their environment? How do we think about the patterns where historically lineages of agents have gone extinct... to hopefully avoid this, or if it is inevitable, to delay or come to terms with it? This is the questions posed to the field of agent foundations. What would a theory that tells us this look like? Or better question, different theories (?)
It would have to somehow take observational data on a particular agent and environment and make some prediction of how that agent adapts to its environment. One potential RSI-relevant example here might be "take data on the informational dynamics of LLM training and use and predict how it will improve itself". This seems hopeless on its face, but there are some meaningful patterns. The most hopeful of which is on highlight in Scott's The Goddess of Everything Else, which observes that multi-agent systems seem to tend towards becoming joint/"super"-agents over time. The progression historically has been something like:
There does seem to be a difference or perhaps a continuum here. I.e. perhaps the evolution process hasn't quite been the same for prokaryotes/eukaryotes vs eukaryotes/organisms vs organism/multi-organism communities?
Prokaryotes inside eukaryotes seem to be a rather close-knit symbiotic system where the outer organism has full control over the inner organism's reproduction. Thus, you don't quite have issues like cancer (or at least I am unaware of analogies at the moment of writing). Multi-celled systems seem to be similarly close-knit (in that it's rather hard for cells to exit), but there are more instances in my mind of cells escaping a multi-cellular context compared to a single-celled one. Both of these contexts are much different from human societies (or to be fair, even some meaningfully multi-cellular or collective prokaryotic organizations). But perhaps because of environment (or in this case, geography) human super-agents are much more fuzzy than the lower down superagent configuration.
It is weird patterns like this that I think an agent foundations theory would need to explain. It would need to somehow extract some course-grained structure from the nature of the environments (things possibly like, how easy is information flow? how regularizing on structure or reproduction are things like governments?). There are quite a few parallels between a body and a civilization, the most obvious of which is unique cells/humans that police other cells/humans to make sure behavior is in a certain desirable range. We even have civilizational approximations of autoimmune diseases, where the body thinks a given cell's behavior is harmful but it is ultimately mostly fine and the damage to the body as a whole for fighting is greater (e.g. racism, civil war, minority rights). A first-approximation research question here is why are eukaryotic/prokaryotic systems so tight-knit compared to human civilizations, and is this what we are heading towards? If so, what does tight-knittedness look like in a context where each composing subagent is itself a decently powerful optimizer (as is the case in the multi-celled case?). Can we critically study biology to find patterns that match human civilizations and make meaningful predictions here? Like what contexts create slime molds or colonies vs strongly regularizing superagents (like humans)? Can we predict which we get from the information distribution structure / environmental affordances, and if so, what does that mean for our likely future?
Explorations
These are huge questions, but I've been thinking about potential directions. Watching this video about city development under restrictive land use by Lars Doulcet and realizing that it kinda looked like a softmax got me thinking about potential parallels between capitalism/civilizations and neural networks. Exactly how close of an analogy could there be between humans and cells, police and immune systems, etc.? One thing that I discovered and that Claude seemed to confirm is that there is a direct correspondence between capital allocation on land (the built environment) and the weights of a neural network, and furthermore there is a correspondence between land itself with the directions of the residual stream in a neural network. I explored this correspondence and ended up with the following model (link) that claims to find a rather precise correspondence between spiking neural networks and a city development.
Here's the main idea:
And correspondence table (incomplete):
From the same equilibrium law, it seems to be able to find quite precise correspondences between land use and neuronal connection, such as districting:
Or various kinds of polysemanticity (temporal, spatial):
And of course something akin to a residual stream (?)
This model is a rather complicated functional/energy-minimization model and while it is possible to write any neural network this way, neural network architectures seem to be much more uniform / designed (as opposed to neurons, which I have some intuition that they can just grow together to use the space effectively). I'm kinda curious if like most of the capabilities/architectural development in transformers (MoE, recursions, better attention mechanisms, etc.) is just kinda trying to approximate a more emergent local learning rule but on our somewhat strange Von Neumann-ish pipelined architectures. Maybe most transformer optimization is just a progression to re-create the neuron wiring dynamics of biology that can be faithful to the local nature of computation? (╭ರ_•́)
Speculation
It seems to me the next step here is to try and take models like this that seem to have quite a bit of emergent behavior that can be finagled to match interesting patterns that look like things that might appear in neural networks or land development and see if we can actually get it to fit real data. This might look like searching for a given information topology of a given complexity, seeing what observations it would produce when evolved/simulated over time, and then seeing if any patterns like that could be found in data. This seems rather compute intensive, but the nice thing is it seems like it might be course-grainable, and applicable to smaller things. In the end this probably looks something like a economics foundation model. If we had such a thing, it would be possible to roll out potential futures under different regularizers / policies (like LVT!) and see if we like the properties of those rollouts or not. I suspect this might not explain all of our problems (because there are always limits to the ability we can measure, and the size of search space we can search) but it might help. (Or it might hurt / aggregate power)
Another problem I wish a model like this would explain is—if we go back to the first section—it feels quite intuitive that there's some kind of fundamental progression to life. Life seems to keeps forming agents that keep composing themselves into tighter and tighter-knit organisms with progressively less ability (desire?) to leave their associated super-agents. If we see ourselves, (or perhaps the capital and AI we build) as the latest incarnation of this process, what does that imply? People are scared of being replaced, or being made unnecessary / disempowered / outright disregarded as impediments to a more powerful being. I think an apt analogy here is what we've done to extinct or enslave other species purely out of accident or desire for their delicious meat. (That is a common theme of our AI-related futurist horror). One place we might be able to look for answers is to see if it's possible to make a meaningful model of historical biologically-caused mass extinction. The major one is the great oxygenation event, but I'm sure there are others. Is there a good model for the causes of these extinctions, and does that imply anything about our possibly-impeding one? My gut tells me it has something to do with insufficient regularizers / inequality of capability, (and I suspect some people here agree, as human self-improvement is like basically the original point of this website), but if the choice is extinction or adaptation, how do we make the adaptation as painless and potentially gradual as possible? I for one am a fan of progressive mind-upload/augmentation like The Gentle Romance... Is anyone working on this? Gwern?
Some miscellaneous possible connections
Conclusion
No conclusion yet as this is still an active research space. Thanks for reading my crazy ideas! Feel free to put your own weird hunches in the comments.