Summary: Reinforcement/ Positive feedback loops can be a big deal and looking for them could be a good strategy to both compare and find causes and interventions.
Most of this post feels pretty obvious and intuitive to me, but I haven't seen anyone else writing it. More importantly, there are important conclusions that I didn't arrive at until I decided to write this post, and I think they could be useful.
Understanding reinforcement loops
Reinforcement loops drive systems toward exponential growth trajectories, provided they are not constrained by balancing feedback loops or resource limits. Depending on whether this runaway growth is considered beneficial or harmful, these loops are generally categorized as 'virtuous' or 'vicious'.
Vicious reinforcement loops
In vicious feedback loops, runaway growth poses a central danger: if humanity generally needs time to adapt to emerging problems, the sheer speed of these accelerating processes leaves very little time to do so. You can find some of these at the core of most x-risks:
AI risk: Recursive self-improvement is the classic example of this, where AI progress increases cognitive capability, which is then fed back into accelerating AI research, leading to exponential growth. Personally, I don't believe this is the core of the threat; a slow takeoff would still lead to a superintelligence that thinks faster than us, and that's the real threat. Or even an AGI that can spread fast through devices like a virus. Speaking of which:
Biorisk: Pandemics are notoriously dangerous because of their potential for exponential transmission, which occurs whenever the reproductive rate of a pathogen remains above 1.
Climate change: Runaway climate change is driven by powerful physical feedback loops, such as the ice-albedo feedback (where melting ice reduces reflectivity, increasing heat absorption and causing more ice to melt) and permafrost thawing (which releases methane and carbon dioxide, further warming the atmosphere). This is why it is expected that after a certain tipping point, it becomes a lot harder to stop the snowball from rolling down the hill[1].
Nuclear risk: In nuclear weapons, supercriticality relies on a positive feedback loop of neutron-triggered fission reactions. This exponential release of energy is thankfully arrested by a negative feedback loop known as "hydrodynamic disassembly." At the geopolitical level, nuclear escalation is also a reinforcement loop that would actually make it an x-risk (or at least catastrophic).
Nanotechnology: The classic "grey goo" scenario represents an existential nanotechnology threat where self-replicating assemblers consume material exponentially to create more assemblers, rapidly depleting resources.
Extreme power concentration: Theoretically, power recursively grants access to more power. Yet, when we look at empirical data like wealth inequality and the Gini coefficient, the trend toward global, long-term rising inequality is less straightforward. When it comes to real humans (and groups of them), the real dynamics are more complex.
But that could be changing right now. Automation and new technologies like AI, life extension, social media and globalization could end up empowering certain actors more than ever.
Virtuous reinforcement loops
Virtuous reinforcement loops with leverage points could allow for interventions where efforts have Increasing Marginal Returns instead of Diminishing Marginal Returns, which would be amazing. This post goes more in depth why we might find this leverage in systemic change interventions.
Some potential ones that come to mind:
Economic growth: The problem with increasing this is that I believe it would also accelerate AI capabilities, economic inequality and power concentration.
Democracy: One obvious way for limiting power concentration is to increase the power of everyone else, and the same logic that applies to one person, I believe, applies to a group of them. I do think of democracy as an attractor state, and the more you can increase it either globally or inside a specific country, the better.
Education: But if you have democracy while the people are uneducated, you get bad governments[2]. Besides political power, knowledge can also be distributed and, if done correctly, can also trigger positive feedback loops[3] like AGIs. I mean, that’s the story of human learning in general. If we use a wide meaning of “education” I think this can include improving AI character, social media and journalism.
Self-improvement: Individually, other things can self-improve besides AI, like
EA/RAT/good movement Movement growth: Naively, one could think a movement could grow virally like an internet meme, but in reality people would feel it is some kind of pyramid scheme-ish cult and feel really negatively about it. Furthermore, even if viral growth were possible, there are compelling arguments that rapid growth could be counterproductive.
Jhanas: People[4] have described them as an incredibly positive experience that you can arrive looping it around itself. "It’s pretty reasonable to expect that humans can learn the mental motions to reliably spark a positive emotion — we can obviously do it with negative emotions by simply visualizing a triggering situation, and we can learn how to reliably do the same thing but for positive emotions. (...) Then, hopefully it’s not hard to see how (with enough practice) this can develop into a more stable, continuous positive feedback loop."
Not clearly good nor bad reinforcement loops
Depends on the specific case. Others came to mind, but I think they are not valuable to mention.
Behaviors/ ideas/ OG "memes": Seem to take a longer time to spread than other things in these lists, but they have huge impact if they reach the whole. One specific type of them to have in mind are technologies, and another, more theoretical one, are strategies for agents about how to cooperate with others; sometimes in game theory simulations, some strategies can become dominant and even spread through the whole population.
Social/ traditional media content/ trends/ other type of memes: Quick spread, but the fact that they don't end up reaching everyone may signal something important about human minds.
Conclusions
When comparing causes and interventions, considering virtuous loops in the analysis could be modeled as an increase in the tractability of the cause/ intervention or its impact over time, and considering vicious loops could be modeled as a decrease in them.
Besides this, I believe we could find new cause areas by searching for both virtuous and vicious loops. This can be approached both theoretically, by modeling potential feedback structures, and empirically, by searching for phenomena in the world currently experiencing growth, exponential growth in particular.
Of course, negative feedback loops should be considered too; I just believe it’s more important to look for positive ones.
Do you have any positive feedback loops in mind that you think the community should focus on or investigate further?
Edit (26-10): added the clarification of what I think is the core x-risk in AI, the not clearly good nor bad section, Affective Death Spirals and Jhanas.
(Disclosure: This post was drafted and refined with the assistance of Gemini and Grammarly to help polish some paragraphs, correct grammar and technical terms, and fact-check key arguments.)
Summary: Reinforcement/ Positive feedback loops can be a big deal and looking for them could be a good strategy to both compare and find causes and interventions.
Most of this post feels pretty obvious and intuitive to me, but I haven't seen anyone else writing it. More importantly, there are important conclusions that I didn't arrive at until I decided to write this post, and I think they could be useful.
Understanding reinforcement loops
Reinforcement loops drive systems toward exponential growth trajectories, provided they are not constrained by balancing feedback loops or resource limits. Depending on whether this runaway growth is considered beneficial or harmful, these loops are generally categorized as 'virtuous' or 'vicious'.
Vicious reinforcement loops
In vicious feedback loops, runaway growth poses a central danger: if humanity generally needs time to adapt to emerging problems, the sheer speed of these accelerating processes leaves very little time to do so. You can find some of these at the core of most x-risks:
But that could be changing right now. Automation and new technologies like AI, life extension, social media and globalization could end up empowering certain actors more than ever.
Virtuous reinforcement loops
Virtuous reinforcement loops with leverage points could allow for interventions where efforts have Increasing Marginal Returns instead of Diminishing Marginal Returns, which would be amazing. This post goes more in depth why we might find this leverage in systemic change interventions.
Some potential ones that come to mind:
"It’s pretty reasonable to expect that humans can learn the mental motions to reliably spark a positive emotion — we can obviously do it with negative emotions by simply visualizing a triggering situation, and we can learn how to reliably do the same thing but for positive emotions.
(...) Then, hopefully it’s not hard to see how (with enough practice) this can develop into a more stable, continuous positive feedback loop."
Not clearly good nor bad reinforcement loops
Depends on the specific case. Others came to mind, but I think they are not valuable to mention.
Conclusions
When comparing causes and interventions, considering virtuous loops in the analysis could be modeled as an increase in the tractability of the cause/ intervention or its impact over time, and considering vicious loops could be modeled as a decrease in them.
Besides this, I believe we could find new cause areas by searching for both virtuous and vicious loops. This can be approached both theoretically, by modeling potential feedback structures, and empirically, by searching for phenomena in the world currently experiencing growth, exponential growth in particular.
Of course, negative feedback loops should be considered too; I just believe it’s more important to look for positive ones.
Do you have any positive feedback loops in mind that you think the community should focus on or investigate further?
Edit (26-10): added the clarification of what I think is the core x-risk in AI, the not clearly good nor bad section, Affective Death Spirals and Jhanas.
(Disclosure: This post was drafted and refined with the assistance of Gemini and Grammarly to help polish some paragraphs, correct grammar and technical terms, and fact-check key arguments.)
CFR - How Climate Change Can Get Even Worse
Osho - Democracy
Example: EA Forum - 14 Techniques to Accelerate Your Learning