Modern industrial society is built to make our lives safe, convenient, and comfortable. Electricity is used to moderate the temperature of our homes, keeping us warm through the bitterest winters, and cool through the hottest summers- even through weather extremes we may not otherwise survive. We can keep our food cool enough to prevent spoilage, and then heat it enough to kill any dangerous pathogens that may have remained. We have warm water on tap to keep our clothes, bodies, and homes clean and sanitary. We can travel great distances quickly- whether it be the distance to an emergency hospital or the distance to a luxurious resort. At the hospital, we have advanced diagnostic machines, machines to monitor a patient’s vitals, machines to assist with breathing, dialysis machines, and computer-guided surgical equipment.
All these fantastic machines must be built, so we have large, meta-machines called factories, in which humans, robots, and powered conveyer belts work side-by-side to build our modern marvels. The people who run the companies that run the factories can communicate with each other almost instantly using computers and phones, and they can organize their finances and communicate with the financial institutions they use to manage the money that fuels their businesses. Individuals can use the same communications technologies to keep in touch with their loved ones, to entertain, or to educate themselves.
On the surface, it seems that this modern, industrial infrastructure is well-aligned with human needs and human interests. Ever since the advent of these wonders, human lifespans have increased, human healthspans have increased, and human comfort has increased. Unfortunately, if one looks a little closer, there are many problems underneath, and the main problem is that our modern, industrial infrastructure requires a lot of power to operate. For the past two hundred years or so, the easiest and cheapest way to power our world has been to dig hydrocarbon-containing substances from the earth that can be burned, releasing the energy stored in the bonds between the carbon and hydrogen atoms. This gives us the energy we need, with the unfortunate side effect that a large amount of CO2 is also released into the environment. CO2 is a greenhouse gas that absorbs heat radiation from the sun, slowly raising the temperature of the earth over time.
Humans will find it more difficult to survive as the temperature of their environment increases. Human survival should be the first and most basic consideration before any system is built, and yet, if the danger to humans cannot be immediately and obviously seen, it often is not given priority. There is often a gap between immediate and long-term goals, and between narrow and broad goals. If the gap is wide enough, humanity can fall into it.
#
What is the difference between ensuring a single factory’s product is safe for the use of the consumer over a single lifetime, and ensuring the interactions of each node in the larger system around these factories and their products are all safe for society into an indefinite future?
Let’s start by looking at a very simple system.
One day, someone invents a new product and decides to build a factory to produce the product. When one inputs blue gobbledy and red gook, the factory outputs purple gobbledygook. Gobbledy and gook have been safely grown, harvested, and consumed by people for generations, and a 5 year trial has showngobbledygook is safe for human consumption. Thus, the gobbledygook factory has been given the go-ahead to begin large-scale production. I’ll call this level-1 safety, which represents simple product-level safety.
Gobbledygook becomes a popular product, and can now be found in most people’s homes and offices. Since gobbledygook is a good replacement for balderdash, being cheaper and simpler to produce, the balderdash industry has been decimated. Some people lament the loss of the balderdash industry, since it employed more people than the gobbledygook industry. Most people don’t mind, however, since gobbledygookis a superior product and produces less waste. While gobbledygook remains safe for human consumption, it’s discovered that marmosets become ill when they accidentally consume gobbledygook. Marmosets are a popular pet, and so many marmoset lives are sadly lost. In addition, the communities closest to thegobbledygook factory begin to notice that the manufacture of gobbledygook creates a purple byproduct, which is turning the nearby streams and ponds purple. These problems are emblematic of level-2 safety. Level-2 safety issues were not unanticipated. It’s common wisdom that when products hit a larger and more complex environment outside of the initial testing conditions, unanticipated problems will arise. Even so, no one has figured out how to account for the unanticipated issues in a complex environment, so the gobbledygook factory was forced to take reactive measures after their product was deployed. Warning labels that gobbledygook must be kept away from pet marmosets are added to each package of gobbledygook, and the factory owners create purple filters to deal with the purple waste. These stop-gap measures are probably not sufficient, but though this is a big problem it’s still a visible one.
The popularity of gobbledygook continues, unabated, 10 years into the future. After 10 years, people notice that the purple waste that has been captured and disposed of in landfills has seeped into groundwater, and now most people in a 30 mile radius of each gobbledygook factory have a purple tongue. The marmoset population has declined, and this has caused the decline in the population of the natural predator of the marmoset, as well as the destruction of the once-thriving pet marmoset economy. New studies have shown that balderdash is an excellent cure for the hiccups, but balderdash is now very rare and expensive, so the great hiccup epidemic causes worldwide suffering. These are Level-3 safety issues; issues have become more complex over time because of the initial effects chaining into other systems. Level-3 safety is not talked about as often as level-2, because it’s largely seen as the same issue as level-2 safety. Level-3 safety, however, is distinct- this level represents cascading risk. Level-3 safety is neglected because if we solve the level-2 issues, we think we are done, and do not check back as often as we should as time increases system complexity. After all- things have been running for 10 years and it’s been fine. The system appears stable if problems emerge slowly enough. If there is one place where “the gap” is missed, it is likely missed here.
Now, with so much purple in the groundwater, both gobbledy and gook crops are becoming more purple than their previous blue/red. This has caused gobbledygook to transform from purple to ultrapurple.One amazing property of ultrapurple is to leech purple from its environment. The more ultrapurplegobbledygook becomes, the more it leeches purple from the environment, leading it to become even more ultrapurple. Soon, purple is completely gone from the environment, and gobbledygook becomes so ultrapurple that it can leech purple from the sun, and then nearby stars. There are few people who had foreseen this outcome. These few did not necessarily think gobbledygook would enter an untrapurple feedback loop, but they knew something, some day, would hit a similar feedback loop, and that the results would be devastating. Level-2 safety issues existed in many products, but these safety issues never became self-sustaining. No one knew how to tell which safety issues would reach level-4. No one knew what would enable a feedback loop to sustain itself for this long- to drain the earth and sun of all its purple before it would collapse.
No one saw the gap.
#
If I’m a gobbledygook manufacturer, I want my product to be useful enough to sell. If my product is dangerous in the short term- if the gobbledygook is ultrapurple enough to drain all of the useful purple from my office, or is at least too purple to look at directly without harming my eyes, no one will want to purchase my product. I have to make my product just safe enough to sell in the short term before I can make a profit and have a successful company. But the mere possibility of a dangerous ultrapurple product emerging 10 years in the future will not cause my company to lose profit now.
This is the gap jumper- usefulness vs. safety. A system will be made just safe enough to ensure a thing is useful in the short-term and in a narrow environment. A system only needs to be ‘good enough’ for its particular niche, because anything that goes wrong outside of that market niche, on its current iteration, is only a problem once the usefulness is challenged and not before. Most systems will lose usefulness and collapse before they hit safety level-4. Few systems will become entrenched enough to grow and collapse the systems that exist outside of their niche. Most people will not see level-4 safety issues coming because they do not look ahead to see where the gaps will be, and even those who do look ahead cannot clearly see where the gaps are.
A gap is, by its very nature, invisible. Even those who can anticipate and warn others about the gaps will not be able to say where everything will go wrong. It’s possible that by naming the safety levels, it may be easier to not only forecast but also explain where safety gaps may exist. Also, since level-3 seems to be where the gap is the widest, it may be useful to check and re-check systems as time increases complexity. Unfortunately, a system that grows too quickly, or one unprecedented enough that we can’t anticipate growth rate, will not be easily monitored as time goes by. Such systems should be halted for more intense study before they are allowed to operate.
Modern industrial society is built to make our lives safe, convenient, and comfortable. Electricity is used to moderate the temperature of our homes, keeping us warm through the bitterest winters, and cool through the hottest summers- even through weather extremes we may not otherwise survive. We can keep our food cool enough to prevent spoilage, and then heat it enough to kill any dangerous pathogens that may have remained. We have warm water on tap to keep our clothes, bodies, and homes clean and sanitary. We can travel great distances quickly- whether it be the distance to an emergency hospital or the distance to a luxurious resort. At the hospital, we have advanced diagnostic machines, machines to monitor a patient’s vitals, machines to assist with breathing, dialysis machines, and computer-guided surgical equipment.
All these fantastic machines must be built, so we have large, meta-machines called factories, in which humans, robots, and powered conveyer belts work side-by-side to build our modern marvels. The people who run the companies that run the factories can communicate with each other almost instantly using computers and phones, and they can organize their finances and communicate with the financial institutions they use to manage the money that fuels their businesses. Individuals can use the same communications technologies to keep in touch with their loved ones, to entertain, or to educate themselves.
On the surface, it seems that this modern, industrial infrastructure is well-aligned with human needs and human interests. Ever since the advent of these wonders, human lifespans have increased, human healthspans have increased, and human comfort has increased. Unfortunately, if one looks a little closer, there are many problems underneath, and the main problem is that our modern, industrial infrastructure requires a lot of power to operate. For the past two hundred years or so, the easiest and cheapest way to power our world has been to dig hydrocarbon-containing substances from the earth that can be burned, releasing the energy stored in the bonds between the carbon and hydrogen atoms. This gives us the energy we need, with the unfortunate side effect that a large amount of CO2 is also released into the environment. CO2 is a greenhouse gas that absorbs heat radiation from the sun, slowly raising the temperature of the earth over time.
Humans will find it more difficult to survive as the temperature of their environment increases. Human survival should be the first and most basic consideration before any system is built, and yet, if the danger to humans cannot be immediately and obviously seen, it often is not given priority. There is often a gap between immediate and long-term goals, and between narrow and broad goals. If the gap is wide enough, humanity can fall into it.
#
What is the difference between ensuring a single factory’s product is safe for the use of the consumer over a single lifetime, and ensuring the interactions of each node in the larger system around these factories and their products are all safe for society into an indefinite future?
Let’s start by looking at a very simple system.
One day, someone invents a new product and decides to build a factory to produce the product. When one inputs blue gobbledy and red gook, the factory outputs purple gobbledygook. Gobbledy and gook have been safely grown, harvested, and consumed by people for generations, and a 5 year trial has showngobbledygook is safe for human consumption. Thus, the gobbledygook factory has been given the go-ahead to begin large-scale production. I’ll call this level-1 safety, which represents simple product-level safety.
Gobbledygook becomes a popular product, and can now be found in most people’s homes and offices. Since gobbledygook is a good replacement for balderdash, being cheaper and simpler to produce, the balderdash industry has been decimated. Some people lament the loss of the balderdash industry, since it employed more people than the gobbledygook industry. Most people don’t mind, however, since gobbledygookis a superior product and produces less waste. While gobbledygook remains safe for human consumption, it’s discovered that marmosets become ill when they accidentally consume gobbledygook. Marmosets are a popular pet, and so many marmoset lives are sadly lost. In addition, the communities closest to thegobbledygook factory begin to notice that the manufacture of gobbledygook creates a purple byproduct, which is turning the nearby streams and ponds purple. These problems are emblematic of level-2 safety. Level-2 safety issues were not unanticipated. It’s common wisdom that when products hit a larger and more complex environment outside of the initial testing conditions, unanticipated problems will arise. Even so, no one has figured out how to account for the unanticipated issues in a complex environment, so the gobbledygook factory was forced to take reactive measures after their product was deployed. Warning labels that gobbledygook must be kept away from pet marmosets are added to each package of gobbledygook, and the factory owners create purple filters to deal with the purple waste. These stop-gap measures are probably not sufficient, but though this is a big problem it’s still a visible one.
The popularity of gobbledygook continues, unabated, 10 years into the future. After 10 years, people notice that the purple waste that has been captured and disposed of in landfills has seeped into groundwater, and now most people in a 30 mile radius of each gobbledygook factory have a purple tongue. The marmoset population has declined, and this has caused the decline in the population of the natural predator of the marmoset, as well as the destruction of the once-thriving pet marmoset economy. New studies have shown that balderdash is an excellent cure for the hiccups, but balderdash is now very rare and expensive, so the great hiccup epidemic causes worldwide suffering. These are Level-3 safety issues; issues have become more complex over time because of the initial effects chaining into other systems. Level-3 safety is not talked about as often as level-2, because it’s largely seen as the same issue as level-2 safety. Level-3 safety, however, is distinct- this level represents cascading risk. Level-3 safety is neglected because if we solve the level-2 issues, we think we are done, and do not check back as often as we should as time increases system complexity. After all- things have been running for 10 years and it’s been fine. The system appears stable if problems emerge slowly enough. If there is one place where “the gap” is missed, it is likely missed here.
Now, with so much purple in the groundwater, both gobbledy and gook crops are becoming more purple than their previous blue/red. This has caused gobbledygook to transform from purple to ultrapurple.One amazing property of ultrapurple is to leech purple from its environment. The more ultrapurplegobbledygook becomes, the more it leeches purple from the environment, leading it to become even more ultrapurple. Soon, purple is completely gone from the environment, and gobbledygook becomes so ultrapurple that it can leech purple from the sun, and then nearby stars. There are few people who had foreseen this outcome. These few did not necessarily think gobbledygook would enter an untrapurple feedback loop, but they knew something, some day, would hit a similar feedback loop, and that the results would be devastating. Level-2 safety issues existed in many products, but these safety issues never became self-sustaining. No one knew how to tell which safety issues would reach level-4. No one knew what would enable a feedback loop to sustain itself for this long- to drain the earth and sun of all its purple before it would collapse.
No one saw the gap.
#
If I’m a gobbledygook manufacturer, I want my product to be useful enough to sell. If my product is dangerous in the short term- if the gobbledygook is ultrapurple enough to drain all of the useful purple from my office, or is at least too purple to look at directly without harming my eyes, no one will want to purchase my product. I have to make my product just safe enough to sell in the short term before I can make a profit and have a successful company. But the mere possibility of a dangerous ultrapurple product emerging 10 years in the future will not cause my company to lose profit now.
This is the gap jumper- usefulness vs. safety. A system will be made just safe enough to ensure a thing is useful in the short-term and in a narrow environment. A system only needs to be ‘good enough’ for its particular niche, because anything that goes wrong outside of that market niche, on its current iteration, is only a problem once the usefulness is challenged and not before. Most systems will lose usefulness and collapse before they hit safety level-4. Few systems will become entrenched enough to grow and collapse the systems that exist outside of their niche. Most people will not see level-4 safety issues coming because they do not look ahead to see where the gaps will be, and even those who do look ahead cannot clearly see where the gaps are.
A gap is, by its very nature, invisible. Even those who can anticipate and warn others about the gaps will not be able to say where everything will go wrong. It’s possible that by naming the safety levels, it may be easier to not only forecast but also explain where safety gaps may exist. Also, since level-3 seems to be where the gap is the widest, it may be useful to check and re-check systems as time increases complexity. Unfortunately, a system that grows too quickly, or one unprecedented enough that we can’t anticipate growth rate, will not be easily monitored as time goes by. Such systems should be halted for more intense study before they are allowed to operate.