Yes, "don't pull a broken chain." But how do you know or notice that it is broken? In all of the examples you cited there was one thing missing: the feedback loop. Seeing incremental results based on your actions and adjusting those actions. Open loop doesn't work well, or, at least, is much harder to make work. Noticing that you are in the open loop mode and looking for ways to close it is something that can help a lot but is often overlooked. I think in the ML parlance it is called iterative learning or something. And if you notice that you are open-looping and there is no way you can find to close the loop, adjust your expectation of success down accordingly.
Noticing that chain-1 is broken introduces a meta-level problem: there has to be a cause-and-effect chain from the break in chain-1 to my mental model of chain-1. One could even imagine a cause-and-effect chain-2 from chain-2 itself to my mental model of chain-2; think Yudkowsky's lens which sees its own flaws.
I'm re-posting this now because my post on cartographic processes made too many inferential jumps all at once and caused confusion. But here's where this is all headed: cartographic processes (i.e. cause-and-effect chains which produce maps/models of the world) which produce maps/models of themselves. Before we can deal with that much meta, we need to characterize more basic cartographic processes, i.e. cause-and-effect chains which produce accurate/useful maps of the territory.
Note that mapping is a type of abstraction that's independent of causal chains and feedback/control loops. You can make an excellent thermostat that doesn't understand a thing about thermodynamics, air flow, or control theory (though you may need to know something about these things in order to make the thermostat work well).
Seems like we speak different languages, or come from different epistemologies, since we seem to be talking past each other. The meta-model I was talking about is a rather universal one: close the loop whenever possible and don't expect much from an open loop. I don't understand the chain-1 to chain-2 argument. Then again, I find the whole idea of causal chains underwhelming. Maybe they have something to show in terms of having solved problems that are intractable otherwise, but if so, I am not aware of any,
Here's a real-world example (in fact the example which motivated the original essay). I was at a startup and there was this complicated feature we were working on for our app. I asked "Is this actually going to make us any money? Like, what's the cause-and-effect process from this feature to profits?". And nobody had any answer. There was no imagined chain from building that feature to making money; we had simply lost sight of the goal. We were building the feature because that was what we knew how to do, and making the app profitable was not something we knew how to do - we were looking for the metaphorical quarter under the streetlight.
Sure, feedback would have straightened us out eventually. We would have released the feature, no money would have been made. But that would have taken months. Noticing that the chain was broken was much faster.
Lost purposes is a common occurrence, definitely. And it looks like you had provided that feedback without waiting in the open loop until your customer give you the feedback. Your motivation was, or appeared to be based on a broken causal chain or something, but the resulting action was something that ought to have been built-in from the start: closing the loop early.
This essay originally appeared here.
A village in a remote corner of an empire decided to erect a statue of the emperor. The village sculptor was hired, but the sculptor had no idea how large to make the emperor’s nose.
The elders conferred. Nobody in the village had ever seen the emperor, nor heard anything about his nose, so they all had wildly different estimates of the size of the emperor’s nose. The elders argued about the right size for hours, until one particularly wise elder stood to address the room.
“Our estimates are too noisy,” the wise elder declared, “In order to improve them, we should employ the wisdom of the crowd. We will ask each person in the village to estimate the length of the emperor’s nose. Then, we can average together the results to obtain an estimate of high precision.”
This proposition was put to a vote, and the elders quickly agreed, eager to end the hours of argument before they missed the early-bird special at the village buffet. The next day, each villager was asked to estimate the length of the emperor’s nose, in millimeters. The elders averaged together all the responses, and estimated the emperor’s nose was 15.49 mm long.
During World War II, many remote pacific islands became military bases for the American navy. On one such island, the native inhabitants worked with the sailors in exchange for food, clothing and other supplies. All these supplies were flown in plane, and landed on an airstrip.
After the war, the Americans left the empty airstrip behind. The planes stopped delivering supplies.
The natives wanted to get more supplies. So, they tried to make the planes land. They went out to the airstrip and did everything the sailors had done. They lit small, regularly spaced fires along the sides of the airstrip. They had someone sit at the side of the strip talking into a wooden box while wearing pieces of wood shaped to look like headphones. They had others stand on the airstrip and gesture with sticks. In short, they did everything they had seen the sailors do.
But the planes just didn’t land.
A boy was pacing back and forth at night under a streetlamp, apparently searching the ground, when another boy passed by.
“What are you looking for?” asked the newcomer.
“My quarter. I dropped it,” replied the searcher.
“Oh. I’ll help you look,” offered the newcomer.
The two continued the search for a minute or so before a third boy came along.
“What are you two looking for?” asked the third.
“A quarter he dropped,” replied the second, indicating the original boy.
“Let me help,” said the third, and set to searching.
This continued for some time, and the crowd grew steadily. Finally, a girl showed up.
“What are you all looking for?” she asked.
“My quarter. I dropped it,” replied the original boy.
“Well where did you drop it?” asked the girl.
“Over there,” said the boy, indicating an area off to the side.
“So why is everybody searching over here?” asked the girl.
“Because there’s more light here,” replied the boy.
The central economic planners regularly set targets for each factory under their control. Factories which exceeded their targets were rewarded in various ways. Factories which fell short of their targets… well, it’s the soviets, you can figure it out.
Early on, nail factories each had to produce some number of nails to meet their target. For a while, things went well. Factories produced nails. But there was always an element of competition - the best-performing factories received rewards and the worst-performing were punished, so occasionally people would cut edges in order to get ahead.
In particular, the nail factories found that they could gain an advantage by producing slightly smaller nails than the competition. By producing smaller nails, they could produce a larger number with the same resources. But over time, all the nail factories figured this out, and they had to cheat a little more to gain an edge - the nails became even smaller.
This arms race continued until each factory was producing large numbers of tiny, useless “nails”, better suited to pinboards than to construction.
The central planners heard reports of the tiny nails. They decided to update their targets - henceforth, nail production would be measured by weight, rather than number of nails.
A few years later, all the nail factories were producing just a few giant, useless “nails”, better suited to ballast than to construction.
In everyday life, things are connected by chains of cause and effect.
Suppose I’m driving along at night when a deer wanders into the road ahead. Light from my headlights reflects off the deer’s hide, into my eyes. The light is absorbed by photoreceptors, which trigger a cascade of electrical signals in my brain. My brain pattern-matches what it sees, and concludes that there’s a deer ahead and hitting it would be bad. The chain of cause and effect links the deer in the road, to me realizing there’s a deer in the road.
Once I realize there’s a deer in the road, electrical signals propagate down my spine to neurons in my leg and foot. Those neurons activate muscles, lifting the foot from gas to brake pedal and then pushing. That force depresses the brake pedal, which applies pressure in a hydraulic system, multiplying the force and eventually squeezing disks connected to the wheels. The increased force on the disks increases friction, slowing the wheels, which in turn slows the car. The chain of cause and effect links my decision to brake, to the car slowing down.
In everyday life, we pull on chains of cause and effect, either to gain information or to influence the world around us. But in each of the four parables above, the chain is broken.
In the story of the emperor’s nose, the elders try to estimate the nose length using statistical techniques… but none of the townspeople know anything at all about the emperor’s nose, so the causal chain from the actual emperor’s nose to the elders’ estimate is broken.
In the story of the cargo cult, the locals mimic the surface actions of sailors at an airstrip, but they don’t understand the underlying chain of cause and effect which led planes to land. Absent that underlying chain, the planes don’t land.
In the story of the missing quarter, the first boy searching under the light causes the second boy to search under the light, and the third, and so on. But the first boy is searching in the wrong place - the chain is broken at the very beginning, even before the story starts. In fact, the first boy himself is pulling on a broken chain: light is helpful for searching, because the light might bounce off the quarter and into the boy’s eye etc. But if the light will never bounce off the quarter - because the quarter isn’t under the light - then that chain is broken.
The Soviet nail factory is the most complicated story. In a normal economy, a nail factory produces an economically valuable nail. That nail is sold, and the nail will only be bought if it’s valuable to the buyer (and the more valuable it is to the buyer, the more the buyer is willing to pay for it). The money from the buyer goes back to the nail maker, and serves as incentive. This chain of cause and effect runs from the nail makers producing an economically valuable nail, to the nail makers being rewarded for whatever value the nail provided for the end user.
But once the central planners step in and set targets in terms of number or weight of nails produced, the chain is broken: the nail makers are no longer rewarded based on the economic value of the nail to its end user. So naturally, the nail makers deprioritize economic value in favor of number or weight of nails.
To summarize the lesson: don’t pull a broken chain. When you want to gather information, make sure that the thing you want to know about is causally connected to the thing you’re looking at directly. When you want to influence the world around you, make sure that your action is causally connected to whatever you want to influence. If the causal chain is broken, don’t pull it.