Predictive coding talks about the "handshake" between top-down and bottom-up priors. The upper layers of the brain make a guess as to what's going to happen, and the lower layers (closer to the sensory input stream) report back. Their report either contains some amount of surprise, or just says "Close enough! What's next?".
Suppose you're thirsty. You start the "drink water" process. You're not going to take in the exact right amount of water to optimize your hydration levels. You'll drink some amount which gets you roughly in the right ballpark and then say "Good enough!"
When your boss gives you an instruction, you do it. Most likely, it doesn't go down in the exact precise way that she imagined. If you run into a serious problem, or something goes really weirdly, you'll tell her, but most of the time you'll report back "Done".
The Close-Enough Primitive
"Close enough" appears to be a primitive property in the implementation of these systems, which is somewhat odd because I don't remember "close enough" being a term in Bayes theorem! But an algorithm that approximates a solution will frequently use hacky terms not found in the beautiful mathematical representation. In this case it seems like multi-part intelligent systems have converged on a process which rounds off some small amount of difference to literally zero difference. Why is this?
I think it's often down to constraints, of which one is the bottlenecks in communication between subsystems. One bottleneck is between layers. There's only a finite amount of information which can be passed up or down. If bandwidth is at a premium, then you're always going to have to truncate your messages, and sometimes you'll truncate it to a length of zero: close enough.
Another constraint is resources within a layer, and this one seems especially relevant to multi-agent systems. If there are a dozen tasks which need doing, you can either do them in series, or spin up a dozen agents to do them in parallel. If in series, you have to allocate your time, and if in parallel, you have to allocate compute between agents.
If you choose the serial option, you probably want to switch between tasks infrequently, because there's costs to switching. A good way to do this is to do one task until it's good enough, and then do the next task.
If you choose the parallel option, you need a way to stop your agents from competing for resources too aggressively. A good way for them to do this is for them to do their assigned task until it's good enough, and then stop.
Prediction vs Steering
Ok, but we've been swapping between two kinds of description here. One is about prediction and surprise, and the other is about agents acting in the world. Goal-models let us use the language of surprise to talk about how well our plans are going. The same process which tells us a prediction is close enough to reality can tell us when reality is close enough to our goals.
Fragility
Humans tend to have this primitive when it comes to our high-level decisions. We do a certain amount of stuff, and then relax. This is why most people only work a "normal" number of hours, rather than maximising the amount of money we get.
Most people don't end up in charge of the world. The people who say "mayor of a medium-sized city is fine for me" don't become president. The "close enough" primitive, at least at the highest level of decision-making, seems to be non-essential for basic functioning.
Now it seems like most people actually do say "enough" at a certain point. Most presidents say "enough" at being president, and don't try to become dictators. Even Jeff Bezos spends a lot of time hanging around on his yacht. Maybe it's just that their "enough" threshold is extremely high?
Building AI
I expect AIs trained in multi-agent settings will have something like a "good enough" primitive in their actions, at least at first. I think a drive like this might be a useful approach to building lower-impact AI. It does seem like, when the primitive works well, it is pretty robust.
Unfortunately, I also expect that this primitive will be fragile, and break under adverse optimizing pressure. and the most powerful AIs at open-ended tasks will be ones with the weaker "good enough" drive. Not ideal!
Predictive coding talks about the "handshake" between top-down and bottom-up priors. The upper layers of the brain make a guess as to what's going to happen, and the lower layers (closer to the sensory input stream) report back. Their report either contains some amount of surprise, or just says "Close enough! What's next?".
Suppose you're thirsty. You start the "drink water" process. You're not going to take in the exact right amount of water to optimize your hydration levels. You'll drink some amount which gets you roughly in the right ballpark and then say "Good enough!"
When your boss gives you an instruction, you do it. Most likely, it doesn't go down in the exact precise way that she imagined. If you run into a serious problem, or something goes really weirdly, you'll tell her, but most of the time you'll report back "Done".
The Close-Enough Primitive
"Close enough" appears to be a primitive property in the implementation of these systems, which is somewhat odd because I don't remember "close enough" being a term in Bayes theorem! But an algorithm that approximates a solution will frequently use hacky terms not found in the beautiful mathematical representation. In this case it seems like multi-part intelligent systems have converged on a process which rounds off some small amount of difference to literally zero difference. Why is this?
I think it's often down to constraints, of which one is the bottlenecks in communication between subsystems. One bottleneck is between layers. There's only a finite amount of information which can be passed up or down. If bandwidth is at a premium, then you're always going to have to truncate your messages, and sometimes you'll truncate it to a length of zero: close enough.
Another constraint is resources within a layer, and this one seems especially relevant to multi-agent systems. If there are a dozen tasks which need doing, you can either do them in series, or spin up a dozen agents to do them in parallel. If in series, you have to allocate your time, and if in parallel, you have to allocate compute between agents.
If you choose the serial option, you probably want to switch between tasks infrequently, because there's costs to switching. A good way to do this is to do one task until it's good enough, and then do the next task.
If you choose the parallel option, you need a way to stop your agents from competing for resources too aggressively. A good way for them to do this is for them to do their assigned task until it's good enough, and then stop.
Prediction vs Steering
Ok, but we've been swapping between two kinds of description here. One is about prediction and surprise, and the other is about agents acting in the world. Goal-models let us use the language of surprise to talk about how well our plans are going. The same process which tells us a prediction is close enough to reality can tell us when reality is close enough to our goals.
Fragility
Humans tend to have this primitive when it comes to our high-level decisions. We do a certain amount of stuff, and then relax. This is why most people only work a "normal" number of hours, rather than maximising the amount of money we get.
Most people don't end up in charge of the world. The people who say "mayor of a medium-sized city is fine for me" don't become president. The "close enough" primitive, at least at the highest level of decision-making, seems to be non-essential for basic functioning.
Now it seems like most people actually do say "enough" at a certain point. Most presidents say "enough" at being president, and don't try to become dictators. Even Jeff Bezos spends a lot of time hanging around on his yacht. Maybe it's just that their "enough" threshold is extremely high?
Building AI
I expect AIs trained in multi-agent settings will have something like a "good enough" primitive in their actions, at least at first. I think a drive like this might be a useful approach to building lower-impact AI. It does seem like, when the primitive works well, it is pretty robust.
Unfortunately, I also expect that this primitive will be fragile, and break under adverse optimizing pressure. and the most powerful AIs at open-ended tasks will be ones with the weaker "good enough" drive. Not ideal!
◆◆◆◆◆|◆◆◆◆◆|◇◇◇◇◇
◆◆◆◆◆|◆◆◆◆◆|◇◇◇◇◇