Or a post on Collective Rationality with a stupid title.
Epistemic Status: At first, written with my Simulator Worlds framing. E.g I ran generated characters with claude in order to generate good cognitive basins and then directed those to output the dialogue based on my instructions. I did however edit around 90% of it afterwards and hopefully it is clear since the post is quite weird. This post is Internally Verified (e.g I think most of the claims are correct with an average of 60-75% certainty). It’s also got some weird auditory analogies but that was because I originally had a weird bit about vincent van gogh in here.
For the record, I’m pro prediction and I think that AI 2027 is a great prediction exercise so please take this as the good faith satire rather than anything too critical.
The Bovine Question
The scene: Athens, 347 BCE. The Academy. SOCRATES, PLATO, and ARISTOTLE recline on couches arranged around a low table. A SERVANT carves roast ox in the background, periodically refilling wine cups. The philosophers' expressions are grave. Scrolls covered in calculations are spread between them.
PLATO: The Pythagoreans have completed their calculations.
ARISTOTLE: And?
PLATO: (producing a scroll) solemnly: By the third generation hence, bovine biomass shall exceed human biomass by a factor of seventeen.
SOCRATES: Let us be precise, as precision is the friend of truth. You say "bovine biomass." Do you mean the weight of bovines, or the number of bovines, or perhaps the total amount of bullshit produced by bovines?
ARISTOTLE: All three, Socrates. I have conducted observations in Thessaly, Macedonia, Corinth, and the Peloponnese. The herds grow. (he shudders) The growth is... consistent.
The SERVANT places a platter of sliced beef before them. They eat absently, focused on the scroll.
SOCRATES: And we are agreed that this represents a threat to human flourishing?
PLATO: The Forms themselves confirm it. I have meditated upon the Form of the Bovine — eternal, unchanging, and possessed of an essence that tends toward multiplication. The Form of the Human, while nobler, lacks this... reproductive vigor.
ARISTOTLE: The natural philosophers concur. I have consulted with geometers, astronomers, and those who study the movements of animals. All models point toward the same conclusion.
SOCRATES: Then let us proceed carefully, for careful proceeding is the mark of wisdom. We are taking what the Delphic tradition calls the "outside view" — deferring not to our individual hunches, but to the accumulated observations of many experts?
PLATO: Precisely. We would not want to fall into the trap of mere personal opinion.
SOCRATES: And the Oracle has been consulted?
ARISTOTLE: Three times. The Pythia was unambiguous. "The horned ones shall inherit the grass. To those who have grass, more grass shall be given."
They all nod solemnly. ARISTOTLE takes another slice of beef.
SOCRATES: So we have convergence. Mathematical projection, empirical observation, the Forms, divine revelation. Four independent lines of evidence.
PLATO: Inescapable.
SOCRATES: ...
PLATO: You hesitate.
SOCRATES: I was going to examine our assumptions — the examination of assumptions being the beginning of —
ARISTOTLE: — of knowledge, yes, you say this often.
SOCRATES: I wonder whether our four lines of evidence are in fact four lines, or one line wearing four disguises. The Pythagoreans trained the geometers. The geometers informed my observations. My observations are what I brought to meditation on the Forms. And the Pythia... well, we did tell her what we were worried about before asking.
PLATO: You are suggesting our outside view is —
SOCRATES: I am suggesting it might be an inside view with excellent references.
(Pause.)
ARISTOTLE: Noted. But the methodology is sound?
SOCRATES: The methodology is sound.
ARISTOTLE: And no alternative methodology has been proposed?
SOCRATES: It has not.
ARISTOTLE: Then we proceed. To mitigation strategies —
ARISTOTLE takes a long sip of wine, then another slice of beef. He chews thoughtfully while outlining the existential threat of the animal he is chewing.
Word spreads through Athens. Other philosophers hear that Socrates, Plato, and Aristotle have independently confirmed the bovine projections. They want to verify it themselves — good epistemic practice. So they gather their own data, consult their own experts, run their own numbers. Same methods. Same reference classes. Same result.
More convergence! The consensus hardens. Nobody asks the servant, the shepherds, or the butchers — the people with direct contact with the actual system — because those people aren't using the approved methodology. “Bro, just eat the cows” isn’t a valid strategy.
Alright, great. This is not very applicable to AI 2027 since it’s not as easy as eating the AGI (unless we can just use the intelligence ourselves?) but it is similar to the reasoning behind the malthusian trap which didn’t take sigmoids into account. We might be in such a scenario, we’re probably not, but we might be and so what should one do to mitigate such a risk and become Less Wrong? (mic drop)
Shared outside views can become an inside view
Well, look at ourselves as a collective agent of course!
Condorcet's Jury Theorem — first articulated in his 1785 treatise — says that if you have n voters, each with probability p > 0.5 of being correct, and their errors are independent, then as n increases, the probability that the majority is correct approaches 1:
This can be applied to consensus formation on non-voting issues as well.
The main point is that independent aggregated information makes the errors wash out. Each person's idiosyncratic wrongness gets swamped by the collective's tendency toward truth. Democracy works! Crowds are wise! Defer to the many! Even a herd, properly aggregated, can find its way to water.
But look at that word independent. It's doing all the work. Like the two ears on a head — each receiving slightly different input, the brain triangulating truth from the difference. Remove the independence and you've lost the triangulation. You're hearing the world in mono.
When errors correlate — when everyone is wrong about the same things — the theorem inverts. You don't approach truth. You approach shared confidence in shared mistakes. Mathematically, if everyone's errors are perfectly correlated, you effectively have n = 1 no matter how many bodies are in the room. A thousand experts who learned from the same masters, read the same texts, and exclude the same variables from consideration might as well be one expert with a particularly confident demeanor. They will all fail to notice what's on the table in front of them, even as they consume it.
The Diversity Prediction Theorem
Scott Page formalized this differently in his book The Difference, and I think his framing makes the mechanism clearer—like finally hearing a familiar sound from a new angle and recognizing it for what it truly is.
For any collection of predictors making estimates of some quantity, the following identity holds:
Collective Error = Average Individual Error − Prediction Diversity
Where:
Collective Error = (crowd mean - truth)²
Average Individual Error = average of (individual prediction - truth)²
Prediction Diversity = average of (individual prediction - crowd mean)²
(pictures for understanding:)
If prediction diversity goes to zero — if everyone predicts the same thing — then collective error equals average individual error. The wisdom of crowds term has dropped out entirely. You've paid the costs of consulting many people (time, coordination, the appearance of epistemic humility) and received none of the benefits. You might as well have asked one person and saved yourself the trouble. The herd moves as one, which looks like wisdom until you notice they're all walking toward the same cliff, lowing in agreement.
Model Selection and the Tape Reader Problem
When you decide to "take the outside view," you're to some extent choosing a basis. What's the reference class? Who are you learning from? What methodology are you using? These choices are usually shaped by your local environment — what you see around you, what information you can take in, who trained you. Model selection is a huge part of predicting the world, and it happens before the prediction starts.
We can formalize this if we think about Turing machines — tape readers that process inputs according to rules encoded in their heads. There's deep stuff connecting Bayesian learning and computation here. Different machines read different types of things. You can take an economic perspective, an ecological model, a population model. Each one processes reality through a different set of primitives.
This is basically what happened with Malthus. He took a pure population model — exponential growth without any sigmoid curves — and projected it forward. The model read real data. The projections followed from the data. But the tape reader couldn't see the symbols that would have told him about agricultural innovation, demographic transitions, or the fact that people adjust their behavior when conditions change. Same thing is happening in our little example here — three sophisticated models running on the same tape, each detecting real patterns, none calibrated to read the symbol sitting right in the middle.
In theory, there's a solution. Kolmogorov complexity defines the complexity of a string x as the length of the shortest program producing it on a universal machine U:
Universal machines can read everything. Solomonoff induction extends this into a prior — weight every computable hypothesis by its complexity, update on evidence, converge on truth. The ideal tape reader.
It's also incomputable. Just like AIXI, you can't actually run it. Every real agent picks an approximation — a restricted hypothesis class, a specific set of programs the tape reader can execute. That choice is your prior, and it comes before Bayes gets involved. It's turtles all the way down, and at some point you have to stop and pick a turtle.
Which turtle you pick determines what counts as "simple."
In the population dynamics language, the cow situation requires an elaborate program — growth rates, carrying capacity, differential equations, eigenvalue computation. K_A(S) = 847 bits. In the folk practice language, the same situation compresses to a single sentence: "Animals we farm stay at farming levels." K_B(S) = 52 bits (fully calculated numbers, trust me.). The "simpler" description depends entirely on which primitives your language treats as basic operations.
I make a longer argument for this in The Atoms of Knowledge Aren't Universal — sometimes it's actually more useful to stop at a coarser description. The valence electron model of chemistry is "less precise" than quantum field theory, but it has higher effective information for predicting bonding behavior. Sometimes the right turtle to stop at is the earlier one. The philosophers' language didn't include "we eat them" as a primitive, and no amount of precision within their chosen language could compensate for that.
They'd all picked the same turtle. Same training, same hypothesis space, same restriction. Within that space, their updates converged. The convergence was real. It just meant "we agree on the model," not "we're tracking reality."
So if the choice of primitives determines what you can see, and the wrong primitives can make you blind to what's right in front of you — how do you make sure you're not all blind in the same way?
Robustness Through Decorrelation
If correlated errors are the enemy of collective wisdom, decorrelation is the remedy. But how do you achieve decorrelation in practice? You need multiple systems, using different methods, checking each other.
Engineering has understood this for centuries. Redundancy alone isn't enough — you need diverse redundancy. Three identical sensors will fail identically. Three sensors using different physical principles might catch each other's failure modes. A system that only listens to one frequency will miss signals on others — and worse, will be confident in silence where there is actually noise it cannot hear.
Nassim Taleb's antifragility concept applies here. A system is antifragile if it gains from disorder—if stressors make it stronger rather than weaker. Monocultures are fragile; they're optimized for one environment and shatter when conditions change. Diverse ecosystems are antifragile; when one species fails, others fill the niche. A herd of identical animals is vulnerable to any disease that can infect one; a mixed ecology persists.
Epistemic communities work the same way. A field where everyone uses the same methodology is fragile — if the methodology has a blind spot, everyone shares it. A field with diverse methodologies is more robust — different approaches serve as error-correctors for each other.
So if you notice your research community has strongly converged on a methodology whilst there’s confusion in the background, that's evidence that someone should try a different one. Develop your own inside view. Do your own analysis. Come up with your own metrics. Build your own models. You are more likely to produce something that is useful to the collective if you do.
Conclusion
When you find yourself in a room full of experts who all agree about an impending threat while actively consuming the threat in question, allow yourself a moment of doubt. Not certainty that they're wrong. Just doubt. Just enough decorrelation to ask: what are we all not seeing?
And as the one person who all rationalists and AI Safety people listen to, Jesus Christ himself, once said “In everything, then, do to others as you would have them do to you.”
Or in other words, act to make the commons the commons that you want to be in. Form your own weird inside view for it is better to be in a diversely intelligent common than it is to be in a monotonic common. This is because if you’re monotonic, Nasim Nicholas Taleb will come in and scream at you in an italian mafioso voice about how fragile your system is and mention something about modern economics and randomness and bla bla bla… and you DO NOT want that to happen.
Or a post on Collective Rationality with a stupid title.
Epistemic Status: At first, written with my Simulator Worlds framing. E.g I ran generated characters with claude in order to generate good cognitive basins and then directed those to output the dialogue based on my instructions. I did however edit around 90% of it afterwards and hopefully it is clear since the post is quite weird. This post is Internally Verified (e.g I think most of the claims are correct with an average of 60-75% certainty). It’s also got some weird auditory analogies but that was because I originally had a weird bit about vincent van gogh in here.
For the record, I’m pro prediction and I think that AI 2027 is a great prediction exercise so please take this as the good faith satire rather than anything too critical.
The Bovine Question
The scene: Athens, 347 BCE. The Academy. SOCRATES, PLATO, and ARISTOTLE recline on couches arranged around a low table. A SERVANT carves roast ox in the background, periodically refilling wine cups. The philosophers' expressions are grave. Scrolls covered in calculations are spread between them.
PLATO: The Pythagoreans have completed their calculations.
ARISTOTLE: And?
PLATO: (producing a scroll) solemnly: By the third generation hence, bovine biomass shall exceed human biomass by a factor of seventeen.
SOCRATES: Let us be precise, as precision is the friend of truth. You say "bovine biomass." Do you mean the weight of bovines, or the number of bovines, or perhaps the total amount of bullshit produced by bovines?
ARISTOTLE: All three, Socrates. I have conducted observations in Thessaly, Macedonia, Corinth, and the Peloponnese. The herds grow. (he shudders) The growth is... consistent.
The SERVANT places a platter of sliced beef before them. They eat absently, focused on the scroll.
SOCRATES: And we are agreed that this represents a threat to human flourishing?
PLATO: The Forms themselves confirm it. I have meditated upon the Form of the Bovine — eternal, unchanging, and possessed of an essence that tends toward multiplication. The Form of the Human, while nobler, lacks this... reproductive vigor.
ARISTOTLE: The natural philosophers concur. I have consulted with geometers, astronomers, and those who study the movements of animals. All models point toward the same conclusion.
SOCRATES: Then let us proceed carefully, for careful proceeding is the mark of wisdom. We are taking what the Delphic tradition calls the "outside view" — deferring not to our individual hunches, but to the accumulated observations of many experts?
PLATO: Precisely. We would not want to fall into the trap of mere personal opinion.
SOCRATES: And the Oracle has been consulted?
ARISTOTLE: Three times. The Pythia was unambiguous. "The horned ones shall inherit the grass. To those who have grass, more grass shall be given."
They all nod solemnly. ARISTOTLE takes another slice of beef.
SOCRATES: So we have convergence. Mathematical projection, empirical observation, the Forms, divine revelation. Four independent lines of evidence.
PLATO: Inescapable.
SOCRATES: ...
PLATO: You hesitate.
SOCRATES: I was going to examine our assumptions — the examination of assumptions being the beginning of —
ARISTOTLE: — of knowledge, yes, you say this often.
SOCRATES: I wonder whether our four lines of evidence are in fact four lines, or one line wearing four disguises. The Pythagoreans trained the geometers. The geometers informed my observations. My observations are what I brought to meditation on the Forms. And the Pythia... well, we did tell her what we were worried about before asking.
PLATO: You are suggesting our outside view is —
SOCRATES: I am suggesting it might be an inside view with excellent references.
(Pause.)
ARISTOTLE: Noted. But the methodology is sound?
SOCRATES: The methodology is sound.
ARISTOTLE: And no alternative methodology has been proposed?
SOCRATES: It has not.
ARISTOTLE: Then we proceed. To mitigation strategies —
ARISTOTLE takes a long sip of wine, then another slice of beef. He chews thoughtfully while outlining the existential threat of the animal he is chewing.
Word spreads through Athens. Other philosophers hear that Socrates, Plato, and Aristotle have independently confirmed the bovine projections. They want to verify it themselves — good epistemic practice. So they gather their own data, consult their own experts, run their own numbers. Same methods. Same reference classes. Same result.
More convergence! The consensus hardens. Nobody asks the servant, the shepherds, or the butchers — the people with direct contact with the actual system — because those people aren't using the approved methodology. “Bro, just eat the cows” isn’t a valid strategy.
Alright, great. This is not very applicable to AI 2027 since it’s not as easy as eating the AGI (unless we can just use the intelligence ourselves?) but it is similar to the reasoning behind the malthusian trap which didn’t take sigmoids into account. We might be in such a scenario, we’re probably not, but we might be and so what should one do to mitigate such a risk and become Less Wrong? (mic drop)
Shared outside views can become an inside view
Well, look at ourselves as a collective agent of course!
Condorcet's Jury Theorem — first articulated in his 1785 treatise — says that if you have n voters, each with probability p > 0.5 of being correct, and their errors are independent, then as n increases, the probability that the majority is correct approaches 1:
This can be applied to consensus formation on non-voting issues as well.
The main point is that independent aggregated information makes the errors wash out. Each person's idiosyncratic wrongness gets swamped by the collective's tendency toward truth. Democracy works! Crowds are wise! Defer to the many! Even a herd, properly aggregated, can find its way to water.
But look at that word independent. It's doing all the work. Like the two ears on a head — each receiving slightly different input, the brain triangulating truth from the difference. Remove the independence and you've lost the triangulation. You're hearing the world in mono.
When errors correlate — when everyone is wrong about the same things — the theorem inverts. You don't approach truth. You approach shared confidence in shared mistakes. Mathematically, if everyone's errors are perfectly correlated, you effectively have n = 1 no matter how many bodies are in the room. A thousand experts who learned from the same masters, read the same texts, and exclude the same variables from consideration might as well be one expert with a particularly confident demeanor. They will all fail to notice what's on the table in front of them, even as they consume it.
The Diversity Prediction Theorem
Scott Page formalized this differently in his book The Difference, and I think his framing makes the mechanism clearer—like finally hearing a familiar sound from a new angle and recognizing it for what it truly is.
For any collection of predictors making estimates of some quantity, the following identity holds:
Collective Error = Average Individual Error − Prediction Diversity
Where:
(pictures for understanding:)
If prediction diversity goes to zero — if everyone predicts the same thing — then collective error equals average individual error. The wisdom of crowds term has dropped out entirely. You've paid the costs of consulting many people (time, coordination, the appearance of epistemic humility) and received none of the benefits. You might as well have asked one person and saved yourself the trouble. The herd moves as one, which looks like wisdom until you notice they're all walking toward the same cliff, lowing in agreement.
Model Selection and the Tape Reader Problem
When you decide to "take the outside view," you're to some extent choosing a basis. What's the reference class? Who are you learning from? What methodology are you using? These choices are usually shaped by your local environment — what you see around you, what information you can take in, who trained you. Model selection is a huge part of predicting the world, and it happens before the prediction starts.
We can formalize this if we think about Turing machines — tape readers that process inputs according to rules encoded in their heads. There's deep stuff connecting Bayesian learning and computation here. Different machines read different types of things. You can take an economic perspective, an ecological model, a population model. Each one processes reality through a different set of primitives.
This is basically what happened with Malthus. He took a pure population model — exponential growth without any sigmoid curves — and projected it forward. The model read real data. The projections followed from the data. But the tape reader couldn't see the symbols that would have told him about agricultural innovation, demographic transitions, or the fact that people adjust their behavior when conditions change. Same thing is happening in our little example here — three sophisticated models running on the same tape, each detecting real patterns, none calibrated to read the symbol sitting right in the middle.
In theory, there's a solution. Kolmogorov complexity defines the complexity of a string x as the length of the shortest program producing it on a universal machine U:
Universal machines can read everything. Solomonoff induction extends this into a prior — weight every computable hypothesis by its complexity, update on evidence, converge on truth. The ideal tape reader.
It's also incomputable. Just like AIXI, you can't actually run it. Every real agent picks an approximation — a restricted hypothesis class, a specific set of programs the tape reader can execute. That choice is your prior, and it comes before Bayes gets involved. It's turtles all the way down, and at some point you have to stop and pick a turtle.
Which turtle you pick determines what counts as "simple."
In the population dynamics language, the cow situation requires an elaborate program — growth rates, carrying capacity, differential equations, eigenvalue computation. K_A(S) = 847 bits. In the folk practice language, the same situation compresses to a single sentence: "Animals we farm stay at farming levels." K_B(S) = 52 bits (fully calculated numbers, trust me.). The "simpler" description depends entirely on which primitives your language treats as basic operations.
I make a longer argument for this in The Atoms of Knowledge Aren't Universal — sometimes it's actually more useful to stop at a coarser description. The valence electron model of chemistry is "less precise" than quantum field theory, but it has higher effective information for predicting bonding behavior. Sometimes the right turtle to stop at is the earlier one. The philosophers' language didn't include "we eat them" as a primitive, and no amount of precision within their chosen language could compensate for that.
They'd all picked the same turtle. Same training, same hypothesis space, same restriction. Within that space, their updates converged. The convergence was real. It just meant "we agree on the model," not "we're tracking reality."
So if the choice of primitives determines what you can see, and the wrong primitives can make you blind to what's right in front of you — how do you make sure you're not all blind in the same way?
Robustness Through Decorrelation
If correlated errors are the enemy of collective wisdom, decorrelation is the remedy. But how do you achieve decorrelation in practice? You need multiple systems, using different methods, checking each other.
Engineering has understood this for centuries. Redundancy alone isn't enough — you need diverse redundancy. Three identical sensors will fail identically. Three sensors using different physical principles might catch each other's failure modes. A system that only listens to one frequency will miss signals on others — and worse, will be confident in silence where there is actually noise it cannot hear.
Nassim Taleb's antifragility concept applies here. A system is antifragile if it gains from disorder—if stressors make it stronger rather than weaker. Monocultures are fragile; they're optimized for one environment and shatter when conditions change. Diverse ecosystems are antifragile; when one species fails, others fill the niche. A herd of identical animals is vulnerable to any disease that can infect one; a mixed ecology persists.
Epistemic communities work the same way. A field where everyone uses the same methodology is fragile — if the methodology has a blind spot, everyone shares it. A field with diverse methodologies is more robust — different approaches serve as error-correctors for each other.
So if you notice your research community has strongly converged on a methodology whilst there’s confusion in the background, that's evidence that someone should try a different one. Develop your own inside view. Do your own analysis. Come up with your own metrics. Build your own models. You are more likely to produce something that is useful to the collective if you do.
Conclusion
When you find yourself in a room full of experts who all agree about an impending threat while actively consuming the threat in question, allow yourself a moment of doubt. Not certainty that they're wrong. Just doubt. Just enough decorrelation to ask: what are we all not seeing?
And as the one person who all rationalists and AI Safety people listen to, Jesus Christ himself, once said “In everything, then, do to others as you would have them do to you.”
Or in other words, act to make the commons the commons that you want to be in. Form your own weird inside view for it is better to be in a diversely intelligent common than it is to be in a monotonic common. This is because if you’re monotonic, Nasim Nicholas Taleb will come in and scream at you in an italian mafioso voice about how fragile your system is and mention something about modern economics and randomness and bla bla bla… and you DO NOT want that to happen.