(On reflection, this is more a semi-redundant riff on what you already wrote, and less a good responsive comment, but oh well, I already wrote it so I guess I'll hit publish.)
When I think about the challenges with applying Solomonoff induction in practice, which the scientific method was designed around, I see two main things.
The first, as you point out, is that hypotheses are sufficiently modular that we CAN develop them piecemeal, and sufficiently complicated that we MUST develop them piecemeal. Thus, scientific hypotheses (being just one modular piece of a "real" hypothesis) can "remain agnostic" about certain observations. As I joked here, "if you treat The Law Of Conservation Of Energy as a “hypothesis”, and you “ask” Conservation Of Energy what the half-life of tritium is, then Conservation Of Energy will tell you “Huh? How should I know? Why are you asking me?”" This property (that hypotheses can be agnostic about things) is also characteristic of logical induction / prediction markets (as you point out), and infra-Bayesianism has that property too.
The second is that parsimony / Occam's razor / Solomonoff prior is central to finding the truth, but scientists range from being imperfect at assessing the complexity-vs-parsimony of a theory, to being atrociously bad at it. So the scientific enterprise is set up to rely as little as possible at complexity-assessments. Thus, as you point out, if we could perfectly assess the complexity-vs-parsimony of a hypothesis, then there would be no need to treat prediction and retrodiction differently. The retrodiction problem is about putting too many bits into the hypothesis, and it's only a problem because people are lousy at taking a theory and reading out the number of bits in it (i.e. they don't notice epicycles and special pleading). So again, the scientific institution is set up to minimize reliance on complexity-assessments. But that minimum reliance is still higher than zero. You just can't get away from it entirely. Even "data" is not theory-free, because you need theory to get from "raw" data to so-called observations.
The results are different in different fields (and sometimes pathological), as a field might or might not equilibrate to a state where practitioners with the sharpest discernment of complexity-vs-parsimony command the most respect and power and sway. See my comments here & here for hot-take examples from real-world academic fields.
So anyway, if I were working on this project, the first thing I would try is to say that the "ideal" is Solomonoff induction searching for a true hypothesis which happens to be modular (i.e. it has different pieces covering different, cleanly-separable domains), and then introduce a constraint that you can only measure bits-of-complexity with an extremely noisy ruler, and try to judge truth-seeking setups like LI etc. by how well they approximate the Solomonoff ideal under those assumptions / constraints. (...But I dunno, I didn't think about it very hard.)
1Parsimony should be understood as merely a heuristic for how well a model could have predicted held out data. For example, the AIC approach to penalizing model complexity in statistical modeling is asymptotically equivalent to leave-one-out cross-validation for model selection. This Stone (1977) result should be understood as an explanation for why parsimony seems to be related to truth: post hoc fit of a parsimonious model is mathematically related to how well the model could have predicted held out data. Whereas parsimony has no direct epistemic relevance, predictive accuracy is the actual goal. Since the latter is what we care about, why not just consider predictive ability directly?
However, scientific epistemology has mostly been described informally—e.g. by Popper, Kuhn, Feyerabend, etc.
Isn't both a key aspect of Kuhn and Feyerabend that there's not one scientific epistemology but that different fields use different epistemology and that this is fine?
I think it's pretty bad idea to call your idea of epistemology "scientific epistomology" as it suggests there a single one used more broadly.
When we face difficulties in defining a concept like degrees of truth, a useful question is “what do we want to use the concept for?” One answer is that traders whose models are more true should get more influence over our actions (given some mechanism for hooking up a logical inductor’s outputs to actions, which I’ll leave unspecified here).
This seems to ignore that there are many true models that are useless. Generally, in science you need not only to say something true but also something that's seen as valuable by the readers of the journal in which you want to publish. While a findings that both novel and surprising is more likely to be valuable to readers, I don't think it's sufficient.
Also, unlike Bayesian hypotheses, traders/theories aren’t mutually exclusive: an ideal reasoner would have many of them focusing on different domains. Because of this,
This paragraph ends in the middle of a sentence. Did you have something more that you wanted to say here?
Ooops, good catch. I was going to say "because of this, you can't assign credences to them which sum to one". But I'm not sure that this is quite correct—I think there's a deeper reason why it's hard to assign credences to them at all (because traders/theories are only ever approximately correct). So I've just deleted the sentence fragment.
Truth is multi-faceted notion, when it comes to its relationship with science. Going with the standard structural realism arguments, there’s a) ontology (entities that exist); b) relationships / structure (between entities).
The ontology in our theories is regularly usurped by newer theories (we don’t believe in ether or infinite space time of Newton anymore so we should expect ontology of current theories to not hold true as well). But something does get preserved and it’s the relationship between coarse grained entities (F=ma will more or less hold true even if our notion for what mass is changes over time).
The implication of this distinction in any epistemology is that the “true” reality will not be — could not be —revealed to us. We only have observations to rely on. What we can expect to model is observed regularities. What models get adopted is sociological phenomena to a great extent but the scientific community weighs risky, precise predictions that turn out to be true a lot more because it’s an extremely strong evidence of the model having captured some aspect/view of true regularities of the underlying reality. (But, of course, future prediction is not the only criteria - models get adopted for all sorts of reasons including unification of disparate observations).
Overall, I’m unsure if the notion of truth is needed at all. What we need is useful models and interpretable, extrapolative models are more useful than black box models.
In my post “Why I’m not a Bayesian”, I argued that the Bayesian approach of assigning credences to propositions with binary truth values only works in simple and restricted domains. Instead, I claimed, a better approach to epistemology is to assign degrees of truth to models of the world.
This approach is broadly inspired by science, which is the domain from which we have the most evidence about which epistemological practices allow us to solve very hard problems. We don’t currently have a complete theory of scientific epistemology, but we can identify some important differences between scientific epistemology and Bayesian epistemology. Central examples of Bayesian epistemology (such as Solomonoff induction) assume that the truth lies within the class of hypotheses being considered. Conversely, in central examples of scientific research, the truth is definitely not already under consideration: the main problem is to come up with any hypothesis that explains existing data.
Another way of putting this point: Bayesian epistemology is entirely about empirical updates, whereas science is mostly about the process of constructing new theories. In some cases, once you’ve constructed a theory, you can be confident that it’s close to the truth merely from how well it fits existing data. But scientific theories are only fully accepted after they make successful advance predictions. That’s another difference compared with Bayesian epistemology, which treats retrodictions as equivalent to predictions.
In general I think scientific epistemology is far superior as a guide for thinking about difficult problems (like AI alignment) than Bayesian epistemology. However, scientific epistemology has mostly been described informally—e.g. by Popper, Kuhn, Feyerabend, etc. Popper did attempt to formally define a metric for degrees of truth, but it wasn’t very successful. I’d like to be able to describe scientific epistemology as formally as we can describe Bayesian epistemology (and ideally to unify them in a single framework).
I think that Garrabrant induction (also known as logical induction) is a major step towards formalizing scientific epistemology. This is an update compared with my position in my previous post, in which I critiqued Garrabrant induction in passing for its focus on assigning credences to propositions. However, in the process of assigning credences to propositions, Garrabrant induction also assigns something like degrees of truth (which it calls “wealth”) to something like models of the world (which it calls “traders”). So my critique was pretty off-base, in a way which I’m surprised nobody called out in the LessWrong comments. (Indeed, I’d even identified some of Garrabrant induction’s nice properties in a previous comment. This is a useful lesson on the pitfalls of writing posts about what you’re against rather than what you’re for.)
The key idea of Garrabrant induction is a market mechanism which sets credences for logical statements (including statements about the Garrabrant inductor itself) as the prices in a prediction market on whether those statements will eventually be proved. The traders in this prediction market are simply all polynomial-time algorithms, iteratively enumerated and given some starting wealth. Traders who are more successful will end up with more wealth, giving them greater power to move market prices.
Traders share a number of properties with scientific theories (which Bayesian hypotheses lack). At each point in time, most traders/theories aren’t yet under consideration. The ones that are under consideration don’t need to make predictions about everything that happens—instead, they can focus on making whichever predictions are most surprising and novel. Also, unlike Bayesian hypotheses, traders/theories aren’t mutually exclusive: an ideal reasoner would have many of them focusing on different domains.
While Garrabrant induction was formulated as a way of predicting mathematical theorems, we can imagine the same algorithm predicting a stream of input data about the world. What else would that version of Garrabrant induction need to be a good formal theory of scientific epistemology? Four things seem most prominent:
Abram Demski already touched on many of these points in this post and others in the same sequence. I don’t claim much novelty here, but for some reason it took me a very long time to fit Garrabrant induction into the “replacement for Bayesian epistemology” slot in my ontology—perhaps because it was originally framed more as an extension of Bayesianism than a replacement for it.[1] So further elaboration of this perspective seems helpful.
The problem of old evidence
Garrabrant induction and Solomonoff induction take very different positions on the problem of old evidence. In Solomonoff induction, there’s no distinction between old evidence and new evidence—they’re treated symmetrically. Whereas in Garrabrant induction, traders only ever gain wealth from predicting new evidence—retrodictions of old evidence are irrelevant.
Scientific epistemology takes a middle ground. Advance predictions of new evidence are weighted much more highly than retrodictions, but old evidence can still support a theory. Intuitively speaking, one reason why retrodictions should be discounted is that a theory might have been designed with that old evidence in mind, and therefore crediting it with predicting that evidence is a kind of overfitting or double counting.
Solomonoff induction doesn’t care about this, because it has a mechanism for preventing overfitting: assigning more complex hypotheses lower prior probability. One extra bit of description length might “smuggle in” information which allows the hypothesis to predict old evidence, but it’ll also halve the prior probability of that hypothesis. And if a hypothesis can more than double its probability relative to other hypotheses using just one extra bit, then it must be compressing information more efficiently, which is actually what we want.
In scientific epistemology, however, we don’t have any clear way of measuring the complexity of a given hypothesis, since it’s implemented within a big messy neural network. Even when the theory is described by precise equations, using those equations to make predictions requires the use of “auxiliary hypotheses” in which it’s often possible to hide a lot of complexity. And so in general it’s not possible to mechanistically penalize hypotheses for “smuggling in” old evidence.
However, it seems like this is the kind of thing that Garrabrant induction traders could take into account if given enough information about each other. This seems related to the concept of trading under adverse selection. In normal financial markets, other traders sometimes know more than you. So when market-making you need to set bid-ask spreads, because the expected value of a stock conditional on someone buying it from you is higher than the expected value of a stock conditional on someone selling it to you.
The implementation details do seem tricky, though. In Solomonoff induction, when you add a new hypothesis you can just go back and evaluate how it would have done on all the old evidence, which is equivalent to it having been there all along. But in Garrabrant induction, predictions move the market prices, and so intuitively it seems like you’d need to rerun the whole market. It’s also unclear how traders should be made aware that some of their competitors are “from the future”. It seems like we might need to bake in some notion of situational awareness, which seems complicated. (For more on this, see this post by Abram.)
Traders vs models
Scientific theories need to make predictions, but there’s no standard way to translate those predictions into bets. By contrast, traders in Garrabrant induction need to make bets, but those bets need not be driven by predictions. Traders in Garrabrant induction are any (and eventually every) polynomial-time algorithm. These include very simple algorithms, like ones which notice when “A OR B” is cheaper than either A or B individually, then arbitrage that difference, without having any opinion on how likely A and B actually are. (Analogously, many human actions are driven by reflexes or heuristics rather than explicit beliefs about what outcomes those actions will cause.)
However, in the long term it seems likely that the biggest wins will accrue to traders which implement models containing important insights that other traders lack, then bet that those models are right. This seems particularly true if we focus on the domain of science. Yet those traders might still use a wide range of trading strategies to convert their internally-represented beliefs into actual bets. It would be nice if we could demonstrate that almost all wealth will eventually accrue to traders which use a given kind of trading strategy (e.g. Kelly betting).
Modifying traders
Traders in Garrabrant induction are generated by enumerating every polynomial-time algorithm in order of simplicity. However, an important part of scientific epistemology is the process of identifying which new theories to consider next, especially via improving existing theories. In one sense, traders can already improve via taking their trading history into account when making new trades. But it would be nice if this were more continuous with the process of adding new traders.
One way to augment Garrabrant induction to account for the process of theory design would be if the existing traders could influence which new traders are added each day. But that doesn’t quite capture what we want, because in scientific epistemology new models evolve from old models and inherit much of their credibility. A theory that has one wrong belief can still be patched in a way which allows it to retain most of the credit for its previous correct predictions. So perhaps traders could be allowed to “donate” their wealth to other traders. More generally, if traders are allowed to invest in each other, then this allows them to represent higher-level concepts composed of the concepts represented by other traders, without needing to reimplement those same concepts internally.
However, all of this makes the concept of “trading strategies” much more complicated—now it’s about relationships between different traders. And I’m uncertain which of these suggestions are adding important innovations, versus adding unimportant details that the original formulation of Garrabrant induction correctly abstracted away.
Wealth vs truth
Making a bunch of wealth certainly suggests that a trader has an approximately-true model of the world. But the key difficulty with interpreting wealth as degree of truth is that wealth is rivalrous, whereas degree of truth isn’t. If a mostly-true theory reallocated its wealth between many slightly-different variants of itself, all of them would still be mostly true, but each of them would have much less wealth. More generally, gaining the most wealth requires betting against the consensus, and so contrarian traders might outcompete conformist traders even if they’re less correct overall than any given conformist. We could try to group traders into clusters, and talk about the degree of truth of each cluster—however, that just moves the same problem to a higher level.
When we face difficulties in defining a concept like degrees of truth, a useful question is “what do we want to use the concept for?” One answer is that traders whose models are more true should get more influence over our actions (given some mechanism for hooking up a logical inductor’s outputs to actions, which I’ll leave unspecified here). But this is still a rivalrous criterion, because our actions need to be determined by some set of traders. However, a less rivalrous version of this answer is that a model’s degree of truth affects how much we trust it to influence our actions relative to non-model-based policies. This seems to intuitively track scientific epistemology. When even our best theories of a phenomenon are quite bad, we’d prefer to rely on intuitions, habits, or traditions that have worked in the past (even when we don’t know why they work). Conversely, when we’re confident that our best theories are very close to the truth, we’re willing to follow even very counterintuitive recommendations from them.
I don’t know how to pin down the distinction between model-based and model-free traders, but it seems related to the concept of gears-level understanding. Eliezer also discussed some related points in this post (see also my comment beneath it).
For example, in this post Abram identifies some ways that understanding Garrabrant induction should change how Bayesians think about hypotheses. But Bayesian hypotheses are so different from Garrabrantian traders that using the same term for both seems misleading. In particular, the former are assigned credences, while the latter aren’t! That’s a much more fundamental change than the ones Abram identifies.