Contemporary rationalist thought is very much dependent on evidence. Evidence is sourced from experiments, observations, or logical conclusions that manage to give some degree of probabilistic guarantee that a certain set of statements about the state of the world is true, or at least likely to be correct, in the absence of absolute certainty. Experiments and observations are often themselves sourced from oracles; such oracles could be based on objectivity (i.e., being derived from a certain set of axioms or natural rules of the world) or on individual authority (they come from individuals or dogmas which are highly regarded). More often than not, our tendency to update our rationale about the current state of the world is dependent on finding slightly stronger oracles, such as a better theory or a source of authority with more accolades. A good rationalist simply uses the oracles and sources of evidence at their disposal to update their own predictions about the world.
Bayesian epistemology and, to a degree, a majority of rationalist thought are rooted in the idea of updating an individual set of hypotheses based on circumstantial evidence. However, this line of thought fails to consider that the majority of hyper-competitive real-world actions, or fundamental changes in technology or science, are undertaken by constructing entirely new theories, ones that were not even in the original hypothesis space, and that a lot of the hypotheses in the original space have varying degrees of correctness. This line of thinking is often formalized as scientific epistemology. When considering fundamentally hard problems (such as technical AI alignment or governance/policy), scientific epistemology often motivates a better model of analysis than its Bayesian counterpart. Richard Ngo's Towards a Formal Scientific Epistemology states this quite well: the central object of an epistemology of science should probably not be a probability distribution over a fixed set of propositions or hypotheses. Science is unusual precisely because it manufactures new ways of representing the world. Any formalism which treats hypothesis generation as occurring outside epistemology has abstracted away much of the thing it is trying to explain.
Ngo's primary focus, as the title indicates, is centered around formalizing the notion of scientific epistemology, mainly by extending Garrabrant Induction to become a formalism for scientific predictions by incorporating some degree of variability: while Garrabrant Induction and technical logical induction primarily focus on computable algorithms, a formal theory of science also needs to take into account a gradient of different hypotheses (some of which may be old) and will need to be able to modify the primary action, which typically is a set of algorithms, over time.
The original logical induction framework that inspired this line of thinking assigns probabilities to logical statements using a market mechanism, with efficiently computable traders identifying opportunities to exploit the current prices. The logical induction criterion requires, roughly, that no polynomial-time trading strategy can earn unbounded profits while taking only bounded risk. This allows the framework to account for the discovery of computationally accessible regularities, rather than requiring every relationship between statements to be known in advance
This line of thinking seems to be quite promising: after all, Bayesian epistemology in and of itself has quite a static ontology centered around individual, immutable hypotheses, with individual weights assigned to them that are upgraded based on new observations, which does not lend itself well to formal science. Meanwhile, Garrabrant Induction, with some slight extensions as described by Ngo, can describe a scenario in which an arbitrary set of computational algorithms can exploit various regularities or situations without needing to create mutually exclusive hypotheses about the world.
However, there is yet one thing missing even with the additions: epistemic quality depends not just on the final representation or prediction, but also on the entire history behind it, the information and individual selection pressures that led to that particular prediction. We define this history as epistemic provenance.
This is already discussed somewhat by Ngo when he highlights theories designed around known observations and the difficulty of detecting complexity hidden in auxiliary assumptions; the idea here is to make that complexity a core variable in scientific epistemology rather than an afterthought. The fundamental idea is quite straightforward: agreement with observations can result from discovering a generalizable regularity, but it can also result from repeatedly adapting a theory to those observations. An evaluator who sees only the final fit may be unable to distinguish these explanations. Information about how the individual observations were collected and reasoned over becomes important, such as what was available, what was changed, what was discarded, and what was genuinely withheld.
This does not mean that a theory becomes less true just because the reasoning behind it was subpar, nor does it show that traditional Bayesian inference cannot incorporate selection bias or effects. The distinction is between how accurate a model is and what a particular record of success establishes about its accuracy. Old evidence, for example, becomes more relevant if it can somewhat constrain or fit something that was not necessarily designed to fit it in the first place.
The fundamental question of scientific epistemology therefore becomes: How should we create a new hypothesis or world model upon being given a wide set of observations, each with individual provenances and biases?
A particularly relevant example follows. Assume that you had two independent problem solvers (automated or otherwise) come up with an equation M, designed to effectively predict or solve for some observations. The first problem solver derived M from first principles and then discovered that it predicts observations through intuition, while the second used pure brute force to find M based on retroactively fitting it to the observations at hand. Although the final solution is the same, the solutions have different epistemic provenance. For the first problem solver, the observations tested predictions fixed independently of the evaluation data. The experiment could have produced results that contradicted those predictions. For the second, agreement with the observations was part of the selection criterion: a theory that failed to fit would not have been presented. Evaluating only the successful candidate conceals the search process that made its success possible.
It is worth noting that discovering a theory through existing observations is a legitimate and indispensable part of science. However, it does fundamentally change how we should view epistemic formalization. Rather than merely being derived from a model and a set of observations, it should be dependent on the model and the full history of how it was formed. A theory's evidential status cannot always be reconstructed from the theory and observations alone. We must also know which observations influenced its construction, which parts were inherited from predecessors, which predictions were genuinely exposed to falsification, and which apparent discoveries arose from correlated search.
Therefore, a formal scientific epistemology requires a theory of epistemic provenance in addition to a theory of belief or model quality. This line of thinking leads to some particularly interesting corollaries; for one, the four major problems with traditional Garrabrant induction (old evidence, traders vs. models, trader modification, and wealth vs. correctness) all end up falling under one class of problem, the problem of assigning provenance. In part 2 of this series, we will look at each of these problems and also formalize what a scientific epistemology could look like.
Contemporary rationalist thought is very much dependent on evidence. Evidence is sourced from experiments, observations, or logical conclusions that manage to give some degree of probabilistic guarantee that a certain set of statements about the state of the world is true, or at least likely to be correct, in the absence of absolute certainty. Experiments and observations are often themselves sourced from oracles; such oracles could be based on objectivity (i.e., being derived from a certain set of axioms or natural rules of the world) or on individual authority (they come from individuals or dogmas which are highly regarded). More often than not, our tendency to update our rationale about the current state of the world is dependent on finding slightly stronger oracles, such as a better theory or a source of authority with more accolades. A good rationalist simply uses the oracles and sources of evidence at their disposal to update their own predictions about the world.
Bayesian epistemology and, to a degree, a majority of rationalist thought are rooted in the idea of updating an individual set of hypotheses based on circumstantial evidence. However, this line of thought fails to consider that the majority of hyper-competitive real-world actions, or fundamental changes in technology or science, are undertaken by constructing entirely new theories, ones that were not even in the original hypothesis space, and that a lot of the hypotheses in the original space have varying degrees of correctness. This line of thinking is often formalized as scientific epistemology. When considering fundamentally hard problems (such as technical AI alignment or governance/policy), scientific epistemology often motivates a better model of analysis than its Bayesian counterpart. Richard Ngo's Towards a Formal Scientific Epistemology states this quite well: the central object of an epistemology of science should probably not be a probability distribution over a fixed set of propositions or hypotheses. Science is unusual precisely because it manufactures new ways of representing the world. Any formalism which treats hypothesis generation as occurring outside epistemology has abstracted away much of the thing it is trying to explain.
Ngo's primary focus, as the title indicates, is centered around formalizing the notion of scientific epistemology, mainly by extending Garrabrant Induction to become a formalism for scientific predictions by incorporating some degree of variability: while Garrabrant Induction and technical logical induction primarily focus on computable algorithms, a formal theory of science also needs to take into account a gradient of different hypotheses (some of which may be old) and will need to be able to modify the primary action, which typically is a set of algorithms, over time.
The original logical induction framework that inspired this line of thinking assigns probabilities to logical statements using a market mechanism, with efficiently computable traders identifying opportunities to exploit the current prices. The logical induction criterion requires, roughly, that no polynomial-time trading strategy can earn unbounded profits while taking only bounded risk. This allows the framework to account for the discovery of computationally accessible regularities, rather than requiring every relationship between statements to be known in advance
This line of thinking seems to be quite promising: after all, Bayesian epistemology in and of itself has quite a static ontology centered around individual, immutable hypotheses, with individual weights assigned to them that are upgraded based on new observations, which does not lend itself well to formal science. Meanwhile, Garrabrant Induction, with some slight extensions as described by Ngo, can describe a scenario in which an arbitrary set of computational algorithms can exploit various regularities or situations without needing to create mutually exclusive hypotheses about the world.
However, there is yet one thing missing even with the additions: epistemic quality depends not just on the final representation or prediction, but also on the entire history behind it, the information and individual selection pressures that led to that particular prediction. We define this history as epistemic provenance.
This is already discussed somewhat by Ngo when he highlights theories designed around known observations and the difficulty of detecting complexity hidden in auxiliary assumptions; the idea here is to make that complexity a core variable in scientific epistemology rather than an afterthought. The fundamental idea is quite straightforward: agreement with observations can result from discovering a generalizable regularity, but it can also result from repeatedly adapting a theory to those observations. An evaluator who sees only the final fit may be unable to distinguish these explanations. Information about how the individual observations were collected and reasoned over becomes important, such as what was available, what was changed, what was discarded, and what was genuinely withheld.
This does not mean that a theory becomes less true just because the reasoning behind it was subpar, nor does it show that traditional Bayesian inference cannot incorporate selection bias or effects. The distinction is between how accurate a model is and what a particular record of success establishes about its accuracy. Old evidence, for example, becomes more relevant if it can somewhat constrain or fit something that was not necessarily designed to fit it in the first place.
The fundamental question of scientific epistemology therefore becomes: How should we create a new hypothesis or world model upon being given a wide set of observations, each with individual provenances and biases?
A particularly relevant example follows. Assume that you had two independent problem solvers (automated or otherwise) come up with an equation M, designed to effectively predict or solve for some observations. The first problem solver derived M from first principles and then discovered that it predicts observations through intuition, while the second used pure brute force to find M based on retroactively fitting it to the observations at hand. Although the final solution is the same, the solutions have different epistemic provenance. For the first problem solver, the observations tested predictions fixed independently of the evaluation data. The experiment could have produced results that contradicted those predictions. For the second, agreement with the observations was part of the selection criterion: a theory that failed to fit would not have been presented. Evaluating only the successful candidate conceals the search process that made its success possible.
It is worth noting that discovering a theory through existing observations is a legitimate and indispensable part of science. However, it does fundamentally change how we should view epistemic formalization. Rather than merely being derived from a model and a set of observations, it should be dependent on the model and the full history of how it was formed. A theory's evidential status cannot always be reconstructed from the theory and observations alone. We must also know which observations influenced its construction, which parts were inherited from predecessors, which predictions were genuinely exposed to falsification, and which apparent discoveries arose from correlated search.
Therefore, a formal scientific epistemology requires a theory of epistemic provenance in addition to a theory of belief or model quality. This line of thinking leads to some particularly interesting corollaries; for one, the four major problems with traditional Garrabrant induction (old evidence, traders vs. models, trader modification, and wealth vs. correctness) all end up falling under one class of problem, the problem of assigning provenance. In part 2 of this series, we will look at each of these problems and also formalize what a scientific epistemology could look like.