The Metaphysical Structure of Pearl's Theory of Time

Epistemic status: metaphysics

I was reading Factored Space Models (previously, Finite Factored Sets) and was trying to understand in what sense it was a Theory of Time.

Scott Garrabrant says "[The Pearlian Theory of Time] ... is the best thing to happen to our understanding of time since Einstein". I read Pearl's book on Causality^[1], and while there's math, this metaphysical connection that Scott seems to make isn't really explicated. Timeless Causality and Timeless Physics is the only place I saw this view explained explicitly, but not at the level of math / language used in Pearl's book.

Here is my attempt at explicitly writing down what all of these views are pointing at (in a more rigorous language)—the core of the Pearlian Theory of Time, and in what sense FSM shares the same structure.

Causality leave a shadow of conditional independence relationships over the observational distribution. Here's an explanation providing the core intuition:

1. Suppose you represent the ground truth structure of [causality / determination] of the world via a Structural Causal Model over some variables, a very reasonable choice. Then, as you go down th

Thoughtdump on why I'm interested in computational mechanics:

• one concrete application to natural abstractions from here: tl;dr, belief structures generally seem to be fractal shaped. one major part of natural abstractions is trying to find the correspondence between structures in the environment and concepts used by the mind. so if we can do the inverse of what adam and paul did, i.e. 'discover' fractal structures from activations and figure out what stochastic process they might correspond to in the environment, that would be cool
  • ... but i was initially interested in reading compmech stuff not with a particular alignment relevant thread in mind but rather because it seemed broadly similar in directions to natural abstractions.
• re: how my focus would differ from my impression of current compmech work done in academia: academia seems faaaaaar less focused on actually trying out epsilon reconstruction in real world noisy data. CSSR is an example of a reconstruction algorithm. apparently people did compmech stuff on real-world data, don't know how good, but effort-wise far too less invested compared to theory work
  • would be interested in these reconstruction algorithms, eg what are the b

Epistemic status: literal shower thoughts, perhaps obvious in retrospect, but was a small insight to me.

I’ve been thinking about: “what proof strategies could prove structural selection theorems, and not just behavioral selection theorems?”

Typical examples of selection theorems in my mind are: coherence theorems, good regulator theorem, causal good regulator theorem.

• Coherence theorem: Given an agent satisfying some axioms, we can observe their behavior in various conditions and construct $U$, and then the agent’s behavior is equivalent to a system that is maximizing $U$.
  • Says nothing about whether the agent internally constructs $U$ and uses them.
• (Little Less Silly version of the) Good regulator theorem: A regulator $R$ that minimizes the entropy of a system variable $S$ (where there is an environment variable $X$ upstream of both $R$ and $S$) without unnecessary noise (hence deterministic) is behaviorally equivalent to a deterministic function of $S$ (despite being a function of $X$).
  • Says nothing about whether $R$ actually internally reconstructs $S$ and uses it to produce its output.
• Causal good regulator t

Learning algebraic topology, homotopy always felt like a very intuitive and natural sort of invariant to attach to a space whereas for homology I don't think I have anywhere as close of an intuitive handle or sense of naturality of this concept as I do for homotopy. So I tried to collect some frames / results for homology I've learned to see if it helps convince my intuition that this concept is indeed something natural in mathspace. I'd be very curious to know if there are any other frames or Deeper Answers to "Why homology?" I'm missing:

1. Measuring "holes" of a space
   • Singular homology: This is the first example I encountered, which will serve as intuition / motivation for the later abstract definitions.
   • Fixing some notations (feel free to skip this bullet point if you're familiar with the notations):
     • Let's fix some space $X$, and recall our goal associating to that space an algebraic object invariant under homeomorphism / homotopy equivalence.
     • First, a singular $p$-simplex is a map $\sigma :{\Delta }^p\to X$, intuitively representing a simplex living inside the space $X$. So there is a natural ${\sigma }^{\left(i\right)}:{\Delta }^{p-1}\to X$ map which represents each of the $i$ faces. Then, it is natural to co

Non-Shannon-type Inequalities

The first new qualitative thing in Information Theory when you move from two variables to three variables is the presence of negative values: information measures (entropy, conditional entropy, mutual information) are always nonnegative for two variables, but there can be negative triple mutual information $I(X;Y;Z)$.

This so far is a relatively well-known fact. But what is the first new qualitative thing when moving from three to four variables? Non-Shannon-type Inequalities.

A fundamental result in Information Theory is that $I(X;Y\mid Z)\ge 0$ always holds.

• Given $n$ random variables $X_1,\dots ,X_n$ and $\alpha ,\beta ,\gamma \subseteq \left[n\right]$, from now on we write $I(\alpha ;\beta \mid \gamma )$ with the obvious interpretation of the variables standing for the joint variables they correspond to as indices.

Since $I(\alpha ;\beta |\gamma )\ge 0$ always holds, a nonnegative linear combination of a bunch of these is always a valid inequality, which we call a Shannon-type Inequality.

Then the question is, whether Shannon-type Inequalities capture all valid information inequalities of $n$ variable. It turns out, yes for $n=2$, (approximately) yes for $n=3$, and no for $n\ge 4$.

Beho... (read more)

The 3-4 Chasm of Theoretical Progress

epistemic status: unoriginal. trying to spread a useful framing of theoretical progress introduced from an old post.

Tl;dr, often the greatest theoretical challenge comes from the step of crossing the chasm from [developing an impractical solution to a problem] to [developing some sort of a polytime solution to a problem], because the nature of their solutions can be opposites.

Summarizing Diffractor's post on Program Search and Incomplete Understanding:

Solving a foundational problem to its implementation often takes the following steps (some may be skipped):

1. developing a philosophical problem
2. developing a solution given infinite computing power
3. developing an impractical solution
4. developing some sort of polytime solution
5. developing a practical solution

and he says that it is often during the 3 -> 4 step in which understanding gets stuck and the most technical and brute-force math (and i would add sometimes philosophical) work is needed, because:

• a common motif in 3) is that they're able to proving interesting things about their solutions, like asymptotic properties, by e.g., having their algorithms iterate through all turing machines, hence somewhat c

A simple sketch of the role data structure plays in loss landscape degeneracy.

The RLCT^[1] is a function of both $q\left(x\right)$ and $p\left(x|\theta \right)$. The role of $p\left(x|\theta \right)$ is clear enough, with very intuitive examples^[2] of local degeneracy arising from the structure of the parameter function map. However until recently the intuitive role of $q\left(x\right)$ really eluded me.

I think I now have some intuitive picture of how structure in $q\left(x\right)$ influences RLCT (at least particular instances of it). Consider the following example.

Toy Example: G-invariant distribution, G-equivariant submodule

Suppose the true distribution is (1) realizable ($p(\cdot |{\theta }^{\ast })=q(\cdot )$ for some ${\theta }^{\ast }$), (2) invariant under some group action, $q\left(x\right)=q\left(gx\right)\forall x$. Now, suppose that the model class is that of exponential models, i.e. $p\left(x|w\right)\propto \exp (⟨\theta ,T\left(x\right)⟩)$. In particular, suppose that $T$, the fixed feature map, is $G$-equivariant, i.e. $\exists \rho :G\to \mathrm{GL}\left(R^d\right)$ such that $T\left(gx\right)=\rho \left(g\right)T\left(x\right)$.

Claim: There is a degeneracy of the form $p\left(x|{\theta }^{\ast }\right)=p\left(x|\rho \right(g{)}^{\ast }\left({\theta }^{\ast }\right))$, and in particular if $G$ is a Lie group, the rank upper bound of RLCT decreases by $\frac{1}{4}\dim G$.

This is nothing nontrivial. The fir... (read more)

Just read through Robust agents learn causal world models and man it is really cool! It proves a couple of bona fide selection theorems, talking about the internal structure of agents selected against a certain criteria.

• Tl;dr, agents selected to perform robustly in various local interventional distributions must internally represent something isomorphic to a causal model of the variables upstream of utility, for it is capable of answering all causal queries for those variables.
  • Thm 1: agents achieving optimal policy (util max) across various local interventions must be able to answer causal queries for all variables upstream of the utility node
  • Thm 2: relaxation of above to nonoptimal policies, relating regret bounds to the accuracy of the reconstructed causal model
  • the proof is constructive - an algorithm that, when given access to regret-bounded-policy-oracle wrt an environment with some local intervention, queries them appropriately to construct a causal model
    • one implication is an algorithm for causal inference that converts black box agents to explicit causal models (because, y’know, agents like you and i are literally that aforementioned ‘regret-bounded-policy-oracle‘)
  • These selec

Quick paper review of Measuring Goal-Directedness from the causal incentives group.

tl;dr, goal directedness of a policy wrt a utility function is measured by its min distance to one of the policies implied by the utility function, as per the intentional stance - that one should model a system as an agent insofar as doing so is useful.

Details

• how is "policies implied by the utility function" operationalized? given a value $u$, we define a set containing policies of maximum entropy (of the decision variable, given its parents in the causal bayes net) among those policies that attain the utility $u$.
• then union them over all the achievable values of $u$ to get this "wide set of maxent policies," and define goal directedness of a policy $\pi$ wrt a utility function $U$ as the maximum (negative) cross entropy between $\pi$ and an element of the above set. (actually we get the same result if we quantify the min operation over just the set of maxent policies achieving the same utility as $\pi$.)

Intuition

intuitively, this is measuring: "how close is my policy $\pi$ to being 'deterministic,' while 'optimizing $U$ at the competence level $u($... (read more)

Becoming Stronger™ (Sep 13 - Sep 27)

Notes and reflections on the things I've learned while Doing Scholarship this week (i.e. studying math)^[1].

• I am starting to see the value of categorical thinking.
  • For example from [FOAG], it was quite mindblowing to learn that stalk (the set of germs at a point) can be equivalently defined as a simple colimit of sections of presheaf over open sets of $X$ containing a point, and this definition made proving certain constructions (eg inducing a map of stalks from a map $\varphi :X\to Y$) very easy.
  • Also, I was first introduced the concept of presheaf as an abstraction of a map that takes open sets and returns functions over it, abstracting properties like there existing a restriction map that composes naturally. Turns out (punchline, presumably) this is just a functor $Open(X{)}^{Op}->Set$!
• Yoneda lemma is very cool. I recall seeing some of the ideas from Programs as Singularities (paper), where there are ideas of embedding programs (for the analogue in Yoneda lemma, the Yoneda embedding being a fully faithful functor from some category $C$ ... ) into a different space (... to the category $Se{t}^C$ ...) that contains "almost progr

EDIT: I no longer think this setup is viable, for reasons that connect to why I think Critch's operationalization is incomplete and why boundaries should ultimately be grounded in Pearlian Causality and interventions. Check update.

I believe there's nothing much in the way of actually implementing an approximation of Critch's boundaries^[1] using deep learning.

Recall, Critch's boundaries are:

• Given a world (markovian stochastic process) $W_t$, map its values $W$ (vector) bijectively using $f$ into 'features' that can be split into four vectors each representing a boundary-possessing system's Viscera, Active Boundary, Passive Boundary, and Environment.
• Then, we characterize boundary-ness (i.e. minimal information flow across features unmediated by a boundary) using two mutual information criterion each representing infiltration and exfiltration of information.
• And a policy of the boundary-posessing system (under the 'stance' of viewing the world implied by $f$) can be viewed as a stochastic map (that has no infiltration/exfiltration by definition) that best approximates the true $W_t$ dynamics.
  • The interpretation here (under low exfiltration and infiltrati

Any thoughts on how to customize LessWrong to make it LessAddictive? I just really, really like the editor for various reasons, so I usually write a bunch (drafts, research notes, study notes, etc) using it but it's quite easy to get distracted.

moments of microscopic fun encountered while studying/researching:

• Quantum mechanics call vector space & its dual bra/ket because ... bra-c-ket. What can I say? I like it - But where did the letter 'c' go, Dirac?
• Defining cauchy sequences and limits in real analysis: it's really cool how you "bootstrap" the definition of Cauchy sequences / limit on real using the definition of Cauchy sequences / limit on rationals. basically:
  • (1) define Cauchy sequence on rationals
  • (2) use it to define limit (on rationals) using rational-Cauchy
  • (3) use it to define reals
  • (4)

Update to my last shortform on "Why Homology?"

My current favorite frame for thinking about homology is as fixing Poincare's initial conception of "counting submanifolds up to cobordism". (I've learned this perspective from this excellent blog post, and I summarize my understanding below.)

In Analysis Situs, Poincare sought to count m-submanifolds of a given a n-manifold up to some equivalence relation - namely, being a boundary of some (m+1)-submanifold, i.e. cobordism. I personally buy cobordism as a concept that is as natural as homotopy for one to have c... (read more)

Any advice on reducing neck and shoulder pain while studying? For me that's my biggest blocker to being able to focus longer (especially for math, where I have to look down at my notes/book for a long period of time). I'm considering stuff like getting a standing desk or doing regular back/shoulder exercises. Would like to hear what everyone else's setups are.

I am curious as to how often the asymptotic results proven using features of the problem that seem basically practically-irrelevant become relevant in practice.

Like, I understand that there are many asymptotic results (e.g., free energy principle in SLT) that are useful in practice, but i feel like there's something sus about similar results from information theory or complexity theory where the way in which they prove certain bounds (or inclusion relationship, for complexity theory) seem totally detached from practicality?

• joint source coding theorem is of

Complaint with Pugh's real analysis textbook: He doesn't even define the limit of a function properly?!

It's implicitly defined together with the definition of continuity where $\forall ϵ>0\exists \delta >0|x-x_0|<\delta ⟹|f\left(x\right)-f\left(x_0\right)|<ϵ$, but in Chapter 3 when defining differentiability he implicitly switches the condition to $0<|x-x_0|<\delta$ without even mentioning it (nor the requirement that $x_0$ now needs to be an accumulation point!) While Pugh has its own benefits, coming from Terry Tao's analysis textbook backgrou... (read more)

Having lived ~19 years, I can distinctly remember around 5~6 times when I explicitly noticed myself experiencing totally new qualia with my inner monologue going “oh wow! I didn't know this dimension of qualia was a thing.” examples:

• hard-to-explain sense that my mind is expanding horizontally with fractal cube-like structures (think bismuth) forming around it and my subjective experience gliding along its surface which lasted for ~5 minutes after taking zolpidem for the first time to sleep (2 days ago)
• getting drunk for the first time (half a year ago)
• feeli

To me, the fact that the human brain basically implements SSL+RL is very very strong evidence that the current DL paradigm (with a bit of "engineering" effort, but nothing like fundamental breakthroughs) will kinda just keep scaling until we reach point-of-no-return. Does this broadly look correct to people here? Would really appreciate other perspectives.

What's a good technical introduction to Decision Theory and Game Theory for alignment researchers? I'm guessing standard undergrad textbooks don't include, say, content about logical decision theory. I've mostly been reading posts on LW but as with most stuff here they feel more like self-contained blog posts (rather than textbooks that build on top of a common context) so I was wondering if there was anything like a canonical resource providing a unified technical / math-y perspective on the whole subject.

i absolutely hate bureaucracy, dumb forms, stupid websites etc. like, I almost had a literal breakdown trying to install Minecraft recently (and eventually failed). God.

God, I wish real analysis was at least half as elegant as any other math subject — way too much pathological examples that I can't care less about. I've heard some good things about constructivism though, hopefully analysis is done better there.

Does anyone know if Shannon arrive at entropy from the axiomatic definition first, or the operational definition first?

I've been thinking about these two distinct ways in which we seem to arrive at new mathematical concepts, and looking at the countless partial information decomposition measures in the literature all derived/motivated based on an axiomatic basis, and not knowing which intuition to prioritize over which, I've been assigning less premium on axiomatic conceptual definitions than i used to:

• decision theoretic justification of probability > C

Mildly surprised how some verbs/connectives barely play any role in conversations, even in technical ones. I just tried directed babbling with someone, and (I think?) I learned quite a lot about Israel-Pakistan relations with almost no stress coming from eg needing to make my sentences grammatically correct.

Example of (a small part of) my attempt to summarize my understanding of how Jews migrated in/out of Jerusalem over the course of history:

They here *hand gesture on air*, enslaved out, they back, kicked out, and boom, they everywhere.

(audience nods, giv

Why haven't mosquitos evolved to be less itchy? Is there just not enough selection pressure posed by humans yet? (yes probably) Or are they evolving towards that direction? (they of course already evolved towards being less itchy while biting, but not enough to make that lack-of-itch permanent)

this is a request for help i've been trying and failing to catch this one for god knows how long plz halp

tbh would be somewhat content coexisting with them (at the level of houseflies) as long as they evolved the itch and high-pitch noise away, modulo disease risk considerations.

Is there a way to convert a LessWrong sequence into a single pdf? Should ideally preserve comments, latex, footnotes, etc.

I don't know if this is just me, but it took me an embarrassingly long time in my mathematical education to realize that the following three terminologies, which introductory textbooks used interchangeably without being explicit, mean the same thing. (Maybe this is just because English is my second language?)

X => Y means X is sufficient for Y means X only if Y

X <= Y means X is necessary for Y means X if Y

Unidimensional Continuity of Preference $\approx$ Assumption of "Resources"?

tl;dr, the unidimensional continuity of preference assumption in the money pumping argument used to justify the VNM axioms correspond to the assumption that there exists some unidimensional "resource" that the agent cares about, and this language is provided by the notion of "souring / sweetening" a lottery.

Various coherence theorems - or more specifically, various money pumping arguments generally have the following form:

If you violate this principle, then [you are rationally re

Damn, why did Pearl recommend readers (in the preface of his causality book) to read all the chapters other than chapter 2 (and the last review chapter)? Chapter 2 is literally the coolest part - inferring causal structure from purely observational data! Almost skipped that chapter because of it ...

One of the rare insightful lessons from high school: Don't set your AC to the minimum temperature even if it's really hot, just set it to where you want it to be.

It's not like the air released gets colder with lower target temperature, because most ACs (according to my teacher, I haven't checked lol) are just a simple control system that turns itself on/off around the target temperature, meaning the time it takes to reach a certain temperature X is independent of the target temperature (as long it's lower than X)

... which is embarrassingly obvious in hindsight.

Useful perspective when thinking of mechanistic pictures of agent/value development is to take the "perspective" of different optimizers, consider their relative "power," and how they interact with each other.

E.g., early on SGD is the dominant optimizer, which has the property of (having direct access to feedback from U / greedy). Later on early proto-GPS (general-purpose search) forms, which is less greedy, but still can largely be swayed by SGD (such as having its problem-specification-input tweaked, having the overall GPS-implementation modified, etc). ... (read more)

I've noticed during my alignment study that just the sheer amount of relevant posts out there is giving me a pretty bad habit of (1) passively engaging with the material and (2) not doing much independent thinking. Just keeping up to date & distilling the stuff in my todo read list takes up most of my time.

• I guess the reason I do it is because (at least for me) it takes a ton of mental effort to switch modes between "passive consumption" and "active thinking":
  • I noticed then when self-studying math; like, my subjective experience is that I enjoy both "p

Random alignment-related idea: train and investigate a "Gradient Hacker Enzyme"

TL;DR, Use meta-learning methods like MAML to train a network submodule i.e. circuit that would resist gradient updates in a wide variety of contexts (various architectures, hyperparameters, modality, etc), and use mechanistic interpretability to see how it works.

It should be possible to have a training setup for goals other than "resist gradient updates," such as restricting the meta-objective to a specific sub-sub-circuit. In that case, the outer circuit might (1) instrumental... (read more)