Scott, thanks for writing this! While I very much agree with the distinctions being drawn, I think the word "boundary" should be usable for referring to factorizations that do not factor through the physical separation of the world into objects. In other words, I want the technical concept of «boundaries» that I'm developing to be able to refer to things like social boundaries, which are often not most-easily-expressed in the physics factorization of the world into particles (but are very often expressible as Markov blankets in a more abstract space, I claim).
Because of this, instead of using "boundary" only for partitions, and "frame" only for factorizations, I propose to instead just use "part"/"partition" for referring to partitions, and "factor"/"factorization" for referring to factorizations. E.g.,
Otherwise, we are using up four words (partition, boundary, factorization, frame) to refer to two things (parts and factors).
Then, in my proposed language, a
Overall, I suspect this language convention to be more expressive than what you are proposing.
What do you think?
I agree completely. I am not really happy with any of the language in this post, and I want it to have scope limited to this post. I will for the most part say boundary for both the additive and multiplicative variants.
Going further, my proposed convention also suggests that "Cartesian frames" should perhaps be renamed to "Cartesian factorizations", which I think is a more immediately interpretable name for what they are. Then in your equation , you can refer to and as "Cartesian factors", satisfying your desire to treat and as interchangeable. And, you leave open the possibility that the factors are derivable from a "Cartesian partition" of the world into the "Cartesian parts" and .
There is of course the problem that for some people "Cartesian" just means "factoring into coordinates" (e.g., "Cartesian plane"), in which case "Cartesian factorization" will sound a bit redundant, but for those people "Cartesian frame" is already not very elucidating.
Note that I wrote this post a month ago, while seeing an earlier draft of the sequence post 3a (before active/passive distinction was central) and was waiting to post it until after that post. I am posting it now unedited, so some of the thoughts here might be outdated. In particular, I think this post does not respect enough the sense in which the FFS ontology is wrong in that it does not have space for expressing the direction of entanglement.
I only vaguely understand both Cartesian frames and «boundaries», but I want to take a shot at explaining what feels to me like some confusions in this post. Also a bunch of this might be invalidated by your disclaimer that this post was published before the final version of Crtich's boundaries sequence.
It seems to me that your work in Cartesian frames is about finding an ontology in which we can make progress on agent foundations in general, whereas Critch's concept of boundaries is about formalizing a necessary condition for identifying when an agent exists in a particular system.
While every boundary can be recast as a frame, the converse is not also true.
Indeed! We wouldn't want a theory of agents (or boundaries) to be applicable to a state in thermal equilibrium, for example, whereas we definitely want our ontology to be able to handle it.
Getting Past the Physical Frame
This whole section is actually a little confusing. It sounds like you're interpreting Critch's sequence to be saying that boundaries are inherently physical. But aren't all the definitions in terms of information theory? The nodes in the network are course-grained, but aren't we free to decide what about the world the are abstracted from?
This also further motivates frames over boundaries, as we might want to reason about multiple different frames simultaneously with no underlying factorization that multiplicatively-refines both of them.
It seems entirely possible to me for a formalization of boundaries to allow for two different but equally-valid boundaries to be drawn within the state.
Actually, now that I think about it, is there any reason that Critch's definition of boundaries isn't fully compatible with the ontology of factored sets?
I continue to be excited about this line of work. I feel like you're slowly figuring out how to formalize ontology in a way reflective of what we actually do and generalizing it. This is something missing from a lot of other approaches.
I've compiled all of the current «Boundaries» x AI safety thinking and research (like this post) that I could find here: «Boundaries» and AI safety compilation
Maintaining Boundaries is about Maintaining Free Will and Privacy
I really like this conceptualization! Especially "privacy". I've written a post about the finer details of this wrt to infiltration and exfiltration.
Here’s a peak at how I summarize this at the end:
This post is partially in response to Critch's boundaries sequence. My best guess is that he would agree with most of it in theory, but disagree with some of it in practice due to tradeoffs with other considerations in defining the concepts.
Imagine a world consisting of some atomic objects . For example, you can think of the as physical atoms or cells in the game of life. Each object comes along with a collection of states it can be in .
(I am not committing now on whether or not we are thinking about our model as timeless. Maybe should be thought about as a cell in the game of life that passes through time, or maybe it should be thought about as a (cell, time) pair.)
There are two sets we naturally want to associate with our world. First, we have , which I will call the object space. Second, we have , which is the set of all ways to assign state to each object in . I will call this the state space. Note that partitions of correspond to factorizations of .
If I want to point at (for example) an agent in , I might tell you what atoms are inside that agent, and thus express in the form , where is the set of all atoms that are in the agent and is the set of all atoms that are outside of the agent (and thus in the environment).
The agent then has its own state space , while the environment has its own state space, . Now, to point at this agent in , I can express in the form .
Instead of specifying a list of objects in the agent, I specify the state space of the agent. Instead of thinking of the environment as the result of subtracting the agent object out of the world, I think of the environment as the result of quotienting out the agent's state space from the world's state space.
Instead of defining a Cartesian Boundary, I am defining a Cartesian Frame.
Playing on this naming scheme, in general (when not necessarily talking about agents), I will use the word boundary when talking about partitioning the object space into a disjoint inside and outside, and I will use the word frame when talking about factoring the state space into an independent inside and outside. (This naming might end up only being used within the scope of this post. I'm not sure.)
I usually prefer to think in terms of frames (but not always!). While every boundary can be recast as a frame, the converse is not also true. If we view the state space as primary, then by imagining the world as a collection of objects, we are essentially factoring the state space. The frames that correspond to boundaries are basically those that carve along the joints given by the factorization into basic objects.
If the world is given to you pre-factored as a collection of objects, then it makes sense to partition those objects into larger objects and draw boundaries around them. However, I think for interesting problems, this is rarely the case. Further, even if the world is pre-factorized, that factorization could be wrong!
Also, the act of drawing a boundary feels similar to me to the act of factoring the state space into atomic objects. Both are carving out interesting features of state space. The initial factorization into atomic objects is identifying microscopic objects, while the boundaries are identifying macroscopic objects. Thus, starting from a pre-factored world feels especially bad when thinking about identifying boundaries, since it is starting with half the problem already done. (Note: I think this argument is weaker than it sounds. The factorization into objects is a different type than the drawing of the boundaries.)
The main practical reason I think we need to work with frames is that we are simply not given a pre-factored world. If you look at the physical world, it feels like we have objects with which we can define boundaries. These atoms are the United States. These atoms are Canada. These atoms are neither. These atoms are on the boundary and are ambiguous/both/neither.
However when you look at math it becomes messier. (When you look at physics more deeply, it also becomes messy, but that is not my main point.) And I think to navigate the future, we are going to have to look at math.
When defining the boundaries on a physical organism, we can use the factorization we get from the physical world. However when defining the boundaries of a gene or a meme or an algorithm, (or even a physical organism that is being predicted by its environment) we don't have that luxury.
I predict that as time goes, informational organisms will become more and more powerful relative to physical organisms, and we will thus want to be able to model them. Or, put another way, frames that factor the world into informational organisms will be more predictive/productive than frames that factor the world into physical organisms.
(This also further motivates frames over boundaries, as we might want to reason about multiple different frames simultaneously with no underlying factorization that multiplicatively-refines both of them. (This was a major update for me in the last year. Finite factored sets was largely about discovering the one true underlying factorization, and unfortunately I now think things won't be that simple.) )
Note: this can be thought of as the motivation for a bunch of agent foundations questions. Logical uncertainty and decision theory are about taking concepts that we understand for physical organisms, and generalizing them to apply to informational organisms.
Note: This is not a disagreement with Critch's pressure towards thinking about living organisms. A meme is a central example of an organism that is living, but not physical.
In my previous work, and thus far in this post, I have been focusing on what I will call thin boundaries, but my model of Critch largely wants to talk about thick boundaries (for good reason). I will explain when I mean first in terms of boundaries, and then make it in terms of frames.
A boundary partitions the world into an inside and an outside. If the boundary is thin, then everything is either inside or outside, there is no ambiguous middle ground. This is the simplest kind of boundary, because the boundary isn't actually there! However, we are going to have to get past this over-simplification.
Above, I said: "I will use the word boundary when talking about partitioning the object space into a disjoint inside and outside."
Now, I want to instead say we are approximately partitioning the object space into an inside and outside, which would be disjoint if not for the thick boundary. So the thick boundary is the intersection of the inside with the outside.
Here is a nice visualization/example:
Imagine closed subset as the inside or agent, and the closure of the complement of will be the outside or environment, . The thick boundary is the intersection .
I like thinking of the thick boundary as both inside and outside, rather than neither/ambiguous, but you could define the inside/outside as the interior of my inside/outside instead.
Now, lets see what happens to this concept when thinking about frames.
Above, I said: "I will use the word frame when talking about factoring the state space into an independent inside and outside."
Now, I want to instead say that we are approximately factoring the state space into an inside and outside that would be independent if not for the thick frame. By this, I mean that the inside and outside are conditionally orthogonal given the thick frame. (recall that we are working in the state space, so "inside," "outside," and "thick frame" are properties of the world rather than physical objects, so we are the right type conditional orthogonality.)
I use the word "orthogonal" rather than "independent" because I don't want to restrict myself to a probability-based ontology, but you can just think of it as independent if you want. I am speaking loosely, and I am talking about the concepts from Finite Factored Sets, but only roughly.
Going back to our visualization, imagine that the state space of the world is actually the set of continuous functions , and our and are subsets of the domain as defined above. We can think of the state space of the agent as the set of continuous functions , and similarly think of the state space of the environment as the set of continuous functions . Finally, let be the boundary, and let B be the set of continuous functions .
Note that , since the agent and environment must agree on the boundary. However we do have , where is the set of all function in A that restrict to on , and similarly for (The shape of this formula should be strongly reminding you of conditional orthogonality/independence.)
The state of the boundary screens off the state of the inside from the state of the outside. When the frame is thin, the boundary is empty, and so only has one possible state, so the inside and outside are already independent. Here "independent" is referring to what is possible, not referring to probability.
I think that this ontology illustrates how important boundaries are for decision theory. You can think of the main problem of decision theory as figuring out how to think of the world as a function of your actions. Factoring the world as Agent and Environment is exactly doing this. By currying, we can see environment as a function from the agent to the world.
When thinking of boundaries as frames, now we see the definition of a boundary instead as the the definition of a factorization into agent and environment, which then gives us all our counterfactuals.
I am not a big fan of reasoning about agents/minds/anything as embedded in a pre-existing time. (I have avoided systems embedded in linear time in basically all my thoughts since logical induction in 2016.) However I claim that this is especially important for thinking about boundaries. Boundaries are about breaking the world up into orthogonal properties, (and maintaining their orthogonality) and I basically think of orthogonality and time as the same concept (orthogonal:disjoint::before:subset).
Unfortunately this is not very workable as advice, I am mostly just making the prediction that the textbook from the future that would make us change our ontology on time would also make us change our ontology on boundaries in a similar way. But in the meantime, if you can use simple notions of time to say something interesting about boundaries, that is better than saying nothing. (Also, I tend to manically say everything is the same as everything else, so advice like this should maybe be taken with a grain of salt.)
For example, since we are thinking of the thick frame as that which screens of an agent from its environment, we might want to think of the history of the development of the agent as part of its boundary.
A physical boundary can be thought of as keeping physical objects in or out. This has the consequence of keeping the stuff on the inside independent from the stuff on the inside. If we take the physical objects out of our ontology, all that is left is the independence. Instead of keeping physical stuff inside, we are keeping information inside, and thus maintaining the inside's privacy. Instead of keeping physical stuff outside, we are stopping the outside features from controlling the inside, and thus maintaining the inside's free will.