For the goal of maintaining the safety and autonomy of particular agents (and potential moral patients), there seems to be something important about maintaining the membranes and boundaries of those agents. A few examples:

A bacterium uses its membrane to protect its internal processes from external influences.

A nation maintains its sovereignty by defending its borders.

A human protects their mental integrity by selectively filtering the information that comes in and out of their mind.

What is the membrane of an agent?

Membranes are the things that maintain themselves despite environmental perturbation. (This is one reason I prefer the term “membranes” over “boundaries”. I think agent membranes are best understood as being living.)

Membranes are one way that agents can causally distance themselves from their environment and threats. 

Think of a bacterium: it has a membrane that separates it from its environment, and the bacterium can exert a great deal of control on what comes in and what goes out using its membrane (via ion channels, gap junctions, etc.).

The bacterium uses its membrane to attempt to keep bad things (e.g.: toxins, potential pathogens) out, and keep good things (e.g.: energy, sovereignty) in.

When an agent fails to causally distance themselves from their environment and threats, it dies.

When an agent maintains causal distance from its environment— when that agent is able to control how the environment affects it, and also how it affects the environment— then it tends to live.

Agents that persist through time tend to maintain membranes.

Causal distance

If you’re an agent, one way to keep yourself safe from threats is to put physical distance between you and the potential threats. It’s hard to get shot by other people when you’re a thousand miles away from everyone else.

Physical distance can therefore often provide causal distance.

Similarly, membranes can also provide causal distance.

In the absence of physical distance from threats, membranes can help provide causal distance.

Again, consider a bacterium: a bacterium has a membrane that separates it from its environment.

Through this membrane, the bacterium can exert a significant amount of control on what comes in and what goes out (via ion channels, gap junctions, etc.).

“Causal distance” can also be useful for conceptualizing interactions in information space. For example, computer systems can be constructed such that only the processes that need to have read and/or write access to particular resources have the privilege^[1] to do so. These privileges can be enforced with cryptography/encryption.

Such a computer system would be secure to the extent that it verifies the information that is received and signs the information that it sent. It’s a computational membrane.

Embedded agents attempt to de-embed themselves

Membranes are one way that embedded agents can try to de-embed themselves from their environment.^[2]

For example: I would love to be physically bulletproof, have a perfect immune system, etc. (as long as there were no downsides). Wouldn’t you?

Membranes are not about preferences

Common misunderstanding: "boundaries are just preferences".

Correction: Boundaries/membranes do not necessarily depend on preferences. 

Andrew Critch, «Boundaries» Sequence, Part 3b:

my goal is to treat boundaries as more fundamental than preferences, rather than as merely a feature of them.  In other words, I think boundaries are probably better able to carve reality at the joints than either preferences or utility functions, for the purpose of creating a good working relationship between humanity and AI technology

Formalize "safety"?

See Agent membranes and formalizing “safety” 

Mathematical formalism

How might membranes/boundaries be formalized mathematically?

Markov blankets seem to be a fitting abstraction. 

A Markov blanket between agent and its environment:

In which case,

• Infiltration of information across this Markov blanket measures membrane piercing, and low infiltration indicates the absence of such piercing.
• (And it may also be useful to keep track of exfiltration across the Markov blanket?^[3])

For more details, see distillation Formalizing «Boundaries» with Markov blankets. 

Also, there are probably other information-theoretic measures that are useful for formalizing membranes/boundaries.

Subscribe to the boundaries/membranes LessWrong tag to get notified of new developments.

Thanks to Jonathan Ng, Alexander Gietelink Oldenziel, Alex Zhu, and Evan Miyazono for reviewing drafts of this post. Thanks to Scott Garrabrant for a conversation that got me thinking about causal distance again. 

1. ^{^}

   See Principle of least privilege - Wikipedia, a.k.a. Principle of Least Authority

2. ^{^}

   Something to think about: relationship to instrumental convergence? ~“Embedded agents want to become de-embedded agents.”

3. ^{^}

   exfiltration, i.e.: privacy and the absence of mind-reading. But I need to think more about this. Related section: “Maintaining Boundaries is about Maintaining Free Will and Privacy” by Scott Garrabrant.