In this post I outline every post I could find that meaningfully connects the concept of «Boundaries/Membranes» (tag) with AI safety.^[1] This seems to be a booming subtopic: interest has picked up substantially within the past year. 

Update (2023 Dec): we're now running a workshop on this topic!

Perhaps most notably, Davidad includes the concept in his Open Agency Architecture for Safe Transformative AI alignment paradigm. For a preview of the salience of this approach, see this comment by Davidad (2023 Jan):

“defend the boundaries of existing sentient beings,” which is my current favourite. It’s nowhere near as ambitious or idiosyncratic as “human values”, yet nowhere near as anti-natural or buck-passing as corrigibility. 

This post also compiles work from Andrew Critch, Scott Garrabrant, Mark Miller, and others. But first I will recap what «Boundaries» are:

«Boundaries» definition recap:

You can see «Boundaries» Sequence for a longer explanation, but I will excerpt from a more recent post by Andrew Critch, 2023 March:

By boundaries, I just mean the approximate causal separation of regions in some kind of physical space (e.g., spacetime) or abstract space (e.g., cyberspace).  Here are some examples from my «Boundaries» Sequence:

• a cell membrane (separates the inside of a cell from the outside);
• a person's skin (separates the inside of their body from the outside);
• a fence around a family's yard (separates the family's place of living-together from neighbors and others);
• a digital firewall around a local area network (separates the LAN and its users from the rest of the internet);
• a sustained disassociation of social groups (separates the two groups from each other)
• a national border (separates a state from neighboring states or international waters).

Also, beware:

When I say boundary, I don't just mean an arbitrary constraint or social norm.

Update: see Agent membranes and causal distance for a better exposition of the agent membranes/boundaries idea.

Posts & researchers that link «Boundaries» and AI safety

All bolding in the excerpts below is mine.

Davidad’s OAA

Saliently, Davidad uses «Boundaries» for one of the four hypotheses he outlines in An Open Agency Architecture for Safe Transformative AI (2022 Dec)

• Deontic Sufficiency Hypothesis: There exists a human-understandable set of features of finite trajectories in such a world-model, taking values in $(-\infty ,0]$, such that we can be reasonably confident that all these features being near 0 implies high probability of existential safety, and such that saturating them at 0 is feasible^[2] with high probability, using scientifically-accessible technologies.
  • I am optimistic about this largely because of recent progress toward formalizing a natural abstraction of boundaries by Critch and Garrabrant. I find it quite plausible that there is some natural abstraction property $Q$ of world-model trajectories that lies somewhere strictly within the vast moral gulf of

$$\text{All Principles That Human CEV Would Endorse}\Rightarrow Q\Rightarrow \text{Don't Kill Everyone}$$

Further explanation of this can be found in Davidad's Bold Plan for Alignment: An In-Depth Explanation (2023 Apr) by Charbel-Raphaël and Gabin:

Getting traction on the deontic feasibility hypothesis 

Davidad believes that using formalisms such as Markov Blankets would be crucial in encoding the desiderata that the AI should not cross boundary lines at various levels of the world-model. We only need to “imply high probability of existential safety”, so according to davidad, “we do not need to load much ethics or aesthetics in order to satisfy this claim (e.g. we probably do not get to use OAA to make sure people don't die of cancer, because cancer takes place inside the Markov Blanket, and that would conflict with boundary preservation; but it would work to make sure people don't die of violence or pandemics)”. Discussing this hypothesis more thoroughly seems important.

Also: 

(*) Elicitors: Language models assist humans in expressing their desires using the formal language of the world model. […] Davidad proposes to represent most of these desiderata as violations of Markov blankets. Most of those desiderata are formulated as negative constraints because we just want to avoid a catastrophe, not solve the full value problem. But some of the desiderata will represent the pivotal process that we want the model to accomplish.

Also see this comment by Davidad (2023 Jan):

Not listed among your potential targets is “end the acute risk period” or more specifically “defend the boundaries of existing sentient beings,” which is my current favourite. It’s nowhere near as ambitious or idiosyncratic as “human values”, yet nowhere near as anti-natural or buck-passing as corrigibility. 

Reframing inner alignment by Davidad (2022 Dec):

I'm also excited about Boundaries as a tool for specifying a core safety property to model-check policies against—one which would imply (at least) nonfatality—relative to alien and shifting predictive ontologies.

I’ve also collected all of Davidad’s tweets about «Boundaries» into this twitter thread.

Update 2023 May: I've written a post about how Davidad conceives of «boundaries» applying to alignment: «Boundaries» for formalizing a bare-bones morality.

Update 2023 August: Davidad explains this most directly in A list of core AI safety problems and how I hope to solve them:

9. Humans cannot be first-class parties to a superintelligence values handshake.

[…]

OAA Solution: (9.1) Instead of becoming parties to a values handshake, keep superintelligent capabilities in a box and only extract plans that solve bounded tasks for finite time horizons and verifiably satisfy safety criteria that include not violating the natural boundaries of humans. This can all work without humans ever being terminally valued by AI systems as ends in themselves.

Update 2024 Jan 28: See Davidad's reply to this comment about specific examples of boundary violations.

Andrew Critch

Andrew Critch has written «Boundaries» Sequence with four posts to date: 

• «Boundaries», Part 1: a key missing concept from utility theory (2022 Jul)
• «Boundaries», Part 2: trends in EA's handling of boundaries (2022 Aug)
  • This post discusses «boundaries» in social systems
• Part 3a: Defining boundaries as directed Markov blankets (2022 Oct)
  • This post provides mathematical formalization of «boundaries»
  • Scott Garrabrant: “Overall, this is my favorite thing I have read on lesswrong in the last year.”
• Part 3b: «Boundaries», Part 3b: Alignment problems in terms of boundaries (2022 Dec):

AI alignment is a notoriously murky problem area, which I think can be elucidated by rethinking its foundations in terms of boundaries between systems, including soft boundaries and directional boundaries. […] I'm doing that now, for the following problem areas:

• Preference plasticity & corrigibility
• Mesa-optimizers
• AI boxing / containment
• (Unscoped) consequentialism
• Mild optimization & impact regularization
• Counterfactuals in decision theory

[…] 

You many notice that throughout this post that I've avoided saying things like "the humans prefer that {some boundary} be respected".  That's because my goal is to treat boundaries as more fundamental than preferences, rather than as merely a feature of them.  In other words, I think boundaries are probably better able to carve reality at the joints than either preferences or utility functions, for the purpose of creating a good working relationship between humanity and AI technology.

Critch also included «Boundaries» in his plan for Encultured AI (2022 Aug):

boundaries may be treated as constraints, but they are more specific than that: they delineate regions or features of the world in which the functioning of a living system occurs.  We believe many attempts to mollify the negative impacts of AI technology in terms of “minimizing side effects” or “avoiding over-optimizing” can often be more specifically operationalized as respecting boundaries.  Moreover, we believe there are abstract principles for respecting boundaries that are not unique to humans, and that are simple enough to be transferable across species and scales of organization. […]

And most recently, Critch wrote Acausal normalcy (2023 March):

Which human values are most likely to be acausally normal? 

A complete answer is beyond this post, and frankly beyond me.  However, as a start I will say that values to do with respecting boundaries are probably pretty normal from the perspective of acausal society.

Scott Garrabrant

Andrew Critch connects «Boundaries» to Scott Garrabrant’s Cartesian Frames (in Part 3a of his «Boundaries» Sequence):

The formalism here is lot like a time-extended version of a Cartesian Frame (Garrabrant, 2020), except that what Scott calls an "agent" is further subdivided here into its "boundary" and its "viscera". 

See Cartesian Frames (Intro) (2020 Oct) for a related formalization of the «Boundaries» core concept.  

Cartesian frames are a way to add a first-person perspective (with choices, uncertainty, etc.) on top of a third-person "here is the set of all possible worlds," in such a way that many of these problems either disappear or become easier to address.

Note: See this summary by Rohin Shah for a conceptual summary of Cartesian Frames.

Scott Garrabrant also wrote Boundaries vs Frames (2022 Oct) which compares the two concepts.

Note: I suspect Garrabrant’s work on Embedded Agency (pre- Cartesian Frames) and Finite Factored Sets (post- Cartesian Frames) are also related, but I haven’t looked into this myself.

Mark Miller

Mark Miller, Senior Fellow at the Foresight Institute (wiki), has worked on the Object-capability model, which applies «boundaries» to create secure systems (computer security). The goal is to make sure that only the processes that should have read and/or write permissions to a resource have those permissions. This can then be enforced with cryptography.

• Key concepts
  • Computer security, secure requests, principle of least privilege/authority (Wikipedia)
  • Object-capability model - Wikipedia
  • I usually break down an agent’s «boundaries» into two components: the unique ability to observe particular things (e.g.: only you can experience your subjective experience), and the unique ability to control particular things (e.g.: only your can control your actions). Note that “ability to observe” is like read privileges, and “ability to control” is like write privileges.
  • Example lecture: Bringing Object-orientation to Security Programming (Mark S. Miller, Google)
• Boundaries-based security and AI safety approaches — LessWrong by Allison Duettmann (2023 Apr)
  • Davidad: “I like a lot of the ideas here”
• AFAICT, Mark thinks that the problem of AI safety has little intrinsically to do with artificial intelligence, but instead to do with gross power imbalance. My understanding is that he thinks that safety comes not from the absence of threats, but from good security (eg in computer systems). See his 2023 talk on Misdiagnosing AGI risk.
• «Boundaries»-like ideas are also central in Gaming the Future

Other researchers interested:

John Wentworth (@johnswentworth)

John Wentworth lists boundaries in a comment addressing “what's my list of open problems in understanding agents?”:

I claim that, once you dig past the early surface-level questions about alignment, basically the whole cluster of "how do agents work?"-style questions and subquestions form the main barrier to useful alignment progress. So with that in mind, here are some of my open questions about understanding agents (and the even deeper problems one runs into when trying to understand agents)

[…]

• What's up with boundaries and modularity?
  • To what extent do boundaries/modules typically exist "by default" in complex systems, vs require optimization pressure (e.g. training/selection) to appear?
  • Why are biological systems so modular? To what extent will that generalize to agents beyond biology?
  • How modular are trained neural nets? Why, and to what extent will it generalize?
  • What is the right mathematical language in which to talk about modularity, boundaries, etc?
  • How do modules/boundaries interact with thermodynamics - e.g. can we quantify the negentropy/bits-of-optimization requirements to create new boundaries/modules, or maintain old ones?
  • Can we characterize the selection pressures on transboundary transport/information channels in a general way?
  • To what extent do agents in general form internal submodules? Why?

[…]

He also wrote in this comment that he considers boundaries to be prerequisite for understanding ‘agenty’ phenomena (2023 Apr).

Also see: Content and Takeaways from SERI MATS Training Program with John Wentworth: Week 4, Day 1 - Boundaries Exercises (2022 Dec) where the «Boundaries» concept is used as a SERI MATS training exercise.

[There is likely to be other content I’ve missed from John Wentworth.]

Vladimir Nesov (@Vladimir_Nesov) 

• eg this comment and this comment and also this comment

Miscellaneous connections

• Consequentialists: One-Way Pattern Trap: Principled Cartesian Boundaries Keep the Good Stuff Inside by David Udell (2023 Jan)
• The Learning-Theoretic Agenda: Status 2023: Agent Detection by Vanessa Kosoy (2023 Apr)
• Causal Incentives Working Group (site)
  • Agency from a causal perspective 

I’ve also created a “Boundaries [technical]” tag, and tagged all of «Boundaries»-related^[2] LW posts I could find. 

What I may have missed

There are surely many topics which I haven’t yet looked into which deserve to be linked in this post. I have noted those that I think are likely to be related below. 

• Embedded Agency
  • (a nice ELI5 hand-drawn explanation)
• Finite Factored Sets
• Natural Abstractions (John Wentworth)
• Free Energy Principle and Active Inference (see Critch 3a)
• Corrigibility (also see: Arbital page — particularly, “side effects”)
  • «Boundaries» may be a clearer way to think about the thing that “corribility” is pointing at? 
  • Also see Critch Part 3b:
    • > "The AI must not interfere too much with a human's thinking about whether to correct the system.  This amounts to the AI system respecting a boundary around the human's ability to independently make decisions about correcting it, and not violating that boundary by messing with the human's perceptions, actions, or viscera (thoughts) in a way that diminishes the human's autonomy as an agent to decide to make corrections."
• Markov blankets
  • update: see Formalizing «Boundaries» with Markov blankets + Criticism of this approach 

If you know of any other posts I should link in this post, let me know and I’ll add them. 

Closing notes

I’m personally extremely excited about this topic, and I will be covering further developments. 

I am also writing several more posts on the topic. Subscribe to my posts and/or the boundaries [technical] tag to get notified.

Please contact me with any «Boundaries»-related tips, ideas, or requests.

Post last edited: 2023-05-30.
 

1. ^{^}

   Here's why I use the word "membranes" as opposed to "boundaries": "Membranes" is better terminology than "boundaries" alone.

2. ^{^}

   («Boundaries»/boundaries [technical]-related posts, not necessarily “boundaries”-related posts.)