This work was supported by OAK, a monastic community in the Berkeley hills. It could not have been written without the daily love of living in this beautiful community. The work involved in writing this cannot be separated from the sitting, chanting, cooking, cleaning, crying, correcting, fundraising, listening, laughing, and teaching of the whole community.

This write-up benefited from feedback from David Kristofferson, Andrew Critch, Jason Crawford, Abram Demski, and Ben Pence. Mistakes and omissions are entirely the responsibility of the author.

---

How is it that we solve engineering problems? What is the nature of the design process that humans follow when building an air conditioner or computer program? How does this differ from the search processes present in machine learning and evolution?

We study search and design as distinct approaches to engineering. We argue that establishing trust in an artifact is tied to understanding how that artifact works, and that a central difference between search and design is the comprehensibility of the artifacts produced. We present a model of design as alternating phases of construction and factorization, resulting in artifacts composed of subsystems that are paired with helpful stories. We connect our ideas to the factored cognition thesis of Stuhlmüller and Christiano. We also review work in machine learning interpretability, including Chris Olah’s recent work on decomposing neural networks, Cynthia Rudin’s work on optimal simple models, and Mike Wu’s work on tree-regularized neural networks. We contrast these approaches with the joint production of artifacts and stories that we see in human design. Finally we ponder whether an AI safety research agenda could be formulated to automate design in a way that would make it competitive with search.

Introduction

Humans have been engineering artifacts for hundreds of thousands of years. Until recently, we seem to have mostly solved engineering problems using a method I will call design: understanding the materials at hand and building things up incrementally. This is the approach we use today when building bridges, web apps, sand castles, pencil sharpeners, air conditioners, and so on.

But a new and very different approach to engineering has recently come online. In this new approach, which I will call search, we specify an objective function, and then set up an optimization process to evaluate many possible artifacts, picking the one that is ranked highest by the objective function. This approach is not the automation of design; it’s internal workings are actually nothing like design, and the artifacts it produces are very unlike the artifacts produced by design.

The design approach to engineering produces artifacts that we can understand through decomposition. A car, for example, is decomposable into subsystems, each of which are further decomposable into parts, and so on down a long hierarchy. This decomposition is not at all simple, and low quality design processes can produce artifacts that are unnecessarily difficult to decompose, yet understanding how even a poorly designed car works is much easier than understanding how a biological tree works.

When we design an artifact, we seem to factorize it into comprehensible subsystems as we go. These subsystems are themselves artifacts resulting from a design process, and are constructed not just to be effective with respect to their intended purpose, but also to be comprehensible: that is, they are structured so as to permit a simple story to be told about them that helps us to understand how to work with them as building blocks in the construction of larger systems. I will call this an abstraction layer, and it consists of, so far as I can tell, an artifact together with a story about the artifact that is simultaneously simple and accurate.

Not all artifacts permit a story that is both simple and accurate. An unwieldy artifact may not be understandable except by considering the artifact in its entirety. In design, we do not produce artifacts and then come up with stories about them afterwards, but instead we seem to build artifacts and their stories simultaneously, in a way where the existence of a simple and accurate story to describe an artifact becomes a design goal in the construction of the artifact itself.

Search, on the other hand, does not operate on the basis of abstraction layers. When we use a machine learning system to search a hypothesis space for a neural network that correctly differentiates pictures of dogs from pictures of cats, the artifact we get back is not built up from abstraction layers, at least not nearly to the extent that artifacts produced by human design are. And we wouldn’t expect it to be, because the search processes used in machine learning have neither the intent nor the need to use explicit abstraction layers.

The absence of abstraction layers in artifacts produced by search is no impediment to the effectiveness of search in finding a solution to the specified problem. But this absence of abstraction layers is an impediment to our human ability to trust the artifacts produced by search. When I find some computer code on stackoverflow for solving some problem, I may copy it and use it within my own program, but not before understanding how it works. Similarly, when FAA approves a new aircraft for flight, it does so not just on the basis of empirical tests of the new aircraft, but also of a description provided by the manufacturer of each of the aircraft’s subsystems and how they work. For simple artifacts we may be willing to establish trust on the basis of empirical tests alone, but for sophisticated artifacts, such as an aircraft or autonomous car or artificial intelligence, we must be able to understand how it works in order to decide whether to trust it.

Design produces decomposable artifacts, for which trust can be built up by reading the stories attached to each abstraction layer. We can verify these stories by further decomposing the subsystems underneath each abstraction layer. Search does not produce decomposable artifacts and for that reason we have no way to establish trust in the artifacts produced by it, except by black-box testing, which is only appropriate for simple artifacts.

Unfortunately, we have discovered how to automate search but we have not yet discovered how to automate design. We are therefore able to scale up search by bringing to bear enormous amounts of computing power, and in this way we are solving problems using search that we are not able to solve using design. For example, it is not currently known how to use design to build a computer program that differentiates cat pictures from dog pictures, yet we do know how to solve this problem using search. We are therefore rapidly producing artifacts that are effective but not trustworthy, and we are finding ourselves tempted by economic incentives to deploy them without establishing trust in them.

The machine learning sub-field of explainability (and it’s related but distinct cousin, interpretability) is concerned with establishing trust in artifacts produced by search. There is important work in this area of relevance to the thesis presented in this paper, and we review some of it in a later section. Overall, work in this field seems to be concerned with one of the following:

• Models that are simple enough that they can be comprehended without any accompanying stories

• Manual decomposition of trained models

• Post-hoc explanations that aim to persuade but may not be accurate depictions of what’s really going on.

All three of these are distinct from our view that comprehensibility comes from the pairing of artifacts with helpful stories about them.

As we begin to construct sophisticated AI systems using search, we venture into dangerous territory. We have the tools of machine learning that may soon be capable of producing AI systems that are highly effective, yet for which we have no way to establish trust. It will be a great temptation for us to deploy them without establishing trust in them, since enormous economic prizes are on offer.

To resolve this, we here propose that we should work to automate design to such a point that design can scale with computing power to the same extent as search. Under this approach we would investigate the nature of the design process that humans use to construct artifacts based on abstraction layers, and attempt to automate it. We may find it possible to automate the whole process of design, or we may automate just part of the process, leaving humans involved in other parts.

The remainder of this document is organized as follows. First we work through a simple example to make clear the distinction between search and design. Then we describe a model of design as alternating construction and factorization. Following this we argue that search can be viewed as a construction process lacking any factorization step. We then work through definitions of the terms abstraction layer and comprehensible artifact upon which much of the material in this report is predicated. We draw a connection to the factored cognition thesis, and also to the machine learning fields of explainability and interpretability. We conclude by sketching an AI safety research agenda that would aim to allow comprehensible design to scale with computing power, in order for it to become competitive with search as a process for developing advanced artificial intelligence systems. In an appendix we present notes from an informal inquiry in which we observed the author’s own process of writing software over a few hours and compared what we saw to the model presented in this document.

Example: Sorting integers

Suppose I need an algorithm that sorts lists of integers from smallest to largest, but I'm not aware of any good sorting algorithms. Suppose it's critical that the algorithm correctly sort integers in all cases, but that I've never encountered the computer science field of sorting, so I don't have any of the concepts or proofs found in computer science textbooks. I consider two approaches:

• Train a neural network to sort integers

• Write a sorting algorithm from scratch in Python

Let's consider the machine learning approach. This problem has some attractive characteristics:

• I already understand what it means for a sequence of integers to be sorted, so I can provide a perfectly consistent training signal

• I can easily generate unlimited training examples, so I immediately have access to an infinitely large dataset

• I have a precise understanding of the range of possible inputs, so although I cannot train on every possible sequence of integers, I can at least be confident that I have not missed anything in my own understanding

Now let’s say that I succeed in training a neural network to sort integers, and I test it on many test cases and it works flawlessly. Am I ready to deploy this in a safety-critical scenario where a single incorrect output will lead to the death of many living beings?

Those familiar with practical machine learning will shudder at this point. A neural network is an unwieldy thing. It is composed of millions or perhaps billions of parameters. No matter how many test cases I run and verify, there are infinitely more that I didn’t try, and in fact there is always a maximum length list and a maximum size integer among my test set — can I really trust that the neural network will correctly sort integers when the length of the list o size of the integer greatly exceeds this maximum tested size? It is difficult to convey just how uncomfortable I would feel in making the leap from "I tested this on this many test cases" to "I’m ready to deploy this and swear on my life that it absolutely will sort integers correctly in all cases".

If I was given the task of determining whether a neural network correctly implemented sorting, what I would actually do is the following. I would examine the operations and coefficients contained in each layer of the network and attempt to extract an understanding of what each piece was doing. I would feed examples into the network and watch how each integer was processed. I would try to watch closely enough to get some insight into how the network was functioning. I would likely end up jotting down little fragments of pseudocode as I unpacked progressively larger subnetworks, then I'd try to understand the network as a whole on the basis of the pseudocode I'd written down. Ultimately, if I did succeed at extracting an understanding of the network, I’d need to decide whether or not the network constituted a correct sorting algorithm, and I might use some formal or informal verification method to do this.

The point is that to really trust that this network will work in all cases, I would want to decompose it and understand it piece by piece.

Now in this example we have taken a problem (sorting integers) that has a known direct solution, and we have compared that direct solution to learning a solution from training data. It is no surprise that the direct solution is the better choice. It is not that we should replace all instances of machine learning with direct solutions: the whole point of machine learning is that we can solve problems for which we do not have a direct solution. Instead, the point of this example is to provide an intuition for where our trust in artifacts comes from, and specifically why it is so important that artifacts are structured in a way that supports us in decomposing and understanding them.

Design as construction and factorization

In design, we build a thing up to meet some requirements, based on an understanding of the materials we have to work with. For example, when we build a shed, we have wood for framing, wood for sheathing, concrete to anchor it into the ground, and so on. We don’t need to know the details of how the two-by-four sections of wood were cut from their source material, or how they were transported, or how they were priced by the store that sold them: there is an abstraction layer upon which we can consider the two-by-fours as basic building blocks with a few known properties: that they are strong enough to support the structure of a shed; that they can be cut to arbitrary lengths; that they can be cut at an angle; that they cannot be made to bend; and so on.

Similarly, when I write some software that uses the Postgres database software to store and retrieve information, I do not need to understand the full internal workings of Postgres. I can consider the database system to be a kind of material that I am working with, and a good database system is designed such that I can understand how to use it without understanding everything about it. I know that when I execute such-and-such an SQL statement, a "row" gets added to a "table", and when I execute such-and-such an SQL statement, I get that same "row" back. But the concepts of "row" and "table" are high-level stories that we use because they are helpful, not because they give a complete description of which bits will be in which of the computer’s memory locations at which time.

One part of design consists of using existing materials to build something. We might call this "construction". If I write a python program containing a single "main" function that pulls in a bunch of libraries and does some computation, then I’m doing construction. But very quickly I will start to factor my program out into functions so that I can more easily test it, grow it, and understand it in a way that allows me to see that it is correct. We might call this "factorization". In factorization, I’m looking for ways that I can take, for example, a relatively complicated gradient descent algorithm and say "This chunk of code finds a local minima of a convex objective. You must give it an objective function and a starting point for optimization and it will give you back the optimum of the function." In this way I can stop remembering the details of the implementation of the algorithm, which are many and can be very subtle, and instead remember only the story about the implementation. This helpful story about an artifact, together with the artifact itself, is what I call an abstraction layer.

We are familiar with looking for simple explanations of existing things from our theory of epistemics. This is different. Here we are constructing something in a way that makes it possible to tell a simple story about it. We are designing the artifact together with an explanation/concept/story about it all within the same process. This makes it possible to do sophisticated engineering much more quickly than if we built unwieldy artifacts and then tried to come up post-hoc stories about them, because we design our artifacts with the goal of making them comprehensible. We have been working for centuries to tease apart the mechanisms making up, for example, biological trees (and we are not done yet), whereas we have successfully built search engines comprising billions of lines of code in just a few years.^[1]

Construction and factorization proceed in a loop. When I begin a new software project, I often work entirely within a single function for a while. Once I have a few basic pieces in place, I run them in order to test that they work and find out more about the materials I’m working with — for example, I might write code that performs a call to a remote API and then prints the server’s response in order to find out some things about it that are not covered in the documentation. Once this is all working I might factor this code out into smaller functions with succinct documentation strings such as "performs such-and-such an API call, does such-and-such upon error, returns data in such-and-such a format". Then I return to construction and start putting more pieces in place, perhaps using my existing factorized code as building blocks in the construction of larger pieces.

When there is too much construction and not enough factorization, we end up with unwieldy artifacts. We forget about the details of how the artifacts work and begin to mis-use them, introducing bugs. There is a kind of proliferation, like spaghetti code, or like a house that has had electrical cables strung endlessly from place to place without ever removing older cables. It becomes impossible to make sense of what’s going on. As we start counting the cables and trying to make sense of things, we get lost and forget about the cables we counted at the beginning. What is happening here is that we have a highly complex artifact without handholds, and it is very difficult to design things on top of it because in order to do so we need to somehow fit some story about the artifact into our minds, and human minds cannot work with arbitrarily unwieldy stories. Software engineers experience great pain and frustration in encountering such unwieldy artifacts, and their work tends to slow to a crawl as they give up on having any clear insight into what’s going on and instead proceed by dull trial-and-error^[2].

It is also possible for there to be too much factorization and not enough construction. Sometimes in software companies there will be an attempt to pre-factorize a system before anything at all has been constructed. The way this plays out is that we come up with some elaborate set of stories for abstraction layers before we really understand the problem, and begin implementing these abstraction layers. We forget that it is the process of construction that gives us evidence about the materials we are working with, and also about the solution we are seeking. We try to do all the construction in our minds, imagining what the artifact will ultimately look like, and pre-factorize it so that we don’t ever have to backtrack during design. But this over-factorization fails for fundamentally the same reason that over-construction fails: we cannot fit very much complexity into our minds, so our imagined picture of what construction will yield is inaccurate, and therefore our factorization is based on inaccurate stories.

So there is a design loop between construction and factorization in which construction gives us evidence about the nature of the materials and factorization rearranges our construction into a form that permits a compact description of what it is and how it works. In a healthy design loop, construction and factorization are balanced in a way that strictly bounds the amount of complexity that we need to fit into our minds to understand any subset of the system, while maximizing the inflow of evidence about the materials we are working with.

Search as construction without factorization

Search is pure construction, with no factorization. In search, we set up an engineering problem in a way that allows us to perform massive experimentation: trying millions of possible solutions until we find one that meets our requirements. The basic search loop consists of construction and evaluation. In machine learning, for example, evaluation corresponds to computing the objective function and its gradient, while construction corresponds to updating our artifact by moving it a little in the direction suggested by the gradient of the objective function.

This process — construction without factorization — leads to the production of unwieldy artifacts. The reason is that in the space of possible artifacts, the vast majority are very unwieldy, so any process that doesn’t explicitly optimize for comprehensibility leads by default to artifacts that are unwieldy. Imagine stepping through the set of all text files containing syntactically valid python programs. The vast majority of these contain unwieldy spaghetti code. We would have to step through many, many such text files before we found the first one that we might describe as comprehensible, or that would pass code review at a typical software company. Similarly, among the policies representable within the neural network architectures used in machine learning, there is presumably some subset that are internally well-factored in a way that would support human comprehension. But this subset is a tiny manifold within the overall hypothesis space, and even if our starting point for optimization were on this manifold, each gradient step is with high probability going to take us further away from that manifold unless we are explicitly working to stay on the manifold.

And why should a search process factorize its constructions? It has no need for factorization because it does not operate on the basis of abstraction layers. It operates on the basis of trial and error, and under trial-and-error it doesn’t matter whether an artifact is comprehensible or not. This is neither a feature nor a bug of search, it is just the way things are.

But although the absence of factorization is no barrier to search, it is certainly a barrier to our comprehension of the artifacts produced by search. Without handholds in the form of abstraction layers, we have no way to understand how an artifact works, and without understanding how it works it is very difficult to establish trust in it.

Defining comprehensible design

When we design some artifact, we want it to be both effective for its intended purpose, and comprehensible to ourselves and other humans. Being comprehensible really means that there is a story that can be told about the artifact that is both simple and accurate.

Definition. A helpful story is a story about an artifact that is both simple and accurate

What does simple mean? In this context when I say simple, I am referring to a concept that has a shape that is convenient for a human to understand. This may differ from an abstract notion of algorithmic simplicity such as description length, because humans seem to understand concepts through analogies to already existing concepts, so one may be able to quite easily understand certain complex concepts that map onto conceptual foundations already in place, such as a collection of interlinked database tables, while struggling to understand concepts without any pre-existing conceptual foundations, such as the notion of a ring in abstract algebra.

What does accurate mean? It means a story such that interacting with the artifact as though it were really as simple as described in the story does not cause harm or surprise. It means a story that is useful without being manipulative. It means a story that reveals, as far as possible, the direction of its own necessary imprecisions. This is different to pure predictive accuracy: what we care about is stories that make possible the use of the artifacts they refer to in the construction of larger systems, while trimming off details not necessary for this purpose. A story that is accurate only in the sense that it tells us how an artifact will behave may not give us any affordances for using that artifact in further construction.

Next we come to abstraction layers. We have already used the concept of an abstraction layer in discussing construction and factorization above. The definition I will use is:

Definition. An abstraction layer is an artifact together with a helpful story about it

It must be stressed once again that the terrain here is subtly different to that of epistemics, in which we observe some natural artifact and come up with a simple model to explain its behavior. In that domain we end up with a similar pairing between some artifact that is complex and a model or story about it that is simple. But in epistemics we are typically studying some already-existing phenomena, and engage in a process of hypothesizing about it. In design we get to construct the phenomena, and we can therefore shape it such that there exists a story about it that is both simple and accurate. We are allowed to ignore regions of the design space that contain effective artifacts, simply because those artifacts are difficult to understand and do not lend themselves to the construction of stories that are simultaneously simple and accurate. Finding helpful stories is a great challenge! Whenever we have a choice between finding a helpful story for an artifact produced by some black-box process, versus designing the artifact from the ground up to be amenable to a helpful story, we should certainly choose the latter!

Finally we come to a recursive definition for comprehensibility:

Definition. A comprehensible artifact is an abstraction layer that is built up from parts that are themselves comprehensible artifacts, using only a limited amount of construction to bridge the gap between the parts and the whole.

A car is a comprehensible artifact. Considered as a single artifact, a car comes with a set of very simple and very accurate stories about how to use it to drive from one place to another. Additionally, because my car was produced by an engineering process in which the design work needed to be distributed across many human engineers, it is internally structured in a way that supports decomposition. I may consider the car’s engine: it, too, is well-encapsulated, meaning it has a shape that permits helpful stories to be told about it, and the owner’s manual provides many such helpful stories about the car’s engine. Similarly, the other parts that make up the car — chassis, wheels, transmission, and so on — each come with helpful stories that make it possible to understand them without considering all of their details. And these parts are in turn decomposable: the engine is itself made from parts, which, due to the engineering process by which the car was designed, are again decomposable.

A tree is not a comprehensible artifact. We have been trying to map out how trees work for centuries, and we have made much progress, but we are not done. The human body is not a comprehensible artifact. Trees and human bodies both contain subsystems, presumably because natural selection is itself driven somewhat towards decomposability by the limited amount of information that can be stored in the genome, but they are not nearly so easy to understand. Of course this is no criticism of the many wonders produced by natural selection, it is just the way things are.

Consider a neural network trained to classify images as containing either dogs or cats. It is actually quite easy to turn this artifact into an abstraction layer: our helpful story is simply that you pass in an image in some appropriate format, and get back a label that tells you, with some probability, whether the image contains a dog or a cat. We can now use the neural network as a function without considering any of its internal details. But this neural network is certainly not a comprehensible artifact. We do not know how to decompose neural networks into crisp subsystems. There is some work in the field of inspectable machine learning that attempts to do this (see survey below), but this work is far from complete, and a full understanding of how image recognition works in neural networks remains elusive to us.

In the definition of comprehensible artifacts we said that only a limited amount of construction be used to bridge the gap between the parts and the whole. Each time we construct some artifact out of parts, there is some construction necessary to "glue" the parts together. In some cases we can factorize the glue itself, but there is always some glue remaining because the decomposition of parts into sub-parts has to bottom out somewhere with raw machinery that does work, or else our artifacts would be nothing more than empty abstraction layers composed endlessly. In order to use an artifact we generally do not need to understand this glue — that is the whole point of an abstraction layer — but in order to understand how an artifact works, we ultimately need to understand this glue. It is therefore critical to keep an upper bound on the amount of construction per abstraction layer, in order that we can recursively decompose our artifacts and verify their stories without ever needing to understand more than a fixed amount of glue.

Connection to factored cognition

Andreas Stuhlmüller and Paul Christiano have proposed the factored cognition hypothesis: that the thinking processes that constitute human intelligence can be broken down into thought episodes of perhaps just a few minutes in length, with limited communication between episodes. If true, this would open the door to scaling human intelligence by scaling these short thinking episodes, such as in their iterated distillation and amplification proposal.

At the surface level, these ideas are quite distinct from those presented here, for we have not taken any stance on the nature of the computational processes in a potential solution to the comprehensible design problem, and we certainly don’t assume that those processes could be factored in any particular way. We have discussed the factoring of artifacts in our model of comprehensible design but this is quite different from the factoring of cognition as Stuhlmüller and Christiano propose (just as the factoring of a car into assemblies of subsystems is distinct from the factoring of a mind that is designing a car).

Yet at a deeper level, it is often said that the structure of software products reflects the structure of the companies that produce them, and more broadly there is a way in which the internal organization of artifacts produced by minds reflects the internal organization of those minds. This may point to a deeper connection between what could be termed a "factored artifact hypothesis" advanced this document, and the factored cognition thesis of Stuhlmüller and Christiano.

Connection to explainability and interpretability in machine learning

The field of interpretability in machine learning is concerned with techniques for making machine learning models understandable to humans. Out of four literature reviews we read on the topic, all four mentioned trust as one of the central motivators for the field. This aligns closely with our concerns. The field has grown quickly over the past few years and we do not have any intention to cover all of it within this short subsection. We refer readers to the recent and much more comprehensive series of posts here on lesswrong by Robert Kirk and Tomáš Gavenčiak (1, 2, 3), as well as to the literature reviews published by Došilović et al^[3], Arrieta et al^[4], Adadi et al^[5], and Gilpin et al^[6]. We also found much value in Christopher Molnar’s textbook Interpretability in Machine Learning, which is freely available online.

The field is broadly concerned with tools that allow humans to develop trust in black box machine learning models by giving humans insight into the internal workings of the black-box models. As per Adadi et al^[7], approaches can be categorized along the following three axes:

• Local vs global. Local approaches aim to help humans to understand a single prediction made by a machine learning model, whereas global approaches aim to help humans understand the model itself. For example, a system that can explain a decision to assign a low credit rating to a particular individual is a local approach, whereas a visualization technique that shows that each layer of a convolutional neural network is building up successively larger models of visual parts is a global approach. In this report we are mostly concerned with global approaches since we want to develop sophisticated AI systems that can be trusted in general to take beneficial actions, rather than having a human review each individual action.

• Intrinsic vs post-hoc. Intrinsic approaches involve modifying the original learning algorithm in some way, whereas post-hoc approaches do work to improve interpretability after learning has already concluded. Imposing a sparsity prior to encourage the generation of simple models is an example of an intrinsic approach, whereas training a shallow neural network to approximate the predictions made by a deeper neural network after that deep network has already been trained is an example of a post-hoc approach. We have argued in this write-up that intrinsic approaches are more attractive for sophisticated AI systems, due to the opportunity to construct models in a way that permits interpretability. We would be excited to discover post-hoc approaches that work for sophisticated and general AI systems, but we expect that route to be much more challenging.

• Model-agnostic vs model-specific. Model-agnostic approaches can be used with any type of machine learning model (neural networks, decision trees, linear or logistic regressors, kernel machines, etc), whereas model-specific approaches are predicated on some particular type of model. Model-agnostic approaches tend to treat the underlying model as a black box and therefore tend to fall into the post-hoc dimension from the preceding axis. For advanced AI systems, we believe intrinsic approaches are more promising, so on this axis we expect energy to be correspondingly focused on model-specific approaches.

A classic interpretability method is that of Shapely explanations^[8], which assigns, for a model that makes a prediction y based on a set of features x1, … xn, a "contribution" to each feature y1, … yn such that the sum of the individual feature contributions is equal to the original prediction. For linear models, computing such contributions is straightforward: we just take the coefficients of the model as the per-feature contributions. But for nonlinear models it is not so clear how best to assign such contributions. The authors show that there is only one way to assign such contributions if one wishes to adhere to certain accuracy and consistency desiderata. This kind of method seems like a useful tool for checking simple predictive models, but is not going to provide deep insights into models that we suspect implement sophisticated algorithms internally.

In the remainder of this section we review three papers that we examined in detail. We chose these three papers based on citations from the review papers above, and based on recommendations from friends.

Rudin, Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead (2019)^[9]

Cynthia Rudin has argued persuasively that it is a mistake for us to construct black-box models and then construct further explanation-producing models that are optimized merely to persuade humans, rather than to be true accounts of how the original model actually works. We very much agree! Instead, she argues that we should train models that have a structure that allows us to understand them directly. She identifies the former (post-hoc explanation production) with "explainable machine learning" and the latter (intrinsically comprehensible models) with "interpretable machine learning". This differs terminologically from several review papers we have read, most of which take these terms to refer to approximately the same overall body of research, but we find it helpful none-the-less.

Rudin offers the provocative hypothesis that most black-box models can be replaced by equally or near-equally accurate interpretable models, and that there is in fact no trade-off between interpretability and accuracy. This is a bold and important claim, and she formalizes the case for it. The gist of her argument is that, due to the finite size of the dataset on which any model is trained, there will generally be many models within a small performance margin of the globally optimal model, and that among these one is likely to find an interpretable model. Put another way: no finite dataset contains enough information to pick out an infinitely precise region within model space, so the question becomes whether the set of close-to-optimal models is large enough to contain an interpretable model. A series of set-approximation theorems in A study in Rashomon curves and volumes^[10] argues that one is indeed likely to find an interpretable model within this set.

Rudin’s lab at Duke University is all about training intrinsically interpretable models. Of particular interest are rule lists, which are sequences of logical conditions of the following form:

IF       age between 18-20 and sex is male         THEN predict arrest (within 2 years)
ELSE IF  age between 21-23 and 2-3 prior offenses  THEN predict arrest
ELSE IF  more than three priors                    THEN predict arrest
ELSE     predict no arrest.

A second model category of interest is scoring systems, in which an integer score is computed by summing integer "points" associated with binary conditions, and the final model output is determined by a lookup table indexed by scores:

It was news to us that there are algorithms for the global optimization of such models! These algorithms differ substantially from the heuristic methods best-known in data science. In our experience it is not usually considered important to construct such logical models optimally, since one is usually working within some boosting or bagging framework in which many models of the same type are combined to produce a final output and any deficiency in one particular model is compensated for in the ensuing gradient steps. But this leads to a proliferation of models, which in turn lends to an uninterpretable overall model. Rudin’s approach is to instead focus on achieving optimality in the construction of a single small model, paying a price in terms of implementation difficulty, but reping rewards in terms of models that are simultaneously accurate and interpretable.

This highlights the real trade-off in machine learning, which is not between interpretability and accuracy but between interpretability and ease of implementation. On the one hand we can work with hypothesis classes such as neural networks that have optimization algorithms that are easy^[11] to understand and implement but produce models that are difficult to interpret. On the other hand we can work with hypothesis classes such as sparse logical models, which are difficult to optimize but easy to interpret. Yet this difficulty of optimization is a one-time cost to be paid in algorithms research and software engineering. Once effective algorithms have been discovered and implemented we can use them as many times as we want. Furthermore, it may take less computing power to find an optimal logical model than to train a neural network, and it will almost certainly require less expertise to use such algorithms since local optimizers are sensitive to all kinds of initialization and stepping details, whereas global optimizers either find the true global optimum or fail to do so in a reasonable amount of time.

It is exciting to consider what it would look like to use optimal simple models in computer vision or reinforcement learning. An enticing array of difficult optimization problems beckon, with the prize for their solution being the ability to construct simultaneously interpretable and accurate models.

Rudin speculates on such advances herself, in particular with prototype networks in computer vision. These are neural networks in which early layers are ordinary convolution feature-extraction layers, and then later layers reason by explicitly finding correspondences between regions in the input image and similar regions in training images. The final prediction is then made on the basis that if there are many correspondences between an input image and training images labelled "bird", then the input image itself probably contains a bird. This makes the model somewhat interpretable because one can visually inspect the correspondences and understand how the prediction was made on that basis. However, the early convolutional layers of the network remain opaque.

Rudin’s work is exciting and insightful, but we believe she misses the following two points.

First, Rudin’s work focuses on building models that are interpretable by being so simple that we can understand how they work just by looking at them. This is like being handed a tool such as a screwdriver or hammer that is so simple that we understand it immediately without needing to refer to any instruction manual. But other tools — say, a 3D printer — may be difficult to understand just by looking at them, yet easy to understand with the help of an instruction manual. We should be willing to produce complex models if they are shaped in such a way that a simple and accurate instruction manual can be written about them, and if we have methods for producing such instruction manuals. Post-hoc explanations are not good enough here; what we are suggesting is to build models and instruction manuals together in such a way that (1) the instruction manual accurately describes how the model really works, and (2) the instruction manual makes it easy for a human to understand the model. Achieving both of these aims simultaneously will require the model itself to be constrained in its complexity since not all models (presumably) permit any such instruction manual to be written about them, but it should impose less of a constraint than requiring our models to be interpretable on-sight without any instruction manual.

Second, Rudin’s work focuses on simplicity as a proxy for interpretability. But algorithmic simplicity is only a weak proxy for how readily a model can be understood by a human. There are concepts that are quite complex when measured by any abstract measure of complexity (such as description length) that humans will nevertheless reason about quite intuitively, such as complex social situations. On the other hand there are concepts from, for example, abstract algebra, that are simple according to abstract measures of complexity, yet require significant training to understand. For this reason we tend to explain sophisticated concepts by analogy to the concepts most intuitive to us. A better measure of a model’s interpretability to a particular person would be the length of the shortest description of that model in terms of concepts this person has already acquired, where the already-acquired concepts are treated as primitives and do not contribute to the length of a description.

Olah et al., Zoom In: An Introduction to Circuits (2020)^[12]

One thread of research within interpretability of great interest and relevance to the present work is Chris Olah’s investigation of circuits in convolutional neural networks trained to perform image classification. This thread of work initially gathered attention with Olah’s 2017 article on visualizing individual neurons as well as whole features (one channel within a layer) by optimizing images and image patches to maximally activate these neurons and features^[13]. This produced beautiful dream-like images that gave some insight into what the network was "looking" for within different layers and channels.

Olah now proposes to go beyond mere visualization and initiate what he foresees as a "natural science of interpretability" — studying the structure of trained neural networks in the way we might study the inner workings of plants or animals in biology. In Olah’s words:

Most work on interpretability aims to give simple explanations of an entire neural network’s behavior. But what if we instead take an approach inspired by neuroscience or cellular biology — an approach of zooming in? What if we treated individual neurons, even individual weights, as being worthy of serious investigation? What if we were willing to spend thousands of hours tracing through every neuron and its connections? What kind of picture of neural networks would emerge?

In contrast to the typical picture of neural networks as a black box, we’ve been surprised how approachable the network is on this scale. Not only do neurons seem understandable (even ones that initially seemed inscrutable), but the "circuits" of connections between them seem to be meaningful algorithms corresponding to facts about the world. You can watch a circle detector be assembled from curves. You can see a dog head be assembled from eyes, snout, fur and tongue. You can observe how a car is composed from wheels and windows. You can even find circuits implementing simple logic: cases where the network implements AND, OR or XOR over high-level visual features.

While we find this work to decompose trained neural networks very exciting, it is worth noting just how immense this research program is. After years of work it has become possible to begin to identify neural network substructures that implement elementary logical operations. Decomposing entire networks is still a long way off. And even at that point we are still in the realm of feed-forward networks with no memory, trained to solve conceptually straightforward supervised learning tasks in which the input is a single image and the output is a single label. If we build sophisticated artificial intelligence systems by training neural networks, we can expect that they will contain logic many levels deeper than this.

We very much hope this works proceeds quickly and successfully, but it's apparent difficulty does lend credence to the thesis advanced in this document that post-hoc interpretation of artifacts not optimized for comprehensibility is difficult, and that we should find ways to design artifacts on the basis of abstraction layers, not because they would perform better, but because we would be able to understand and therefore trust (or distrust) them.

Wu et al., Beyond Sparsity: Tree Regularization of Deep Models for Interpretability^[14]

In this paper, Wu et al. train neural networks on medical diagnosis tasks, in such a way that the neural networks are well-approximated by decision trees. The idea is that humans can interpret the decision trees, thereby gaining some insight into what the neural network is doing. The contribution of the paper is a regularizer that can be used to train neural networks that have this property.

At a high level this idea was exciting to us. We hoped for a demonstration that neural networks can be trained with regularizers that cause them to take on a form that is well-approximated by a "simple story", if one is willing to take the decision tree as a story.

But on closer inspection the paper is disappointing. The regularization does not cause the network to have an internal structure that approximates a decision tree, it merely causes the outputs of the network to be well-approximated by a decision tree. The decision tree therefore gives no real insight into how the network works, which is the kind of understanding we should be demanding before trusting a statistical model. Furthermore, the authors report that on a binary prediction task the neural network "has predictions that agree with its corresponding decision tree proxy in about 85-90%!o(MISSING)f test examples". That means that in 10-15%!o(MISSING)f training examples, and presumably at least 10-15%!o(MISSING)f the time in the real world, the neural network produces the opposite answer from the decision tree. One might wish to understand why these cases differ in this way, but these are precisely the cases where the decision tree offers no help.

Conclusion: AI safety via automated comprehensible design?

As we consider how to navigate towards a safe and beneficial future of artificial intelligence, we face the following dilemma. On one hand we have search which is moving forward quickly but produces untrustworthy artifacts. On the other hand we have design, which under good conditions can produce trustworthy artifacts, but is moving forward very slowly in the domain of artificial intelligence. Search is automated and is therefore accelerating as more and more compute power becomes available to it; design remains stuck at the fixed pace of human cognition.

In the AI safety community there are two basic views on how to resolve this dilemma. On one hand there are those who seek to rescue search: to find ways to harness the power of modern machine learning in a way that produces trustworthy artifacts. Paul Christiano’s work on iterated distillation and amplification seeks to resolve the dilemma by using imitation learning to gradually amplify human capabilities. CHAI’s work on assistance games seeks to resolve the dilemma by relaxing the requirement that an objective be specified before optimization begins. Geoffrey Irving’s work on debate seeks to resolve the dilemma by having powerful AI systems scrutinize one another’s claims in front of a human judge. We applaud these efforts and wish them success.

On the other hand there are those who say that our best bet is to throw as much human thought as possible at the design approach; that despite the slow progress of manual design to-date it is critical that we acquire a foundational theory of intelligent agency and that we should therefore pursue such a theory with whatever resources we have. This is the perspective of researchers at the Machine Intelligence Research Institute (as we understand it). We applaud these efforts, too, and very much hope for research breakthroughs on this front.

But there is a third option: we could automate design, making it competitive with search in terms of its effectiveness at producing powerful artificial intelligence systems, yet retaining its ability to produce comprehensible artifacts in which we can establish trust based on theories and abstraction layers.

We do not currently know how to automate design. We do not really know what design is, although we hope the ideas presented in this essay are helpful. This is therefore a call for research into the nature of comprehensible design and its possible automation.

To automate design, our fundamental task is to build computer systems capable of producing artifacts together with helpful stories about them. To do this, we will need to understand what makes a story helpful to humans (we have proposed simplicity and accuracy, but this surely just scratches the surface), and we will need to discover how computer systems can produce such stories.

We will need to discover how to deeply integrate story production with artifact production. If we try for post-hoc story production — constructing the artifact first and then fitting a story to it after-the-fact — then we will be solving a much more difficult problem than we need to. We should shape our artifacts so as to make story-writing as easy as possible.

Between construction and factorization, it is factorization that seems at present most mysterious. How might we automate the "carving at the joints" of a messy bit of construction? How might we do this such that an elegant abstraction layer is produced?

To do this, we could start with existing search algorithms (i.e. machine learning) and modify them so that they produce stories together with artifacts. Perhaps such an approach would fit within the existing field of interpretability in machine learning.

Or we could start with existing design processes, carefully examining humans engaged in engineering and attempt to automate some or all of their labor.

Or we could start somewhere else entirely, perhaps with some stroke of genius that points the way towards a different and more trustworthy approach. Let us hope that such a stroke of genius is forthcoming soon.

Appendix: An informal practical investigation

I undertook a small investigation into the nature of engineering by working on a personal software project myself, while noting how I was navigating and problem-solving. I set a timer at 10 minute intervals, at which point I would stop and write down what I was working on, how I knew to work on that, and how I was going about solving the problem.

The project I worked on was a small script in the Go language to manage the creation of Google Cloud Platform (henceforth "GCP") projects, as well as to enable and disable APIs. In this section I will refer to the specifics of the various technologies I was using to solve this problem because I find it helpful to be very concrete when performing investigations such as this. However, if you are unfamiliar with these particular technologies then please know that their specifics are not really central to this section.

The investigation took place over the course of about 5 hours total over 3 days. I made no attempt to formally test any particular hypothesis, although I was interested in whether my experience matched the framework presented in this essay.

The script I wrote was intended to allow one to configure a GCP project by writing a single configuration file specifying the project’s name, ID, billing account, and a list of APIs to be enabled on the project. The script would then make the necessary API calls to GCP to create or update the project as necessary. This was something that I’ve wanted to build for a while because every time I set up a new GCP project I find that I’ve forgotten these finicky project creation steps and have to once again read through the documentation and discover how to do it again. I therefore was excited to build a small tool to automate this.

I began by defining the structure of the configuration file and parsing it in YAML format, then I wrote API calls to create the project if it didn’t already exist, then I wrote API calls to enable and disable APIs based on the configuration file, then I wrote API calls to link a billing account to the project, then finally I wrote a helper command to invoke the standard "gcloud" tool with the relevant project automatically selected.

My findings were as follows.

I was astounded by the wealth and depth of the concepts needed to build this simple tool. In the first few minutes of working I wrote some code to parse some simple command-line arguments to the script. Already here I was operating on the basis of powerful stories about how a computer program is invoked from the command line, what the command line is, and how one typically passes in options on the command line. I was using a particular command-line processing library that is based on defining a struct in Go and tagging the various fields with information about how they are to be mapped to strings passed on the command line. I was already familiar with this library so I could work through this part very quickly. I could just "see" the obvious "right way" to solve the sub-problem of passing in command line arguments.

It did not feel as though my cognition consisted of any kind of "search" over a hypothesis space (although of course I didn’t have full access to all the things happening beneath the subconscious level). It felt more like I was rolling out a recipe that I was already familiar with.

Later, as I was constructing the API calls to create projects and enable and disable APIs, there were an even larger number of concepts in play: concepts about what a project is, what a REST API is, how errors are typically returned from REST APIs, how the GCP client libraries typically wrap these APIs, what a context is in Go, to say nothing of the concepts involved in formulating the Go code itself — which involves understanding functions, variables, structures, if statements, for statements, and so on. And these are really only the low-level concepts that one is concerned with during implementation. There are also the high level concepts of declarative infrastructure and software tooling in general that was very much guiding my approach to solving the problem.

When we write such software, we are building on top of a huge mountain of sophisticated materials (languages, remote APIs, cloud services, and so on). The only way we can make sense of these enormously complex materials is via abstraction layers and the concepts and stories they are predicated on. Navigating this landscape without concepts would be utterly impossible.

At one point I encountered an API call that was returning a 403 Forbidden error code. I initially believed that this was due to an authentication problem, since there is a general concept in REST APIs about which error codes should be used to indicate which kinds of problems, and I expected that the GCP APIs would follow this standard concept. I spent some time trying to debug this by changing the way I was doing authentication, but then I later realized that this error code is returned when one attempts to get information about a project that doesn’t exist yet. This is an interesting example of what happens when one uses a concept that does not match the reality of the materials being described. It’s also interesting that I was the one that created this pairing between material (the API call itself) and concept (the standard conventions for HTTP error codes) in my head. It was not that I read the GCP documentation and it turned out to be incorrect. In fact I did not read the GCP documentation on this particular matter, I simply assumed that the API would conform to standard conventions. I therefore created a kind of abstraction layer of my own over the top of the lower-level abstraction layer provided by GCP. This abstraction layer that I created did not consist of me writing any code but just of me taking a concept I was familiar with (conventions for HTTP error codes) and pairing it with the artifact of these particular GCP APIs.

The way I both discovered and resolved this mismatch between concept and artifact was through experimentation. After I had written the code to call this API, I put in place an "end cap" that allowed me to run my script even though it was far from finished. I did this by writing code to print the data returned from the API call and then exiting. In this way I could look directly at the materials I was working with by inspecting the actual data returned from the server, and see any ways in which the raw materials differed from the concepts I was using to formulate my expectations. I discovered the problem this way, and then later resolved the problem this way also, by trying various IDs of projects that did and did not exist, and noticing that the 403 Forbidden error code was returned for non-existent projects. This is a good example of the way that construction yields "evidence" about the nature of the materials we are working with. While engaging in design it is critical to establish a channel for regularly receiving this evidence.

Most of my work on this project consisted of construction. The APIs I was working with came with client libraries that made the code to perform the API call fairly succinct, and I found no reason to add further abstraction layers. Furthermore I perceived some risk of running into various show-stoppers that would force me to abandon the project entirely (such as there not being any published API to do the things I needed to do in order to have the script operate the way I wanted it to), so I was eager to see the whole thing work end-to-end before engaging in a lot of factorization.

There were places where I did engage in factorization though. One was in writing a poll loop to check on the long-running operation of creating a project. This involved a loop that would repeatedly call the API that checks on the status of a long-running operation on GCP, and return either success or an error message in the case of failure. This piece needed logic so that it would give up after a timeout expired to prevent the possibility of the script executing in a loop indefinitely. The code to perform this poll loop was sufficiently complicated, and there was a sufficiently elegant abstraction layer that could be wrapped around it, that I quickly factored this code out into its own function.

Overall, I was struck by how little this process resembled any kind of optimization process, at any level. Perhaps there was a sophisticated and general optimization process happening at a subconscious level in my brain, but it didn’t appear that way. It appeared that I was rolling out a set of known recipes; that I basically knew what to do at most points and my time was occupied with looking up documentation and trying API calls to discover the specifics of how to do each piece.

Perhaps this was because I was working on a from-scratch project starting with a blank slate, for which I already had a clear goal. Perhaps it was the planning stage, which happened prior to the time during which I was recording my activities at 10 minute intervals, where optimization took place. I did not actually do any formal planning but I had encountered the need for this script several times over a time period of year or so, and each time this happened I further crystalized a rough plan for the script I was implementing here.

Or perhaps it was because the project involved lots of API calls with very little algorithmic complexity. It would be interesting to repeat the experiment while solving some algorithmic programming puzzles.

I was also struck by my ability to work towards a goal that was only vaguely specified. Although I did, as mentioned above, have a sense of what the finished product should look like, this sense was very rough. During construction phases I regularly felt refinements to this rough overall picture snap into place, as I grappled directly with the materials at hand. For example, I decided on the exact structure of the configuration file while writing the code to parse the configuration file, and I made decisions about what would happen if no billing account was specified while writing the API calls to link billing accounts. In this way it felt like my overall plan was itself a kind of high-level concept, and I incrementally refined it to lower-level conceptual clarity as I built up each piece of the puzzle.

In this way, the construction phases brought in evidence not just about the nature of the problem (the nature of the materials I was working with, etc), but also about the nature of the goal. It seemed that I was collecting evidence about the objective as I proceeded. This is very different from a pure search approach in which the objective is specified upfront.

---

1. I do not mean to make any claim about the relative complexity of trees versus modern operating systems. I suspect trees are profoundly more sophisticated. ↩︎

2. We might say that when software engineering becomes search rather than design, it becomes dull and painful. ↩︎

3. Došilović, F.K., Brčić, M. and Hlupić, N., 2018, May. Explainable artificial intelligence: A survey. In 2018 41st International convention on information and communication technology, electronics and microelectronics (MIPRO) (pp. 0210-0215). IEEE. ↩︎

4. Arrieta, A.B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R. and Chatila, R., 2020. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion, 58, pp.82-115. ↩︎

5. Adadi, A. and Berrada, M., 2018. Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI). IEEE Access, 6, pp.52138-52160. ↩︎

6. Gilpin, L.H., Bau, D., Yuan, B.Z., Bajwa, A., Specter, M. and Kagal, L., 2018, October. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA) (pp. 80-89). IEEE. ↩︎

7. Adadi, A. and Berrada, M., 2018. Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI). IEEE Access, 6, pp.52138-52160. ↩︎

8. Lundberg, S.M. and Lee, S.I., 2017. A unified approach to interpreting model predictions. In Advances in neural information processing systems (pp. 4765-4774). ↩︎

9. Rudin, C., 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5), pp.206-215. ↩︎

10. Semenova, L. and Rudin, C., 2019. A study in Rashomon curves and volumes: A new perspective on generalization and model simplicity in machine learning. arXiv preprint arXiv:1908.01755. ↩︎

11. In the sense that it is conceptually easy to understand and implement gradient descent, not that it is computationally easy, or easy to find a global optimum ↩︎

12. Olah, et al., "Zoom In: An Introduction to Circuits", Distill, 2020. ↩︎

13. Olah, et al., "Feature Visualization", Distill, 2017. ↩︎

14. Wu, M., Hughes, M.C., Parbhoo, S., Zazzi, M., Roth, V. and Doshi-Velez, F., 2018, April. Beyond sparsity: Tree regularization of deep models for interpretability. In Thirty-Second AAAI Conference on Artificial Intelligence. ↩︎