(warning: this comment is significantly longer than the post)

I want to being by saying that I appreciate the existence of this post. Truly and honestly. I think it's important to praise those who at least try explaining difficult topics or hard-to-communicate research intuitions, even when those explanations are imperfect or don't fulfill their intended purpose. For it's only by incentivizing genuine attempts that we have a chance of obtaining better ones or dispelling our confusions. And, at the very least, this post represents a good vehicle for me to express my current disagreement with/disapproval of agent foundations research.^[1]

Nevertheless, in the interest of honesty, I will say this post leaves me deeply unsatisfied. Kind of like... virtually every post ever made on LW that tries to explain agent foundations? At this point I don't think there's anything about the authors^[2] that causes this, but rather the topic itself which doesn't lend itself nicely to this type of communication (but see below for an alternate perspective^[3]).

Let's start with the "Empirics" section. Alex Altair writes:

From where I'm standing, it's hard to even think of how experiments would be relevant to what I'm doing. It feels like someone asking me why I haven't soldered up a prototype. That's just... not the kind of thing agent foundations is. I can imagine experiments that might sound like they're related to agent foundations, but they would just be checking a box on a bureaucratic form, and not actually generated by me trying to solve the problem.

It's... hard to see how experiments would be relevant to what you're doing? Really? The point of experiments is to ensure that the mathematical frameworks you are describing actually map onto something meaningful in reality as opposed to being a nice, quaint, self-consistent set of mathematical symbols and ideas that nonetheless reside in their own separate magisterium without predicting anything important about how real life shapes out. As I have said before:

There's a famous Eliezer quote about how for every correct answer to a precisely-stated problem, there are a million times more wrong answers one could have given instead. I would build on that to say that for every powerfully predictive, but lossy and reductive mathematical model of a complex real-world system, there are a million times more similar-looking mathematical models that fail to capture the essence of the problem and ultimately don't generalize well at all. And it's only by grounding yourself to reality and hugging the query tight by engaging with real-world empirics that you can figure out if the approach you've chosen is in the former category as opposed to the latter.

Conor Leahy has written:

Humans are really, really bad at doing long chains of abstract reasoning without regular contact with reality, so in practice imo good philosophy has to have feedback loops with reality, otherwise you will get confused.

I have written:

Idk man, some days I'm half-tempted to believe that all non-prosaic alignment work is a bunch of "streetlighting." Yeah, it doesn't result in the kind of flashy papers full of concrete examples about current models that typically get associated with the term-in-scare-quotes. But it sure seems to cover itself in a veneer of respectability by giving a (to me) entirely unjustified appearance of rigor and mathematical precision and robustness to claims about what will happen in the real world based on nothing more than a bunch of vibing about toy models that assume away the burdensome real-world details serving as evidence whether the approaches are even on the right track. A bunch of models that seem both woefully underpowered for the Wicked Problems they must solve and also destined to underfit their target, for they (currently) all exist and supposedly apply independently of the particular architecture, algorithms, training data, scaffolding etc., that will result in the first patch of really powerful AIs.

I fear there is a general failure mode here that people who are not professional mathematicians tend to fall into when they think about this stuff. People who have read Wigner's Unreasonable Effectiveness of Mathematics and Hamming's follow-up to it and Eliezer's Sequences vibing about the fundamentally mathematical nature of the universe, and whose main takeaway from them is that elegance and compact-descriptiveness in pure mathematics is some sort of strong predictor of real-world applicability. They see all these examples of concepts that were designed purely to satisfy the aesthetic curiosities of pure mathematicians, but afterwards became robustly applicable in concrete, empirical domains.

But there is a huge selection effect here. You only ever hear about the cool math stuff that becomes useful later on, because that's so interesting; you don't hear about stuff that's left in the dustbin of history. It's difficult for me to even put into words a precise explanation meant for non-mathematicians^[4] of how this plays out. But suffice it to say the absolute vast majority of pure mathematics is not going to have practical applicability, ever. The vast majority of mathematical structures selected because they are compact and elegant in their description, or even because they arise "naturally"^[5] in other domains mathematicians care about, are cute structures worth studying if you're a pure mathematician, but almost surely irrelevant for practical purposes.

Yes, the world is stunningly well-approximated by a relatively compact and elegant set of mathematical rules.^[6] But there are infinitely more short and "nice" sets of rules which don't approximate it.^[7] The fact that there is a solution out there doesn't mean other posited solutions which superficially resemble it are also correct, or even close to correct. There is no "theory of the second-best" here. And those rules were found through experiments and observations and rigorous analysis of data, not merely through pre-empirical daydreaming and Aristotelian "science."

Eliezer loves talking about the story of how Einstein knew his theory was correct based on its elegance, and how when he was asked by journalists what he'd do if Eddington falsified his theory, he would say “Then I would feel sorry for the good Lord. The theory is correct.” But that's one story! One. And it's cool and memorable and fun and you give it undue weight for how Deeply Awesome it feels. But it still generates the same selection effect I mentioned before. If you peer through the history of science, even prior to Einstein during the time of Galileo and Kepler, and especially after Einstein and further developments we've had in the field of physics, you'll see the story is not representative of the vast, vast majority of how science is done. That empirics reigns, and approaches that ignore it and try to nonetheless accomplish great and difficult science without binding themselves tight to feedback loops almost universally fail.

But also pay attention to the reference class it's in! Physics, I said above. Why physics? How do we know that's at all representative of the type of science we're interested in? If this didn't have an effect, meaning the histories of different fields of inquiry were fundamentally similar, then it wouldn't matter much which specific subfield we focus on. And yet it does! 

Shankar Sivarajan has written:

The opening sounds a lot like saying "aerodynamics used to be a science until people started building planes."

The idea that an area of study is less scientific because the subject is inelegant is a blinkered view of what science is. A physicist's view. It is one I'm deeply sympathetic to, and if your definition of science is Rutherford's, you might be right, but a reasonable one that includes chemistry would have to include AI as well.

Richard Ngo has written:

Villiam: I have an intuition that the "realism about rationality" approach will lead to success, even if it will have to be dramatically revised on the way.

To explain, imagine that centuries years ago there are two groups trying to find out how the planets move. Group A says: "Obviously, planets must move according to some simple mathematical rule. The simplest mathematical shape is a circle, therefore planets move in circles. All we have to do is find out the exact diameter of each circle." Group B says: "No, you guys underestimate the complexity of the real world. The planets, just like everything in nature, can only be approximated by a rule, but there are always exceptions and unpredictability. You will never find a simple mathematical model to describe the movement of the planets."

The people who finally find out how the planets move will be spiritual descendants of the group A. Even if on the way they will have to add epicycles, and then discard the idea of circles, which seems like total failure of the original group. The problem with the group B is that it has no energy to move forward.

The right moment to discard a simple model is when you have enough data to build a more complex model.

Richard Ngo: In this particular example, it's true that group A was more correct. This is because planetary physics can be formalised relatively easily, and also because it's a field where you can only observe and not experiment. But imagine the same conversation between sociologists who are trying to find out what makes people happy, or between venture capitalists trying to find out what makes startups succeed. In those cases, Group B can move forward using the sort of "energy" that biologists and inventors and entrepreneurs have, driven an experimental and empirical mindset. Whereas Group A might spend a long time writing increasingly elegant equations which rely on unjustified simplifications.

Instinctively reasoning about intelligence using analogies from physics instead of the other domains I mentioned above is a very good example of rationality realism.

jamii has written:

Uncontrolled argues along similar lines - that the physics/chemistry model of science, where we get to generalize a compact universal theory from a number of small experiments, is simply not applicable to biology/psychology/sociology/economics and that policy-makers should instead rely more on widespread, continuous experiments in real environments to generate many localized partial theories.

Anyway, enough on that.^[8] Let's move on to "What makes agent foundations different?" In the very first paragraph, Alex Altair writes:

One thing that makes agent foundations different from science is that we're trying to understand a phenomenon that hasn't occurred yet (but which we have extremely good reasons for believing will occur). I can't do experiments on powerful agents, because they don't exist.

The first sentence is false. Science routinely tries to understand phenomena it has good reason to believe exist, but it hasn't been able to pinpoint exactly and concretely yet. The paradigmatic and illustrative example of this is the search for room-temperature superconductors. This is primarily done by scientists, not by engineers.

But more to the point, the second sentence also reads as substantively dubious. You don't have all-powerful ASI to experiment on, but here's what you do have:

• the sole example of somewhat-aligned generally-intelligent occasionally-agentic beings ever created, namely humans
• somewhat agentic, somewhat intelligent, easy-to-query-and-interact-with AI models that you can (very cheaply!) run recurrent experiments on to test your theories

Does your theory of agency have nothing to say about either of them? Then why on Earth would you assume any partial results you obtain are anywhere close to reliable? Are you assuming a binary dichotomy between something that's "smart and agentic, so our theories apply" on one end, and "dumb and unagentic, so our theories don't apply" on the other end?^[9] 

If that's so, and even humans fall into the latter, then I also don't see why your theories would have any applicability in the most safety-critical regime, i.e., when the first powerful models are created. Nate Soares has written:

By default, the first minds humanity makes will be a terrible spaghetti-code mess, with no clearly-factored-out "goal" that the surrounding cognition pursues in a unified way. The mind will be more like a pile of complex, messily interconnected kludges, whose ultimate behavior is sensitive to the particulars of how it reflects and irons out the tensions within itself over time.

If your theory of Agent Foundations has nothing to say about current AI, and nothing to say about current generally-intelligent humans,^[10] does it have anything to say about the actual AGI we might create?

Alex Altair also writes:

So, I don't think that what we're lacking is data or information about the nature of agents -- we're lacking understanding of the information we already have.

Well, actually, you are lacking some data or information about something you care about when it comes to agency. Namely, to what extent are the models we're interested in aligning actually well-modeled as agents. When I look at other humans around me (and at myself), I see beings that are well-approximated as agents in some ways, and poorly-approximated as agents in other ways. Figuring out which aspect of cognition falls on which side of this divide seems like an absolutely critical (maybe even the absolutely critical) question I'd want agent foundations to give me a reliable answer to. Are you not even trying to do that?

Let me give a concrete example. A while ago, spurred on by observing many instances of terrible arguments in favor of treating relevant-agents-as-utility-maximizers, I wrote a question post on "What do coherence arguments actually prove about agentic behavior?" Do you think you have an complete answer to this question? And do you think you don't even need any new data or information to answer it?

And finally, Alex Altair talks about how "It's kinda like computer science." And he writes:

One needs to have some kind of life experiences that points your mind toward the relevant concepts, like "computing machines" at all. But once someone has those, they don't necessarily need more information from experiments to figure out a bunch of computability theory.

One does need information from experiments to know that computability theory is at all useful/important in the real world. And also to know when it matters, and when it doesn't or is an incomplete description of what's going on.^[11] Which is what I hope agent foundations is about. Something useful for AI safety. Something useful in practice. If it's a cute branch of essentially-math that doesn't necessarily concern itself with saving the world from AI doom, why should anyone give you any money or status in the AI safety community?

In any case, the analogy to computations also feels like the result of a kind of selection effect as well. Conor Leahy has written:

It is not clear to me that there even is an actual problem to solve here. Similar to e.g. consciousness, it's not clear to me that people who use the word "metaphilosophy" are actually pointing to anything coherent in the territory at all, or even if they are, that it is a unique thing. It seems plausible that there is no such thing as "correct" metaphilosophy, and humans are just making up random stuff based on our priors and environment and that's it and there is no "right way" to do philosophy, similar to how there are no "right preferences". I know the other view ofc and still worth engaging with in case there is something deep and universal to be found (the same way we found that there is actually deep equivalency and "correct" ways to think about e.g. computation).

Yes, we have found a deep mathematical way of thinking about computation. Something simple, compact, elegant. But saying agent foundations is like the study of computation... kind of hides the fact that the theory of computation might be sort of a one-off in terms of how nice it is to formalize properly? As I see it, and as Leahy writes above, we have examples of things that make sense intuitively and we could formalize nicely (computation). And we also have examples of things that make sense intuitively and we couldn't formalize nicely (everything else he talks about in that comment). Saying agent foundations is like computation is putting the cart before the horse, it's assuming it falls into the category of nicely-formalizable things, which feels to me like it isn't a representative subset of the set of things-we-try-to-formalize.

1. ^{^}

   Disclaimer: as an outsider who is not working on AI safety in any way, shape, or form

2. ^{^}

   There are many of them, they use different writing styles and bring attention to different kinds of evidence or reasoning, etc.

3. ^{^}

   Spoiler alert: it's hard to communicate why agent foundations makes sense... because agent foundations doesn't make sense

4. ^{^}

   By which I mean, people literally not in math academia

5. ^{^}

   In some hard-to-define aesthetic sense

6. ^{^}

   But that's not even fully correct, frankly. Not to get into the nerdy weeds of this too much, but modern QFT, for instance, requires an inelegant and mathematically dubious cancellation of infinities to allow 

7. ^{^}

   And yet scientists thought they did, for a long time!

8. ^{^}

   For now

9. ^{^}

   It's difficult to believe you'd actually hold this view, since frankly it's really dumb, but I also would have had a difficult time believing you'd say you don't have any experiments to run... and yet you're saying it regardless!

10. ^{^}

    Since, again, you're not running any experiments to check your theories against them

11. ^{^}

    As an illustrative example, proving an algorithm can be computed in polynomial time is cool, but maybe the constants involved are so large you can't actually make it work in practice. If all complexity theory did was the former, without also telling me what the domain of applicability of its results is when it comes to what I actually care about, then I'd care about complexity theory a lot less

When Alan Turing figured out computability theory, he was not doing pure math for math's sake; he was trying to grok the nature of computation so that we could actually build better computers.

Are you sure this is true? I am not a Turing expert but my impression is that his practical work on building computing machinery was downstream of his theoretical work on the nature of computation, not vice versa.

Thanks for writing this up! I strong-upvoted because, as you say, these ideas are not well-communicated, and this post contributes an explanation that I expect to be clarifying to a significant subset of people confused about agent foundations.

Initially, I wasn't quite buying the claim that we don't need any experiments (or more generally, additional empirics) to understand agency and all we need is to "just crunch" math and philosophy. The image I had in mind was something like "This theorem proves something non-trivial — or even significantly surprising — about a class of agents that includes humans, and we are in a position to verify it experimentally, so we should do it, to ensure that we're not fooling ourselves.".

Then, this passage made it click for me and I saw the possibility that maybe we are in a position where armchairs, whiteboards, tons of paper, and Lean code are sufficient.

It's noteworthy that humanity did indeed deliberately invent the first Turing-complete programming languages before building Turing-complete computers, and we have also figured out a lot of the theory of quantum computing before building actual quantum computers.

When Alan Turing figured out computability theory, he was not doing pure math for math's sake; he was trying to grok the nature of computation so that we could actually build better computers. And he was not doing typical science, either. He obviously had considerable experience with computers, but I seriously doubt that, for example, work on his 1936 paper involved running into issues which were resolved by doing experiments. I would say agent foundations researchers have similarly considerable experience with agents.

(Another example/[proof of concept]/[existence proof of the reference class] is Einstein's Arrogance.)

However, the reference class that includes the theory of computation is one possible reference class that might include the theory of agents.^[1] But for all (I think) we know, the reference class we are in might also be (or look more like) complex systems studies, where you can prove a bunch of neat things, but there's also a lot of behavior that is not computationally reducible and instead you need to observe, simulate, crunch the numbers. Moreover, noticing surprising real-world phenomena can serve as a guide to your attempts to explain the observed phenomena in ~mathematical terms (e.g., how West et al. explained (or re-derived) Kleiber's law from the properties of intra-organismal resource supply networks^[2])$.$

I don't know what the theory will look like; to me, its shape remains an open a posteriori question.

1. ^{^}

   Or whatever theory we need to understand agents as the theory that we need to understand agents need not be a theory of agents (but maybe something broader like IDK adaptivity or powerful optimization processes or maybe there's a new ontology that cuts across our intuitive notion of agency and kinda dissolves it for the purpose of joint-carving understanding).

2. ^{^}

   The explanation of their proof that I was able to understand is the one in this textbook.

So is Agent Foundations primarily about understanding the nature of agency so we can detect it and/or control it in artificial models, or does it also include the concept of equipping AI with the means of detecting and predictively modeling agency in other systems? Because I strongly suspect the latter will be crucial in solving the alignment problem.

The best definition I have at the moment sees agents as systems that actively maintain their internal state within a bounded range of viability in the face of environmental perturbations (which would apply to all living systems) and that can form internal representations of arbitrary goal states and use those representations to reinforce and adjust their behavior to achieve them. An AGI whose architecture is biased to recognize needs and goals in other systems, not just those matching human-specific heuristics, could be designed to adopt those predicted needs and goals as its own provisional objectives, steering the world toward its continually evolving best estimate of what other agentic systems want the world to be like. I think this would be safer, more robust, and more scalable than trying to define all human preferences up front.

These are just my thoughts. Take from them what you will.