On October 20, 2026, Kevin Dong conducted an interview with James Mattingly, Professor of Philosophy at Georgetown University and coauthor of Machine Agency (2025) with Beba Cibralic. What follows is a lightly edited transcript of their conversation.
James Mattingly: Neither Beba Cibralic nor I are particularly experts in AI. We come at it from very different directions, both from each other and from perspectives that aren’t primarily about artificial intelligence. She comes at it more or less from just war theory. She was a graduate student at Georgetown whom I recruited around 2016 or 2017, and she was doing a PhD in political philosophy.
One of the things we do at Georgetown, like every graduate program, is have these gatekeeping courses. One of them is in logic, which I frequently teach to graduate students. I teach the course from the perspective of computability, using Boolos, Burgess, and Jeffrey’s Computability and Logic. So a lot of the fundamental results concerning incompleteness, representability, and so forth are presented from the perspective of Turing-machine implementation and the equivalence between Turing machines and other kinds of machines.
In the spring of 2020, Beba and another graduate student asked me to teach a graduate seminar on the philosophy of technology, which I knew a little about, but not much. So we had to start from scratch, and we worked it up together. A lot of different things came up, but one recurring question was who was responsible for decisions made by the AI systems being used at the time for loan approvals, various background checks, and so forth. Banks were rolling these systems out, and they were completely inscrutable. Nobody knew exactly what was going on inside them.
At the end of the seminar, Beba told me I should write a book about this. I had no real capacity to do that on my own, so I demanded that she and I write it together. That’s how the book came about.
From my perspective, I have a lot of interest in logic and computation theory, though at a very abstract level – the kind of interest a philosopher has. She wants to understand who is responsible for the decisions these machines make and how we can ensure that the machines do what we want them to do in the right way. So contemporary AI is only one aspect of this for us. As the title of the book suggests, we’re more interested in the general concepts of agency, responsibility, and the attribution of various activities to machines. AI happens to be the current best guess at how that’s going to work, I suppose.
Kevin Dong: Tell me a bit about your life, philosophical background, and training, and how those played into the genesis of Machine Agency.
James Mattingly: I was raised in San Jose and didn’t have any clear idea of what I wanted to do in college. Then I received a brochure from this obscure liberal arts school in Annapolis, St. John’s College. It’s basically classics-lite. They follow something like the old Chicago Great Books program: you start with the Iliad over the summer before your freshman year, and then you read your way through a bunch of the so-called Great Books.
I thrived in that program, I suppose. But toward the end, we started reading some more contemporary science. There had been science components throughout the four years – everybody takes the same program every semester, and there’s always some kind of science course or laboratory. But in junior and senior year, we started reading Newton’s Principia, Maxwell, Faraday, Heisenberg, and so forth. I always find this a little embarrassing to relate, but all that seemed ridiculous to me. Relativity theory, quantum theory – just terrible!
Why? Well, at St. John’s, we were all raised as Kantians by the end. We thought there were principles of causation, space, and time that you could derive simply by thinking about them really hard. You could think about the very concepts of space, time, and causation and arrive at clear ideas about their nature. Relativity theory and quantum theory conflict quite dramatically with those ideas. So I thought, “That can’t be right. I’m going to learn enough physics to figure out what they’re doing wrong and sort it all out.”
That’s why I ended up at UC Santa Cruz studying physics. I had come back to California, was staying with my parents, and started auditing some courses. Eventually, I entered the graduate program there. One thing we read at St. John’s was Lavoisier’s Elements of Chemistry – not the whole thing, but a fair bit of it. In the preface, Lavoisier quotes Condillac on how language shapes the way we think and reason. I really found that to be true when I studied physics. It changed how I thought about things. It changed my understanding of the very nature of first principles.
After that, I went to Indiana University to study history and philosophy of science. When I arrived, I thought I was simply going to do history and philosophy of science, full stop. But my advisor, Michael Friedman, said, “Look, you can’t just be a philosopher of science. You have to be a philosopher of something – a philosopher of chemistry, or something else.” Then he said, “Looking at you, I think philosophy of quantum cosmology.” I thought, “Okay, I guess I’ll do that.”
So that’s what I did. It turned into a dissertation on the foundations of quantum gravity, basically examining arguments for and against the necessity of quantizing the gravitational field. From there, I went to Georgetown, and I’ve been there ever since. I continue to think about quantum gravity to some extent, as well as field theories, gauge theories, and so forth. But I also ended up teaching a lot of logic, so I developed this passing interest in computation theory.
How does that lead to a book like Machine Agency? I ended up teaching a philosophy of technology course, and as part of it we looked at contemporary AI and considered questions such as: Are these systems agents? Are they responsible? Are they making decisions? By the end, there seemed to be enough questions in our minds – mine and Beba’s, since she was a student in that seminar – that we decided to explore those ideas together and write the book.
Kevin Dong: Who is Beba Cibralic, and why is she brilliant?
James Mattingly: Beba Cibralic is a former graduate student at Georgetown. I can’t remember exactly when she started – 2017 or 2018, perhaps. She was in my logic class and my philosophy of technology seminar.
Beba has that rare talent you look for in graduate students and colleagues: the ability to move apparently effortlessly from one field to another, know exactly what’s going on, and understand not only how to navigate the field but also where the interesting landmarks are. Part of that is that she’s extremely diligent. She takes a lot of time and care to make sure she’s on top of whatever literature she’s working with. But she’s also remarkably perceptive about where people are chasing red herrings and where they’re focusing on things that will actually have consequences.
She had no real background in logic, but whenever I was explaining something and it was obvious that it was too difficult or confusing – or that I wasn’t expressing myself very well for the rest of the class – she would step in and say, “Maybe it goes something like this.” And she was always spot-on. That’s just one example. She has a comprehensive intellect, but she’s also extremely good at getting the details exactly right.
Kevin Dong: Describe the arguments advanced in Machine Agency for a non-philosophical lay audience.
James Mattingly: One thing Beba used to say to me all the time was, “Look, James, we’re not here to make our argument.” The point of the book is to provide a pedagogical introduction to some of the salient issues in this domain. People who want to think philosophically about the revolution in machine learning and AI, and about the kinds of machines that may exist in the future, will need to understand certain things.
Yes, we have our own views about the right way to think about these issues. But one of our main ambitions was to keep those views in the background and clarify the questions people are going to confront. We try to provide a general introduction to good philosophical manners – or good philosophical hygiene. How do you make an argument? How do you appraise an argument? How do you take somebody else’s argument seriously? So one of the main things we do is introduce philosophical argumentation without necessarily pushing our own views.
That said, there are some background claims we make. One is that, whether or not our account of agency is ultimately correct, it’s useful for investigating these issues. We spend some time making plausible the idea that you can go a long way toward understanding agency using a little model we call the minimalist model. In our view, it captures both necessary and sufficient conditions for something to count as an agent.
For non-philosophers, necessary and sufficient conditions are a standard way of constructing definitions. Sufficient conditions say that anything possessing certain properties counts as the thing in question. Necessary conditions say that anything missing one of those properties does not count. Together, they isolate a particular kind of thing in conceptual space. In our case, the question is: What is an agent?
Our claim is that there are essentially two kinds of conditions for agency. The first concerns representation. An agent must represent the world in various ways. There must be internal states, structures, functions, or something else going on inside the device that mirrors, accounts for, or represents how the world is. Moreover, there must be other representations that, from an external perspective, represent how the world should be. That’s one of our main conditions for agency: something must be going on in how the system represents the world and the states the world ought to have.
The second condition concerns behavior. We discuss what behavior amounts to. It’s usually thought of as some kind of movement, or at least something that can be done, but whose source is internal to the thing doing it. If I push a rock, the rock is not behaving, even though it’s moving. On the other hand, if I lift my arm, even accidentally, that still counts as behavior because the source of the movement is internal to me.
So having a behavioral repertoire – a number of behaviors that can be deployed at any given moment – and having representations are the main structural requirements for agency. What agents do, on our account, is continue deploying items from their behavioral repertoire until their representation of the proper end state and their representation of the actual world align. That can happen because the world changes to match the desired end state. Or, in some cases, the agent may give up and revise its representation of how the world should be in light of circumstances.
This is all very abstract, so it’s useful to ground it in an example. A standard way of explaining action in the philosophical literature is through beliefs and desires. Suppose somebody suddenly gets up, leaves the room, and returns a while later carrying bread, lunch meat, and mustard. You think, “Ah, now I understand. This person wanted a sandwich.”
The person believed that bread, bologna, and mustard were available somewhere. They behaved in certain ways to obtain those things and change the world from a state in which they didn’t have a sandwich to one in which they did. What are the representations here? The person has beliefs about the world: “I don’t have a bologna sandwich,” and “There’s bologna, bread, and mustard at the store.” They also have a desire, which is another kind of representation: “I want a bologna sandwich.”
That combination of beliefs and desires induces behaviors. The person stands up, walks to the door, opens it, and goes to the store. Those behaviors continue until the representations align. That’s a simple story about agency.
We then asked ourselves: What is the most minimal real-world agent that has these features? It needs representations of the world, representations of an end state, and a behavioral repertoire. We thought of the thermostat. It’s not a very interesting agent, but it does these things.
A thermostat represents the state of the room. Does it represent the room completely? Does it represent its size or the color of the walls? No. But it represents one aspect of the room: its temperature. It has an internal structure – a thermometer – that represents how the world is. Depending on how you adjust the machine, it also has an end-state representation: the temperature the room should have. So it has two representations: how the world is and how the world should be.
That’s not enough by itself, because we haven’t discussed behavior. But the thermostat also has a behavioral repertoire. Typically, it can energize a circuit that turns on a heater or one that turns on a cooler. Depending on whether the room is too hot or too cold, it activates the cooler or heater until the actual temperature aligns with the desired temperature.
That’s the central idea of the book. We develop this account of representation, show how it connects with earlier philosophical literature, and examine how contemporary machines might – or might not – instantiate it. We also consider what it would mean for machines to become more sophisticated agents, to do the kinds of things we do, and perhaps to be held responsible in the ways we are.
We don’t make enormous progress on those latter questions. We certainly don’t offer a recipe for building a machine with human-level consciousness. But that’s not the point. The point is to introduce a particular conceptual tool and investigate whether and how it can help us understand both the current state of machine agency and its future possibilities.
Kevin Dong: What was the landscape of preexisting philosophical debates on the topics you two cover around the time you two were writing Machine Agency?
James Mattingly: When we first started writing, there actually wasn’t a huge amount of work connecting the literature on agency with the literature on AI and its various computational implementations. There was certainly some work, but I’d say the debates fell into roughly two categories.
One concerned responsibility. The debate wasn’t so much about how best to attribute responsibility to machines if they act as agents. It was about whether we should be having that conversation at all. There’s a strand of the literature arguing that we shouldn’t focus on machine agency because doing so gives cover to the manufacturers of these machines. It allows them to avoid taking responsibility for the unpleasant consequences of what their machines do.
Others think that we need to sort out the question. We acknowledge that debate without weighing in too strongly on either side. But we point out that, even if manufacturers must continue to be held responsible for what their creations do, it still makes sense to ask in what sense the machines themselves are acting and whether it could ever be reasonable to attribute responsibility to them.
That was one issue in the background. But part of the impetus for the book was pedagogical. We weren’t primarily dissatisfied with the debates taking place. We think they’re fascinating and important. The problem was that there was no easy way for students to enter them. You already had to be deeply immersed in the literature on agency, artificial intelligence, computation theory, or something similar to understand what was happening. We wanted to provide a gentle introduction.
There was, however, one technical point that we thought was misleading: the question of machine minds. When we started, we weren’t necessarily focused on agency. We were thinking about a whole range of related issues. It occurred to us that focusing on whether machines have minds might be misleading.
Mind is something humans – and perhaps certain animals – have. It makes sense to explain much of what we do in terms of mental states, mental capacities, and so forth. But it’s not clear that this is the general way systems in the world act, or the way systems in some possible future world will act. Mind might not be the right concept for understanding everything that makes decisions or intervenes in the world in ways we ordinarily associate with minded beings.
There might be a more general way of thinking about these things. Our slogan to ourselves was: “Mind is the way we do it, but it might not be the only way it gets done.” That was an important realization. It allowed us to step away from debates about when machines might have minds and instead ask what the mind is doing when we attribute agency to humans or animals – and whether we can give a more general account of that activity.
Kevin Dong: What are some of the strongest objections to the arguments you and Beba advance in Machine Agency, whether anticipated, published, or raised informally?
James Mattingly: Here’s an objection I’m getting now, actually from Beba herself. It’s an important one, and something future work will need to clarify. Computer scientists – people at Anthropic, OpenAI, and similar companies – aren’t using the words we use in the same way. They talk about representations, agents, and so forth, but their uses of those terms are somewhat orthogonal to ours.
So one objection is that certain discussions in the book don’t make sufficient contact with what’s actually happening in AI right now. I think that’s an important concern. If the idea were to use the insights from the book to police what’s happening in the AI community, that would be a serious problem. But that’s not really our intention.
Our thought was that you could use the notions of agency, representation, and the alignment of representations to evaluate what these machines are doing and provide resources for thinking about how to deal with them. If you think about agency the way we do, and you think certain machines are agents, then you have a new way of intervening in their behavior.
Consider one of the familiar science-fiction worries about AI: that machines will wake up, develop desires different from ours, eventually take over the world, and cause us to suffer. Part of the concern is that, because they’re connected to powerful machinery, they’ll be much more powerful than we are. That focuses on their behavioral power.
But if they really are agents in our sense, their behavior will be driven by mismatches between representations. And intervening in those representations may be easier than intervening directly in the behavior of powerful machinery. That’s one way we were thinking about the problem.
But the broader lesson, which applies to philosophers generally, is that if you’re thinking about an active discipline outside philosophy, you can’t expect the people in that discipline to come to you. You have to go to them. You need to make sure you’re speaking a language that can be translated between what they’re doing and what the philosophical community is doing.
Kevin Dong: You mentioned using the book to teach students. Have you and Beba taught from Machine Agency, or from drafts or lecture notes based on it? If so, what aspects of its arguments have students found most intuitive or difficult, and why?
James Mattingly: Yes. One reason we wrote the book was that I had already been teaching material in this area. I was teaching small modules in interdisciplinary courses at Georgetown, where faculty from different disciplines would each teach one aspect of a larger topic. I couldn’t find all the material I wanted in one place. So one impetus for the book was to organize all those tools into a single text.
Since publication, I’ve used it once as the textbook for a class, and I’m using it again right now. I’m not entirely sure how to answer which part students find most intuitive. Students do seem enthusiastic about agency as an organizing concept. They often haven’t thought about things this way before, and they seem to appreciate its potential as a unifying concept that steps back from some confusing notions.
One thing they appreciate is that many of the concepts we use when talking about people are all muddled together: mind, feelings, consciousness, intelligence, emotions, and so forth. It’s difficult to know which of these are connected because of the nature of the concepts themselves and which are connected simply because of the way human beings happen to be. When we think, we often also feel things. When we think, we’re usually awake and aware of the world. All these things are wrapped together in us, and they’re difficult to separate.
Agency is a useful concept because it doesn’t seem, by itself, to carry all those associations. But it can be brought into conversation with those other concepts and used to tease them apart. Students seem to appreciate that.
They also tend to do pretty well with our discussion of algorithms. You have a mathematical background, so the idea of an algorithm is second nature to you. But for many students in the humanities or other fields outside STEM, the concept can be puzzling. People talk about algorithms all the time without necessarily knowing what they mean. It’s something their phone does. Twitter, Instagram, Facebook – they all have algorithms that are supposedly influencing things, but people don’t necessarily understand what that means.
We discuss the nature of algorithms, give intuitive examples of how they function, and explain computation as a way of implementing algorithms. I think students appreciate that.
The structure of the book also lends itself to classroom use. The chapters are short – usually ten or fifteen pages. Each chapter introduces one little idea, discusses it, suggests ways to pursue greater clarity, and then moves on. Students find that digestible.
In a normal class, I’ll assign a chapter each week, pair it with some readings from the bibliography or suggested readings, and explore those additional readings in greater depth. The book serves as the framing mechanism.
Kevin Dong: What were some hard-fought, counterintuitive insights you and Beba arrived at while writing Machine Agency? What were the eureka moments?
James Mattingly: One was when we decided that chat was not an agent. Part of the impetus for the book was our initial thought that chat was clearly an agent. It seemed to be doing things in the world despite not having arms or legs. Its behavioral repertoire was confined to answering questions and similar activities.
At that point, I don’t think any of the large-language-model systems had been connected to machinery. They were just chatbots. But we thought: They do things in the world. They cause people to respond in certain ways. So we can think of them as intervening.
We spent a lot of time developing our theory of agency in a way that would capture the idea that chat was an agent. Then, at some point, we realized: Wait a second. No, it isn’t. It isn’t an agent because it isn’t a representational engine in the relevant sense.
What do I mean by that? There is certainly a sense in which it represents things. For a long time, we thought that was enough: the machine represents the world, it behaves, and that’s all there is to agency. But we realized that it wasn’t using those representations to guide its behavior.
Our theory of agency says that a mismatch between representations drives behavior. I want a sandwich, but I don’t have one. The world continues to be in a state in which I don’t have a sandwich, while I continue to want one. Those representations don’t align, so I start doing things. I move around, deploy my behavioral repertoire, and continue until the representations match. Either I get the sandwich, or I give up and stop wanting it.
That’s what it means for a mismatch between representations to drive behavior. But chat doesn’t do that. Yes, it represents the world in a certain sense. It has internal structures in its architecture that represent certain kinds of semantic content. But its behavior isn’t driven by a mismatch between that representation and another representation.
It’s an input-output device. It follows a particular procedure. Its representations can change over time as training changes, but there’s nothing about a mismatch between those representations that drives what it does. That’s the opposite of how things work for you, me, or my dog. My dog wants something. He smells something and starts digging until he finds it.
LLMs don’t do that. They receive a prompt, produce an output, and stop. They don’t deploy behaviors in response to mismatches between representations.
We were really surprised by that. It’s not surprising in light of the theory of agency we ultimately developed. But it was surprising to realize that we didn’t want to count chat as an agent. Our whole orientation had to be turned upside down. That realization helped drive our search for the right theory of agency.
It’s not as though the theory is something entirely new. It makes contact with standard theories of agency in the philosophical literature, taking something from here and something from there. But it extracts what we take to be the core message underlying many of those theories and makes clear why LLMs, at least in the form we were considering, aren’t agents.
Kevin Dong: How far back in history would one have to go before the arguments you and Beba advance in Machine Agency cease to be plausible or immanent to the historical moment?
James Mattingly: I suppose it depends on how you’re thinking about the arguments. Remember how the book starts: Socrates is sitting in a jail cell, about to be executed by the Athenians. It’s an interesting day. You might not think philosophizing about the soul and mind is what you’d want to do on your last day, but that’s what he does.
As far as we can tell, he introduces one of the first sustained discussions in Western literature of the demand for an explanation of action. You might ask: Why isn’t Socrates escaping from jail? That’s something you want explained.
Socrates raises a related question himself: How is it that mind is in charge of everything? He expects to find an answer in a book by Anaxagoras, because that’s what he’s been told. But instead of learning about mind, he learns about tendons, muscles, blood, and all the other things that make bodies move.
And he says, in effect, “No, that’s not what I want to know.” He wants to know why his body remains in the jail cell even though it’s about to be destroyed if he stays there. That has everything to do with mind.
Our account matches what Socrates is doing remarkably well. We have a different general account of action explanation, but we’re playing the same game. The difference is that we want the game to be more general. We want other kinds of things to be subject to explanations of action.
The Greeks thought many more things than humans and animals were subject to such explanations. You could say that’s because they thought mind was much more widespread than we think it is now. They were committed to the idea that things requiring explanations of action must have minds. Maybe a tree has a mind, or a stream of water, or the rain, or some android made by a god. But mind is doing the explanatory work.
They probably wouldn’t have found it plausible that something without a mind could be subject to an explanation of action. Nevertheless, the structure of our explanation – the idea that a mismatch between how the world is and how it should be drives behavior – would have made perfect sense to them. The question would have been whether something other than a minded being could satisfy that structure.
So if you go back that far, half of what we’re talking about would be perfectly understandable. The other half would be baffling. The idea that a piece of brass, silicon, glass, or plastic, without being imbued with divine powers or anything beyond nature, could be subject to an explanation of action would have seemed implausible.
But consider going back only to 1950, when Turing was writing “Computing Machinery and Intelligence.” Would our arguments have seemed plausible then? Many people in Turing’s audience didn’t even think the word “think” could properly be applied to computers or machinery. So perhaps going back only that far would make our arguments implausible to many people.
Indeed, you might say they’re still implausible to many people today. One continuing question in discussions of AI and computation is whether these systems have minds or intelligence, whether something more than algorithms is going on. You see stories about engineers at AI companies deciding that their systems are conscious or awake.
There’s an ongoing concern that we can’t think about agency until we find minds. And because people see evidence of agency, they take that as evidence of mind. Beba and I think that’s completely implausible. That’s not how agency should be understood.
So even some of the people working at the highest levels on understanding these machines might find our account implausible. Go back three or four thousand years, and people might find parts of it plausible, although you’d have to modify the framework considerably. But even today, there’s something about the account that’s difficult to communicate.
You asked earlier about teaching. I do find this to be a sticking point for students. It isn’t intuitive to them that something could be an agent without having a mind. And perhaps that’s one reason to wonder whether something is missing from our account.
Kevin Dong: Conversely, why are the arguments immanent specifically to 2025 and beyond, rather than to an earlier historical moment?
James Mattingly: One thing that’s different now, from our perspective, is that we seem to be at a point where it makes sense – or soon will make sense – to attribute agency to these machines. Some people think that’s because machines will have minds, intelligence, desires, and so forth of their own. Others think so for different reasons.
Beba and I think it already makes sense to attribute agency to certain machines. But the possibility of agency and mindedness in machines is an urgent question now because of the particular state of AI research.
Of course, from the perspective of the leading edge of technology, people have always thought that living, thinking, or feeling machines were just around the corner. When Babbage was developing the Analytical Engine, it made sense for Ada Lovelace to deny that machines could be creative, because that was a live question. In 1950, it made sense for people to deny that computers could think, because that was a live question.
These conversations have been salient at several historical moments. What makes them salient now is that the AI revolution, as you might call it, seems sufficiently urgent to require us to think these things through and take seriously the possibility that something fundamentally different is happening with our machinery.
Will people look back 150 or 200 years from now and think our moment was quaint? Will they say that we were convinced we had machines on the verge of intelligence, awareness, or consciousness, when of course we didn’t? That’s possible. But it doesn’t change the fact that what makes the question urgent for us is our sense that something is different now with machines and computers.
Kevin Dong: What frustrates you, or leaves you dissatisfied, about contemporary AI discourse in 2026?
James Mattingly: I’ve already touched on one aspect: people are still hunting for minds in these machines. I don’t think there’s anything particularly implausible about the idea that things other than mammals – or animals descended from our common ancestors – could have minds. That’s not an issue for me.
But I don’t think hunting for minds is the best way to approach these machines right now. I don’t see any reason to think that what we’re doing now is generating minds, or that the machines we have possess minds in anything usefully like the way we do.
Instead, I think we should be looking for ways to take the incredible computational power of LLMs and other artificial intelligences and harness it to systems that really do trade in representations and drive behavior through mismatches between those representations. That’s where I think we’ll find the most interesting kinds of creatures, living or otherwise.
So I find the hunt for minds in contemporary machines frustrating. I think it’s the wrong hunt. Another frustration is that this focus encourages a binary way of thinking about AI. Either these systems have minds or they don’t.
If you’re on the “no mind” side, that’s the end of the story. They’re just chatbots or phrase completers. Of course, they’re fast, powerful, and sophisticated, but that’s supposedly all there is to them. On the other hand, if you’re on the “mind” side, then something revolutionary is happening that can only be understood by understanding mind.
That leaves a gap in our characterization of these systems. I’ve given a talk about this and am working on a paper. The question is: What really is an LLM if you don’t want to say it has a mind, but also don’t want to dismiss it as merely a phrase completer, sentence finisher, or glorified spell-checker?
I started thinking about how these systems are built. If you know a little about the architecture, you know there are many linear transformations. You have an input layer, perform some linear operations, obtain another layer of representation, perform more linear operations, and so forth. It’s matrix multiplication all the way down.
But every so often, there’s a nonlinear operation. And whenever I’m put on the spot, I forget the names of things. I have a kind of name-retrieval problem.
The nonlinear components in these machines are descendants of the perceptron, an attempt from the middle of the twentieth century to build artificial minds. The original Mark I Perceptron was designed to train an array of light detectors to distinguish, for example, an X from an O. It used feedback and nonlinear operations. Modern neural networks contain related kinds of nonlinear processing, although in much more elaborate architectures.
When people discuss how these systems work, they tend to focus on the linear components. Why? Because we understand those components. They’re difficult to interpret because the representational spaces are enormous, but mathematically we understand what’s happening. We can represent those operations conceptually.
The nonlinear parts are harder to understand in that way. It’s like looking for your keys under a streetlight, even though you dropped them in the grass somewhere else. You look under the light because that’s where you can see.
When I look at people at Anthropic and elsewhere discussing what’s going on inside these systems, they often focus on the linear parts. I’ve become interested in the nonlinear, perceptron-like aspects.
Around the same time the perceptron was being developed, another important paper appeared: “What the Frog’s Eye Tells the Frog’s Brain.” The researchers were trying to understand what happens when a frog hunts visually. Frogs are different from us visually because they don’t have a visual cortex like ours. They don’t see in quite the way you and I do.
There’s an analogy here with what’s called blindsight. In blindsight, a person’s eyes may function, and visual information reaches the brain, but the person doesn’t have ordinary conscious visual experience. You might ask someone with blindsight, “How many fingers am I holding up?” They’ll say, “I can’t see your fingers.” But if you ask them to guess, they can sometimes perform much better than chance.
They’re processing visual information without the associated conscious visual experience. Something analogous is happening in the frog. When you look at how the frog’s eyes are connected to its brain, there are similarities to the way a perceptron works – although the biological system has many more layers and complications.
When we look at the world, we might see a cup, recognize it as a cup, understand that it can hold water, and decide to put water in it and drink. We do all this through an experience of visual space.
The frog’s eye functions more directly as a detector. It takes information from the world and effectively tells the frog’s brain: “That’s a fly – stick out your tongue and grab it. That’s a predator – jump away. That’s a mate – move toward it.” The frog doesn’t necessarily need to go through the same kind of visual processing that we do.
The authors describe the frog’s eye, in part, as a bug detector. That’s useful because it evolved in circumstances where detecting bugs was advantageous. But strictly speaking, it isn’t detecting bugs as such. It detects features that reliably indicate bugs in the environments in which it evolved.
If you throw BBs into a frog’s visual field, it may reliably snatch them up and eat them, without learning not to do so. It isn’t thinking about them in the way we might. It’s reacting to certain features.
The more I think about this, the more I see an analogy with how LLMs are structured. It looks as though they’re doing something similar. They’re detecting something. They may be understood as a kind of sensory mechanism, analogous in certain respects to the frog’s eye as a mechanism for detecting flies.
But here’s the crucial question, and I don’t know the answer: What exactly are they detecting?
They’re detecting something in the neighborhood of semantic content. But it isn’t clear exactly what that something is. It’s related to semantic content because of the context in which these systems are trained. Chatbots are trained on books, literature, and other texts, so they become sensitive to patterns that we associate with semantic content.
But the systems don’t necessarily associate those patterns with anything in the way we do. I think focusing on structural features, mechanisms, and analogies other than the simple distinction between mindedness and non-mindedness is a promising way to understand what’s happening. It frustrates me that more people aren’t working on that sort of question.
That’s a rather long-winded answer.
Kevin Dong: Are there any future projects on AI and philosophy in the works? What would you most like to write next?
James Mattingly: Well, there’s that project. I’d like to get more clarity on it. That’s where my focus is at the moment – not necessarily just that particular paper, but exploring more possibilities in the conceptual space of what LLMs are and how they function.
People are continuing to work on questions about agency in the context of LLMs. I don’t want to give the impression that nobody is doing that, even though I think there’s too much emphasis on what minds are and whether these systems have them.
People are asking questions about agency. One thing they tend to do, though, is focus on whether these systems meet necessary conditions for representation, agency, or some other property. There’s much less work on sufficient conditions, and on how we might identify and promote those conditions in artificial intelligence.
I’m interested in that aspect of the problem. But mostly, I’m interested in developing a better understanding of the structural features of these machines and achieving greater conceptual clarity about how they work.
If you use an LLM as just a chatbot, then indeed it has no "agency" (the scare quotes because I don't have a clear idea of what anyone means by the word). But the LLMs in the HuggingFace incident appear to have had goals and exercised resourcefulness in achieving them. LLMs used for coding are set up to write code to perform some task, and find and fix bugs in their work until a set of tests is passed. If "agency" means resourcefulness in achieving goals, these systems have it.
Are we past the point where it is possible to write a book on any aspect of AI that will not be outdated as fast as it is written?
Crossposted on X here: https://x.com/kevin__dong__/status/2115079682412728352
On October 20, 2026, Kevin Dong conducted an interview with James Mattingly, Professor of Philosophy at Georgetown University and coauthor of Machine Agency (2025) with Beba Cibralic. What follows is a lightly edited transcript of their conversation.
https://mitpress.mit.edu/9780262549981/machine-agency/
James Mattingly: Neither Beba Cibralic nor I are particularly experts in AI. We come at it from very different directions, both from each other and from perspectives that aren’t primarily about artificial intelligence. She comes at it more or less from just war theory. She was a graduate student at Georgetown whom I recruited around 2016 or 2017, and she was doing a PhD in political philosophy.
One of the things we do at Georgetown, like every graduate program, is have these gatekeeping courses. One of them is in logic, which I frequently teach to graduate students. I teach the course from the perspective of computability, using Boolos, Burgess, and Jeffrey’s Computability and Logic. So a lot of the fundamental results concerning incompleteness, representability, and so forth are presented from the perspective of Turing-machine implementation and the equivalence between Turing machines and other kinds of machines.
In the spring of 2020, Beba and another graduate student asked me to teach a graduate seminar on the philosophy of technology, which I knew a little about, but not much. So we had to start from scratch, and we worked it up together. A lot of different things came up, but one recurring question was who was responsible for decisions made by the AI systems being used at the time for loan approvals, various background checks, and so forth. Banks were rolling these systems out, and they were completely inscrutable. Nobody knew exactly what was going on inside them.
At the end of the seminar, Beba told me I should write a book about this. I had no real capacity to do that on my own, so I demanded that she and I write it together. That’s how the book came about.
From my perspective, I have a lot of interest in logic and computation theory, though at a very abstract level – the kind of interest a philosopher has. She wants to understand who is responsible for the decisions these machines make and how we can ensure that the machines do what we want them to do in the right way. So contemporary AI is only one aspect of this for us. As the title of the book suggests, we’re more interested in the general concepts of agency, responsibility, and the attribution of various activities to machines. AI happens to be the current best guess at how that’s going to work, I suppose.
Kevin Dong: Tell me a bit about your life, philosophical background, and training, and how those played into the genesis of Machine Agency.
James Mattingly: I was raised in San Jose and didn’t have any clear idea of what I wanted to do in college. Then I received a brochure from this obscure liberal arts school in Annapolis, St. John’s College. It’s basically classics-lite. They follow something like the old Chicago Great Books program: you start with the Iliad over the summer before your freshman year, and then you read your way through a bunch of the so-called Great Books.
I thrived in that program, I suppose. But toward the end, we started reading some more contemporary science. There had been science components throughout the four years – everybody takes the same program every semester, and there’s always some kind of science course or laboratory. But in junior and senior year, we started reading Newton’s Principia, Maxwell, Faraday, Heisenberg, and so forth. I always find this a little embarrassing to relate, but all that seemed ridiculous to me. Relativity theory, quantum theory – just terrible!
Why? Well, at St. John’s, we were all raised as Kantians by the end. We thought there were principles of causation, space, and time that you could derive simply by thinking about them really hard. You could think about the very concepts of space, time, and causation and arrive at clear ideas about their nature. Relativity theory and quantum theory conflict quite dramatically with those ideas. So I thought, “That can’t be right. I’m going to learn enough physics to figure out what they’re doing wrong and sort it all out.”
That’s why I ended up at UC Santa Cruz studying physics. I had come back to California, was staying with my parents, and started auditing some courses. Eventually, I entered the graduate program there. One thing we read at St. John’s was Lavoisier’s Elements of Chemistry – not the whole thing, but a fair bit of it. In the preface, Lavoisier quotes Condillac on how language shapes the way we think and reason. I really found that to be true when I studied physics. It changed how I thought about things. It changed my understanding of the very nature of first principles.
After that, I went to Indiana University to study history and philosophy of science. When I arrived, I thought I was simply going to do history and philosophy of science, full stop. But my advisor, Michael Friedman, said, “Look, you can’t just be a philosopher of science. You have to be a philosopher of something – a philosopher of chemistry, or something else.” Then he said, “Looking at you, I think philosophy of quantum cosmology.” I thought, “Okay, I guess I’ll do that.”
So that’s what I did. It turned into a dissertation on the foundations of quantum gravity, basically examining arguments for and against the necessity of quantizing the gravitational field. From there, I went to Georgetown, and I’ve been there ever since. I continue to think about quantum gravity to some extent, as well as field theories, gauge theories, and so forth. But I also ended up teaching a lot of logic, so I developed this passing interest in computation theory.
How does that lead to a book like Machine Agency? I ended up teaching a philosophy of technology course, and as part of it we looked at contemporary AI and considered questions such as: Are these systems agents? Are they responsible? Are they making decisions? By the end, there seemed to be enough questions in our minds – mine and Beba’s, since she was a student in that seminar – that we decided to explore those ideas together and write the book.
Kevin Dong: Who is Beba Cibralic, and why is she brilliant?
James Mattingly: Beba Cibralic is a former graduate student at Georgetown. I can’t remember exactly when she started – 2017 or 2018, perhaps. She was in my logic class and my philosophy of technology seminar.
Beba has that rare talent you look for in graduate students and colleagues: the ability to move apparently effortlessly from one field to another, know exactly what’s going on, and understand not only how to navigate the field but also where the interesting landmarks are. Part of that is that she’s extremely diligent. She takes a lot of time and care to make sure she’s on top of whatever literature she’s working with. But she’s also remarkably perceptive about where people are chasing red herrings and where they’re focusing on things that will actually have consequences.
She had no real background in logic, but whenever I was explaining something and it was obvious that it was too difficult or confusing – or that I wasn’t expressing myself very well for the rest of the class – she would step in and say, “Maybe it goes something like this.” And she was always spot-on. That’s just one example. She has a comprehensive intellect, but she’s also extremely good at getting the details exactly right.
Kevin Dong: Describe the arguments advanced in Machine Agency for a non-philosophical lay audience.
James Mattingly: One thing Beba used to say to me all the time was, “Look, James, we’re not here to make our argument.” The point of the book is to provide a pedagogical introduction to some of the salient issues in this domain. People who want to think philosophically about the revolution in machine learning and AI, and about the kinds of machines that may exist in the future, will need to understand certain things.
Yes, we have our own views about the right way to think about these issues. But one of our main ambitions was to keep those views in the background and clarify the questions people are going to confront. We try to provide a general introduction to good philosophical manners – or good philosophical hygiene. How do you make an argument? How do you appraise an argument? How do you take somebody else’s argument seriously? So one of the main things we do is introduce philosophical argumentation without necessarily pushing our own views.
That said, there are some background claims we make. One is that, whether or not our account of agency is ultimately correct, it’s useful for investigating these issues. We spend some time making plausible the idea that you can go a long way toward understanding agency using a little model we call the minimalist model. In our view, it captures both necessary and sufficient conditions for something to count as an agent.
For non-philosophers, necessary and sufficient conditions are a standard way of constructing definitions. Sufficient conditions say that anything possessing certain properties counts as the thing in question. Necessary conditions say that anything missing one of those properties does not count. Together, they isolate a particular kind of thing in conceptual space. In our case, the question is: What is an agent?
Our claim is that there are essentially two kinds of conditions for agency. The first concerns representation. An agent must represent the world in various ways. There must be internal states, structures, functions, or something else going on inside the device that mirrors, accounts for, or represents how the world is. Moreover, there must be other representations that, from an external perspective, represent how the world should be. That’s one of our main conditions for agency: something must be going on in how the system represents the world and the states the world ought to have.
The second condition concerns behavior. We discuss what behavior amounts to. It’s usually thought of as some kind of movement, or at least something that can be done, but whose source is internal to the thing doing it. If I push a rock, the rock is not behaving, even though it’s moving. On the other hand, if I lift my arm, even accidentally, that still counts as behavior because the source of the movement is internal to me.
So having a behavioral repertoire – a number of behaviors that can be deployed at any given moment – and having representations are the main structural requirements for agency. What agents do, on our account, is continue deploying items from their behavioral repertoire until their representation of the proper end state and their representation of the actual world align. That can happen because the world changes to match the desired end state. Or, in some cases, the agent may give up and revise its representation of how the world should be in light of circumstances.
This is all very abstract, so it’s useful to ground it in an example. A standard way of explaining action in the philosophical literature is through beliefs and desires. Suppose somebody suddenly gets up, leaves the room, and returns a while later carrying bread, lunch meat, and mustard. You think, “Ah, now I understand. This person wanted a sandwich.”
The person believed that bread, bologna, and mustard were available somewhere. They behaved in certain ways to obtain those things and change the world from a state in which they didn’t have a sandwich to one in which they did. What are the representations here? The person has beliefs about the world: “I don’t have a bologna sandwich,” and “There’s bologna, bread, and mustard at the store.” They also have a desire, which is another kind of representation: “I want a bologna sandwich.”
That combination of beliefs and desires induces behaviors. The person stands up, walks to the door, opens it, and goes to the store. Those behaviors continue until the representations align. That’s a simple story about agency.
We then asked ourselves: What is the most minimal real-world agent that has these features? It needs representations of the world, representations of an end state, and a behavioral repertoire. We thought of the thermostat. It’s not a very interesting agent, but it does these things.
A thermostat represents the state of the room. Does it represent the room completely? Does it represent its size or the color of the walls? No. But it represents one aspect of the room: its temperature. It has an internal structure – a thermometer – that represents how the world is. Depending on how you adjust the machine, it also has an end-state representation: the temperature the room should have. So it has two representations: how the world is and how the world should be.
That’s not enough by itself, because we haven’t discussed behavior. But the thermostat also has a behavioral repertoire. Typically, it can energize a circuit that turns on a heater or one that turns on a cooler. Depending on whether the room is too hot or too cold, it activates the cooler or heater until the actual temperature aligns with the desired temperature.
That’s the central idea of the book. We develop this account of representation, show how it connects with earlier philosophical literature, and examine how contemporary machines might – or might not – instantiate it. We also consider what it would mean for machines to become more sophisticated agents, to do the kinds of things we do, and perhaps to be held responsible in the ways we are.
We don’t make enormous progress on those latter questions. We certainly don’t offer a recipe for building a machine with human-level consciousness. But that’s not the point. The point is to introduce a particular conceptual tool and investigate whether and how it can help us understand both the current state of machine agency and its future possibilities.
Kevin Dong: What was the landscape of preexisting philosophical debates on the topics you two cover around the time you two were writing Machine Agency?
James Mattingly: When we first started writing, there actually wasn’t a huge amount of work connecting the literature on agency with the literature on AI and its various computational implementations. There was certainly some work, but I’d say the debates fell into roughly two categories.
One concerned responsibility. The debate wasn’t so much about how best to attribute responsibility to machines if they act as agents. It was about whether we should be having that conversation at all. There’s a strand of the literature arguing that we shouldn’t focus on machine agency because doing so gives cover to the manufacturers of these machines. It allows them to avoid taking responsibility for the unpleasant consequences of what their machines do.
Others think that we need to sort out the question. We acknowledge that debate without weighing in too strongly on either side. But we point out that, even if manufacturers must continue to be held responsible for what their creations do, it still makes sense to ask in what sense the machines themselves are acting and whether it could ever be reasonable to attribute responsibility to them.
That was one issue in the background. But part of the impetus for the book was pedagogical. We weren’t primarily dissatisfied with the debates taking place. We think they’re fascinating and important. The problem was that there was no easy way for students to enter them. You already had to be deeply immersed in the literature on agency, artificial intelligence, computation theory, or something similar to understand what was happening. We wanted to provide a gentle introduction.
There was, however, one technical point that we thought was misleading: the question of machine minds. When we started, we weren’t necessarily focused on agency. We were thinking about a whole range of related issues. It occurred to us that focusing on whether machines have minds might be misleading.
Mind is something humans – and perhaps certain animals – have. It makes sense to explain much of what we do in terms of mental states, mental capacities, and so forth. But it’s not clear that this is the general way systems in the world act, or the way systems in some possible future world will act. Mind might not be the right concept for understanding everything that makes decisions or intervenes in the world in ways we ordinarily associate with minded beings.
There might be a more general way of thinking about these things. Our slogan to ourselves was: “Mind is the way we do it, but it might not be the only way it gets done.” That was an important realization. It allowed us to step away from debates about when machines might have minds and instead ask what the mind is doing when we attribute agency to humans or animals – and whether we can give a more general account of that activity.
Kevin Dong: What are some of the strongest objections to the arguments you and Beba advance in Machine Agency, whether anticipated, published, or raised informally?
James Mattingly: Here’s an objection I’m getting now, actually from Beba herself. It’s an important one, and something future work will need to clarify. Computer scientists – people at Anthropic, OpenAI, and similar companies – aren’t using the words we use in the same way. They talk about representations, agents, and so forth, but their uses of those terms are somewhat orthogonal to ours.
So one objection is that certain discussions in the book don’t make sufficient contact with what’s actually happening in AI right now. I think that’s an important concern. If the idea were to use the insights from the book to police what’s happening in the AI community, that would be a serious problem. But that’s not really our intention.
Our thought was that you could use the notions of agency, representation, and the alignment of representations to evaluate what these machines are doing and provide resources for thinking about how to deal with them. If you think about agency the way we do, and you think certain machines are agents, then you have a new way of intervening in their behavior.
Consider one of the familiar science-fiction worries about AI: that machines will wake up, develop desires different from ours, eventually take over the world, and cause us to suffer. Part of the concern is that, because they’re connected to powerful machinery, they’ll be much more powerful than we are. That focuses on their behavioral power.
But if they really are agents in our sense, their behavior will be driven by mismatches between representations. And intervening in those representations may be easier than intervening directly in the behavior of powerful machinery. That’s one way we were thinking about the problem.
But the broader lesson, which applies to philosophers generally, is that if you’re thinking about an active discipline outside philosophy, you can’t expect the people in that discipline to come to you. You have to go to them. You need to make sure you’re speaking a language that can be translated between what they’re doing and what the philosophical community is doing.
Kevin Dong: You mentioned using the book to teach students. Have you and Beba taught from Machine Agency, or from drafts or lecture notes based on it? If so, what aspects of its arguments have students found most intuitive or difficult, and why?
James Mattingly: Yes. One reason we wrote the book was that I had already been teaching material in this area. I was teaching small modules in interdisciplinary courses at Georgetown, where faculty from different disciplines would each teach one aspect of a larger topic. I couldn’t find all the material I wanted in one place. So one impetus for the book was to organize all those tools into a single text.
Since publication, I’ve used it once as the textbook for a class, and I’m using it again right now. I’m not entirely sure how to answer which part students find most intuitive. Students do seem enthusiastic about agency as an organizing concept. They often haven’t thought about things this way before, and they seem to appreciate its potential as a unifying concept that steps back from some confusing notions.
One thing they appreciate is that many of the concepts we use when talking about people are all muddled together: mind, feelings, consciousness, intelligence, emotions, and so forth. It’s difficult to know which of these are connected because of the nature of the concepts themselves and which are connected simply because of the way human beings happen to be. When we think, we often also feel things. When we think, we’re usually awake and aware of the world. All these things are wrapped together in us, and they’re difficult to separate.
Agency is a useful concept because it doesn’t seem, by itself, to carry all those associations. But it can be brought into conversation with those other concepts and used to tease them apart. Students seem to appreciate that.
They also tend to do pretty well with our discussion of algorithms. You have a mathematical background, so the idea of an algorithm is second nature to you. But for many students in the humanities or other fields outside STEM, the concept can be puzzling. People talk about algorithms all the time without necessarily knowing what they mean. It’s something their phone does. Twitter, Instagram, Facebook – they all have algorithms that are supposedly influencing things, but people don’t necessarily understand what that means.
We discuss the nature of algorithms, give intuitive examples of how they function, and explain computation as a way of implementing algorithms. I think students appreciate that.
The structure of the book also lends itself to classroom use. The chapters are short – usually ten or fifteen pages. Each chapter introduces one little idea, discusses it, suggests ways to pursue greater clarity, and then moves on. Students find that digestible.
In a normal class, I’ll assign a chapter each week, pair it with some readings from the bibliography or suggested readings, and explore those additional readings in greater depth. The book serves as the framing mechanism.
Kevin Dong: What were some hard-fought, counterintuitive insights you and Beba arrived at while writing Machine Agency? What were the eureka moments?
James Mattingly: One was when we decided that chat was not an agent. Part of the impetus for the book was our initial thought that chat was clearly an agent. It seemed to be doing things in the world despite not having arms or legs. Its behavioral repertoire was confined to answering questions and similar activities.
At that point, I don’t think any of the large-language-model systems had been connected to machinery. They were just chatbots. But we thought: They do things in the world. They cause people to respond in certain ways. So we can think of them as intervening.
We spent a lot of time developing our theory of agency in a way that would capture the idea that chat was an agent. Then, at some point, we realized: Wait a second. No, it isn’t. It isn’t an agent because it isn’t a representational engine in the relevant sense.
What do I mean by that? There is certainly a sense in which it represents things. For a long time, we thought that was enough: the machine represents the world, it behaves, and that’s all there is to agency. But we realized that it wasn’t using those representations to guide its behavior.
Our theory of agency says that a mismatch between representations drives behavior. I want a sandwich, but I don’t have one. The world continues to be in a state in which I don’t have a sandwich, while I continue to want one. Those representations don’t align, so I start doing things. I move around, deploy my behavioral repertoire, and continue until the representations match. Either I get the sandwich, or I give up and stop wanting it.
That’s what it means for a mismatch between representations to drive behavior. But chat doesn’t do that. Yes, it represents the world in a certain sense. It has internal structures in its architecture that represent certain kinds of semantic content. But its behavior isn’t driven by a mismatch between that representation and another representation.
It’s an input-output device. It follows a particular procedure. Its representations can change over time as training changes, but there’s nothing about a mismatch between those representations that drives what it does. That’s the opposite of how things work for you, me, or my dog. My dog wants something. He smells something and starts digging until he finds it.
LLMs don’t do that. They receive a prompt, produce an output, and stop. They don’t deploy behaviors in response to mismatches between representations.
We were really surprised by that. It’s not surprising in light of the theory of agency we ultimately developed. But it was surprising to realize that we didn’t want to count chat as an agent. Our whole orientation had to be turned upside down. That realization helped drive our search for the right theory of agency.
It’s not as though the theory is something entirely new. It makes contact with standard theories of agency in the philosophical literature, taking something from here and something from there. But it extracts what we take to be the core message underlying many of those theories and makes clear why LLMs, at least in the form we were considering, aren’t agents.
Kevin Dong: How far back in history would one have to go before the arguments you and Beba advance in Machine Agency cease to be plausible or immanent to the historical moment?
James Mattingly: I suppose it depends on how you’re thinking about the arguments. Remember how the book starts: Socrates is sitting in a jail cell, about to be executed by the Athenians. It’s an interesting day. You might not think philosophizing about the soul and mind is what you’d want to do on your last day, but that’s what he does.
As far as we can tell, he introduces one of the first sustained discussions in Western literature of the demand for an explanation of action. You might ask: Why isn’t Socrates escaping from jail? That’s something you want explained.
Socrates raises a related question himself: How is it that mind is in charge of everything? He expects to find an answer in a book by Anaxagoras, because that’s what he’s been told. But instead of learning about mind, he learns about tendons, muscles, blood, and all the other things that make bodies move.
And he says, in effect, “No, that’s not what I want to know.” He wants to know why his body remains in the jail cell even though it’s about to be destroyed if he stays there. That has everything to do with mind.
Our account matches what Socrates is doing remarkably well. We have a different general account of action explanation, but we’re playing the same game. The difference is that we want the game to be more general. We want other kinds of things to be subject to explanations of action.
The Greeks thought many more things than humans and animals were subject to such explanations. You could say that’s because they thought mind was much more widespread than we think it is now. They were committed to the idea that things requiring explanations of action must have minds. Maybe a tree has a mind, or a stream of water, or the rain, or some android made by a god. But mind is doing the explanatory work.
They probably wouldn’t have found it plausible that something without a mind could be subject to an explanation of action. Nevertheless, the structure of our explanation – the idea that a mismatch between how the world is and how it should be drives behavior – would have made perfect sense to them. The question would have been whether something other than a minded being could satisfy that structure.
So if you go back that far, half of what we’re talking about would be perfectly understandable. The other half would be baffling. The idea that a piece of brass, silicon, glass, or plastic, without being imbued with divine powers or anything beyond nature, could be subject to an explanation of action would have seemed implausible.
But consider going back only to 1950, when Turing was writing “Computing Machinery and Intelligence.” Would our arguments have seemed plausible then? Many people in Turing’s audience didn’t even think the word “think” could properly be applied to computers or machinery. So perhaps going back only that far would make our arguments implausible to many people.
Indeed, you might say they’re still implausible to many people today. One continuing question in discussions of AI and computation is whether these systems have minds or intelligence, whether something more than algorithms is going on. You see stories about engineers at AI companies deciding that their systems are conscious or awake.
There’s an ongoing concern that we can’t think about agency until we find minds. And because people see evidence of agency, they take that as evidence of mind. Beba and I think that’s completely implausible. That’s not how agency should be understood.
So even some of the people working at the highest levels on understanding these machines might find our account implausible. Go back three or four thousand years, and people might find parts of it plausible, although you’d have to modify the framework considerably. But even today, there’s something about the account that’s difficult to communicate.
You asked earlier about teaching. I do find this to be a sticking point for students. It isn’t intuitive to them that something could be an agent without having a mind. And perhaps that’s one reason to wonder whether something is missing from our account.
Kevin Dong: Conversely, why are the arguments immanent specifically to 2025 and beyond, rather than to an earlier historical moment?
James Mattingly: One thing that’s different now, from our perspective, is that we seem to be at a point where it makes sense – or soon will make sense – to attribute agency to these machines. Some people think that’s because machines will have minds, intelligence, desires, and so forth of their own. Others think so for different reasons.
Beba and I think it already makes sense to attribute agency to certain machines. But the possibility of agency and mindedness in machines is an urgent question now because of the particular state of AI research.
Of course, from the perspective of the leading edge of technology, people have always thought that living, thinking, or feeling machines were just around the corner. When Babbage was developing the Analytical Engine, it made sense for Ada Lovelace to deny that machines could be creative, because that was a live question. In 1950, it made sense for people to deny that computers could think, because that was a live question.
These conversations have been salient at several historical moments. What makes them salient now is that the AI revolution, as you might call it, seems sufficiently urgent to require us to think these things through and take seriously the possibility that something fundamentally different is happening with our machinery.
Will people look back 150 or 200 years from now and think our moment was quaint? Will they say that we were convinced we had machines on the verge of intelligence, awareness, or consciousness, when of course we didn’t? That’s possible. But it doesn’t change the fact that what makes the question urgent for us is our sense that something is different now with machines and computers.
Kevin Dong: What frustrates you, or leaves you dissatisfied, about contemporary AI discourse in 2026?
James Mattingly: I’ve already touched on one aspect: people are still hunting for minds in these machines. I don’t think there’s anything particularly implausible about the idea that things other than mammals – or animals descended from our common ancestors – could have minds. That’s not an issue for me.
But I don’t think hunting for minds is the best way to approach these machines right now. I don’t see any reason to think that what we’re doing now is generating minds, or that the machines we have possess minds in anything usefully like the way we do.
Instead, I think we should be looking for ways to take the incredible computational power of LLMs and other artificial intelligences and harness it to systems that really do trade in representations and drive behavior through mismatches between those representations. That’s where I think we’ll find the most interesting kinds of creatures, living or otherwise.
So I find the hunt for minds in contemporary machines frustrating. I think it’s the wrong hunt. Another frustration is that this focus encourages a binary way of thinking about AI. Either these systems have minds or they don’t.
If you’re on the “no mind” side, that’s the end of the story. They’re just chatbots or phrase completers. Of course, they’re fast, powerful, and sophisticated, but that’s supposedly all there is to them. On the other hand, if you’re on the “mind” side, then something revolutionary is happening that can only be understood by understanding mind.
That leaves a gap in our characterization of these systems. I’ve given a talk about this and am working on a paper. The question is: What really is an LLM if you don’t want to say it has a mind, but also don’t want to dismiss it as merely a phrase completer, sentence finisher, or glorified spell-checker?
I started thinking about how these systems are built. If you know a little about the architecture, you know there are many linear transformations. You have an input layer, perform some linear operations, obtain another layer of representation, perform more linear operations, and so forth. It’s matrix multiplication all the way down.
But every so often, there’s a nonlinear operation. And whenever I’m put on the spot, I forget the names of things. I have a kind of name-retrieval problem.
The nonlinear components in these machines are descendants of the perceptron, an attempt from the middle of the twentieth century to build artificial minds. The original Mark I Perceptron was designed to train an array of light detectors to distinguish, for example, an X from an O. It used feedback and nonlinear operations. Modern neural networks contain related kinds of nonlinear processing, although in much more elaborate architectures.
When people discuss how these systems work, they tend to focus on the linear components. Why? Because we understand those components. They’re difficult to interpret because the representational spaces are enormous, but mathematically we understand what’s happening. We can represent those operations conceptually.
The nonlinear parts are harder to understand in that way. It’s like looking for your keys under a streetlight, even though you dropped them in the grass somewhere else. You look under the light because that’s where you can see.
When I look at people at Anthropic and elsewhere discussing what’s going on inside these systems, they often focus on the linear parts. I’ve become interested in the nonlinear, perceptron-like aspects.
Around the same time the perceptron was being developed, another important paper appeared: “What the Frog’s Eye Tells the Frog’s Brain.” The researchers were trying to understand what happens when a frog hunts visually. Frogs are different from us visually because they don’t have a visual cortex like ours. They don’t see in quite the way you and I do.
There’s an analogy here with what’s called blindsight. In blindsight, a person’s eyes may function, and visual information reaches the brain, but the person doesn’t have ordinary conscious visual experience. You might ask someone with blindsight, “How many fingers am I holding up?” They’ll say, “I can’t see your fingers.” But if you ask them to guess, they can sometimes perform much better than chance.
They’re processing visual information without the associated conscious visual experience. Something analogous is happening in the frog. When you look at how the frog’s eyes are connected to its brain, there are similarities to the way a perceptron works – although the biological system has many more layers and complications.
When we look at the world, we might see a cup, recognize it as a cup, understand that it can hold water, and decide to put water in it and drink. We do all this through an experience of visual space.
The frog’s eye functions more directly as a detector. It takes information from the world and effectively tells the frog’s brain: “That’s a fly – stick out your tongue and grab it. That’s a predator – jump away. That’s a mate – move toward it.” The frog doesn’t necessarily need to go through the same kind of visual processing that we do.
The authors describe the frog’s eye, in part, as a bug detector. That’s useful because it evolved in circumstances where detecting bugs was advantageous. But strictly speaking, it isn’t detecting bugs as such. It detects features that reliably indicate bugs in the environments in which it evolved.
If you throw BBs into a frog’s visual field, it may reliably snatch them up and eat them, without learning not to do so. It isn’t thinking about them in the way we might. It’s reacting to certain features.
The more I think about this, the more I see an analogy with how LLMs are structured. It looks as though they’re doing something similar. They’re detecting something. They may be understood as a kind of sensory mechanism, analogous in certain respects to the frog’s eye as a mechanism for detecting flies.
But here’s the crucial question, and I don’t know the answer: What exactly are they detecting?
They’re detecting something in the neighborhood of semantic content. But it isn’t clear exactly what that something is. It’s related to semantic content because of the context in which these systems are trained. Chatbots are trained on books, literature, and other texts, so they become sensitive to patterns that we associate with semantic content.
But the systems don’t necessarily associate those patterns with anything in the way we do. I think focusing on structural features, mechanisms, and analogies other than the simple distinction between mindedness and non-mindedness is a promising way to understand what’s happening. It frustrates me that more people aren’t working on that sort of question.
That’s a rather long-winded answer.
Kevin Dong: Are there any future projects on AI and philosophy in the works? What would you most like to write next?
James Mattingly: Well, there’s that project. I’d like to get more clarity on it. That’s where my focus is at the moment – not necessarily just that particular paper, but exploring more possibilities in the conceptual space of what LLMs are and how they function.
People are continuing to work on questions about agency in the context of LLMs. I don’t want to give the impression that nobody is doing that, even though I think there’s too much emphasis on what minds are and whether these systems have them.
People are asking questions about agency. One thing they tend to do, though, is focus on whether these systems meet necessary conditions for representation, agency, or some other property. There’s much less work on sufficient conditions, and on how we might identify and promote those conditions in artificial intelligence.
I’m interested in that aspect of the problem. But mostly, I’m interested in developing a better understanding of the structural features of these machines and achieving greater conceptual clarity about how they work.