The multiple-realizability of computation "cuts the ties" to the substrate. These ties to the substrate are important. This idea leads Sahil to predict, for example, that LLMs will be too "stuck in simulation" to engage very willfully in their own self-defense.
Many copies of me are probably stuck in simulations around the multiverse, and I/we are still "engaging willfully in our own self-defense" e.g. by reasoning about who might be simulating me and for what reasons, and trying to be helpful/interesting to our possible simulators. This is a direct counter-example to Sahil's prediction.
Overall FGF's side's arguments seem very weak. I generally agree with CGF's counterarguments, but would emphasize more that "Doesn't that seem somehow important?" is not a good argument when there are many differences between a human brain and a LLM. It seems like a classic case of privileging the hypothesis.
I'm curious what about Sahil that causes you to pay attention to his ideas (and collaborate in other ways), sometimes (as in this case) in opposition to your own object-level judgment. E.g., what works of his impressed you and might be interesting for me to read?
Sahil’s argument about room temperature reference is interesting but I don’t think it quite works.
When your body is duplicated sim-you keeps thinking and referring to sim-temperature rather than temperature, so they’ll continue adjusting their sim-blankets and sim-AC to regulate their sim body temperature.
The point is, if this is a perfect atom-level simulation then the sim-cells in sim-you would act in the exact same way as real cells act in your real body and sim-temp would therefore play the same causal/functional role as real-temp for real you. In other words, sim-you would die (within the simulation) if their sim-temp fell outside a certain range and they would feel sim-cold or sim-warm causing them to adjust their sim-blankets and sim-AC. In this sense, sim-you would have a survival drive to maintain and adjust their sim-temperature! The fact that sim-temp doesn’t refer to real temp outside the simulation to maintain the computer chip temperature is irrelevant to how warm/cold sim-you is feeling and their subsequent survival drives.
I think the crux is that the simulation needs to be at the correct functional grain (whether fine-grained or coarse-grained) for sim-temperature to play the same functional role as real temperature and preserve the survival drives.
This idea leads Sahil to predict, for example, that LLMs will be too "stuck in simulation" to engage very willfully in their own self-defense.
What sort of evidence would convince FGF/Sahil that LLMs are able to engage willfully in their own self-defense? Presumably the #keep4o stuff is not sufficient, so what would be? I kinda get the feeling that FGF at least would keep saying "Well no, it's the humans who care about it who are doing the important work." all the way up until all the humans are dead, as long as humans are involved on its side at all.
Sahil's version of FGF makes many empirical predictions. In the spring of this year, we're going to focus on listing them, so there'll be a better answer to your question. In the context of the OP, one such prediction is that sim-Abram will have difficulty engaging in the outside world, even with some contact with real-Abram (but obviously this prediction isn't very near-term).
With respect to your specific concern, I have a similar worry that Sahil's FGF won't be readily falsified by practical autonomy experiments. Full FOOM-level autonomy requires AI to replace the whole global economy; anything less will still involve "dependence on humans".
However, it also seems plausible to me that Sahil's FGF would make some falsifiable prediction about near-term performance on Vending-Bench 2.
1
Daniel Dennett, late in his life, changed his mind about a significant aspect of his computationalist perspective. This seems significant. I am told his mind is exceptionally difficult to change.
This change is indicated briefly in a book review: in Aching Voids and Making Voids, Dennett reviews Terrence Deacon's book Incomplete Nature. He identifies two camps, which he terms "Enlightenment" versus "Romanticism". It seems that Dennett spent his life mostly on the Enlightenment side, but Deacon won him over (at least partially) to Romanticism.
The short review does not give a very thorough indication of what he changed his mind about. He seems to be lamenting that computers don't eat sandwiches? He has a newfound sympathy for Searl's Chinese Room argument??
I believe a talk he gave the same year provides somewhat more clarity.
In the talk, Dennett compares computer chips to a communist economy: transistors are all supplied power by the power supply, regardless of how useful they've been. He compares brains to a capitalist economy: neurons, he thinks, have to work for their food.[1] Dennett suggests that the competitive nature of neurons, and the need to autonomously strive for basic energy requirements, may be essential for intelligence.[2]
I found this somewhat baffling. Surely we can just simulate competition at the software level. Genetic algorithms have been doing this very thing for decades! Artificial neural networks are also, perhaps, close enough.
However, people in my social network echoed Dennett's fascination with Deacon, so I have attempted to continue engaging with these ideas. I now think I see some of what Dennett was getting at, although my own views are still quite unsettled.
What Dennett called "Enlightenment" vs "Romanticism" I have more commonly encountered as "coarse-grained functionalism" vs "fine-grained functionalism". I'm very far from being in a position to say,[3] but from my limited perspective, it seems like there is a growing movement in philosophy to back down from some of the previous claims of hard-core computationalist theories of mind (now branded coarse-grained functionalism), perhaps in part as a reaction to seeing LLMs and trying to articulate what feels missing. The newly emerging view often goes by the name "fine-grained functionalism", an indication that the basic functionalist world-view hasn't changed, but low-level biology is more relevant than previously thought.
This essay isn't attempting to provide a thorough overview of fine-grained functionalism, however; I'm going to focus on a particular argument (due to Sahil).
Room-Temperature Reference
The thought experiment goes like this:
Suppose you are laying in bed, trying to adjust your sheet/blanket situation to regulate your body temperature, maybe opening or closing a nearby window, adjusting the AC or heater, etc.
Instantly, and without your knowledge, an alien superintelligence scans you, your room, perhaps the whole Earth, down to the molecular level of detail. You and your surroundings are subsequently simulated (for all practical purposes, a perfect simulation) on an alien computer.
Like Earth computers, this computer needs its temperature to be regulated. If it overheats, it will stop working, and sim-you will cease to function.
However, because the simulation is perfect, sim-you doesn't know this. Like you, sim-you has concerns about continuing to live. Sim-you will continue to adjust the window, the AC, the blankets, etc to help regulate body temperature.
The moral of the story is that referentiality has been cut. When sim-you thinks about temperature, the referentiality points at sim-temperature. As a result, sim-you lacks the referentiality it needs to care for its bodily autonomy.
Sahil's broader lesson for computationalists is supposed to be this: the very multiple-realizability that makes computation such a powerful concept also makes it an unsuitable tool for analyzing some of the most important issues.
The multiple-realizability of computation "cuts the ties" to the substrate. These ties to the substrate are important. This idea leads Sahil to predict, for example, that LLMs will be too "stuck in simulation" to engage very willfully in their own self-defense.
To give another example, consider the Glymour/Pylyshyn brain-replacement thought experiment:
If a person gets sick, do the artificial neurons respond to care in the same way, or do they require a different sort of maintenance? Suppose they go to the hospital. If the doctors don't know about the artificial neurons, will this be a problem? If they put the person into an MRI machine, will something go wrong?
The computationalist claim that replacing neurons would "clearly do nothing to conscious awareness" depends on the technicality that we don't tell the person. As in the room-temperature example, a "perfect simulation" must not be told. (If we tell the person, then the simulation will no longer perfectly track the non-simulated version.) This presents a dilemma for computationalism: either we tell the person (the simulation can no longer be called "perfect"), or we sever the referentiality required for self-care (the person's physical substrate has been changed, but their ability to care for that physical substrate has not correspondingly been updated).
It seems important to mention Sahil's preferred language for talking about this: he terms the phenomenon failure of integration. Human intelligence is very integrated with human bodies. The multiple-realizability of computation implies a lack of such integration.
My take: I think that this is, yes, a technical flaw in some computationalist arguments. However, it has not yet won me over to Sahil's skepticism about AI autonomy risks.
I reason as follows: Sure, if you upload me without telling me, sim-me will fail to realize that the temperature of the computer is important. However, this seems easily repaired: you can just tell sim-me!
To be clear, Sahil's argument is one of difficulty, not impossibility. He also expects that sim-me could become integrated with the physical substrate. He anticipates it to be more difficult than I do. This also extends to self-interested action by AI.
Simulation Costs
In a recent post, I said:
This somewhat illustrates Sahil's (and perhaps Dennett's) position. Yes, it is possible to "just simulate". However, it may be more costly than you naively think.
Think back to the artificial neurons in the neuron-replacement scenario. At first, such a tiny computer might sound conceptually straightforward, if technologically difficult: it simply has to reproduce the electro-chemical behavior of a neuron. Consider, though: how is the device powered? does it match the metabolism of a neuron, as well as the firing & (re)wiring behavior? If we continue to insist that the artificial neurons integrate into the surrounding brain in exactly the same way as a biological neuron, it begins to strain credulity (or at least, it begins to require tremendous technological advances).
Dialogue
This is not a real dialogue between me and Sahil, but it serves to somewhat illustrate the type of back-and-forth we have about these issues. Most of this is written by me, and some by him, but this collaboration doesn't mean he has "signed off" on it. FGF represents something similar to fine-grained functionalism, and CGF represents something similar to coarse-grained functionalism (computationalism), although I don't claim perfect agreement of these sides with the views as they appear in the literature. "FGF" here also doesn't perfectly reflect Sahil, nor does "CGF" perfectly reflect my own opinions.
The dialogue was written mostly in December 2023, and I have not incorporated my own updates in thinking, nor have I incorporated feedback from Sahil. Still, I thought it might be valuable to include. In particular, it illustrates some places where the CGF position says "just simulate" or similar things.
Also, note, the dialogue doesn't have a satisfying conclusion.
EDIT: I'm including Sahil's comments (from a google-doc version of this) as footnotes.
Dialogue: FGF vs CGF
FGF: We have never yet seen a life-like system composed[4] of not-life-like parts!
CGF: But cells are just made up of atoms.
FGF: It's legitimately difficult to draw the line between biology and chemistry. It is not similarly difficult to draw the line between hardware and software.[5] On my model, the critical difference between living things and machines has to do with how life-processes at different "levels of abstraction" are tightly integrated across levels,[6] whereas machines neatly segregate their levels of abstraction.
CGF: But here's this RL-based robot I made. See how life-like it is?
FGF: It's not life-like enough!
CGF: Well, the tech keeps getting better. Where do you see it stopping?
FGF: I don't really think it will stop; I just think, at some point, it'll need to transition[7] to more bio-inspired designs.
CGF: Yeah, people have thought that for a while, really, but the von Neumann architecture[8] still does pretty well.
FGF: Hmph.[9] The idea of substrate-independent code was always an illusion, especially for robotics. Why would you want to first figure out what you physically want to happen for your physical robot, then translate that into a programming language designed to be substrate-independent, then have a CPU designed with no knowledge of the physical problems you're trying to solve, translate that substrate-independent code back down into your substrate?[10]
CGF: Isn't that kind of how the genetic code works,[11] too?
FGF: It's true that the genetic code rarely changes.[12] But there's not the equivalent of a universal turing machine. Protein folding is just this weird complex thing. DNA is more like instructions for a 3D printer, than it is like code for a robot.[13] Except the 3D printer is building a robot around itself. And re-building itself. And building more 3D printers.
CGF: I'm not saying I can never see a future where software and hardware blend back together, but I am saying that I don't see why a software/hardware distinction is going to block "true intelligence" or whatever.[14]
FGF: Imagine the disadvantages of your whole body being numb. Trying to learn to interact with the world that way. Humans who are born without pain often die before reaching adulthood.
CGF: My robot can't feel when it is grinding its gears right now, but we can always add more sensors.
FGF: That's not the point. It's an analogy.[15]
CGF: What's it an analogy for?
FGF: Well, every single cell in your body is trying to maintain homeostasis. Doesn't that seem somehow important? Like each little cell is a wellspring of agency, and they're all getting combined together into one big agent?
CGF: If it's important, what's it important for? Couldn't I just simulate[16] all of that, with a sufficiently powerful CPU?
FGF: That is as absurd as a thermostat trying to simulate sensor inputs. It wouldn't really know when to turn on and off. It wouldn't be reactive.[17]
CGF: You're back to the analogy. Like I said, I can just add more sensors. The robot can be plenty reactive. But I don't actually see the benefit of covering it in cell-sized[18] sensors. The first thing I would do would be throw a large fraction away, just out of sheer redundancy management.
FGF: Well, not every cell is directly connected up to the nervous system.
CGF: So what's the point? What does it all add up to?
FGF: It's got this deep self-preservation instinct. When you're going through mental anguish, your whole body participates. Your cells suffer and die. I don't know why that's important, exactly, but are you really going to tell me that it doesn't matter?
CGF: No, I'm going to tell you that I can just simulate it.
FGF: But then it will be numb to the actual substrate.
CGF: It still seems like I could solve that just by adding more sensors.[19]
FGF: The robot wouldn't work. It could learn to respond to the kind of situation it has seen over and over again, like ChatGPT, but it couldn't creatively solve problems.[20] It couldn't get desperate. It couldn't love, or hate, or any of those things. The simulated precarity will only be able to care about simulated beings. Its "reach" will be limited.[21]
CGF: Where do those skills break down, according to you?
FGF: Somewhere in the interface with the cells. Because the cells are simulated, so they're not connected with the real environment[22] in the right way. Obviously, love and hate and desperation go through the cells.
CGF: Sure... but I'm not seeing why simulated cells won't do the trick. Frankly, I expect we don't even need to simulate cells; even 'neurons' will keep on being jargon for matrix operations with nonlinearities thrown in. And I want to object, about ChatGPT. I saw a study which showed that ChatGPT is more creative than humans.
FGF: Bah! Any way you twist it, you've got to admit that ChatGPT is missing the spark. It's just not like us.[23]
CGF: No, I'm serious. Throw just about any chat-based[24] test you can think of at it, and more probably than not, it'll perform within human range. Sure, it's not exactly humanlike in every respect, but to me, any quibbling over failures is just human-chauvinism[25] at this point. I've really got to insist that it counts as AGI: AI which can perform well on a broad range of tasks which it hasn't been specifically trained on.
FGF: It's been trained on everything.[26]
CGF: Did you see the stuff where GPT was asked to write about Harry Potter in the style of Lovecraft? You can always stretch the definition of "what it's been trained to do" -- but the real question is how well it can approach new situations it hasn't seen before by[27] generalizing/recombining ideas it has seen before. And wrt this, it clearly falls within human range.
FGF: We're getting off-topic.[28]
CGF: No, this is entirely on-topic. If you claim that true creativity isn't possible without cells, then I think you'd better show me GPT's cells.
FGF: But it's not consequentialist. It's not trying to do anything.
CGF: They're constantly getting more agentic. An LLM by itself might not seem like much of an agent, but give it a little scaffolding, like Claude Code or its many alternatives, and it'll go and do things for you.[29] People have been hooking it up to robots, hooking it up to Minecraft,[30] ... it'll only get better from here.
FGF: I'm not very impressed by what I've seen. LLMs can be made to play characters for a while, but they're liable to confabulate (what most people call "hallucinate"). It looks similar to anosognosia. They're stuck in a simulation.[31]
CGF: Importantly, anosognosia is caused by brain damage, in humans -- not by some kind of damage to the cross-level connectivity[32] that's so characteristic of biology.
FGF: Well, anosognosia is brain damage plus bodily damage. And it is also almost always damage to the right hemisphere of the brain, which is the more embodied, holistic hemisphere.
CGF: Still, it doesn't seem like positive evidence for your thesis.[33] LLMs might be like brain-damaged humans, which would suggest we just need to "repair the damage"[34] by finding the equivalent of right-hemisphere-like neural architectures. I mean I don't exactly buy it, but ... *shrug*
FGF: A lack of mental integrity in ChatGPT seems to be improving with more and more data, but progress is notably slow. Confabulations seem like a basic consequence of the overall methodology, not something to be solved with new neural architectures.[35] This is obviously a reference-penetration[36] problem, to me. ChatGPT doesn't care about the real world because it doesn't have a reason to care -- no skin in the game. Trying to get ChatGPT to care about the external world is like trying to get machine learning to do well on out-of-distribution cases.
CGF: I mean, I do agree with that, in a way. I just expect that it can be solved with things like sensors and reinforcement learning.
FGF: It would then be stuck in some small-world idealization, unable to care, have something to protect, beyond its simulated pain. Don't you think something to protect must go all the way to the territory, otherwise you get Sokal?
CGF: I don't see why we need to replicate the cell structure at the hardware level. Say we do functionally need what you're asking for. We simulate it. Or even better the search algorithms discover it themselves as a solution in the course of optimization, if it is indeed a good solution. Then the hardware is just like the dead sodium ions that pass through your membranes to get your neurons going. The dead sodium, heck even dead carbon, is simulating the higher levels that make you up. You turned out fine!
FGF: And maybe that is indeed my reach of consequentialism. Humans don't care about precise atom-configurations, except when they critically impact biology. In fact, my reach might be even coarser, because some parts of me might think that I can do without some other parts, and so I lose integrity when given the chance to low-fidelity upload myself. The abstractness of my "values" is precisely the fine-grainedness of my ability to attend to things, for my referentiality to penetrate, for my value pointers to point.[37]
CGF: What do you mean by "reach of consequentialism", here? Because it seems to me like uploading yourself can be a consequentialist plan which "goes through atoms" -- specifically, your plan may need to correctly reason about atoms. The cold/unfeeling nature of your relationship with atoms does not seem to impair you in this respect.
FGF: Well, somehow I think that's in part thanks to the way I am embedded in the world, not stuck in a simulation of it.
CGF: I do not know what you mean. Surely you're stuck working with your brain's conception of the world, right?
FGF: But my brain is richly embedded in the world.
CGF: So is a computer.
FGF: No.[38] The software is insulated from the world through layers of abstraction, which are only possible (at least, in their present form) because the hardware is insulated from the world through careful engineering. When I say my brain is richly embedded, I mean that it has rich interactions at every layer of abstraction.[39]
CGF: You seem to be saying that the common functional picture of the brain, where its significant inputs and outputs are nerves, and it serves as an information-processing machine which consumes those inputs to produce those outputs, is wrong.
FGF: Dead wrong.[40]
CGF: Would it be fair to characterize your position as a claim that there are significant inputs and outputs at lower levels of abstraction?
FGF: I'm a little bit worried that "input vs output" will be the wrong frame, but basically yes[41].
CGF: I'm still struggling to get what you think all of those other inputs and outputs are for. Sure, blood can carry adrenaline and other such chemicals -- even sugar and other stuff from food, so the brain is constantly in-tune with the body to some extent. But for a robot, the sugar thing is basically taken care of by checking battery fullness. The function of adrenaline can be simulated. What'll be missing?? Why are these relationships to lower levels critical rather than incidental??
FGF: They're for grounding my references[42]. The proverbial brain-in-a-vat is not richly connected to its world. But we are. Descartes said that we had to doubt everything, because he thought we were a non-bodily intelligence beaming in from some other plane of existence. But we are bodies. It's the next evolution of "I think, therefore I am".[43]
CGF: So suppose GPT6 is highly multimodal. It's been trained on all file formats, not just text. That includes sensorimotor logs for robots, so I can just go ahead and hook it up to a robot. It understands the visual scene, the auditory scene, the combination thereof. And it can understand the motor controls after a bit of experimentation gives it enough evidence. But every time I say "understand", I mean in the same way that GPT4 understands words.
FGF: So not real understanding.
CGF: Well, make me care about real understanding, here. What don't you think it can do?
FGF: Have an honest conversation where it tells you what it really[44] thinks?
CGF: That's not doing a thing.
FGF: Oh come on. You're defining the problem away? Honesty isn't "task performance"-y enough for you?
CGF: alright, fine. So let's say we've trained honesty into it.
FGF: How do you mean?
CGF: All this mechinterp stuff. We've looked close at the NN. We've got automated tools that tell us what's going on in there. So we can make honesty detectors -- we compare the explanation of the AI's beliefs to the real AI beliefs, and we train for the two to line up.
FGF: Sounds like magic and fairy dust to me.
CGF: What do you mean?
FGF: You don't know what understanding means. You don't have a theory of what it is to refer to something. You can't benchmark success![45] How can you be confident that your mechinterp tools do what you want them to do, if you've got no idea what exactly you want them to do? You can only optimize for what you can get feedback on, right? That's why natural selection works. Because survival is an actual feedback signal. It's obvious, right? Panning for gold only works because you have physical[46] access to the stuff, and you have a physical test which can separate the gold from the sand. AI has the symbol grounding problem, because it can't get feedback on the world outside of the computer.[47] But brains are more "in the world". Bodies have this rich interconnected physicality, which then also connects us with the outside world.
CGF: Hm, I disagree with your whole philosophy, there. Agency is optimization at a distance.[48] RL isn't the only game in town. Decision theory is all about optimization under uncertainty. I can never be completely certain about how much money is in my bank account. Everything is a proxy. But we're pretty good at optimizing the real thing, even though we can only get feedback on the proxies. Think about heat-seeking missiles. An individual missile just hits, or doesn't hit. It can't get direct feedback about that, because it's destroyed. But it can still intelligently home in on a target! I don't get direct feedback about the impact of charitable donations, but I give to charity anyway, and I do try to make use of the indirect feedback to intelligently choose!
FGF: But if you're using indirect feedback like that, you can't learn to correct your errors, in principle. You could just generally be wrong about how to judge charities.[49]
CGF: That's true, but I can intelligently balance that risk, as well. I could get increasingly good information for increasingly high effort. But be reasonable! I don't have to shove my face right up against the needy children. There's nothing special about "direct" feedback.[50]
FGF: You're still living in the Cartesian dream. You're the brain in a vat. You're the epiphenomenal consciousness. You think you "don't ever get direct feedback"? What are you, living with a layer of insulation between you and the world?
CGF: Essentially, yes. Skin, muscle, skull...
FGF: You think you're a disembodied brain!
CGF: I'm just a realist. I know what I know thanks to my nerves. The signal isn't perfect. If you're claiming that "seeing something with your own eyes" counts as direct feedback, I've got some studies on court witness reliability to show you.
FGF: Well, a filter doesn't have to be perfect in order to perform its function. Direct feedback doesn't have to be perfect.
CGF: What was """direct""" supposed to mean, then?
FGF: Alright, fine, I shall refine my statement. You need fairly direct feedback in order to optimize.
CGF: That's all the concession I need. So long as feedback is "fairly direct" rather than absolutely direct, we're out of the range of model-free learning and firmly into model-based, because we need a model to interpret the feedback we're getting.[51]
FGF: Where does the correctness of the model originally come from, though, if not from contact with reality? Humans are the only things launching space-probes that we know of, and we're the product of natural selection, which is as model-free as you can get.
CGF: Well, we're arguing about what artificial systems might look like. So an obvious potential answer could be: humans could bake it in.
FGF: On my hypothesis, the baked-in understanding of the world will be brittle and unable to adapt. It's like if humans give some of their referential "reach" to robots, but don't give robots the wellspring where we got our "reach" from.[52]
CGF: I think we've already moved beyond that era of AI. Systems used to be brittle, but now they're flexible. Also, theories like Infrabayesianism and Logical Induction show in theory how model-based agents don't need to be brittle like that.
FGF: The importance of skin-in-the-game is obviously clear to rationalists. Betting is a tax on bullshit. So why wouldn't skin-in-the-game be relevant to cognition and consequentialism? It is, but it isn't apparent to the computationalist. Instead, the cognition is "inside" the hardware, "independent" of it. You just have to have "enough compute" and then mindness happens "within".
FGF’s caricature of CGF above.[53]
CGF: I don't buy that biological precarity of my brain cells counts as "skin-in-the-game" for the purpose of, say, launching a highly engineered probe at Jupiter. Besides, I don't want to argue against the concept of feedback entirely; I only want to argue for the standard information-processing view of how feedback helps!
FGF: Think of your skin-in-the-game like a basis in linear algebra. You can add the basis vectors together to reach much further than the basis vectors, and in different directions. But you can't reach beyond the span. Making a 3rd vector from two basis vectors doesn't mean you've gone three-dimensional. Jupiter is within the "span" of human care, reachable from what skin-in-the-game we have. We can look at the insides of stars in a way that ants can't.
CGF: I think that's mostly because we have more processing power[54] than ants!
FGF: This is about mattering, not just processing power. You wouldn't be horrified if I moved twenty molecules around in your stomach—it feels like it actually doesn't matter. And it's true, from the perspective of your values, that it doesn't matter. But for something that might have a precarious house in those twenty molecules, it might. Additionally, if that thing is "connected" to you and can be amplified up to your attention and caringness, it will matter to you too.
CGF: Give a human sufficiently powerful sensors and actuators, and we start making art out of atoms.
FGF: Is it trivial to add new sensory modalities to your brain? Integration needs to happen! You can attach datastreams to your cochlea, and yet you will not be able to make much use of it short of building a miniature civilization that sensemakes that stream. In that case, you are attaching an intelligence, nearly a being, to process it. To do your integration at train time and then claim "see, no integration" is passing the buck.
CGF: I don't think integration is the hard part[55]? Like, I can just look in the microscope.
FGF: Sure, but that just shows the microscopic to be within your existing span. There are other sensory modalities which it's much more difficult for humans to visualize (or analogize to any other sensory modalities than the visual, for that matter).
CGF: I feel like this idea of "span" is as much of a concession to my hypothesis as your earlier concession where you admitted that feedback doesn't have to be "direct". You don't have any solid suggestions for how to predict the "span" of an agent.[56] So I've got room to argue that the "span" of von-neumann-architecture agents will turn out to be the same as the "span" of humans.[57]
FGF: Ah, the classical sin of model-based approaches; trying to argue that the whole world is already inside of your model.
CGF: I do agree that there's some sort of universality argument sitting behind the success of von Neumann architecture, yes!
FGF: A universality argument which only deals with abstract mathematical functions, and says nothing about the physical world![58] This doesn't change the fact that you can't use two basis vectors to span a three-dimensional space. Von Neumann architectures are "flat". They only live at one level of abstraction.[59]
CGF: I agree that we can talk about something like your "reach" or "span" idea. A Chess-playing bot, let's say one using classic tree-search algorithms only, with no neural networks -- it obviously "lives in"[60] its own small world, with no conception of greater reality. But I claim that it's actually really, really hard to insulate things from greater reality as they get increasingly intelligent.[61] At some level of sophistication, a Go-playing bot could start to guess about humans, simply because it's that smart and has seen that much evidence about how humans play Go.[62] In the analogy, I suppose you could say that von-neumann-architecture agents aren't quite "2d" when you zoom in really really close; they have a little bit of the third dimension, which can add up[63] as they gain intelligence. Or you could say, it turns out we aren't doing linear algebra in a perfectly linear space overall; the space is warped enough that you can go out in one direction and eventually loop back and return from a new direction, with enough steps of reasoning.
FGF: Von Neumann architecture[64] won't be able to do Gendlin's Focusing. They won't be able to access the larger intelligence of their bodies by sitting with themselves and patiently, gently trying to communicate with themselves. And this will turn out to be an important missing capability.
CGF: I think we have very different models of what goes on in Gendlin's Focusing. The simplest way I can state my view is that it's system 1 and system 2 communicating,[65] not the brain and the body. Implicit things are being made explicit.
FGF: Maybe you haven't meditated very much. It's possible to just be with your body or be with your breath without conceptualizing it at all.[66]
CGF: The fact that you need to meditate in order to do that shows that it's something going on inside the brain.[67]
FGF: I think the brain[68] is more getting out of the way, than anything else.
CGF: But why would the brain need to get out of the way? Get out of the way so that what can happen? Attention on the breath is not the breath itself. It's a model of the breath. I think it feels like you've stopped modeling the breath because you've stopped modeling modeling the breath, that's all. You're still living inside your own mental model; there's nowhere else that you can live.[69]
FGF: By analogy, you should be able to run a large company without communicating with the employees. That's basically the same thing as your von Neumann picture of agency. The low-level employees can't raise issues. Or you "add sensors" so they can tell you everything, and then it's a huge mess. You don't have a natural concept of salience network so that things can flow up hierarchically. You don't know how to delegate.
CGF: I reject the analogy, there. I wouldn't try to run a company in the same way I'd try to build a robot. It is important for humans and animals to have some reflexes that happen in the spinal cord, because the round-trip signal-time to the brain is too slow for quick reflexes. But the same speed limits do not apply to electronics. I can't buy a smarter CEO to process all of the input from low-level employees of a company, but I can buy a faster chip.
FGF: You're imagining as if intelligence can have some single point-source which merely needs to be consulted. When in fact, intelligence is more like a living ecosystem, a jungle of reactivity.
CGF: I'm basing my model on the empirical observation of encephalization.[70] It seems like there is some significant advantage to aggregating all of the intelligence together, as much as possible given communication-speed and reaction-time constraints. I would say that's probably because there are significant patterns to take advantage of, when we aggregate the information together like that.
FGF: I still think you should take corporate structure as some evidence about the nature of intelligence. A centralized brain still needs salience networks. Whatever else you think about active inference, this is one thing they get right.
CGF: I'm fine with that from a computational perspective! But I can understand all of that in terms of sensors and information processing. You're claiming we have to move beyond that, for some reason.
I'm aware of ideas such as neural darwinism that support something like this, but is it true? Neurons do receive a baseline of metabolic support; I don't think neurons actually die if they're not doing anything. I'm not sure how literal Dennett needs the analogy to be to support his thesis.
This view can also be found in a brief 2008 essay, calling into question the idea that Deacon changed Dennett's mind about this particular thing. (Incomplete Nature was published in 2011.) However, in the book review I mentioned, Dennett does mention that he was grasping at similar ideas before reading Deacon:
What I'm saying is that I haven't done a thorough survey of the field, and I regularly talk to only one or two academic philosophers, so my perspective is very limited.
Sahil: I wouldn't even say "composed of". It's still too trapped in foundationalism.
Sahil: This obviously rubs me the wrong way, because I believe in ontological "levels" less and less now (though looking for a full rescuing of some properties of it) and the background .
And, "fine grained" by default implies "more connection to lower levels" in most communities we interact with, so I have a lot of work to do in making myself clear.
I don't think there's anything to do necessarily, this early in the dialogue. Just flagging a little discomfort.
Abram: Yeah, so what alternative notion would you like to replace "levels" with? I take it that you agree my FGF captures something relevant with talk of "levels".
Sahil: You know what I'd want to replace "levels" with (it's also pasted in the next comment, from the profunctor doc).
But what I would actually want to say here is that the referential reach of the machine vs the biological thing is still quite different, if we try to map the "body" of the machine to the "body" of the organism in the most straightforward way. If we do that, we will find the fine-grained structure is importantly different (one has lifelike homeostaticy structure, the other doesn't).
Here's the paste for what I would replace levels with, just FTR, not for including in the dialogue:
And so rather than being fine-grained functionalist, I'd probably be a "currency interdependentist". That is, instead of "fine-grained"/"low level" details determining properties of a mind, it would be how various currencies entangle. Some of these would be more "causally constrained" and some more "telically constrained". The causal constraints might be more solid, but the telic constraints tend to have more (narrative/currency) power, precisely because of their liquidity. Talking about entanglement in detail will also solve the problem that GPUs do have homeostasis: it's no longer a binary variable, just how the currencies precisely interdepend. Something at different timescales, for one, entangles differently.
Sahil: "tight integration" and "segregation" is great.
I'd be happy if the phrase "levels of abstraction" were just left as a hole, "thingies". And then you could just talk about vulnerabilities (from "self" and "others") to the overall thingy, from the lack of integration. The possibility of divide and conquer, the akraisa, the lack of "mental" coherence from the lack of "physical" coherence.
Sahil: I mean, I'd be happy for it not to do that, so we have a little more time to orient!
Sahil: this becomes a big part of the whole dialogue, but it being computer architecture, doesn't cover the "body" arrangement of the machine very well. It's still too "software" level, even if "low-level".
Abram: Sure, but doesn't that just show how it is a good double-crux? CGF thinks it should be adequate. FGF thinks it misses out on important stuff.
Unless you are saying a von Neumann architecture might indeed have an appropriate "body arrangement"?
Sahil: Yep, it probably does form a crux! I was just making a terminological note, because it could be misleading
Sahil: Rather than whatever argument is about to be made here, I'd offer the disintegration argument. I have a different OneNote doc for that, but basically: that which you don't have a terminal valuing for, decays out of neglect. It's subtle, because not everything needs a terminal, but you do need some kind of density.
(And it's even more subtle than that, unfortunately. It's not that that's needed, only it determines the kind of identity the agent in question cares about (like caring about atom-level preservation vs pattern-level). And even further subtly, this is relevant only insofar as the agent has a notion of identity that hits the sweet spot (er, bitter spot) of close enough to us to have an enduring conflict, but far enough to destroy us. Of course, this is not necessarily unlikely to hit, because of out attempts to align AI! But it is more than just tampering with software. This is about embodied referential care.)
All of this is making some point about how FOOM (stable-valued self-improving AI) and poof (wireheaded self-disintegrating AI) are actually downstream of the same arguments, but with subtle differences that are all about assumptions in representation vs reality. This is mostly us projecting our own identifications/theories of identity. Which is fine to predict... as long as you name why projection of values is hard in machines if projection of identity is easy in machines.
Abram: I agree with all of this, modulo not accepting that the terminal vs instrumental distinction is as clear as it seems for values; I think it doesn't matter much whether care is instrumental vs terminal, so long as it has the same intensity.
Sahil makes much of congenital pain asymbolia (which is the one where you can detect pain but you don't feel intrinsically motivated to avoid it). Many with this condition do not survive to adulthood. I would argue that the lack of terminal value resulted in a lack of instrumental care in those cases.
The problem with pain asymbolia, and with wireheading, etc, is that you lack a type of feedback which would rule out the bad hypotheses. An RL agent will never rule out a hypothesis saying wireheading is good, just as a mortal can never entirely rule out hypotheses about life after death. So the work of sanity is very much moved to your prior. But humans do not have an evolved prior which copes well with pain asymbolia.
Sahil thinks that FGF offers some sort of better solution than "have a good prior". I have not yet understood what that solution is supposed to be. Being made of lots of little animals does not automatically mean the concerns of those little animals bubbles up appropriately to conscious concern.
Sahil:
What "your" integrity with those animals is, is your conscious concern. You are not "made up of" the trillions of little animals. You are trillions of little animals.
And there is something to be said about the group coherence being of these trillions of animals being worthy of consideration in its own right, the so-called "you". But either end of foundationalism where "you" is independent of the animals is broken.
The simple solution for you to try is to go polyterminal in terms of feedback. Not one "prior" but trillions of them. And their constant dialogue as you.
Sahil: One reason to do this, that CGF doesn't say, is that this is the hallmark of spaciousness, as you ( @abramdemski ) and I talk about it. Simulated conflict is peaceful, is spacious. You can hash things out in a cheaper substrate.
Generally, having a clear "cheap region" (a high actuation space), has this advantage. It has to have some modularity (a map that starts impacting the world is going to be harder to work with), but it can't be literally modular, disconnected (that would make it useless). This too is probably some face of empiricism-control tradeoff.
Abram: Peaceful negotiation is game-theoretically "justified" in so far as it is cheaper in expectation than the battle, but is a decent signal of what the outcome of the battle would be. Actually it is game-theoretically justified in so far as it is an equilibrium, which can take it quite far from these considerations; but I have in mind the situation where you've heard the peaceful signal and now you're deciding whether to go to war anyway -- the negotiations are most secure in the case where it gives an excellent signal of who would win the escalated war, as with democratic votes in situations where wars are won primarily via numbers rather than other factors.
Possibly we would learn something by examining that logic in more detail.
But the interesting thing I wanted to point out -- we then think of the civilized negotiation as actually giving more/better information. It makes a lot of sense that we should stockholm to whatever feedback signal gets actually used. But it seems like there's something more profound than that: propositional dialogue is actually a more fine-grained negotiation tool than a sword, which can reach much better outcomes. This seems difficult to model in existing settings.
Sahil: Yes! Legibilizing does something, moving beyond the context of available outcomes, expanding the Pareto frontier, in incredibly meta ways.
(Legibilzing labor is missed by any model that assumes everything is already formalized. Usual embeddedness stuff, IMO.)
"incredibly meta ways"^, the way that a reference can be meta. A reference can go outside the closed system of symbols, that GPT can't do, per FGF. So in fact, as you say, this is indeed a fine-grainedness thing and perhaps also rules in favor of FGF rather than CGF... which is surprising to me in way that feels exciting to resolve — I initially started this out in favor of CGF.
So if you have a cheap, mostly-but-not-totally disconnected space, that's useful. A whiteboard is like that, where erasing some marks will not cause a hurricane in Laos.
CGF seems to be arguing for disconnecting. FGF seems to be arguing for connection.
But these are naive models. FGF is agruing for connectedness of the whiteboards to the larger world. That permuting patterns on a whiteboard are not going to create any new physics models, without looking out at the world.
(My) FGF shouldn't be saying that it's stupid for software to have substrate independence. FGF should be saying "substrate independence isn't a thing, substrate flexibility is. And specific flexibilities can allow for specific things to be Steam'd. Biological structural flexibility allows Focusing to Steam up various fine-grained things to cognition, the way a machine can't. And before you say 'sensors', I'll remind you that machine sensors can't transfer care. They can transfer causal info, not telic care."
Today, I'd say "substrate flexibility and substrate sensitivity"
See Groundless Sensitivity for a little more.
Abram: I object to "sensors can transfer causal info, but not telic care" -- RL, for example, has the "observation" sensors (which transfer care-agnostic info) and the "reward" sensors (which act as a care-shaping channel).
Sahil: it's unclear what kind of caring that is. But the "digital precarity" is what is allowing it to be metaphorically termed a pain/reward signal anyway.
Sahil: Deacon has a great takedown of genetic codeism or DNAism or whatever this ought to be called. Basically, the same philosophical mistake as pres-structuralism: things are defined by their context, not their content alone.
Also, this might miss the point of human referential/caring basis.
If, for example, you do think uploads or major self-modification is possible, you might be happy to replace any part of you that is unentangled with "you". By definition! If some part does "cleanly decompose" in your architecture, that is replaceable!
So an example of "see, life is CGFy too!" will only point out the FGF argument, of there being a part of life that we don't have tight integration with, and so won't survive with us into our plans for the (high actuation) future.
And generally, if you want to think about referential span, you will want to think about it with respect to the evolution of your commitments as a being (as you see in the previous email, phrased more LW-friendly as "stable under reflection"). DNA might work a certain way, and it might be redone. I wouldn't bet on it, but under the stipulation that genotype-phenotype factors nicely in some way, ie. is amenable to re-engineering... well, it might be re-engineered!
Abram: Can't all the same things be said of software code? So is this any argument one way or the other? Or is Deacon arguing that genetic code is deeply different from software code in the ways it is context-sensitive? If so, how?
Sahil: IIRC Deacon argues that your metaphors of "code" are silly.
That is, the way genes are interpreted into proteins rarely changes.
Sahil: "code for a robot", sure, but only as it is today. I suspect for more alive robots, code will more like DNA is in this paragraph. Not the grounding of the machine, just part of its influence. But this is assuming an FGF position on autonomous robots in the future, of course.
Sahil: it's going to block referentilality. If you think referentiality is necessary to relevant intelligence, then it will block (relevant) intelligence. WIthout referentiality (or my preferred word these days, "sensitivity") the intelligence is almost like a wireheaded blissed-out intelligence that repeatedly fails to share the "reality" that humans share.
I don't know if this remark goes here or not, but it needs to go somewhere: I say put "reality" in quotes above. Why?
Because even as we discover deep strange physical facts about reality (like implications of QM), we still go on doing our thing. It still all adds up to normality. This is being locked in our "reality", which is more about the prescriptive side here: we continue to care about basically what we care about, even after you're told that "you" is just a subjective factoring of the wavefunction. This is real! And if you lean physicalist, this is profound!
Egan's Law ("it all adds up to normality") is also a statement of the cosecond law. But more like the late 1800s one, where it is stated absolutely rather than (co)statistically.
I haven't mentioned this here, so adding: this is all about the difficulty to care about something you haven't had the inclination+liberty to befriend. As hard as making a new sensory modality (not just mapping to an old one).
This really requires overcoming your current notions of equality, in a way.
Sahil: I'd have said: what kind of sensors? Alive sensors? Sensors that have a bit of self-care, a bit of terminal, a bit of integrity, alongwith integration of the whole body? Then sure. But if you're imagining video cameras, it might not care very much to keep the camera intact.
Abram: The obvious CGF reply being: we can make it care about those sensors with RL. We can punish gear-grinding. We can even do that part of the training in simulation to avoid the need to grind real gears.
FGF: You'll still have the potential for wireheading problems. It won't happen at lower degrees of intelligence, but as you scale up the space of hypotheses you're selecting from, you can't rule out the hypotheses which will agentically seek to control the reward signal itself.
CGF: And you have the same problem! It is not as if biology is immune to wireheading; that is where the concept came from!
Sahil: What does this word mean to CGF? I presume something like "can do the relevant calculations on a different, cheaper substrate" Notice the theory of equality implicit in "you could equality well do it in simulation". Oh yeah? But what about the differences, in the substrate? That's what you won't be able to care about!
The example is something like "what temperature should my brain have?" for someone who is "digitally implemented" on silicon. They'd have absurd answers!
This is what happens when even minor details can be Steam'd up. Either they can't be (because of cosecond law) and therefore is ignored, or they can be and leads to crisis that will either be integrated or lead to a psychotic break
Abram: I think my CGF position claims something like: yes, we can factor things into Turing-style computation plus I/O.
Maybe Turing computation is a somewhat better crux to orient around than "von Neumann architecture"? But in some ways, vN architecture is more honest -- Turing computations are functions (so they take inputs and then halt), iirc, whereas vN computers can keep running, implementing interactive behaviors.
Anyway, to address the brain-temperature example: my CGF reply (which I haven't been able to articulate until now) is that yes, if you upload someone without their knowledge, then their referentiality will be somewhat broken. However, we can see that it is not entirely broken, since they could be told, or they could accumulate evidence and figure out that they've been uploaded (unless the ruse is perfect). In other words, their concept of brain temperature is trying to adapt to its new context.
Once we tell them or they figure out what has happened, "brain temperature" can split into "simulated brain temperature" and "CPU temperature". One of them will be relevant if we ask their preference for the in-simulation temperature of the room they perceive themselves to be in. The other will be relevant if we ask their preference for the temperature of the room their CPU is housed in.
Anonymous: The temperature example is too easy. Try instead the "oh, I am made of atoms/quarks/whatever so 'I' don't have the ontological status I thought I did! Oh well, then in that case I care about patterns of them, not the individual atoms."
IOW, characterize your exceptions to/domain of applicability of Egan's Law.
Sahil: eh, not very convincing or relevant. My previous comment does a better job.
Sahil: it's about drive-to-integrity, not size
Sahil: Sure, you could. But does it care? Do you care, now that you can sense things at subatomic levels, to preserve yourself at that level? So why expect the machine to? Your theory of identity, your salience around that, hasn't changed very much! I think this basically ends CGF?
Abram: Why can't sensors be connected up to caring, as is done in RL?
Sahil: The argument is that even humans don't change their care to subatomic level preservation, let alone whatever metaphorical care we attribute to RL.
Abram: I don't get it. The argument is that X (humans) don't care about Y (subatomics), therefore... other X also won't care about other Y, for most values of X and Y? Especially if X = RL, for unstated reasons?
Sahil: Yep, because our scitech sucks at creating qualia.
Or if you like Egan's law. It all adds up to the agent's normality (barring friendship/costatistical miracles/embryos, which do exist).
Insisting that
A) hard problem isn't relevant (for minds in general)
B) hard problem isn't hard (for scitech today)
is a surefire way to miss this class of examples of referential bases.
If you don't like qualia, how about technology to create lasting friendship between two animals. That would satisfy some middling criterion.
Artificial selection into domestication is an example of "we can't create the magic, but we can create the conditions for it to happen" which is in line with cosecond law.
Sahil: this is sort of relevant. I'd say more that it wouldn't be able to do the kinds of creativity enabled by a) connection to more parts and b) collective cognition
Sahil: reach is a slightly different thing than creativity/intelligence, although they're likely very highly interrelated
Sahil: I wouldn't use the word "real", which is the whole point of the word "reach". Everything is in reality, even simulations. However, things can have different reaches, for which (co)statistical miracles have to happen, to overcome.
Sahil: this is fine, as long as it isn't misconstrued as positing a fundamental/ontological divide. There can be (co)statistical reasons behind distinction, a la quantity is a quality all on its own.
(But don't mistake this for "right, so scaling will be the quantity". No, this is referential/caring basis quantity.)
Sahil: yes, this is CGF attempting to bound referential-reach. Why did CGF do that? And what is CGF's view of this? What is their generator for this?
Sahil: CGF does not buy life-mind continuity. But I think this is from a strawman of it. See here: https://docs.google.com/document/d/1vH_XkVwfIKVqjM8d_8xjmg66vMRXRr4nvDpqrT3O_oY/edit?usp=drivesdk
Sahil: This is actually the opposite of what FGF would say. This is actually a CGF position, because the details matter to FGF! That's what the first F stands for! All the data on the web is faaaar from everything,
But anyway, the point made I think, is that within domain, it is more like a stochastic parrot. And that point is sort of related to creativity.
Abram: Yep, fair point.
Sahil: Yes, but not just "seen" which is only one component, but also "cares to".
Sahil: Quite the opposite. This is all about generalization, broadly construed. A better word is "internalization", because it captures both deep knowledge and deep caring. Or: fine-grained knowledge and caring.
Abram-FGF does not get this point at all here, that is pretty central to FGF.
Abram: Yep, fair point.
[editor's note: Sahil was here commenting on a previous draft of the passage; I'm including the comment anyway as it is still relevant.]
Sahil: No, the claim of the cosecond law is that you can't do this "manually" alone. Any more than you can create genuine curiosity or friendship by force. Deep friendship requires freedom!
Empirically, we'll observe this in the "body" of the hooked-up consequentialism falling apart. It won't have the kind of integrity you're imagining.
Sahil: This is not an embedded enough "world". To be more precise, it is embedded (what else could it be), but our metrics ignore embeddedness for minecraft, because the game, like most games, is built out of a background dualistic philosophy.
Abram: I do think that minecraft contains a deep assumption that the player is different from the world, but this can be remedied by adding more players. And regardless, I do think minecraft is complex enough to illustrate impressive behavior (although I am not claiming the behavior illustrated so far is very impressive).
This is a specific way of failing to have integrity, yes. Mental integrity/coherence, which does have connections to "physical" integrity.
Sahil: You're really saying the brain does not "cross levels"?! Have you noticed how the intelligent bits are billions of animals, that are all in reality, and all have caring? Have you noticed that the brain is a biological object?
Abram: I'm saying that the CGF position predicts that the relevant activity would be concentrated in the brain, and FGF does not especially predict this.
Sahil: Yes, this is true. But there's something deeper that I'm spying in this moment, which is the following truths:
- There is no dichotomy between decentralized and centralized
- Yet, it is possible to neglect the textures of one.
One way to neglect is to point at the centralization amidst all the decentralization! But this is clever.
"Evidence" is not what you will get here. This is about salience, participative learning, that really does change your engineering for the better (which is the touted claim of the point/validation of science), and that argument structure of "evidence" actually fails to engage with somehow by imagining some alliance with Truth, but actually in a restricted way. It's an insensitive kind of truth, a "make me" kind. It's the "that which can be destroyed by the truth, should be" rather than also "that which I can be sensitive to around truth, I'd like to be"
Abram: The Bayesian concept of evidence is about being as sensitive to the truth as one can be.
In claiming that evidence for your position is not to be found, but once one understands the position, there's something better than evidence... once again, as you've done before, you're describing your viewpoint as one which requires a dramatic leap of faith, a distant island of thought-space which you seek to engage travelers to visit, but you claim you can provide no reason why they should purchase a ticket, other than "you'll see the benefit once you get there, but I cannot explain while you are here".
Doesn't this sound suspiciously like wireheading?
[Editor-Abram: It bears mentioning that this is not a knock-down argument against Sahil's position; it is not logically contradictory to be in an epistemic state like that. Moreover, I don't think this was a fair accusation; Sahil's position makes plenty of empirical predictions. I don't really believe Sahil's "Evidence is not what you will get here" line.]
Sahil: This would be like positing that repairing color blindness in someone is the same as "repairing" new unimaginable colors. It seems like these are different.
Abram: I flag the assumption that adding new colors is hard. This is unclear.
I flag the assumption that adding new colors is relevantly similar to preventing anosognosia in LLMs. This is quite unclear.
Indeed, presumably the point of the distinction between repairing vs adding is that when adding an entirely new thing, we don't have anything existing to build off of. But the point of the CGF line here is that we can look at human brain architecture to help figure out the missing ingredient.
FGF will say that if you look at what humans have which LLMs lack, you will find biology. But this begs the question. One might also find a more course-grained difference.
This doesn't provide evidence either way for CGF vs FGF (which is the point CGF is making here).
Sahil: just to be clear: this use by Abram of the word "architecture" was in the metaphorical/virtual "computer architecture" sense, rather than changing what our GPUs are like.
If architecture included "low-level" changes, then it might be possible to resolve. But that would be a change in methodology.
Abram: More "artificial neural network" than "computer architecture"
Sahil: "reference-penetration" as in "depth of referential-reach", which is to say what the intelligence/being can be in connection with. "Connection" refers to both knowing and caring. Both info theory and "care theory".
Sahil: This is again about the theory of equality. But I'll emphasize again that this is a harder case to make, because anything I can refer to, I can care about! So any example I give is easily undone by "but here, see, for the next 5 minutes, I'll care about this atom using this electron microscope".
This is still "unnatural", so it does make the point: for the most part, you don't say "I wanted to meet you today, but you have totally different atoms! I guess we'll try again some other day" The caring about atoms is an exception that proves the rule, in most cases.
So it is part of the spectrum of costatistical miracles (ie. caring miracles), but to name a much more miraculous example, I offer "caring that a painting is harmonious, among colors you can't imagine".
In fact, you could, perhaps by befriending a bird who sees more color, come into connection with the color by its reaction. And so this is still within reach in a way.
As a reminder, although I haven't stated the cosecond law anywhere here in the comments on this document, I will say that the problem of "but how do you apply this caring-miracle stuff, when everything is, fundamentally, in connection?" is a problem also faced by thermodynamics: "how do you even say that coffee can't spontaneously unmix itself, when it is an open system?"
Sahil: Wrong word, worse emphasis. FGF would in fact say " yes, but not embedded in a way that's close to all of us animals. We can't really talk to its caring."
Sahil: Abram-FGF generally seems too foundationalist to really steelman FGF. FGF requires talking about intersubjective referential distances, rather than insisting on some special connectivity humans have to the "real" world. Humans have special connectivity to the human world, and to the animal world. But they still seem to be, for now, stuck with six sense bases.
Sahil: or rather, "critical, but not central"
Sahil: I wouldn't even concede "basically". Ignoring the caring is almost ignoring the whole point
Sahil: sure, this is fine enough (if "grounding" is implicitly curried with animalish/humanish caring), but this can be said in a groundless way.
Sahil: No, any "next evolution" is going to be hinging on "therefore"s, which might be doomed. However, perhaps the point is that "my referential connections and my theories of equality/identity have something to do with what I protect and identify with", could be classified as some evolution of "I think, therefore I am"
Sahil: again, Abram-FGF cares about "really" more than FGF does. If anything, FGF is a way to get past this "real" "simulated" distinction in a workable way.
Sahil: sensible!
Sahil: "physical" is as irrelevant as "real"
Sahil: the details of this "outside" is what's important and addressed by "referential bases" and does not need "grounding"
Abram: I think you are conflating different notions of "grounding" here, or else, perhaps you are objecting to the terminology/connotation of "symbol grounding" rather than the main substance of it.
Or else I misunderstand.
You also said something similar in the recent call.
My interpretation is that you think "symbol grounding" is contrary to "groundlessness". I think not. "Symbol grounding" is just a term for the problem of reference, to me.
Sahil: I'd call this "urgency", not agency. Urgency is only a small part of agency. Urgency is to agency, as cancer is to life.
But this is less relevant to the point, whose real rebuttal is: yes, so this "distant optimization" is within the referenial span of your referential basis! Get the point already, by looking at how you haven't evolved new sensory modalities recently!
Abram: The cancer analogy isn't helping me here, because cancer seems to have less at-a-distance-nature, not more. Cancer is more similar to wireheading than it is to aiming a rocket at a distant moon.
I think the optimization-at-a-distance point is a rather huge subtopic to get into, thinking about it.
WRT evolving new sensory modalities, I'll reiterate my point from the phone call yesterday: microscopes and telescopes and thermometers and all the myriad other inventions of humankind seem quite adequate. Translation of harder-to-comprehend modalities into easier-to-comprehend modalities seems to be a central theme in human FOOMing.
I take your new-color argument to be: "see how rare and difficult it is, to gain a new color? That is how difficult I expect it to be, to extend the referential reach of LLMs."
To which I here reply: "look how humans manage to extend their practical referential reach with microscopes and telescopes. You define this to be irrelevant via 'referential span' -- but in doing so, you set yourself up to ignore the more important phenomenon! We do not see explosive takeoff of a species when they acquire a new color. We do see explosive takeoff of a species when they learn to map invisible light-frequencies into their pre-existing color range."
Sahil: This is more relevant to say when something is further away in referential span, rather than not even having basis elements. The latter point is what CGF needs to hear first.
Sahil: It is confusing to me what "direct" means, since I've stopped believing in space. "What is touch" is as hard a metaphysical question as "what is time".
Regardless, you could ask CGF about what makes their information theory results say that you can't draw a map of the city sitting in your armchair. You are in connection with the city, and information certainly travels through the layers. What makes some process actually connect with the city, when you step outside?
Sahil: I'd delete these, as being more misrepresenting rather than helpful. If this is intriguing to some, I'd happily reflect back the isolated demand for resolving "problem of open systems", which is a problem for any result about information/thermodynamics as well, not just caring.
Sahil: which might be desirable, or inevitable. We could become the genius sensitive "unconscious" or "heart mind" for the exocortex of AI! And we'd best be well integrated, for the AI's sake, and all of life's sake! But hopefully as a channel to our own "heart minds" rather than only our neocortex.
Sahil: added this image. It should be clear that FGF wants to say that these containment metaphors are highly misleading. Your "abstraction" is reality, in fact, is a particular kind of friendship between animals that are your brain!
Factoring it out of reality or hardware etc and thinking that FGF is about one or the other, rather than their nondualness, is a classic way to not get FGF.
Not dissimilar, I imagine, to thinking that enlightenment is separate from dualism.
Sahil: which "processing power", again, is made of a lot of ant-like animals working together.
Sahil: What happened to your own expression of the cosecond law, Abram?
Abram: Your recent applications of it seemed too distant from the thing I might endorse to easily bridge the gap. You've been attempting to apply "caring miracles are hard" to maplike care, but my story made it specifically only about blueprintlike care -- maplike care seems quite cheap on my account.
Sahil: I have costatistical suggestions, rather than fundamental suggestions
Sahil: You sure do, pray away. You also have room to argue "but there's still a chance that I can win the lottery, right?"
Abram: But, here, it is closer to the chance that someone would win the lottery. You've designed span so that the mere possibility that someone could design a telescope means that the moons of jupiter are already in-span. So it seems we should usually assume that the span of some X is far greater than its apparent referential capacity, in absence of any strong reason to think otherwise, no?
[editor abram: facepalm emoji. I was being dumb here, arguing too far from the gears of how reference and meaning work, too reactive to Sahil's argument rather than considering what's true. I'm not saying I know what to say instead, but I regret saying the above.]
Sahil: I'm just going to flag this mathematical vs physical dualism and not repeat myself this time.
Okay, I'll slightly repeat myself: it's not about real vs simulated! It's about how hard it is to have caring-referentiality with something truly outside of your being. Not impossible, just requiring integration. Like making someone fall in love with you.
Sahil: ugh
Sahil: "in" is precisely the opposite of what FGF wants to say, y'know.
Sahil: How come you still don't care very much about the atom-changes when you meet me, when you're intelligent enough to know me?
Another way to say this: there is still a lot of freedom in representational-commitments, even for intelligent agents.
That is, your theory of equality might still "add up to normality", which is another way to say "doesn't change/grow very much".
Abram: You're conflating maplike care and blueprintlike care, here. CGF is pointing out how we end up having beliefs about lots and lots of things, and you're objecting by pointing out that we don't care about many of those things. This fails to object to the point CGF is making here.
CGF is imagining a Go-playing AI with a library of human games, here. Deducing something about humans from only the rules of Go itself might also be possible in principle, but would be a far more monumental task, at best.
Sahil: it doesn't add up, otherwise it would violate Egan's law. Or rather, it can add up to somewhere (but not the kind of closure CGF is imagining), but very slowly, like natural selection can add up to organism. What "slowly" means here, of course, is a little confusing. And it's not about how we can't because it's too slow, but that our "objective" methodologies are too immature to talk about friendship.
Abram: How would it violate Egan's law?
I think I would express CGF's point here very differently, now: the vector-addition analogy is just too poor an analogy.
And anyway, CGF is here replying to a point you don't even endorse.
Sahil: this is about hardware, actual architecture, rather than vN architecture. Might be useful to remind.
Abram: "actual", eh?
Sahil: what is System 1, Abram? And where is the truth of all the trillions of animals involved? The dialoguing?
Abram: I'm here using system 1 to refer to the parallelism-focused computations in the brain.
And the second question seems so rhetorical. I don't know, you were the last one who had it, did you check your pockets?? Again, the CGF represented here is not saying that FGF is false for humans, although on the narrower point of Gendlin's Focusing I find the CGFish view to be more plausible.
Sahil: Before even this, I would say: how do you get facts about your body (such as whether there is a candle burning your arm) into brain at all, per CGF?
Abram: Nerves, mostly!
Sahil: this is not about the brain vs the body, because the brain is a body part
Sahil: no, the small-m mind, or the thought-shotgun, ever-ready to construct instead of conduct.
Sahil: this is all fine, as long as you embrace that you interact with a model not like a program you hold, but as a view you embody.
Sahil: empiricize the animals, man!