It seems inconceivable that an LLM isn't planning where it's going when it writes the middle of a long argument, but that's sure what it looks like it's doing.
Actually it seems like more advanced LLMs do plan ahead, at least sometimes. See Anthropic's "On the Biology of a Large Language Model" here, talking about writing poetry:
the model often activates features corresponding to candidate end-of-next-line words prior to writing the line, and makes use of these features to decide how to compose the line.
They are amusing, clever, and self-indulgent. They often have explanatory power. Sometimes I feel like they unnecessarily padded the word count.
I think this is an artifact of GEB's age - it had to be written as a physical book. Imagine if it had been written today, as a Sequence, with hyperlinks. You could have the exact same content, but organized so you could easily jump around, back, and forth, in various orders and portions that are optimal for different readers.
I will also say that, 20 years after I first read it, the dialogs are the pieces that let me remember the technical content at all, for the same reason I remember lyrics and poetic quotes better than sentences that lack those extra layers of structure and meaning.
They also serve as convenient mental shorthands. When I see people talking about the idea of LLMs using steganography to hide messages in their CoT or output, and doubting whether that is viable in practice, my thoughts jump almost immediately to Contracrostipunctus. When I talk to (or read things by) people who doubt that LLMs (or any other digital construct) could be 'intelligent' by whatever definition, or conscious, etc., I have a lot of reasons I disagree, but one of the first places my mind goes is Six Part Ricercar. I can then reconstruct more detailed explanations if I need to give them to others, but my thinking is faster because I don't need to recreate them for myself.
I think this same idea is the main source of value I get from EY's and Scott Alexander's fiction, having read their nonfiction writing. Understanding all the detailed arguments is valuable, but calling them to mind is slow. It's much faster to be able to think of Moloch, whale cancer, Fnargl, Ebborians, Baby Eaters, or beisutsukai, and then take the time if needed to figure out why I thought that. I think this is also similar to the skill Feynman talked about for spotting flaws in arguments he didn't fully understand, by creating a concrete mental visualization that encoded some of the essential structure.
The higher-level language contains things like "for loops," which are much easier for a human to understand than an incomprehensible string of binary which you feed directly into a wire as changes in voltage, but I don't think it contains any information that the machine code does not.
As trivial examples: comments, variable naming, etc. are information contained in HLL programs, but not in the corresponding machine code programs.
The other way around seems closer to right, but even there the compiler can provide additional information.
That's a good point.
The compiler ignores comments, so saying that they are program information is like saying that a sticky note stuck on a book is book information. The addition may or may not be relevant to the thing, but it's not the thing.
Variable names, on the other hand, are extremely the thing. You are absolutely correct that variable names contain information that the machine code does not. They are also a functional part of the code, in that changing one variable somewhere will usually change the function of the code, whereas changing the sticky note on a book will not change the contents of the book at all (although it might change the meaning of the book to a person who reads the note). That said, you could exchange every instance of each variable name one-for-one for a randomly chosen Latin word and the program would act exactly the same, but the program might make much less sense to a human reading the source code. The variable names are explanations for the intent of the program which are also themselves the program. However, they are not logically bound to the program in a way that the Gödel string is logically bound to the natural numbers. You can, in fact, change all of the variable names of a program and remove their explanatory power without changing the function of the program. You cannot change the Gödel numbering for a typographical number theory and lose the explanatory power of Gödel's theorem, you just have to make the Gödel string out of different symbols. The explanatory power of the variable names is largely contained in the associations that a human reader has with the strings which make up the variable names.
More generally speaking: every way you can change a source program without changing the compiler output is a way it contains information that the machine code doesn't. And compilers do a lot of non-injective transformations, that is, transformations which have same outputs for different inputs. Loop unrolling, replacing for with while (or vice versa), etc.
Look, nothing I can say about GEB will be news to anyone reading LessWrong, but I wrote all this up to organize my thoughts, so I might as well post it here for anyone who hasn't read the book or wants a wave of nostalgia about that time they got obsessed with it as a teenager. I have been trying to read Douglas Hofstadter's famous book Gödel, Escher, Bach (henceforth GEB) for ten years. I decided that I was going to try reading it from the beginning for the third or fourth time a few months ago, and I finally read the whole thing this time. Every previous time I tried to read it, I found the narrative intellectually compelling and delightfully playful, and yet I would have difficulty picking it up again for some reason. The thing that kept me coming back was the formal logic. I love a surprising logical consequence, and the set theory in the early chapters stoked my curiosity. Intellectually, I wanted to know where the setup of typographical formal systems was going, but I didn't have the willpower to get the words into my brain for the longest time. I'm glad that I finally did.
To explain why the book is so difficult to get through, it might help to explain its structure. The book alternates between dialogues and chapters. The dialogues read like scripts for a play and consist of fantastical dialogues between Achilles, Tortoise, and others which are relevant to the following chapter. The dialogues are explicitly structured after Bach fugues. They are amusing, clever, and self-indulgent. They often have explanatory power. Sometimes I feel like they unnecessarily padded the word count. Many things in the book do this, but I don't think Hofstadter could have written this book without all of the tangents he explores. For example, he has an extended chapter about DNA and protein synthesis which doesn't really demonstrate anything that the typographical number theory chapters did not already do, but it certainly carves more deeply the outlines of what he means by strange loops. As you'll see later in this post, it's incredibly useful as an analogy. The Bach elements do not do much for me, although they clearly assisted the author in his thinking. The Escher bits are much more compelling to me, as they provide obviously self-referential reference analogies for self-referential arguments. When he writes the final chapters which contain and restate his most important arguments, you feel like it all rests upon the previous chapters and even the dialogues. Everything definitely feels relevant, but it's not the most concise way to make the argument he makes.
The computer science in this book is fifty years out of date, but I think the core insight on explaining minds holds up, in that it is still plausible regardless of whether it is correct. In my read, the book is an exploration of the idea that you can build something out of easily understood components on a low level that becomes incomprehensibly tangled up in self-reference but also beautiful or true or functional if understood at a higher level of organization. He has a number of case studies here. The one for which the book is named is Gödel's Incompleteness Theorem. Hofstadter does a masterful job of pointing out the parts of the theorem which are actually clever and glossing over the minute details while still giving some of the structure. I cannot do his explanation justice, but here is my understanding of how the theorem works: Gödel's theorem creates a "sentence" which indirectly describes itself using language which is designed to describe only numbers. The trick is to define a way to turn sentences into numbers and then make a sentence which describes a number which from the outside clearly refers to the sentence itself. The sentence itself does not declare its own truth; it (indirectly) declares a statement that there isn't a proof of itself in its own language. When you apply formal logic to the declaration, you find that any assumption you make inevitably forces the declaration to be true. This probably all sounds like nonsense if you haven't read the book or aren't intimately familiar with the proof for some other reason, and that's why I forced myself to read the book. I had to know what the nonsense was about.
The thing Hofstadter seems to actually want to talk about is the nature of consiousness. Here is an exerpt from the book explaining why Hofstadter went to so much trouble to explain an esoteric mathematics proof in a philosophy of mind book:
Gödel’s proof offers the notion that a high-level view of a system may contain explanatory power which simply is absent on the lower levels. By this I mean the following. Suppose someone gave you G, Gödel’s undecidable string, as a string of TNT[1].
...
The only way to explain G's nontheoremhood is to discover the notion of Gödel-numbering and view TNT on an entirely different level. It is not that it is just difficult and complicated to write out the explanation on the TNT-level; it is impossible. Such an explanation simply does not exist. There is, on the high level, a kind of explanatory power which simply is lacking, in principle, on the TNT-level. G's nontheoremhood is, so to speak, an intrinsically high-level fact.
...
Looked at this way, Gödel’s proof suggests-though by no means does it prove!-that there could be some high-level way of viewing the mind/brain, involving concepts which do not appear on lower levels, and that this level might have explanatory power that does not exist-not even in principle-on lower levels. It would mean that some facts could be explained on the high level quite easily, but not on lower levels at all.
I don't know if I buy this! This does not merely assert that consciousness makes far more sense on a higher level of abstraction than the level of raw neurons[2]. I think he is saying that there are true statements about the arrangement of the physical materials which support consciousness (neurons in a brain for example) that there is no way to explain using only the mechanisms of those materials. I have a couple of metaphors which I will try to use to understand this assertion, and they happen to be mostly things which Hofstadter talks about at length in the book[3].
The strong claim by Hofstadter is that you won't understand consciousness simply by looking at neurons. He has a variety of weaker claims which are more interesting. One is that he thinks that the self-referential nature of Gödel's theorem will be directly analogous to a self-referential explanation which will be the best way to understand consciousness. Here is an exerpt which comes soon after the last one in GEB:
This act of translation from low-level physical hardware to high-level psychological software is analogous to the translation of number-theoretical statements into metamathematical statements. Recall that the level-crossing which takes place at this exact translation point is what creates Godel's incompleteness and the self-proving character of Henkin's sentence. I postulate that a similar level-crossing is what creates our nearly unanalyzable feelings of self.
In order to deal with the full richness of the brain/mind system, we will have to be able to slip between levels comfortably. Moreover, we will have to admit various types of "causality": ways in which an event at one level of description can "cause" events at other levels to happen.
This is the point of the book. I'm going to briefly slip into Hofstadter's nomenclature, where the word "symbol" is used to refer to a recognizable pattern of neurons in the brain doing something. Hofstadter talks about how consiousness, the feeling of being yourself, might be explained by a "self-symbol" which interacts with other symbols in your brain which are entangled with your environment (which is to say that they are shaped the way they are because of signals from your sensory organs). The feeling of "free will" is what happens when the self-symbol interacts with the mental processes by which your body does things. The self-symbol abstractly represents the body which contains the mind which is made up of neurons, some of which compose the self-symbol. The self-symbol can be said to contain itself via moving through a chain of abstraction layers in what feels like a single direction until you end up back where you started.
This is in opposition to my earlier musings on machine code. You can understand a high-level computer program on its own terms: if you click the button or type the command, it does a thing. You can understand the source code on its own terms: the for loop iterates the variable and slightly changes what the commands do each time it repeats. You can understand the corresponding machine code on its own terms: when this binary number is read into register 2, the CPU add together the numbers in registers 3 and 4 to be output to register 5. Hofstadter isn't saying that you can't understand what a mind is doing by looking at the neurons. I quote GEB: "We should remember that physical law is what makes it all happen-way, way down in neural nooks and crannies which are too remote for us to reach with our high-level introspective probes." The brain is a physical object, and you can explain everything that happens with things like "this neuron has had more neurons connect to it than the threshold amount that makes it connect to this other neuron." But Hofstadter (in GEB at least) doesn't think you'll understand consciousness that way, and even if you find a more convenient representation of the brain which allows you to represent huge collections of neurons as an entity that performs functions. He doesn't really make clear what that explanation would look like beyond the sort of argument in the last paragraph, which I suppose is something.
Now we can finally talk about the outdated parts of the book. Hofstadter thinks that his strange loops are at the heart of understanding consiousness and at the heart of creating artificial intelligence. That didn't pan out with modern artificial intelligence regimes. As an unfair example, here is possibly the least accurate prediction in the book:
Question: Will there be chess programs that can beat anyone?
Speculation: No. There may be programs which can beat anyone at chess, but they will not be exclusively chess players. They will be programs of general intelligence, and they will be just as temperamental as people. "Do you want to play chess?" "No, I'm bored with chess. Let's talk about poetry." That may be the kind of dialogue you could have with a program that could beat everyone. That is because real intelligence inevitably depends on a total overview capacity-that is, a programmed ability to "jump out of the system", so to speak-at least roughly to the extent that we have that ability. Once that is present, you can't contain the program; it's gone beyond that certain critical point, and you just have to face the facts of what you've wrought.
This is arguably a decent description of LLMs, but LLMs aren't that good at chess. The best chess algorithms are based on a similar neural network architecture to LLMs, but they will repeatedly destroy any human chess player until the heat death of the universe without getting "bored with chess." Rather than guessing what Hofstadter got wrong, I will paste in his words from the 90s when it became clear that chess algorithms were surpassing human chess prodigies: "They're just overtaking humans in certain intellectual activities that we thought required intelligence. My God, I used to think chess required thought. Now, I realize it doesn't. It doesn't mean Kasparov isn't a deep thinker, just that you can bypass deep thinking in playing chess, the way you can fly without flapping your wings." Getting back to LLMs, the nice thing about being the author of arguably the most famous book ever on artificial intelligence is that, as seen above, people interview you once it becomes clear that computers are becoming capable of doing the things which you once argued that they would eventually do. I don't need to guess what Hofstadter would say about ChatGPT, because I can just go listen to what he did say about it. To that end, I see little reason to ramble about what GEB gets wrong about human and artificial intelligence. I will simply quote 2023 Hofstadter, who muses that "maybe the human mind is not so mysterious and complex and impenetrably complex as I imagined it was when I was writing GEB...I felt at those times...that, as I say, we were very far away from reaching anything computational that could possibly rival us."
I could argue that Hofstadter is underestimating the complexity of minds here. I think that he expected that we wouldn't be able to build something remotely intelligent without understanding it. It turns out that we could, and, in this sense, intelligence was not as complex as he thought. Something truly complex shouldn't be able to be created by tweaking numbers in a giant matrix in whatever direction makes words come out prettier for a few weeks.[5] If LLMs can create anything remotely like deep thought, which Hofstadter and I agree that LLMs can do, then deep thought wasn't as difficult as he thought it was, much like chess didn't require as much deep thought as he thought it did. That said, we don't really understand how the ability to answer questions in syntactically correct English is encoded into giant matrices just like we can't explain the process by which brains do the same thing in enough detail to make a blueprint for such a brain. It may be that the explanation that would actually make a human understand LLMs or their own brain at that level will require the sort of level-hopping explanations he envisions. The thing he wants to explain may not be as complex as he thought it was, but it still might be complex enough to require the sort of explanations he thought it would.
I was previously somewhat conflicted about calling this a philosophy book, but I'm rather attached to the label now. Robert Miles says that "philosophy" is appoximately a synonym for "thinking about things we don't know how to think about" until we know how to think about it well enough that we call it something else. Isaac Newton was a philosopher in his own time, but now we understand motion well enough that we call the theories he wrote down "physics" along with all the other theories we use to describe where undirected lumps of matter will be at what time. GEB is philosophy in the fullest sense of this definition. The arguments for consciousness are barely there. It's a pile of metaphors and frameworks which the author says might help understand consciousness someday. Not even decades-older Hofstadter thinks that what he wrote in GEB is accurate, but he was trying to think about something which we still don't actually know how to think about: consiousness and what it means for "intelligent" computers, whatever that means. He was careful even at the time to declare that he was speculating, but it's a beautifully justified chain of speculation.
TNT is Hofstadter's typographical number theory. He shows earlier in the book how you can encode a small set of basic principles of the natural numbers into a set of typographical rules which let you show things like a+b=b+a for any two natural numbers a and b. The upside is that the typographical rules are objective and unambiguous. Some mathematical proofs point at concepts using natural language, and the typographical theory is one way to avoid the ambiguities inherent in language that can result in apparent paradox or result in a proof which has hidden consequences. One can mechanistically check whether a series of well-formed TNT string follows from the axioms of number theory and then interpret the string into a truth about the natural numbers if it follows. ↩︎
Or other substrates that minds could run on. He does assert elsewhere in the book that minds are best interpretted via higher level "symbols" (more on that soon) rather than trying to understand them on the physical level. Ant Fugue is my favorite of the dialogues in the book, and it explores this principle. I think it is the one dialogue which makes a complete argument that chapters in the book do not make on their own. It might be worth reading even if you don't read the rest of the book. ↩︎
I would have reached for these examples even if Hofstadter had not. Maybe this means that I should not have bothered with the book. Or maybe it means that this book was made for me. Or that I have self-sorted into only engaging intellectually with people who think like me, and my individuality is a cherished lie. Who knows? ↩︎
In theory, the dependence on prior life to propagate life has been violated at least once, but you can take it as given for the past 3 billion years or so. I once tried to make an argument that there is no need to consider explanations of how life evolves which considers things on a lower level than DNA and the protein synthesis chain. Yes, DNA and proteins are made of atoms and you can interact with them as atoms, but I argued that the explanation for life was best carried out on the DNA level or higher. In principle, I said, the DNA codes for all of the proteins, so there is a well-founded stack that rests on DNA and adds up to an organism. My wife disabused me of this by talking about epigenetic inheritence. You really do have to consider what the environment of atoms does to DNA and proteins in different scenarios without modifying the underlying genetic sequence. This seems to argue that higher-level abstractions lose explanatory power relative to lower-level ones, which I think is the obverse of Hofstadter's point, but I also think it is true. It can be true that higher-level abstractions make for better explanations of reality and also that they lose information about reality relative to lower-level abstractions. ↩︎
One could argue that this is a silly position to take because evolution created human intelligence, and evolution is just the repeated tweaking of DNA until it creates an organism that keeps replicating its DNA. But evolution doesn't understand the intelligence it made. Evolution is just the name we give to how a self-perpetuating system changes over time. Of course, that's still an argument that you don't need to understand intelligence to create it, so maybe it would be better to say that evolution implicitly understood intelligence. Evolution was a gigantic program running on entire biosphere worth of parallelization for 3 billion years, and it finally created a set of DNA strands that encode a structure capable of answering questions with syntactically correct English. Even then, a baby can't understand English until it has had several years worth of training by intelligent humans that do have some understanding of how English works (although one can argue that the earliest users of language did not in fact know or care about explicit rules of grammer or syntax or whatever). You also have to understand that GEB uses DNA as a prime example of strange loopiness. The structures which read DNA go on to read DNA that tells them how to build more of the structures that read DNA. If intelligent life was created by accident, at least it is obscenely complicated to sort out what is happening inside intelligent life. I think Hofstadter thinks that LLMs aren't that complicated. For example, Hofstadter expresses shock that LLMs are feed-forward. I'm not entirely sure what he's referring to when he says this, but I assume he's referring to the fact that LLMs generate one token at a time based off of previous tokens. You can feed half of a sentence to an LLM and it will happily spit out the next word, and then you can feed that new sentence with one more word to a fresh instance of the LLM and get the next word and so on, and you will get something just as good as if you asked for the whole sentence from one LLM to begin with. It seems inconceivable that an LLM isn't planning where it's going when it writes the middle of a long argument, but that's sure what it looks like it's doing. If I correctly understand the state of the art, the current most impressive models generate a bunch of answers and pick the best one, but this is a hack built on top of many instances generating one feed-forward response each. ↩︎