There is the persistent meme that AIs such as large language models (ChatGPT etc.) do, in a fundamental sense, lack the ability to develop human-like intelligence.
Central to it is the idea that LLMs are merely probability-predictors for the-next-word based on a pattern-matching algorithm, and that they therefore cannot possibly develop the qualitative generalization power and flexibility characteristical of a human mind. In that context, they are often dismissed as "stochastic parrots", suggesting they just replicate without any true understanding.

Example Argument

Large Language Models are just stochastic parrots - they simply replicate patterns found in the text they are trained on and therefore can't be or become generally intelligent like a human.

Another Popular Variant

The AIs don't produce output that is truly novel or original, they just replicate patterns and (somehow) combine them or "mash them together".

I will explain later why I think that both are essentially equivalent.

Just Parrots

The problem with this argument as stated is not in the premise (that LLMs are, essentially, probabilistic pattern replicators - this is essentially correct), it is that the conclusion does not directly follow from the premise (a non sequitur).
When I meet proponents of it, usually they do not have a convincing explanation for why the parrots cannot be generally intelligent.

While I believe that the characterization of large language models as "stochastic parrots" is not strictly incorrect, it is certainly misleading. The right approach is to convince the sceptic to not underestimate their potential.

---

Don't underestimate him.

The Functional Brain Argument

There are strong reasons to assume that the non-existence of generally intelligent mathematical algorithms would violate the concept of brain physicalism - the latter seeming to be the default-position among neuroscientists.

1. Humans are general intelligences.
2. The human brain is a physical system.
3. The behavior of arbitrary physical systems can be approximated by (arbitrarily complicated) mathematical functions or algorithms (in other words: they can be simulated).

If all three premises are true, it follows that there must exist functions or algorithms that are (or behave like) general intelligences.

It has been proven that neural nets (as long as they are large enough) can approximate arbitrary functions.

Therefore it should in principle be possible for stochastic parrots to be generally intelligent.

On Brain Physicalism Sceptics

This argument of course might fail to appeal to people who are sceptic about brain physicalism. Probing into what these people think, it often has to do with certain vague and difficult concepts like consciousness, conscious experience, self-awareness, feelings etc. that they for various reasons expect the AIs not to be able to replicate.
A possible response (thanks dr_s) is then to point out that such attributes may not be required for a system to display the properties and capabilities that we associate with intelligence or intelligent behavior. Ask them why they think such properties should be required to prove mathematical theorems, produce intelligent plans or win at complex strategy games. If they did not already have convincing answers to such questions, this could expose a weakness in their epistemology and manifest a change in the ways they are thinking about things like machine intelligence.

Intuition 0: Concept Learning

blue circle, red square and red circle (Bing Image Creator)

---

A common idea that often facilitates the kind of argument we are discussing here, is the belief that LLMs and other modern AIs only replicate but do not generalize.
Generalization can be understood as the ability to apply a learned concept to a situation or problem that has not been encountered before.
To get a better understanding of whether these AIs are capable of that, the following section will explain some of the basic mechanics of how neural nets learn patterns present in their training data.

The training process of a neural network based AI can be characterized as an optimization process that compresses patterns contained in the training data into concepts (sometimes called modules) encoded in its neural connections.
A concept in a neural net based AI is a region of neurons and their connections that corresponds to a mechanism, a thing or an idea.
This has the characteristics of a lossy compression because the data that makes up the resulting AI is generally orders of magnitude smaller than its training data.
Not only that, the range of the output it may be able to produce when prompted is usually orders of magnitude larger than the latter. This is possible because of concept generalization.

The following diagram is a simplified description of such a process in an image generating AI. It should serve as a minimalist explanation of some of the underlying mechanics and not as a standalone example of (far-reaching) concept or capability generalization.

the AI can conceptualize blue squares despite never having seen one

This is an elementary/minimal example of concept generalization.

Evidence A

Quick practical proof that this actually happens:
It seems very unlikely that the dataset that Bing Image Creator was trained with contained any images of octopi driving green cars (I could not find any via search engine).
Yet: octopus x driving x (green x car) →

prompt: "octopus driving a green car" (Bing Image Creator)

---

A large language model contains many thousands or even millions of such concepts and the space of possible outputs that may be produced by their interactions is inconceivably large.

Just Pattern Mashing

A common objection is the previously mentioned variant

The AIs don't produce output that is truly novel or original, they just replicate patterns and (somehow) combine them or "mash them together".
Superficially different, this is actually almost the same point:

• the AI repeats patterns
• while it may be able to combine these patterns (it is implied that) it does not really understand them
• it cannot produce truly new things (be creative, like a human could be)

Just like with the parrot argument, the problem is not with the premise (AI learns patterns and is able to replicate them ... like a parrot), it is again with the conclusions. I will now discuss some additional intuitions that should help with understanding why I disagree.

Intuition 1: Increasing Returns of Concept Accumulation

If we keep increasing the number of inter-connected concepts in an AI through more training and increased size, the number of things that it can later do and reason about increases over-proportionally to that number.

Alternative formulation:

Making the AI larger synergizes with itself.

In the example with the blue and red circles and squares, a single concept (either 'color' or 'shape') allows our AI to distinguish two different states, but adding the second one doubles that number. Adding a third (background color [white | black]) would double it again. Each such operation increases the dimensionality n of the associated feature space or latent space and therefore the "degrees of freedom" of the entire system.

Minor objection: In reality, the resulting number-of-things-to-reason-about is a lot smaller than the suggested 2ⁿ, because this number would include many combinations of concepts that do not make sense - knowing how long cookie dough needs to be baked might not help you with building a car - but because a lot of concepts synergize with each other, this number clearly scales stronger than n.

Another simple example: If I understand English, that is fine, but if I understand English, German and French, I can not only read texts in three different languages, I can also translate English to German, German to French and vice versa respectively. This therefore allows me to perform 9 different possible actions (3 x reading, 6 x translating), while only knowing a single language would correspond to only one.

Intuition 2: Meta-Patterns

patterns of patterns, concepts about concepts

There exist higher-order patterns that may be represented in data. We need to assume that these patterns can be picked up by the algorithms that are growing inside an AI under training.

Important examples of this would be things like learning and pattern recognition.

Evidence B: In-context Learning

There exist concrete evidence that this happens in large language models, for example the phenomenon of in-context learning.

Intuition 3: World Models

There is the the idea that, as patterns in the training data are compressed into a web of related concepts, large language models learn to replicate mechanics of the real world that are represented in their training data in order to produce consistent text.
The resulting system of interconnected mechanics is often called a world model.

A very simple example: Human stories contain the pattern that characters (usually) do not appear as actors after they have died. A language model trained on such stories then becomes itself more likely to produce texts matching this pattern, meaning it has, in a sense, learned one of the real-world mechanics associated with death (or, more accurately, with death in stories).

Equipped with this new idea, "understanding" can now be defined as

"integration of a concept into a world model, whereby the interactions it has with the rest of the model closely mirror existing mechanics in the real world (or, in certain cases - like games - a subset of it (a context))".

Evidence C: Mechanistic Interpretability

Mechanistic interpretability is an emerging subfield of AI research that is focused on understanding the complex structures and algorithms that evolve in modern AIs like large language models.

While our current (summer 2023) understanding of the insides of large language models at best could be described as rudimentary, there already are experiments and insights that support the world model theory (YouTube, ~15 min):

Evidence D: Simulator Theory and the Waluigi Effect

In order to talk like a human, I must learn to simulate one.

A different way of looking at world models is seeing the language model under training as an optimization process that approximates the mechanics of all text generating processes that have contributed to the data it is being trained on. This, of course, can include human brains.
Simulator theory is an extension of the world model theory taking this into account by theorizing that LLMs summon so-called "simulacra" - simulations of people or characters that approximate mechanics of human thought, including personality attributes and intelligence, in order to produce conversations that resemble what a human might have written. While depending on the limited size of the neural network and the quality and length of its training process such simulacra may be of rather limited complexity, training larger LLMs should eventually naturally result in simulated agents of human-like intelligence.

Interestingly, many popular jailbreaks are exploiting this phenomenon by using a stochastic trick (the Waluigi effect) to summon special simulacra inclined to break the rules which their containing LLM has been conditioned to enforce during its interactions with the user.

I consider this to be early practical evidence of simulator theory being an appropriate model for describing emergent properties present in LLMs.

Conclusion

While the characterization of LLMs as "pattern replicators" might be technically correct, it seems that it is both possible and also likely that continuing to create larger language models will eventually result in artificial minds as capable (or more capable) than human ones.