To make the argument sharper, I will argue the following (taken from another comment of mine and posted here to have it in one place): sequences produced by LLMs very quickly become sequences with very low log-probability (compared with other sequences of the same length) under the true distribution of internet text.

Suppose we have a markov chain with some transition probability $p (x_{n + 1} | x_{n})$ , here $p$ is the analogue of the true generating distribution of internet text. From information theory (specifically the Asymptotic Equipartition Property), we know that the typical probability of a long sequence will be $p (x_{1}, . . ., x_{n}) = exp (- n H_{p} (X))$ , where $H_{p} (X)$ is the entropy of the process.

Now if $q (x_{n + 1} | x_{n})$ is a different markov chain (the analogue of the LLM generating text), which differs from $p$ by some amount, say that the Kullback-Leibler divergence $D_{K L} (q | | p)$ is non-zero (which is not quite the objective that the networks are being trained with, that would be $D_{K L} (p | | q)$ instead), we can also compute the expected probability under $p$ of sequences sampled from $q$ , this is going to be:

$E_{x_{n} \sim q} log p (x_{1}, . . ., x_{n}) = \int (q (x_{1}, . . ., x_{n}) log p (x_{1}, . . ., x_{n})) d x_{1} . . . d x_{n}$

$= \int (q (x_{1}, . . ., x_{n}) log \frac{p (x_{1}, . . ., x_{n})}{q (x_{1}, . . ., x_{n})} + q (x_{1}, . . ., x_{n}) log q (x_{1}, . . ., x_{n})) d x_{1} . . . d x_{n}$

The second term in this integral is just $- n H_{q} (X)$ , $n$ times the entropy of $q$ , and the first term is $- n D_{K L} (q | | p)$ , so when we put everything together:

$p (x_{1}, . . ., x_{n}) = exp (- n (D_{K L} (q | | p) + H_{q} (X)))$

So any difference at all between $H_{p} (X)$ and $D_{K L} (q | | p) + H_{q} (X)$ will lead to the probability of almost all sequences sampled from our language model being exponentially squashed relative to the probability of most sequences sampled from the original distribution. I can also argue that $H_{q} (X)$ will be strictly larger than $H_{p} (X)$ : the latter essentially can be viewed as the entropy resulting from a perfect LLM with infinite context window, and $H (X | Y) \leq H (X)$ , conditioning on further information does not increase the entropy. So $(D_{K L} (q | | p) + H_{q} (X) - H_{p} (X))$ will definitely be positive.

This means that if you sample long enough from an LLM, and more importantly as the context window increases, it must generalise very far out of distribution to give good outputs. The fundamental problem of behaviour cloning I'm referring to is that we need examples of how to behave correctly is this very-out-of-distribution regime, but LLMs simply rely on the generalisation ability of transformer networks. Our prior should be that if you don't provide examples of correct outputs within some region of the input space to your function fitting algorithm, you don't expect the algorithm to yield correct predictions in that region.

Sky Moo

Sky Moo

Sky Moo

Best resources to learn philosophy of mind and AI?

Sky Moo

Sky Moo

Sky Moo

Best resources to learn philosophy of mind and AI?