This is a special post for quick takes by J Bostock. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Comment Permalink
What LLMs are doing doesn't look like "interpolation" of the training data to me, in the most commonly-used sense of the word.
Interpolation usually means taking the convex hull over a set of points, which looks like this:
In this interpolation, every point in the set can be expressed as a weighted average of the training datapoints, with coefficients summing to a value between 0 and 1.
But actually, if models have been trained on data A and on data B, they can often produce outputs containing both all of A, and all of B, instead of just, like, half of A and half of B. So I think this is the wrong analogy.
It looks more like a kind of Minkowski sum (or maybe a restricted generalized Minkowski sum) of the set with itself:
In this case, every point can be expressed as a weighted sum of the training datapoints, with all the coefficients individually being between 0 and 1. This is a very different shape, especially in very high dimensions, I think, because the sum of vectors
3
I'd say that even if it was interpolation, this would still mean we're interpolating within an absolutely huge space of tens of thousands of dimensions, the overwhelming majority of which is entirely unexplored. Interpolation can seem a bit basic when you look at low-dimensional cases, but I just don't think that applies the same way to higher-dimensional spaces.
20
The volume of a cross-polytope (i.e. diamond, octahedron ...) compared to the volume of the hypercube which encloses it (i.e. square, cube, ...) scales as 1/n!
