"Deep Learning" Is Function Approximation

[-]tin4822y3817

This does a great job of importing and translating a set of intuitions from a much more established and rigorous field. However, as with all works framing deep learning as a particular instance of some well-studied problem, it's vital to keep the context in mind:

Despite literally thousands of papers claiming to "understand deep learning" from experts in fields as various as computational complexity, compressed sensing, causal inference, and - yes - statistical learning, NO rigorous, first-principles analysis has ever computed any aspect of any deep learning model beyond toy settings. ALL published bounds are vacuous in practice.

It's worth exploring why, despite strong results in their own settings, and despite strong "intuitive" parallels to deep learning, this remains true. The issue is that all these intuitive arguments have holes big enough to accommodate, well, the end of the world. There are several such challenges in establishing a tight correspondence between "classical machine learning" and deep learning, but I'll focus on one that's been the focus of considerable effort: defining simplicity.

This notion is essential. If we consider a truly arbitrary function, there is no need for a relationship between the behavior on one input and the behavior on another - the No Free Lunch Theorem. If we want our theory to have content (that is, to constrain the behavior of a Deep Learning system whatsoever) we'll need to narrow the range of possibilities. Tools from statistical learning like the VC dimension are useless as is due to overparameterization, as you mention. We'll need a notion of simplicity that captures what sorts of computational structures SGD finds in practice. Maybe circuit size, or minima sharpness, or noise sensitivity... - how hard could it be?

Well no one's managed it. To help understand why, here are two (of many) barrier cases:

Sparse parity with noise. For an input bitstring x, y is defined as the xor of a fixed, small subset of indices. E.g if the indices are 1,3,9 and x is 101000001 then y is 1 xor 1 xor 1 = 1. Some small (tending to zero) measurement error is assumed. Though this problem is approximated almost perfectly by extremely small and simple boolean circuits (a log-depth tree of xor gates with inputs on the chosen subset), it is believed to require an exponential amount of computation to predict even marginally better than random! Neural networks require exponential size to learn it in practice.
Deep Learning Fails
The protein folding problem. Predict the shape of a protein from its amino acid sequence. Hundreds of scientists have spent decades scouring for regularities, and failed. Generations of supercomputers have been built to attempt to simulate the subtle evolution of molecular structure, and failed. Billions of pharmaceutical dollars were invested - hundreds of billions were on the table for success. The data is noisy and multi-modal. Protein language models learn it all anyway.
Deep Learning Succeeds!

For what notion is the first problem complicated, and the second simple?

Again, without such a notion, statistical learning theory makes no prediction whatsoever about the behavior of DL systems on new examples. If a model someday outputted a sequence of actions which caused the extinction of the human race, we couldn't object on principle, only say "so power-seeking was simpler after all". And even with such a notion, we'd still have to prove that Gradient Descent tends to find it in practice and a dozen other difficulties...

Without a precise mathematical framework to which we can defer, we're left with Empirics to help us choose between a bunch of sloppy, spineless sets of intuitions. Much less pleasant. Still, here's a few which push me towards Deep Learning as a "computationally general, pattern-finding process" rather than function approximation:

Neural networks optimized only for performance show surprising alignment with representations in the human brain, even exhibiting 1-1 matches between particular neurons in ANNs and living humans. This is an absolutely unprecedented level of predictivity, despite the models not being designed for such and taking no brain data as input.
LLM's have been found to contain rich internal structure such as grammatical parse-trees, inference-time linear models, and world models. This sort of mechanistic picture is missing from any theory that considers only i/o
Small changes in loss (i.e. function approximation accuracy) have been associated with large qualitative changes in ability and behavior - such as learning to control robotic manipulators using code, productively recurse to subagents, use tools, or solve theory of mind tasks.

I know I've written a lot, so I appreciate your reading it. To sum up:

Despite intuitive links, efforts to apply statistical learning theory to deep learning have failed, and seem to face substantial difficulties
So, we have to resort to experiment where I feel this intuitive story doesn't fit the data, and provide some challenge cases

[-]Quintin Pope2y197

NO rigorous, first-principles analysis has ever computed any aspect of any deep learning model beyond toy settings

This is false. From the abstract of Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer

Hyperparameter (HP) tuning in deep learning is an expensive process, prohibitively so for neural networks (NNs) with billions of parameters. We show that, in the recently discovered Maximal Update Parametrization (muP), many optimal HPs remain stable even as model size changes. This leads to a new HP tuning paradigm we call muTransfer: parametrize the target model in muP, tune the HP indirectly on a smaller model, and zero-shot transfer them to the full-sized model, i.e., without directly tuning the latter at all. We verify muTransfer on Transformer and ResNet. For example, 1) by transferring pretraining HPs from a model of 13M parameters, we outperform published numbers of BERT-large (350M parameters), with a total tuning cost equivalent to pretraining BERT-large once; 2) by transferring from 40M parameters, we outperform published numbers of the 6.7B GPT-3 model, with tuning cost only 7% of total pretraining cost. A Pytorch implementation of our technique can be found at this http URL and installable via `pip install mup`.

muP comes from a principled mathematical analysis of how different ways of scaling various architectural hyperparameters alongside model width influences activation statistics.

[-]tin4822y93

I was trying to make a more specific point. Let me know if you think the distinction is meaningful -

So there are lots of "semi-rigorous" successes in Deep Learning. One I understand better than muP is good old Xavier initialization. Assuming that the activations at a given layer are normally distributed, we should scale our weights like 1/sqrt(n) so the activations don't diverge from layer to layer (since the sum of independent normals scales like sqrt(n)). This is exactly true for the first gradient step, but can become false at any later step once the weights are no longer independent and can "conspire" to blow up the activations anyway. So not a "proof" but successful in practice.

My understanding of muP is similar in that is "precise" only if certain correlations are well controlled (I'm fuzzy here). But still v. successful in practice. But the proof isn't airtight and we still need that last step - checking "in practice".

This is very different from the situation within statistical learning itself, which has many beautiful and unconditional results which we would very much like to port over to deep learning. My central point is that in the absence of a formal correspondence, we have to bridge the gap with evidence. That's why the last part of my comment was some evidence I think speaks against the intuitions of statistical learning theory. I consider Xavier initialization, muP, and scaling laws, etc. as examples where this bridge was successfully crossed - but still necessary! And so we're reduced to "arguing over evidence between paradigms" when we'd prefer to "prove results within a paradigm"

[-]Zack_M_Davis2y31

For what notion is the first problem complicated, and the second simple?

I might be out of my depth here, but—could it be that sparse parity with noise is just objectively "harder than it sounds" (because every bit of noise inverts the answer), whereas protein folding is "easier than it sounds" (because if it weren't, evolution wouldn't have solved it)?

Just because the log-depth xor tree is small, doesn't mean it needs to be easy to find, if it can hide amongst vastly many others that might have generated the same evidence ... which I suppose is your point. (The "function approximation" frame encourages us to look at the boolean circuit and say, "What a simple function, shouldn't be hard to noisily approximate", which is not exactly the right question to be asking.)

[-]tailcalled2y40

I think it's also a question of the learning techniques used. It seems like generalizable solutions to xor involve at least one of the two following:

noticing that many of the variables in the input can be permuted without changing the output and therefore it's only the counts of the variables that matter
noticing how the output changes or doesn't change when you start with one input and flip the variable one at a time

But current neural networks can't use either of these techniques, partly because it doesn't align well with the training paradigm. They both kind of require the network to be able to pick the inputs to see the ground truth for, whereas training based on a distribution has the (input, output) pairs randomized.

To "humanize" these problems, we can:

Break intuitive permutation invariance by using different symbols in each slot. So instead of an input looking like 010110 or 101001 or 111000, it might look like A😈4z👞i or B🔥😱Z😅0 or B😈😱Z😅i.
Break the ability to notice the effects of single bitflips by just seeing random strings rather than neighboring strings.

This makes it intuitively much harder to me.

[-]ErickBall2y327

Downvoted because I waded through all those rhetorical shenanigans and I still don't understand why you didn't just say what you mean.

[-]Zack_M_Davis2y1511

This comment had been apparently deleted by the commenter (the comment display box having a "deleted because it was a little rude, sorry" deletion note in lieu of the comment itself), but the ⋮-menu in the upper-right gave me the option to undelete it, which I did because I don't think my critics are obligated to be polite to me. (I'm surprised that post authors have that power!) I'm sorry you didn't like the post.

[-]Algon2y510

I am suprised that you have that affordance. I want to know I can delete my comments and be sure they won't get read by anyone after I delete them.

[-]habryka2y241

Oh, hmm, this is an edge-case we've never ran into. The point of giving authors the ability to undelete comments they can delete is so that they can reverse deletions they made (or admins made on their post trying to help them enforce their norms) not the deletions other people made.

I'll look into fixing the permissions here. Definitely not intended, just a side effect of some other things we tried to do.

[-]abramdemski2y70

To me, the lengthy phrases do in fact get closer to "zack saying what zack meant" than the common terms like 'deep learning' -- but, like you, I didn't really get anything new out of the longer phrases. I believe that people who don't already think of deep learning as function approximation may get something out of it tho. So in consequence I didn't downvote or upvote.

[-]RHollerith2y57

As a deep-learning novice, I found the post charming and informative.

[+][comment deleted]2y20

[-]Veedrac2y183

It took me a good while reading this to figure out whether it was a deconstruction of tabooing words. I would have felt less so if the post didn't keep replacing terms with ones that are both no less charged and also no more descriptive of the underlying system, and then start drawing conclusions from the resulting terms' aesthetics.

With regards to Yudkowsky's takes, the key thing to keep in mind is that Yudkowsky started down his path by reasoning backwards from properties ASI would have, not from reasoning forward from a particular implementation strategy. The key reason to be concerned that outer optimization doesn't define inner optimization isn't a specific hypothesis about whether some specific strategy with neural networks will have inner optimizers, it's because ASI will by necessity involve active optimization on things, and we want our alignment techniques to have at least any reason to work in that regime at all.

[-]Closed Limelike Curves2y155

I'm not sure what point this post is trying to make exactly. Yes, it's function approximation; I think we all know that.

When we talk about inner and outer alignment, outer alignment is "picking the correct function to learn." (When we say "loss," we mean the loss on a particular task, not the abstract loss function like RMSE.)

Inner alignment is about training a model that generalizes to situations outside the training data.

[-]Olli Järviniemi2y12-1

I liked how this post tabooed terms and looked at things at lower levels of abstraction than what is usual in these discussions.

I'd compare tabooing to a frame by Tao about how in mathematics you have the pre-rigorous, rigorous and post-rigorous stages. In the post-rigorous stage one "would be able to quickly and accurately perform computations in vector calculus by using analogies with scalar calculus, or informal and semi-rigorous use of infinitesimals, big-O notation, and so forth, and be able to convert all such calculations into a rigorous argument whenever required" (emphasis mine).

Tabooing terms and being able to convert one's high-level abstractions into mechanistic arguments whenever required seems to be the counterpart in (among others) AI alignment. So, here's positive reinforcement for taking the effort to try and do that!

Separately, I found the part

(Statistical modeling engineer Jack Gallagher has described his experience of this debate as "like trying to discuss crash test methodology with people who insist that the wheels must be made of little cars, because how else would they move forward like a car does?")

quite thought-provoking. Indeed, how is talk about "inner optimizers" driving behavior any different from "inner cars" driving the car?

Here's one answer:

When you train a ML model with SGD-- wait, sorry, no. When you try construct an accurate multi-layer parametrized graphical function approximator, a common strategy is to do small, gradual updates to the current setting of parameters. (Some could call this a random walk or a stochastic process over the set of possible parameter-settings.) Over the construction-process you therefore have multiple intermediate function approximators. What are they like?

The terminology of "function approximators" actually glosses over something important: how is the function computed? We know that it is "harder" to construct some function approximators than others, and depending on the amount of "resources" you simply cannot^[1] do a good job. Perhaps a better term would be "approximative function calculators"? Or just anything that stresses that there is some internal process used to convert inputs to outputs, instead of this "just happening".

This raises the question: what is that internal process like? Unfortunately the texts I've read on multi-layer parametrized graphical function approximation have been incomplete in these respects (I hope the new editions will cover this!), so take this merely as a guess. In many domains, most clearly games, it seems like "looking ahead" would be useful for good performance^[2]: if I do X, the opponent could do Y, and I could then do Z. Perhaps these approximative function calculators implement even more general forms of search algorithms.

So while searching for accurate approximative function calculators we might stumble upon calculators that itself are searching for something. How neat is that!

I'm pretty sure that under the hood cars don't consist of smaller cars or tiny car mechanics - if they did, I'm pretty sure my car building manual would have said something about that.

^{^}
(As usual, assuming standard computational complexity conjectures like P != NP and that one has reasonable lower bounds in finite regimes, too, rather than only asymptotically.)
^{^}
Or, if you don't like the word "performance", you may taboo it and say something like "when trying to construct approximative function calculators that are good at playing chess - in the sense of winning against a pro human or a given version of Stockfish - it seems likely that they are, in some sense, 'looking ahead' for what happens in the game next; this is such an immensely useful thing for chess performance that it would be surprising if the models did not do anything like that".

[-]DaemonicSigil2y105

It's the same thing for piecewise-linear functions defined by multi-layer parameterized graphical function approximators: the model is the dataset. It's just not meaningful to talk about what a loss function implies, independently of the training data. (Mean squared error of what? Negative log likelihood of what? Finish the sentence!)

This confusion about loss functions...

I don't think this is a confusion, but rather a mere difference in terminology. Eliezer's notion of "loss function" is equivalent to Zack's notion of "loss function" curried with the training data. Thus, when Eliezer writes about the network modelling or not modelling the loss function, this would include modelling the process that generated the training data.

[-]abramdemski2y3112

The issue seems more complex and subtle to me.

It is fair to say that the loss function (when combined with the data) is a stochastic environment (stochastic due to sampling the data), and the effect of gradient descent is to select a policy (a function out of the function space) which performs very well in this stochastic environment (achieves low average loss).

If we assume the function-approximation achieves the minimum possible loss, then it must be the case that the function chosen is an optimal control policy where the loss function (understood as including the data) is the utility function which the policy is optimal with respect to.

In this framing, both Zack and Eliezer would be wrong:

Zack would be wrong because there is nothing nonsensical about asking whether the function-approximation "internalizes" the loss. Utility functions are usually understood behaviorally; a linear regression might not "represent" (ie denote) squared-error anywhere, but might still be utility-theoretically optimal with respect to mean-squared error, which is enough for "representation theorems" (the decision-theory thingy) to apply.
Eliezer would be wrong because his statement that there is no guarantee about representing the loss function would be factually incorrect. At best Eliezer's point could be interpreted as saying that the representation theorems break down when loss is merely very low rather than perfectly minimal.

But Eliezer (at least in the quote Zack selects) is clearly saying "explicit internal representation" rather than the decision-theoretic "representation theorem" thingy. I think this is because Eliezer is thinking about inner optimization, as Zack also says. When we are trying to apply function-approximation ("deep learning") to solve difficult problems for us -- in particular, difficult problems never seen in the data-set used for training -- it makes some sense to suppose that the internal representation will involve nontrivial computations, even "search algorithms" (and importantly, we know of no way to rule this out without crippling the generalization ability of the function-approximation).

So based on this, we could refine the interpretation of Eliezer's point to be: even if we achieve the minimum loss on the data-set given (and therefore obey decisiot-theretic representation-theorems in the stochastic environment created by the loss function combined with the data), there is no particular guarantee that the search procedure learned by the function-approximation is explicitly searching to minimize said loss.

This is significant because of generalization. We actually want to run the approximated-function on new data, with hopes that it does "something appropriate". (This is what Eliezer means when he says "distribution-shifted environments" in the quote.) This important point is not captured in your proposed reconciliation of Zack and Eliezer's views.

But then why emphasize (as Eliezer does) that the function approximation does not necessarily internalize the loss function it is trained on? Internalizing said loss function would probably prevent it from doing anything truly catastrophic (because it is not planning for a world any different than the actual training data it has seen). But it does not especially guarantee that it does what we would want it to do. (Because the-loss-function-on-the-given-data is not what we really want; really we want some appropriate generalization to happen!)

I think this is a rhetorical simplification, which is fair game for Zack to try and correct to something more accurate. Whether Eliezer truly had the misunderstanding when writing, I am not sure. But I agree that the statement is, at least, uncareful.

Has Zack succeeded in correcting the issue by providing a more accurate picture? Arguably TurnTrout made the same objection in more detail. He summarizes the whole thing into two points:

Deep reinforcement learning agents will not come to intrinsically and primarily value their reward signal; reward is not the trained agent's optimization target.
Utility functions express the relative goodness of outcomes. Reward is not best understood as being a kind of utility function. Reward has the mechanistic effect of chiseling cognition into the agent's network. Therefore, properly understood, reward does not express relative goodness and is therefore not an optimization target at all.

(Granted, TurnTrout is talking about reward signals rather than loss functions, and this is an important distinction; however, my understanding is that he would say something very similar about loss functions.)

Point #1 appears to strongly agree with at least a major part of Eliezer's point. To re-quote the List of Lethalities portion Zack quotes in the OP:

Even if you train really hard on an exact loss function, that doesn't thereby create an explicit internal representation of the loss function inside an AI that then continues to pursue that exact loss function in distribution-shifted environments. Humans don't explicitly pursue inclusive genetic fitness; outer optimization even on a very exact, very simple loss function doesn't produce inner optimization in that direction. [...] This is sufficient on its own [...] to trash entire categories of naive alignment proposals which assume that if you optimize a bunch on a loss function calculated using some simple concept, you get perfect inner alignment on that concept.

However, I think point #2 is similar in spirit to Zack's objection in the OP. (TurnTrout does not respond to the same exact passage, but has his own post taking issues with List of Lethalities.)

I will call the objection I see in common between Zack and TurnTrout the type error objection. Zack says that of course a line does not "represent" the loss function of a linear regression; why would you even want it to? TurnTrout says that "reward is not the optimization target" -- we should think of a reward function as a "chisel" which shapes a policy, rather than thinking of it as the goal we are trying to instill in the policy. In both cases, I understand them as saying that the loss function used for training is an entirely different sort of thing from the goals an intelligent system pursues after training. (The "wheels made of little cars" thing also resembles a type-error objection.)

While I strongly agree that we should not naively assume a reinforcement-learning agent internalizes the reward as its utility function, I think the type-error objection is over-stated, as may be clear from my point about decision-theoretic representation theorems at the beginning.

Reward functions do have the wrong type signature, but neural networks are not actually trained on reward gradients; rather, a loss is defined from the reward in some way. The type signature of the loss function is not wrong; indeed, if training were perfect, then we could conclude that the resulting neural networks would be decision-theoretically perfect at minimizing loss on the training distribution.

What we would not be able to make confident predictions about is what such systems would do outside of the training distribution, where the training procedure has not exercised selection pressure on the behavior of the system. Here, we must instead rely on the generalization power of function-approximation, which (seen through a somewhat bayesian lens) means trusting the system to have the inductive biases which we would want.

[-]DaemonicSigil2y40

To be clear, I'm definitely pretty sympathetic to TurnTrout's type error objection. (Namely: "If the agent gets a high reward for ingesting superdrug X, but did not ingest it during training, then we shouldn't particularly expect the agent to want to ingest superdrug X during deployment, even if it realizes this would produce high reward.") But just rereading what Zack has written, it seems quite different from what TurnTrout is saying and I still stand by my interpretation of it.

eg. Zack writes: "obviously the line itself does not somehow contain a representation of general squared-error-minimization". So in this line fitting example, the loss function, i.e. "general squared-error-minimization" refers to the function , and not $L (f_{θ})$ .
And when he asks why one would even want the neural network to represent the loss function, there's a pretty obvious answer of "well, the loss function contains many examples of outcomes humans rated as good and bad and we figure it's probably better if the model understands the difference between good and bad outcomes for this application." But this answer only applies to the curried loss.

I wasn't trying to sign up to defend everything Eliezer said in that paragraph, especially not the exact phrasing, so can't reply to the rest of your comment which is pretty insightful.

[-]Algon2y32

In both cases, I understand them as saying that the loss function used for training is an entirely different sort of thing from the goals an intelligent system pursues after training.

I think Turntrout would object to that charecterization as it is privileging the hypothesis that you get systems which pursue goals after training. I'm assuming you mean the agent does some sort of EV maximization by "goals an intelligent systems pursues". Though I have a faint suspicion Turntrout would disagree even with a more general interpretation of "pursues goals".

[-]tailcalled2y61

It wasn't obvious beforehand that this would work! You'd expect that if your function approximator has more parameters than you have example input–output pairs, it would overfit, implementing a complicated function that reproduced the example input–output pairs but outputted crazy nonsense for other choices of x—the more expressive function approximator proving useless for the lack of evidence to pin down the correct approximation.

And that is what we see for function approximators with only slightly more parameters than example input–output pairs, but for sufficiently large function approximators, the trend reverses and "generalization" improves—the more expressive function approximator proving useful after all, as it admits algorithmically simpler functions that fit the example pairs.

One way I think of this that makes it more intuitive:

You can understand mean squared error as implying that your regression equation has a Gaussian-distributed error term $\varepsilon$:

\[Y = \beta X + \varepsilon\]

In simple statistical models, this $\varepsilon$ latent is not explicitly estimated, but instead eliminated by pumping up the sample size so much that it can be averaged away. This only works when you have far fewer parameters than data points.

Overparameterized models, on the other hand, use the "texture" of the datapoint (as in, rare, "high-frequency" "noise" characteristics that allow you to relatively uniquely pick out the datapoint from the dataset) to identify the datapoint, and shove the $\varepsilon$ term into that "texture". (You can see this explicitly in 1D regression problems, where the overparameterized models have a sharp deviation from the trend line at each data point.)

I think this works under quite general conditions. I vaguely remember there was a paper that showed that if you train a NN on random labels, then it "learns to memorize" in some sense (can't remember what sense exactly but I think it was something like, it could more quickly be trained on a different random labelling). I suspect this is because it learns to emphasize the "textures" of the datapoints in the training set, finding features that more unqiuely distinguish datapoints in the future.

Real NNs obviously also learn real structure, not just $\varepsilon$, but I assume they do some mixture where at least part of what they learn is just arbitrary features that distinguish unique datapoints.

[-]lemonhope2y60

I think this is a strong starting point but I think the nice crisp "neural net = function approximator" mostly falls apart as a useful notion when you do fancy stuff with your neural net like active learning or RLAIF. Maybe it's not technically the neural net doing that...

I guess we don't have great terms to delineate all these levels of the system:

code that does a forward pass (usually implicitly also describing backward pass & update given loss)
code that does that plus training (ie data fetch and loss function)
that plus RL environment or training set
that plus "training scaffolding" code that eg will do active learning or restart the game if it freezes
just code & weights for a forward pass during inference (presuming that the system has separable training and inference stages)
that plus all the "inference scaffolding" code which will eg do censorship or internet search or calculator integration
that plus the "inference UI". (Consider how differently people use gpt4 api vs chatgpt website.) (This could also eg be the difference between clicking a checkbox for who to kill and clicking the cancel button on the notification!)
the actual final system turned on in the wild with adversarial users and distribution shift and so on

I wonder if some folks are taking past each other by implicitly referring to different items above...

[-]Logan Zoellner2y52

but for sufficiently large function approximators, the trend reverses

Transformers/deep learning work because of built-in regularization methods (like dropout layers) and not because "the trend reverses". If you did naive "best fit polynomial" with a 7 billion parameter polynomial you would not get a good result.

[-]hmys2y58

Other people were commending your tabooing of words, but I feel using terms like "multi-layer parameterized graphical function approximator" fails to do that, and makes matters worse because it leads to non-central fallacy-ing. It'd been more appropriate to use a term like "magic" or "blipblop". Calling something a function appropriator leads to readers carrying a lot of associations into their interpretation, that probably don't apply to deep learning, as deep learning is a very specific example of function approximation, that deviates from the prototypical examples in many respects. (I think when you say "function approximator" the image that pops into most peoples head is fitting a polynomial to a set of datapoints in R^2)

Calling something a function approximator is only meaningful if you make a strong argument for why a function approximator cant (or at least is systematically unlikely to) give rise to specific dangerous behaviors or capabilities. But I don't see you giving such arguments in this post. Maybe I did not understand it. In either case, you can read posts like Gwern's "Tools want to be agents" or Yudkowsky's writings, explaining why goal directed behavior is a reasonable thing to expect to arise from current ML, and you can replace every instance of "neural network" / "AI" with "multi-layer parameterized graphical function approximator", and I think you'll find that all the arguments make just as much sense as they did before. (modulo some associations seeming strange, but like I said, I think thats because there is some non-central fallacying going on).

[-]gwern2y*2319

That deviates from the prototypical examples in many respects.

It basically proves too much because it's equivocation. I am struggling to find anything in Zack's post which is not just the old wine of the "just" fallacy in new 'function approximation' skins. When someone tells you that a LLM is "just" next token prediction, or a neural network is "just some affine layers with nonlinearities" or it's "just a Markov chain with a lot of statistics", then you've learned more about the power and generality of 'next token prediction' etc than you have what they were trying to debunk.

If I use the mean squared error loss function to approximate a set of data points in the plane with a line (which some authors call a "linear regression model" for some reason), obviously the line itself does not somehow contain a representation of general squared-error-minimization. The line is just a line.

I don't think that is obvious at all, and is roughly on the level of saying 'a tiger is just a set of atoms in a 3D volume' or 'programs are just a list of bits'. What are these data points on this hyperplane, exactly...? They could be anything - they could be, say, embeddings of optimization algorithms*. If you had an appropriate embedding of the latent space of algorithms, why can't there be a point (or any other kind of object or structure, such as a line) which corresponds to general squared-error minimization or others? And this doesn't seem 'possible', this seems likely: already an existing LLM like GPT-4 or Claude-3 Opus is clearly mapping its few-shot examples to some sort of latent space, appears to be doing internal gradient descent on higher level embeddings or manifolds, and is quite effective at writing things like 'here are 10 variants of error minimization optimization algorithms as examples; write a Python program using Numpy to write a new one which [...]' (and if it is not, it sure seems like future ones will); something inside that LLM must correspond to a small number of bytes and some sort of mathematical object and represent the combination of the points. Programs and algorithms are 'just' data points, like everything else. ("With an appropriate encoding, an AGI is just an index into the decimal expansion of pi; an index, a simple ordinary natural number, is obviously completely harmless; QED, AGI is completely harmless.") Which means that if you think something is 'just' data, then your assertions are meaninglessly vacuous because if it applies to everything, then it means nothing.

* Or more relevantly, RL algorithms such as specific agents... of course a line or point on an appropriate hyperplane can 'represent' the loss function, and it would be a useful thing to do so. Why would you ever want a system to not be able to do that? We can see in Decision Transformers and research into the meta-reinforcement learning behavior of LLMs that large LLMs prompted with reward functions & scenarios, trained by imitation learning at scale across countless agents, do what must be exactly that, and represent environments and rewards, and can eg. explicitly write the source code to implement reward functions for RL training of sub-agents. I think it's telling that these results generally do not come up in these posts about how 'reward is not the optimization target'.

[-]Zack_M_Davis2y136

I am struggling to find anything in Zack's post which is not just the old wine of the "just" fallacy [...] learned more about the power and generality of 'next token prediction' etc than you have what they were trying to debunk.

I wouldn't have expected you to get anything out of this post!

Okay, if you project this post into a one-dimensional "AI is scary and mysterious" vs. "AI is not scary and not mysterious" culture war subspace, then I'm certainly writing in a style that mood-affiliates with the latter. The reason I'm doing that is because the picture of what deep learning is that I got from being a Less Wrong-er felt markedly different from the picture I'm getting from reading the standard textbooks, and I'm trying to supply that diff to people who (like me-as-of-eight-months-ago, and unlike Gwern) haven't read the standard textbooks yet.

I think this is a situation where different readers need to hear different things. I'm sure there are grad students somewhere who already know the math and could stand to think more about what its power and generality imply about the future of humanity or lack thereof. I'm not particularly well-positioned to help them. But I also think there are a lot of people on this website who have a lot of practice pontificating about the future of humanity or lack thereof, who don't know that Simon Prince and Christopher Bishop don't think of themselves as writing about agents. I think that's a problem! (One which I am well-positioned to help with.) If my attempt to remediate that particular problem ends up mood-affiliating with the wrong side of a one-dimensional culture war, maybe that's because the one-dimensional culture war is crazy and we should stop doing it.

[-]gwern2y*78

I don't object to ironic exercises like "A Modest Proposal" or "On The Impossibility of Supersized Machines", but it can certainly be difficult to write and ensure the message gets across, and I don't think it does here.

[+][comment deleted]2y2-1

[-]tailcalled2y40

It may also be worth adding that transformers aren't piecewise linear. A self-attention layer dynamically constructs pathways for information to flow through, which is very nonlinear.

[-]Algon2y31

The main thing I liked about this post was that you were making a serious attempt at tabooing your words. So I upvoted your post.

I would've strong upvoted this if you were more consistent with tabooing words (e.g. exploit, optimization etc.) and attempted intensive definitions for more of them. I understand that's hard, because a bunch of these words have not been given crisp-definitions, or if they have, those definitions might not be obvious. Still, the added precision would've been a boon for a post talking about how we need to be clearer about what our state of knowledge is so we don't go confusing ourselves.

[-]Gurkenglas2y20

Let's investigate whether functions must necessarily contain an agent in order to do sufficiently useful cognitive work. Pick some function of which an oracle would let you save the world.

LESSWRONG
LW

LESSWRONG
LW

98

"Deep Learning" Is Function Approximation

98

98

A Surprising Development in the Study of Multi-layer Parameterized Graphical Function Approximators

Multi-layer Parameterized Graphical Function Approximators Have Many Exciting Applications

An Example of Applying Multi-layer Parameterized Graphical Function Approximators in Success-Antecedent Computation Boosting

Risks From Learned Approximation

Bibliography