The "maximize all the variables" tendency in reasoning about AGI.

Here are some lines of thought I perceive, which are probably straw to varying extents for some people and real to varying extents for other people. I give varying responses to each, but the point isn't the truth value of any given statement, but of a pattern across the statements:

1. If an AGI has a concept around diamonds, and is motivated in some way to make diamonds, it will make diamonds which maximally activate its diamond-concept circuitry (possible example). 
   1. My response.
2. An AI will be trained to minimal loss on the training distribution. 
   1. SGD does not reliably find minimum-loss configurations (modulo expressivity), in practice, in cases we care about. The existence of knowledge distillation is one large counterexample.
   2. Quintin: "In terms of results about model distillation, you could look at appendix G.2 of the Gopher paper. They compare training a 1.4 billion parameter model directly, versus distilling a 1.4 B model from a 7.1 B model."
3. Predictive processing means that the goal of the human learning process is to minimize predictive loss.^[1]
   1. In a process where local modifications are applied to reduce some locally computed error (e.g. "subconsciously predicting the presence of a visual-field edge where there definitely was not an edge"), it's just an enormous leap to go "and then the system minimizes this predictive error." 
   2. I think this is echoing the class of mistake I critique in Reward!=Optimization Target: "there's a local update on recent data $d$ to increase metric $f\left(d\right)$" -> "now I can use goal-laden language to describe the global properties of this process." I think this kind of language (e.g. "the goal of the human learning process", "minimize") should make you sit bolt upright in alarm and indignation. Minimize in a local neighborhood at the given timestep, maybe.
   3. Even if this claim (3) were true due to some kind of convergence result, that would be derived from a detailed analysis of learning dynamics. Not a situation where you can just look at an update rule and go "yeah kinda looks like it's trying to globally minimize predictive loss, guess I can just start calling it that now."
4. Policy networks are selected for getting as much reward as possible on training, so they'll do that.
   1. Response: "From my perspective, a bunch of selection-based reasoning draws enormous conclusions (e.g. high chance the policy cares about reward OOD) given vague / weak technical preconditions (e.g. policies are selected for reward) without attending to strength or mechanism or size of the trained network or training timescales or net direction of selection." 
   2. SOTA agents often don't get max reward on training.
   3. See more of my comments in this thread for actual detailed responses.
5. Because AI researchers try to make reward number go up because that's a nice legible statistic, we can't anticipate in advance ways in which the trained policy won't get maximal reward, otherwise the designers would as well.
6. An AI will effectively argmax over all plans according to its goals. 
   1. Response: I think this is not how cognition works in reality. There is a deeper set of concepts to consider, here, which changes the analysis. See: Alignment allows "nonrobust" decision-influences and doesn't require robust grading.

I think there are just a bunch of really strong claims like these, about some variable getting set really high or really low for some reason or another, and the claims now seem really weird to me. I initially feel confused why they're being brought up / repeated with such strength and surety. 

I speculate that there's a combination of:

1. It feels less impressive to imagine a training run where the AGI only gets a high average policy-gradient-intensity score (i.e. reward), and not a maximally high number. 
   1. Wouldn't a sophisticated mind not only be really good at realizing its values (like making paperclips), but also get minimal predictive loss? 
   2. (No.)
2. People don't know how to set realistic parameter values for these questions because they're confused about AGI / how intelligence works, and they don't explicitly note that to themselves, so they search under the streetlight of what current theory can actually talk about. They set the <quantity> setting to MAX or MIN in weak part because limit-based reasoning is socially approved of. 
   1. But in math and in life, you really have to be careful about your limit-based reasoning!
   2. IE you can make statements about global minima of the loss landscape. But what if training doesn't get you there? That's harder to talk about.^[2] 
   3. Or maybe some hapless RL researcher spends years thinking about optimal policies, only to wake up from the dream and realize that optimality results don't have to tell us a damn thing about trained policies...
   4. "Argmax" or "loss minimization" feel relatively understandable, whereas "learned decision-making algorithms" are fuzzy and hard. But what if the former set has little to do with the latter?

1. ^{^}

   My response here assumes that PP is about self-supervised learning on intermediate neuronal activations (e.g. "friend neuron") and also immediate percepts (e.g. "firing of retinal cells"). Maybe I don't understand PP if that's not true.  

2. ^{^}

   There can be very legit reasons to search under the streetlight, though. Sometimes that helps you see a bit farther out, by relaxing assumptions.