Thanks for the organized reply, i'll try to keep the same format.
Intrinsic motivation / curiosity:
You are familiar with the serotogenic and dopaminergic pathways and associated learning systems - typically simplified to an unsupervised learning component and a reward learning component.
My main point is that picture is incomplete/incorrect, and the brain's main learning system involves some form of empowerment. Curiosity is typically formulated as improvement in prediction capability, so it's like a derivative of more standard unsupervised learning (and thus probably a component of that system). But that alone isn't so great at learning for the roughly half the brain involved in action/motor/decision/planning. Some form of 'empowerment' criteria - specifically maximization of mutual information between actions and future world state (or observations, but the former is probably better) is a more robust general learning signal for action learning, and seems immune to the problems that plague pure curiosity approaches like the rule 101 type issues you mention.
For example: dopamine release on winning a bet has nothing to do with innate drives, it's purely an empowerment type learning signal. This is actually just the normal learning system at work.
The brain is mostly explained by this core learning system (which perhaps has just two or three main components). The innate drives (hunger,thirst,comfort/pain,sex,etc) are completely insufficient as signals for training the brain. They are instead satisficing drives that quickly saturate. They are secondary learning signals, but moreover they also can directly control/influence behavior in key situations, like the emotional subsystems. (Naturally there are exceptions to typical saturation - humans with a mutation causing perpetual unsatisfiable deep hunger and thus think about food all day long)
Empowerment that operates over learned world state also could support easy modulation - for example by up-weighting the importance of modeling humans/agents.
The altruism/empathic component isn't really like those innate drives (it's not really satisfying/saturating), and so instead is more core, part of the primary utility function and learning systems. (And also probably involves it's own neuromodulator component through oxytocin).
I think that intrinsic motivation in both humans & AGIs needs to be supplemented by a "drive to pay attention to humans", which in humans is based on superficial things like an innate brainstem circuit that disproportionately fires when hearing human speech.
Human infants grow up around humans who spend a large amount of time talking near the child. It's actually a dominant component of the audio landscape human infants grow up in. Any reasonably competent UL system will learn a model of human speech just from this training data (and ML systems prove this). Any innate human-speech brainstem circuit is of secondary importance - perhaps it speeds up learning a bit (like the simple brainstem face detector that helps prime the cortex), but it simply can not be necessary - as that would be incompatible with everything we know about the powerful universal learning capability of the brain.
Then once the brain has learned a recognition model of human speech, empowerment based learning is completely sufficient to learn speech production motor skills, simply by learning to maximize the mutual information between larynx motor actions and future predicted human speech audio world state. Again the brain may use some tricks to speed up learning, but the universal learning system is doing all the heavy lifting.
We also disagree about "drive for having high social status / impressing my friends": You think it's purely a special case of "intrinsic motivation" and thus requires no further explanation,
Once a child has learned a model of other humans - parents, friends, general models of other 'kids', etc, the empowerment system naturally then tries to learn ways to control these agents. This is so difficult that it basically drives a huge chunk of subsequent learning for most people, and becomes social theory of mind and innate 'game theory'. Social status is simply a proxy measure for influence, so it's closely correlated - or even just the same as - maximization of mutual info between actions and future agent beliefs (ie empowerment). If you think of what the word influence means, it's actually just a definition of a specific form of empowerment.
Other low-level drives:
The ancient innate Satisficing drives are what I think of as the low-level drive category (hunger,thirst,pain,sex,etc).
And finally the core emotions (happiness, sadness, fear, anger) are a third category. They are ancient subsystems that are both behavioral triggers and learning modulators. Happiness/sadness are just manifestations of predicted utility, whereas fear and anger are innate high-stress behavior modes (flight and fight responses). Humans then inherit more complex triggers - such as the injustice/righteousness triggers for anger, and more complex derived emotions.
I would put altruism/empathy in its own category, although it's also obviously closely connected to the emotion of love. Implementation wise it results in mixing of the learned utility functions of external agents into the agent's own root utility function. It is essentially evolved alignment. There are good reasons for this to evolve - basically shared genes and disposable somas, and we'll want something similar in AGI. It's a social component in the sense that it needs to connect the learned models of external agents to the core utility function.
I think we agree that AGI can have some or all of those human social instincts, but only if the AGI designers put them in, which would require (1) more research to nail down exactly how they're implemented, (2) advocacy etc.
We want to align AGI, and the brain's empathic/altruistic system could show us a practical way to achieve that. I don't see much role for the other emotional circuitry or innate drives. So we mostly agree here except you seem more interested in various 'social instincts' beyond just empathy/altruism (alignment).
Where does that leave anthropomorphism?
I believe humans (and more specifically high-impact humans) are mostly explained by a universal/generic learning system optimizing for a few things: mainly some mix of empowerment, curiosity, and altruism/empathy. There are many other brain systems (innate drives, emotions, etc), but they aren't so relevant.
I also believe brains are efficient, and thus AGI will end up being brain like - specifically it will also be mostly understandable as a universal neural learning system optimizing for some mix of empowerment, curiosity, and altruism/empathy or equivalents. There may be some other components, but they aren't as important.
Goals and values are complex learned concepts. Initial AGI will not reinvent all of human cultural history, and instead will just absorb human values - as they emerge from a universal learning system training on human world experience data, and AGI will have a similar universal learning system and similar experience training data. This doesn't imply AGI will have the exact same values of some typical mix of humans. Only that it's values will be mostly sampled from within the wide human-set.
From the original comment I was replying to (from Jon Garcia, not you):
There is no reason to think that the first AGIs will have goal/value structures any less alien to humans than would a superintelligent spider
There are deep reasons to believe AGI will be more anthropomorphic than not - mostly created in the image of humans. AGI will be much closer to a human mind than some hypothetical superintelligent spider.
Previously, I argued that we should expect future ML systems to often exhibit "emergent" behavior, where they acquire new capabilities that were not explicitly designed or intended, simply as a result of scaling. This was a special case of a general phenomenon in the physical sciences called More Is Different.
I care about this because I think AI will have a huge impact on society, and I want to forecast what future systems will be like so that I can steer things to be better. To that end, I find More Is Different to be troubling and disorienting. I’m inclined to forecast the future by looking at existing trends and asking what will happen if they continue, but we should instead expect new qualitative behaviors to arise all the time that are not an extrapolation of previous trends.
Given this, how can we predict what future systems will look like? For this, I find it helpful to think in terms of "anchors"---reference classes that are broadly analogous to future ML systems, which we can then use to make predictions.
The most obvious reference class for future ML systems is current ML systems---I'll call this the current ML anchor. I think this is indeed a pretty good starting point, but we’ve already seen that it fails to account for emergent capabilities.
What other anchors can we use? One intuitive approach would be to look for things that humans are good at but that current ML systems are bad at. This would include:
Models sufficiently far in the future will presumably have these sorts of capabilities. While this still leaves unknowns---for instance, we don't know how rapidly these capabilities will appear---it's still a useful complement to the current ML anchor. I'll call this the human anchor.
A problem with the human anchor is that it risks anthropomorphising ML by over-analogizing with human behavior. Anthropomorphic reasoning correctly gets a bad rap in ML, because it's very intuitively persuasive but has a mixed at best track record. This isn't a reason to abandon the human anchor, but it means we shouldn't be entirely satisfied with it.
This brings us to a third anchor, the optimization anchor, which I associate with the "Philosophy" or thought experiment approach that I've described previously. Here the idea is to think of ML systems as ideal optimizers and ask what a perfect optimizer would do in a given scenario. This is where Nick Bostrom's colorful description of a paperclip maximizer comes from, where an AI asked to make paperclips turns the entire planet into paperclip factories. To give some more prosaic examples:
Ideas produced by the optimization anchor are often met with skepticism, because they often contradict the familiar current ML anchor, and they don't benefit from the intuitive appeal of the human anchor. But the differences from these other two anchors are precisely what make the optimization anchor valuable. If you (like me) feel that both the current ML and human anchors paint an incomplete picture, then you should want a third independent perspective.
The optimization anchor does have limitations. Since it abstracts ML into an ideal optimizer, it ignores most on-the-ground facts about neural networks. This can lead to underconstrained predictions, and to ignoring properties that I think will be necessary for successfully aligning ML systems with humans. I'll say more about this later, but some particularly important properties are that neural networks often generalize in "natural" ways, that we can introspect on network representations, and that training dynamics are smooth and continuous. Researchers focused on the optimization anchor don't entirely ignore these facts, but I think they tend to underemphasize them and are overly pessimistic as a result.
The Value of Thought Experiments
The optimization anchor points to the value of thought experiments more generally. While it poses the thought experiment of "What if AI were a perfect optimizer?", there are many other thought experiments that can provide insights that'd be hard to obtain from the ML or human anchors. In this sense thought experiments are not a single anchor but a generator for anchors, which seems pretty valuable.
One thought experiment that I particularly like is: What happens if most of an agent's learning occurs not during gradient descent, but through in-context learning[1]? This is likely to happen eventually, as ML agents are rolled out over longer time horizons (think artificial digital assistants) and as ML improves at in-context learning. Once this does happen, it seems possible that agents' behavior will be controlled less by the "extrinsic" shaping of gradient descent and more by whatever "intrinsic" drives they happen to have[2]. This also seems like a change that could happen suddenly, since gradient descent is slow while in-context learning is fast.
It would be great if we had a community of researchers making thought experiments with clearly stated assumptions, explaining in detail the consequences of those assumptions and ideally connecting it to present-day research.
Other Anchors
There are many other anchors that could be helpful for predicting future ML systems. Non-human animal behavior could provide a broader reference class than humans alone. Evolution and the economy are both examples of powerful, distributed optimization processes. I am most excited about better understanding complex systems, which include biological systems, brains, organizations, economies, and ecosystems and thus subsume most of the reference classes discussed so far. It seems to me that complex systems have received little attention relative to their germaneness to ML. Indeed, emergence is itself a concept from complex systems theory that is useful for understanding recent ML developments.
Limitations of Thought Experiments
I've focused so far on predicting problems that we need to address. But at some point we actually have to solve the problems. In this regard thought experiments are weaker, since while they often point to important big-picture issues, in my view they fare poorly at getting the details right, which is needed for engineering progress. For instance, early thought experiments considered a single AI system that was much more powerful than any other contemporary technologies, while in reality there will likely be many ML systems with a continuous distribution of capabilities. More recent thought experiments impose discrete abstractions like "goals" and "objectives" that I don’t think will cleanly map onto real ML systems. Thus while thought experiments can point to general ideas for research, even mapping these ideas to the ontology of ML systems can be a difficult task.
As a result, while we can't blindly extrapolate empirical trends, we do need a concerted empirically-based effort to address future ML risks. I'll explain why I think this is possible in a later post, but first I'll take us through an example of "taking a thought experiment seriously", and what it implies about possible failure modes of ML systems.
In-context learning refers to learning that occurs during a single "rollout" of a model. The most famous example is GPT-3's ability to learn new tasks after conditioning on a small number of examples. ↩︎
While this statement borders on anthropomorphizing, I think it is actually justified. For instance, depending on the training objective, many agents will likely have a "drive" towards information-gathering, among others. ↩︎