I think there’s benefit from being more specific about what we’re arguing about.
CLAIM 1: If there’s a learning algorithm whose reward function is X, then the trained model will not necessarily explicitly desire X.
I think everyone agrees that this is true, and that evolution provides an example. Most people don’t even know what inclusive genetic fitness is, and those who do, and who also know that donating eggs / sperm would score highly on that metric, nevertheless often don’t donate eggs / sperm.
CLAIM 2: If there’s a learning algorithm whose reward function is X, then the trained model cannot possibly explicitly desire X.
I think everyone agrees that this is false—neither Nate nor almost anyone else (besides Yamploskiy) thinks perfect AGI alignment is impossible. I think everyone probably also agrees that evolution provides a concrete counterexample—it’s a big world, people have all kinds of beliefs and desires, there’s almost certainly at least one person somewhere who knows what IGF is and explicitly wants to maximize theirs.
CLAIM 3: If there’s a learning algorithm whose reward function is X, and no particular efforts are taken to ensure alignment (e.g. freezing the model occasionally and attempting mechanistic interpretability, or simbox testing, or whatever), then the probability that the trained model will catastrophically fail to achieve X at all is: [FILL IN THE BLANK].
Here there’s some disagreement about how to fill in the blank in the context of AGI. (I’ll leave aside evolution for the moment, I’ll get back to it after these bullet points.)
- Nate’s probability for claim 3 in the context of AGI alignment is, IIUC, maybe upwards of 90% (but no more). And then he gets to >>90% probability of doom via disjunctive stuff.
- I say “No, I refuse to fill in the blank at the end of claim 3, until you need to tell me more specifics about this learning algorithm”.
- That said, for the AGI learning algorithms that I personally expect to be used (which are importantly disanalogous to evolution, and also importantly disanalogous to LLM+RLHF), I would fill in the blank at, I dunno, maybe 50%. (And then, like Nate, I get to much higher probability of doom for other reasons.)
- I’m not sure what OP thinks.
Then a separate question is whether evolution provides evidence on how to fill in that blank.
I agree with OP that humans today are doing clearly better than the very low bar of “catastrophically fail to achieve IGF at all”. I think the jury is still out about what will happen in the future. Maybe future humans will all upload their brains into computers, which would presumably count as zero IGF, if we define IGF via literal DNA molecules (which we don’t have to, but that’s another story). Whatever, I dunno.
But I mostly don’t care either way, because again, I don’t expect AGI learning algorithms to resemble evolution-of-humans tightly enough to make evolution a useful source of evidence on claim 3. I think different learning algorithms call for filling in the blank differently.
3
1
1
Some people like to use the evolution of homo sapiens as an argument by analogy concerning the apparent difficulty of aligning powerful optimization processes:
The much confused framing of this analogy has lead to a protracted debate about it's applicability.
The core issue is just misaligned mesaoptimization. We have a powerful optimization process optimizing world stuff according to some utility function. The concern is that a sufficiently powerful optimization process will (inevitably?) lead to internal takeover by a selfish mesa-optimizer unaligned to the outer utility function, resulting in a bad (low or zero utility) outcome.
In the AGI scenario, the outer utility function is CEV, or external human empowerment, or whatever (insert placeholder, not actually relevant). The optimization process is the greater tech economy and AI/ML research industry. The fear is that this optimization process, even if outer aligned, could result in AGI systems unaligned to the outer objective (humanity's goals), leading to doom (humanity's extinction). Success here would be largenum utility, and doom/extinction is 0. So the claim is mesaoptimization inner alignment failure leads to 0 utility outcomes: complete failure.
For the evolution of human intelligence, the optimizer is just evolution: biological natural selection. The utility function is something like fitness: ex gene replication count (of the human defining genes)[1]. And by any reasonable measure, it is obvious that humans are enormously successful. If we normalize so that a utility score of 1 represents a mild success - the expectation of a typical draw of a great apes species, then humans' score is >4 OOM larger, completely off the charts.[2]
So evolution of human intelligence is an interesting example: of alignment success. The powerful runaway recursive criticality that everyone feared actually resulted in an enormous anomalously high positive utility return, at least in this historical example. Human success, if translated into the AGI scenario, corresponds to the positive singularity of our wildest dreams.
Did it have to turn out this way? No!
Due to observational selection effects, we naturally wouldn't be here if mesaoptimization failure during brain evolution was too common across the multiverse.[3] But we could have found ourselves in a world with many archaeological examples of species achieving human general technocultural intelligence and then going extinct - not due to AGI of course, but simply due to becoming too intelligent to reproduce. But we don't, as far as I know.
And that is exactly what'd we necessarily expect to see in the historical record if mesaoptimization inner misalignment was a common failure mode: intelligent dinosaurs that suddenly went extinct, ruins of proto pachyderm cities, the traces of long forgotten underwater cetacean atlantis, etc.
So evolution solved alignment in the only sense that actually matters: according to its own utility function, the evolution of human intelligence enormously increased utility, rather than imploding it to 0.
So back to the analogy - where did it go wrong?
Nate's critique is an example of the naive engineer fallacy. Nate is critiquing a specific detail of evolution's solution, but failing to notice that all that matters is the score, and humans are near an all time high score success[5]. Evolution didn't make humans explicitly just optimize mentally for IGF because that - by itself - probably would have been a stupid failure of a design, and evolution is a superhuman optimizer whose designs are subtle, mysterious, and often beyond human comprehension.
Instead evolution created a solution with many layers and components- a defense in depth against mesaoptimization misalignment. And even though all of those components will inevitably fail in many individuals - even most! - that is completely irrelevant at the species level, and in fact just part of the design of how evolution explores the state space.
And finally, if all else fails, evolution did in fact find some weird way to create humans who rather obviously consciously optimize for IGF! And so if the other mechanisms had all started to fail too frequently, the genes responsible for that phenotype would inevitably become more common.
On further reflection, much of premodern history already does look like at least some humans consciously optimizing for something like IGF: after all, "be fruitful and multiply" is hardly a new concept. What do you think was really driving the nobility of old, with all their talk of bloodlines and legacies? There already is some deeper drive to procreate at work in our psyche (to varying degrees); we are clearly not all just mere byproducts of pleasure's pursuit[6].
The central takeway is that evolution adapted the brain's alignment mechanisms/protections in tandem with our new mental capabilities, such that the sharp left turn led to an enormous runaway alignment success.
Nitpick arguments about how you define this specifically are irrelevant and uninteresting. Homo sapiens is enormously successful! If you really think you know the true utility function of evolution, and humans are a failure according to that metric, you have simply deluded yourself. My argument here does not depend on the details of the evolutionary utility function. ↩︎
We are unarguably the most successful recent species, probably the most successful mammal species ever, and all that despite arriving in a geological blink of an eye. The dU/dt for homo sapiens is probably the highest ever, so we are tracking to be the most successful species ever, if current trends continue (which of course is another story). ↩︎
Full consideration of the observational selection effects also leads to an argument for alignment success via the simulation argument, as future alignment success probably creates many historical sims, whereas failures do not. ↩︎
Condom analogs are at least 5000 years old; there is ample evidence contraception was understand and used in various ancient civilizations, and many premodern tribal people understood herbal methods, so humans have probably had this knowledge since the beginning, in one form or another. (Although memetic evolution would naturally apply optimization pressure against wide usage) ↩︎
Be careful anytime you find yourself defining peak evolutionary fitness as anything other than the species currently smiling from on top a giant heap of utility. ↩︎
I say this as I am about to have a child myself, planned for reasons I cannot fully yet articulate. ↩︎