This is a special post for quick takes by Stephen Fowler. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
There are meaningful distinctions between evolution and other processes referred to as "optimisers"
People should be substantially more careful about invoking evolution as an analogy for the development of AGI, as tempting as this comparison is to make.
"Risks From Learned Optimisation" is one of the most influential AI Safety papers ever written, so I'm going to use it's framework for defining optimisation.
"We will say that a system is an optimiser if it is internally searching through a search space (consisting of possible outputs, policies, plans, strategies, or similar) looking for those elements that score high according to some objective function that is explicitly represented within the system" ~Hubinger et al (2019)
It's worth noting that the authors of this paper do consider evolution to be an example of optimisation (something stated explicitly in the paper). Despite this, I'm going to argue the definition shouldn't apply to evolution.
2 strong (and 1 weak) Arguments That Evolution Doesn't Fit This Definition:
Weak Argument 0:
Evolution itself isn't a separate system that is optimising for something. (Micro)evolution is the change in allele frequency over generations. There is no separate entity you can point to and call "evolution".
Consider how different this is from a human engaged in optimisation to design a bottle cap. We have the system that optimises, and the system that is optimised.
It is tempting to say "the system optimises itself" but then go ahead and define the system you would say is engaged in optimisation. That system isn't "evolution" but is instead something like "the environment", "all carbon based structures complexes on earth" or "all matter on the surface of earth" etc.
Strong Argument 1:
Evolution does not have an explicitly represented objective function.
This is a major issue. When I'm training a model against a loss function I can explicitly represent that loss function. It is possible to physically implement
There is no single explicit representation of what "fitness" is within our environment.
Strong Argument 2:
Evolution isn't a "conservative" process. The thing that it is optimising "toward" is dependent on the current state of the environment, and changes over time. It is possible for evolution to get caught in "loops" or "cycles".
- A refresher on conservative fields.
In physics a conservative vector field is vector field that can be understood of the gradient of some other function. By associating any point in that vector field with the corresponding point on the other function, you meaningfully order each point in your field.
To be less abstract, imagine your field is "slope" which describes the gradient of a mountain range. You can talk meaningfully order the points in the slope field by the height of the point they correspond to on the mountain range.
In a conservative vector field, the curl everywhere is zero. Letting a ball roll down the mountain range (with a very high amount of friction) and the ball will find its way to a local minima and stop.
In a non-conservative vector field it is possible to create paths that loop forever.
My local theme-park has a ride called the "Lazy River" which is an artificial river which has been formed into a loop. There is no change in elevation, and the water is kept flowing clockwise by a series of underwater fans which continuously put energy into the system. Families hire floating platforms and can drift endlessly in a circle until their children get bored.
If you throw a ball into the Lazy River it will circle endlessly. If we write down a vector field that describes the force on the ball at any point in the river, it isn't possible to describe this field as the gradient of another field. There is no absolute ordering of points in this field.
- Evolution isn't conservative
In the ball rolling over the hills, we might be able to say that as time evolves it seems to be getting "lower". By appealing to the function that it is the gradient of, we can meaningfully say if two points are higher, lower or the same height.
In the lazy river, this is no longer possible. Locally, you could describe the motion of the ball as rolling down a hill, but continuing this process around the entire loop tells you that you are describing an impossible MC Escher Waterfall.
If evolution is not conservative (and hence has no underlying goal it is optimising toward) then it would be possible to observe creatures evolving in circles, stuck in "loops". Evolving, losing then re-evolving the same structures.
This is not only possible, but it has been observed. The side-blotched lizard appear to shift throat colours in a cyclic repeating pattern. For more details, See this talk by John Baez.
To summarise, the "direction" or "thing evolution is optimising toward" cannot be some internally represented thing, because the thing it optimises toward is a function of not just the environment but also of the things evolving in that environment.
Who cares?
Using evolution as an example of "optimisation" is incredible common among AI safety researchers, and can be found in Yudkowsky's writing on Evolution in The Sequences.
I think the notion of evolution as an optimiser can do more harm than good.
As a concrete example, Nate's "Sharp Left Turn" post was weakened substantially by invoking an evolution analogy, which spawned a lengthy debate (see Pope, 2023 and the response from Zvi). This issue could have been skipped entirely simply by arguing in favour of the Sharp Left Turn without any reference to evolution (see my upcoming post on this topic).
Clarification Edit:
Further I'm concerned that AI Safety research lacks a sufficiently robust framework for reasoning about the development, deployment and behaviour of AGI's. I am very interested in the broad category of "deconfusion". It is under that lens that I comment on evolution not fitting the definition in RFLO. It indicates that the optimiser framework in RFLO may not be cutting reality at the joints, and a more careful treatment is needed.
To conclude
An intentionally provocative and attention-grabbing summary of this post might be "evolution is not an optimiser", but that is essentially just a semantic argument and isn't quite what I'm trying to say.
A better summary is "in the category of things that are referred to as optimisation, evolution has numerous properties that it does not share with ML optimisation, so be careful about invoking it as an analogy".
On similarities between ML similar to RLHF and evolution:
You might notice that any form of ML that relies on human feedback, also fails to have an "internal representation" of what it's optimising toward, instead getting feedback from humans assessing it's performance.
Like evolution, it is also possible to set up this optimisation process so that it is also not "conservative".
A contrived example of this:
Consider training a language model to complete text if the humans giving feedback exhibited a preference for text that was a function of what they'd just read. If the model outputs dense, scientific jargon the humans prefer lighter prose. If the models output light prose, the humans prefer more formal writing etc.
(This is a draft of a post, very keen for feedback and disagreement)
Another example in ML of a "non-conservative" optimization process: a common failure mode of GANs is mode collapse, wherein the generator and discriminator get stuck in a loop. The generator produces just one output that fools the discriminator, the discriminator memorizes it, the generator switches to another, until eventually they get back to the same output again.
In the rolling ball analogy, we could say that the ball rolls down into a divot, but the landscape flexes against the ball to raise it up again, and then the ball rolls into another divot, and so on.