I summarize my learnings and thoughts on Liron Shapira's discussion with Ken Stanley on the Doom Debates podcast. I refer to them as LS and KS respectively.
Key beliefs of KS:
In contrast, LS does believe that superintelligence will be a super-optimizer and that it will be capable of feats of foresight and ingenuity that allow it to skip many of the stepping stones that we humans would have to take. E.g. LS believes that a superintelligence in early 1800s could have skipped the vacuum tubes and developed modern electronic computers on its first try.
Unfortunately, most of KS' claims are not justified. Partly this is because KS did not explain himself clearly so it took effort to pin down his beliefs, and also because LS chose to drill down into details that - in my opinion - were not the key cruxes.
I do not have fixed views on the nature of super-intelligence, and this is big source of uncertainty for me. I am persuaded by the notion of instrumental convergence. Presumably a non-optimizing open-ended superintelligence will avoid getting turned off. Or, it would accumulate resources to carry out its open-ended explorations. My general sense is that things are extremely unpredictable and that super-intelligence vastly increases the variance of future outcome. I do not have intuition for how to weigh the odds of positive outcomes versus negative.
Based only on this interview, LS has a simplistic view of intelligence, believing that raw intelligence can provide Godly foresight into everything, allowing one to skip doing experiments and interacting with the universe.
Here are some relevant ideas for this discussion:
If you want to learn more about open-endedness:
(I have not yet listened to any of these, but the first few minutes of each sound promising for getting a more concrete understanding of open-endedness and what it means to be interesting.)
I present these highlights and thoughts chronologically as they appear in the podcast.
A potential application of open-endedness is to generate high quality data for LLM training. KS does say later that he believes it is unlikely to be big part of OpenAI's pipeline.
00:03:26 LS. So your open endedness research, did that end up merging with the LLM program or kind of where did it end up?
KS. Yeah, to some extent. If you think about it from their big picture, not necessarily mine, it's aligned with this issue of what happens when the data runs out. [...] It's always good to be able to generate more really high quality data. And if you think about it, that's what open-endedness does in spades.
KS tries to explain how open-endedness is not just a different kind of objective. This is not clear at all. I wish LS pushed KS to be more formal about it, because the vague intuitive description is not convincing.
00:07:35. KS. Open-endedness is a process that doesn't have an explicit objective. Some people will say open-endedness is its own objective, and this can muddy the waters. But first, just want to make this kind of clear what the distinction is, which is that in open-endedness, you don't know where you're going by intent. And the way you decide things is by deciding what would be interesting. And so open-ended processes make decisions without actually a destination in mind. And open-ended processes that exist in the real world are absolutely grandiose. They're the most incredible processes that exist in nature.
And there's really only two. One is natural evolution. So you start from a single cell and then you wait like a billion years, and all of living nature is invented in - what would be from a computer science perspective - a single run.
There's nothing like that algorithmically that we do. It invented photosynthesis, flight, the human mind itself, the inspiration for Al all in one run. We don't do stuff like that with objective-driven processes. It's a very special divergent process.
And the second one is civilization. Civilization does not have a final destination in mind. [...]
What does it mean for evolution to 'decide' something? And what does it mean for something to be interesting? Does evolution find things interesting?
And a bit later:
00:09:32. KS: You could say, for example, that evolution has an objective of survive and reproduce. And this gets little bit hair-splitting, but I like to make a distinction there because I don't think of survive as formally the same kind of objective. I prefer not to call it an objective, because it's not a thing that you haven't achieved yet. When I think of objectives, I'm thinking of a target that I want to get to that I've not yet achieved. With survive and reproduce, the first cell did that. So I think of it more as a constraint. Everybody in this lineage needs to satisfy this constraint, but it's already been achieved and everybody in this lineage needs to satisfy this constraint.
LS argues that random mutations - some of which help and some of which hinder - are doing a kind of gradient descent in the loss landscape. E.g. if you have a patch of cells that are light sensitive, then some mutation might make it more light sensitive and hence more likely to survive. KS believes this is only valid for micro scale evolution but not macro scale.
00:11:47 KS. Yeah, I don't deny that optimization does happen in evolution. But it's important to highlight that the overall accounting for why there's all of the diversity of life on Earth is not well accounted for by just that observation. It's an astounding diversity of amazing inventions. To account for that requires other explanations.
On a macro scale, it is difficult to explain what we mean by better. How are we humans better than single-celled bacteria? We have less biomass, less offspring per generation, lower opulation. There's nothing objective to point to why we're better in terms of optimization. What's better about us is that we're more interesting.
A lot of evolution has to do with escaping competition - like finding a new niche - which is not an optimization problem.
This is doing something different and that's the divergent component. I argue that the convergent optimization subprocesses are less interesting and they don't account for the global macro process of evolution.
KS makes interesting observation that evolutionary algorithms are not divergent and so are not good metaphor for understanding full scope of evolution.
00:14:57. KS. Those kinds of algorithms work the way you describe. You do set an objective and fitness is measured with respect to the objective and you explicitly follow that gradient just as you would in another optimization algorithm. But think about what genetic algorithms do. They do converge. They have almost nothing to do with what's actually happening in nature. The intuition is off.
And so it's unfortunate to become this kind of misleading metaphor that lot of people key into. These are highly convergent algorithms that always converge to a single point or get stuck, just like conventional optimization. That's not what we see in nature.
LS is still generally confused (as am I) and asks a good question: what is the ultimate claim being made?
00:16:48 KS. Evolution is incredibly prolifically creative. And we don't have algorithms like that in computer science. So there's something to account for here that we have not abstracted properly. And yes, this is related to intelligence because civilization also has this property which is built on top of human intelligence.
And it's related to the superintelligence problem because my real deeper claim here is that superintelligence will be open-ended. It must be because the most distinctive characteristic of human intelligence is our prolific creativity. We will not get a super intelligence that's not open-ended and therefore we need to understand divergent processes. All of our optimization metaphors don't account for that property which can mislead us and lead us astray in analyzing what's in store for us in the future.
Finally something concrete to hold on to! KS believes that future superintelligence will be open-ended, and thinking of them as optimizers will lead to incomplete analysis and predictions.
But then oddly, LS goes back to the question of evolution and how evolution is or is not explained by following an objective function. Some interesting points come up but not central in my view. For example:
At around 30 minutes, they discuss a thought experiment of what would happen if you went to 1844 (100 years before ENIAC created) and tried directly optimizing for creation of computers. KS says it would fail because you would miss open-ended curiosity-driven explorations that lead to vacuum tubes, that were a crucial component of computers. LS (~00:32:00) says this is just a matter of lacking intelligence and foresight. With enough intelligence, you could create computer with direct optimization.
KS responds:
00:32:32. KS. A fundamental aspect of my argument is that we cannot foresee the tech tree. It's just complete futility. Omnipotence is impossible. We cannot understand how the universe works without experimentation. We have to try things. But it's important for me to highlight that trying things is not random. They [scientists] were highly informed because they understood that there's very interesting things about the properties of these technologies. And they wanted to see where that might lead, even though they don't know ultimately where it leads. And this is why the tech tree is expanding, because people follow these interesting stepping stones.
But we will not be able to anticipate what will lead to what in the future. Only when you're very close can you do that.
LS says this is a limitation of humans. What about superintelligence?
00:35:36 KS: The AI's hypotheses will be better, but it still needs to make hypotheses and it still needs to test those hypotheses. And it will still need to follow that tech tree and gradually discover things over time. Omnipotence is not on the table, even for AGI or superintelligence.
Great once again! Another concrete claim from KS about superintelligence, and something I would like to see the two discuss to find out why their intuitions disagree and what evidence could change either of their mind. But like last time, LS changes topic...
00:35:51. LS: One thing that you say in your book is that as soon as you create an objective, you ruin your ability to reach it. Unpack that.
KS says that using an objective works if the you have the necessary stepping stones to carry out the optimization and reach your objective. Such objectives are 'modest objectives'. However, for ambitious objectives, direct optimization will not work, because the necessary stepping stones will be things you simply would not even consider researching if you were directly optimizing for the goal.
The discussion moves to explore this in the context of evolution. LS asks whether in KS's framework, there even is a modest objective.
00:40:09. KS. I prefer to put it that way. Your argument caused me to contort myself in a way that I don't prefer, to describe this as an objective process once you're near something. There never is an explicit objective in evolution. My argument is this is why it's so prolifically creative. It's essential to have it for creative processes not to be objectively driven. And evolution is the greatest creative process that we know in the universe.
LS asks whether genetic fitness is an implicit objective that evolution optimizes for, even if that is not the explicit goal that evolution has. KS gives strong and bizarre claim in response about how all that matters is 'what is interesting':
00:42:10. KS. I don't think of inclusive genetic fitness as an objective. There's nothing wrong with something that has lower fitness. What we care about in evolution is what's interesting, ultimately, not necessarily what has higher fitness. Our fitness is probably on an objective basis lower than lots of species that we would consider less intelligent than us. It's an orthogonal issue. Fitness is not a target. Like I said earlier, it's a constraint.
I do not buy the idea that what we care about is what is interesting. I suspect this is KS just not phrasing himself very well, because it seems odd to claim that evolution does not care about fitness but it does care about 'being interesting'. Interesting to who??
KS tries a metaphor with Rube Goldberg machines:
00:43:47. KS. I was watching a TV show about this guy who is making Rube Goldberg machines to open a newspaper. And it was really funny because he invented the most complex things you could possibly imagine to do something completely trivial.
And this is effectively what's happening. [Evolution] is a Rube Goldberg machine generating system. [...] the complexity of those machines itself becomes what's interesting. It's not the fact that they're getting better. In some ways they're getting worse because it's crazy to go through all this complexity just to get another cell.
So we live in a Rube Goldberg machine generating universe, and this is just going on forever. It's a different story. It's a different story than this like deathmatch convergence type of view of evolution.
Discussion veers for several minutes in an unhelpful direction, in my opinion. LS doesn't look good here, saying things that just make KS repeat himself, without progressing the discussion, or digging into any of the previous key cruxes.
At 57 minutes in, LS moves to discussion of P(doom). Key points from KS:
At around 1:11:00, some discussion on what 'interesting means'. KS basically says its subjective but believes there is commonality amongst all humans. Nothing concrete or insightful.
At 1:15:46, LS asks what KS' ideal scenario is. KS is unsure how to tradeoff between well-being and interesting-ness/curiosity. There will be things that are interesting but dangerous.
At 1:20:14, LS asks what the headroom is above human intelligence. This leads to revealing more cruxes, and LS does a valiant job pinning down what KS thinks about whether future AI can be better optimizers than humans.
At 1:38:41, LS asks if KS thinks instrumental convergence is a thing in superintelligence. KS: "I think my answer here is no. There seems to be widely shared assumption that they're going to converge to goal oriented behaviour." And then KS repeats himself that superintelligence will be open-ended, and that this brings its own dangers.
I would have liked at this stage for LS to ask about other examples of instrumental convergence, e.g. would the AI avoid being turned off, or would the AI want to accumulate resources to allow it to carry out its explorations.
There is then some discussion about the example of what would happen if superintelligence found idea of 'going to Pluto' interesting. Conversation gets confusing here with little progress. One insight is that KS says that intelligence will not take 'extreme' actions to satisfy its interests (e.g. enslave all humanity, which is what LS suggests might happen), because it will have many many interests and taking extreme actions will have large opportunity cost. This is one way interests are different from goals.
At 1:55:53, LS asks KS about ideas for mitigating risks from open-ended intelligences. KS once again emphasises that his main point is that we should think about open-endedness. Then tentatively suggests looking at how open-endedness has been controlled in the past - what institutions have we set up. LS pushes KS for a policy that KS would recommend for this year [2024]. KS (weakly) suggests having human sign off on big decisions and likely that we need humans in the loop forever.
Discussion continues for another 30 minutes, but I do not think further insights are uncovered, with mostly repetition of ideas already mentioned. I think LS' attempt at summarizing KS' perspective reveals that the conversation did not bring a lot of clarity:
LS. You have a P(doom). It's significant. You don't know exactly what it is.
Your definition of what's good and bad might be different from mine because you see the future universe as being so likely to have so much good divergence in it. This gives you kind of a default sense that things are probably going to go okay, even thought it could also backfire.
Then there's this other point we disagree on. You think that optimizing toward some goal doesn't work that well. You just have to explore in the present until a solution will reveal itself later.
The second point surprised me. KS at multiple occasions says he thinks AI will be dangerous and that open-endedness is dangerous, and his top concern is that by focussing on optimization, people are misunderstanding the main issues and will not think of appropriate measures.
It is a shame that the discussion ended up being confusing and less fruitful than other Doom Debates interviews, because there is potentially a lot to learn from understanding KS' perspective.