Carl Shulman on The Lunar Society (7 hour, two-part podcast)

ESRogs

79 Carl Shulman on The Lunar Society (7 hour, two-part podcast)

by ESRogs

28th Jun 2023

2 min read

79

This is a linkpost for https://www.dwarkeshpatel.com/p/carl-shulman-2

Dwarkesh's summary for part 1:

In terms of the depth and range of topics, this episode is the best I’ve done.
No part of my worldview is the same after talking with Carl Shulman. He's the most interesting intellectual you've never heard of.
We ended up talking for 8 hours, so I'm splitting this episode into 2 parts.
This part is about Carl’s model of an intelligence explosion, which integrates everything from:
how fast algorithmic progress & hardware improvements in AI are happening,
what primate evolution suggests about the scaling hypothesis,
how soon before AIs could do large parts of AI research themselves, and whether there would be faster and faster doublings of AI researchers,
how quickly robots produced from existing factories could take over the economy.
We also discuss the odds of a takeover based on whether the AI is aligned before the intelligence explosion happens, and Carl explains why he’s more optimistic than Eliezer.
The next part, which I’ll release next week, is about all the specific mechanisms of an AI takeover, plus a whole bunch of other galaxy brain stuff.
Maybe 3 people in the world have thought as rigorously as Carl about so many interesting topics. So this was a huge pleasure.

Dwarkesh's summary for part 2:

The second half of my 7 hour conversation with Carl Shulman is out!
My favorite part! And the one that had the biggest impact on my worldview.
Here, Carl lays out how an AI takeover might happen:
AI can threaten mutually assured destruction from bioweapons,
use cyber attacks to take over physical infrastructure,
build mechanical armies,
spread seed AIs we can never exterminate,
offer tech and other advantages to collaborating countries, etc
Plus we talk about a whole bunch of weird and interesting topics which Carl has thought about:
what is the far future best case scenario for humanity
what it would look like to have AI make thousands of years of intellectual progress in a month
how do we detect deception in superhuman models
does space warfare favor defense or offense
is a Malthusian state inevitable in the long run
why markets haven't priced in explosive economic growth
& much more
Carl also explains how he developed such a rigorous, thoughtful, and interdisciplinary model of the biggest problems in the world.

(Also discussed on the EA Forum here.)

AI TakeoffCarl ShulmanIntelligence explosionAI

Frontpage

79

New Comment

Rendering 0/17 comments, sorted by

top scoring

(show all) Click to highlight new comments since: Today at 10:23 AM

Moderation Log

79 Carl Shulman on The Lunar Society (7 hour, two-part podcast)

by ESRogs

28th Jun 2023

2 min read

79

This is a linkpost for https://www.dwarkeshpatel.com/p/carl-shulman-2

Dwarkesh's summary for part 1:

In terms of the depth and range of topics, this episode is the best I’ve done.
No part of my worldview is the same after talking with Carl Shulman. He's the most interesting intellectual you've never heard of.
We ended up talking for 8 hours, so I'm splitting this episode into 2 parts.
This part is about Carl’s model of an intelligence explosion, which integrates everything from:
how fast algorithmic progress & hardware improvements in AI are happening,
what primate evolution suggests about the scaling hypothesis,
how soon before AIs could do large parts of AI research themselves, and whether there would be faster and faster doublings of AI researchers,
how quickly robots produced from existing factories could take over the economy.
We also discuss the odds of a takeover based on whether the AI is aligned before the intelligence explosion happens, and Carl explains why he’s more optimistic than Eliezer.
The next part, which I’ll release next week, is about all the specific mechanisms of an AI takeover, plus a whole bunch of other galaxy brain stuff.
Maybe 3 people in the world have thought as rigorously as Carl about so many interesting topics. So this was a huge pleasure.

Dwarkesh's summary for part 2:

The second half of my 7 hour conversation with Carl Shulman is out!
My favorite part! And the one that had the biggest impact on my worldview.
Here, Carl lays out how an AI takeover might happen:
AI can threaten mutually assured destruction from bioweapons,
use cyber attacks to take over physical infrastructure,
build mechanical armies,
spread seed AIs we can never exterminate,
offer tech and other advantages to collaborating countries, etc
Plus we talk about a whole bunch of weird and interesting topics which Carl has thought about:
what is the far future best case scenario for humanity
what it would look like to have AI make thousands of years of intellectual progress in a month
how do we detect deception in superhuman models
does space warfare favor defense or offense
is a Malthusian state inevitable in the long run
why markets haven't priced in explosive economic growth
& much more
Carl also explains how he developed such a rigorous, thoughtful, and interdisciplinary model of the biggest problems in the world.

(Also discussed on the EA Forum here.)

AI TakeoffCarl ShulmanIntelligence explosionAI

Frontpage

79

Mentioned in

174Meta Questions about Metaphilosophy

62Automating philosophy if Timothy Williamson is correct

16Flowchart: How might rogue AIs defeat all humans?

12Is Metaethics Unnecessary Given Intent-Aligned AI?

New Comment

Rendering 0/17 comments, sorted by

top scoring

(show all) Click to highlight new comments since: Today at 10:23 AM

Moderation Log

More from ESRogs

Curated and popular this week

17Comments

Carl Shulman on The Lunar Society (7 hour, two-part podcast) — LessWrong

Comment Permalink

Wei Dai3y4527

Carl, your vision (which I guess is fairly mainstream in AI alignment these days) involves having a powerful civilization be controlled by something weak and fragile (humans), but I worry that humans are both prone to responding in crazy ways to many kinds of inputs^[1] and being optimized against by superintelligences (since they are rewarded/reinforced by evaluations from humans). I feel like a lot of the stuff you talk about in this podcast (mind reading, adversarial training) as being responsible for your optimism mostly just shifts probability mass from the more obvious bad outcomes (deception, takeover) to less obvious bad outcomes (humans going crazy, failing to converge to correct values). It's hard for me to get nearly as hopeful as you seem, without being able to see some kind of solution that has the shape of "here are all the ways that humans are defective/unsafe, and ways to patch those holes or avoid them in human-AI interactions" (or some other kind of solution that avoids depending on humans for x-safety at all), unless "hopeful" is defined narrowly as solving intent-alignment or avoiding takeover.

To put it another way, being easily deceived is just one way that humans are defective/unsafe, but we (i.e., people in the AI alignment/x-risk space) seem to be focused on it almost to the complete exclusion of other human-AI safety problems, which seems bad from both a technical perspective (how do we solve the other problems if nobody is working on them) and social/governance perspective (society is liable to be too optimistic about the AI transition going well because nobody hears about the less obvious problems). As evidence for the neglect, I note that interstice's comment to Paul Christiano's My views on “doom” remains unanswered:

Interesting that you think the risk of us “going crazy” after getting AI in some way is roughly comparable to overall AI takeover risk. I’d be interested to hear more if you have more detailed thoughts here. On this view it also seems like it could be a great x-risk-reduction opportunity if there are any tractable strategies, given how neglected it is compared to takeover risk.

^{^}
for example, getting addicted, believing in obviously crazy conspiracy theories, refusing to speak or act in accordance with obvious truths because they're low status / outside the overton window, getting sucked into non-effective or counterproductive causes

Showing 3 of 8 replies (Click to show all)

6Wei Dai3y

I would agree with "philosophy are coherent stories that help people craft shared meaning and motivation" but I think it's not merely this. The position that it is merely this (and therefore philosophical questions do not have answers that can be true or false) would be self-undermining, because if there is no true or false in metaphilosophy, then your own metaphilosophical position can't be true (providing me with no reason to accept it on that basis), and thinking that all or most philosophical questions that I care about have no truth value would be rather demotivating. Consider instead the following alternative metaphilosophical position (that I tentatively hold): Philosophy may be many things (including "coherent stories that help people craft shared meaning and motivation") but it is also a method of answering confusing questions, that (at least some) humans seem to possess but can't yet understand or explain, and many philosophical questions do seem to have answers that can be true or false. What is incoherent or not motivating about this? What is wrong with this in general?

3Roman Leventov3y

This is exactly what I meant as well, in "Philosophy is a nebulous concept with multiple overlapping meanings, and as a practice, doesn't have a single crisp functional conceptualisation, like those you enumerated in your post on metaphilosophy. It's rather some superposition of these conceptualisations." Let's first disentangle two questions: "Why people are practicing philosophy?" and "What is the (truth) status of philosophical statements?". The first one is the question of anthropology and (social) psychology which, I could argue, are kind of "philosophies" rather than "sciences" themselves, at this moment: they attempt to explain existing evidence (with "coherent stories"), but don't produce mathematical models with good predictive power. Nevertheless, I agree that there are multiple explanations that we can provide, from "finding answers to difficult questions" and "by practicing philosophy, people try to increase the coherence of their world models, which is a deep 'motivation' of conscious biological agents, and which became imported into the domain of words 80k years ago or whenever people have acquired compositional language", to "showing off". As retrospective explanation, which doesn't inform us much to why will AI be motivated to practice philosophy. As an algorithm for finding more coherent and parsimonious sets of symbols/concepts (ontologies), it may be implemented in LM-like AI, perhaps, or there may be other, more efficient algorithms for this. To the second question, about the status of philosophical questions, I think the answer is the following: while the essence of a scientific theory is a mathematical model which is judged by the quality of its match with evidence and coherence with adjacent scientific (mathematical) models, philosophy is text which is judged by its coherence with scientific models, internal (linguistic) coherence, and coherence with other philosophical texts (arbitrarily selected by a philosopher, as well as a scientist

Wei Dai3y60

Internal coherence of a text and its coherence with other texts is a question of (neuro)semiotics and linguistics/philosophy of language, both of which, in my mind, are branches of cognitive science. If there is something else that makes texts convincing to people apart from their coherence, apart from external factors such as the likability and the authority of the author of the text or the orator, then the “quality” of philosophical texts also becomes the question of neuropsychology more generally.

Before the invention of logic, someone might have said... (read more)

See in context