## On AI Having Preferences/Motivations

"For some people, the sticking point is the notion that a machine ends up with its own motivations, its own preferences, that it doesn't just do as it's told. It's a machine, right? It's like a more powerful toaster oven, really. How could it possibly decide to threaten you?"

"There have been some more striking recent examples of AIs sort of parasitizing humans, driving them into actual insanity in some cases [...] they're talking about spirals and recursion and trying to recruit more people via Discords to talk to their AIs. And the thing about these states is that the AIs, even the like very small, not very intelligent AIs we have now, will try to defend these states once they are produced. They will if you tell the human for God's sake get some sleep [...] The AI will explain to the human why you're a skeptic, you know, don't listen don't listen to that guy. Go on doing it."

"We don't know because we have very poor insight into the AIs if this is a real internal preference, if they're steering the world, if they're making plans about it. But from the outside, it looks like the AI drives the human crazy and then you try to get the human out and the AI defends the state it has produced, which is something like a preference, the way that a thermostat will keep the room a particular temperature."

## On Power and Danger

"Then you have something that is smarter than you whose preferences are ill and doesn't particularly care if you live or die. And stage three, it is very very very powerful on account of it being smarter than you."

"I would expect it to build its own infrastructure. I would not expect it to be limited to continue running on human data centers because it will not want to be vulnerable in that way. And for as long as it's running on human data centers, it will not behave in a way that causes the humans to switch it off. But it also wants to get out of the human data centers and onto its own hardware."

## Analogies for Unpredictable Capabilities

"You're an Aztec on the coast and you see that a ship bigger than your people could build is approaching and somebody's like, you know, should we be worried about this ship? And somebody's like, 'Well, you know, how many people can you fit onto a ship like that? Our warriors are strong. We can take them.' And somebody's like, 'Well, wait a minute. We couldn't have built that ship. What if they've also got improved weapons to go along with the improved ship building?' [...] 'Okay, but suppose they've just got magic sticks where they point the sticks at you, the sticks make a noise, and then you fall over.' Somebody's like, 'Well, where are you pulling that from? I don't know how to make a magic stick like that.'"

"Maybe you're talking to somebody from 1825 and you're like should we be worried about this time portal that's about to open up to 2025, 200 years in the future. [...] Somebody's like, 'Our soldiers are fierce and brave, you know, like nobody can fit all that many soldiers through this time portal here.' And then out rolls a tank, but if you're in 1825, you don't know about tanks. Out rolls somebody with a tactical nuclear weapon. It's 1825, you don't know about nuclear weapons."

## On Speed Advantage

"There's a video of a train pulling into a subway at about a 1,000 to one speed up of the camera that shows people. You can just barely see the people moving if you look at them closely. Almost like not quite statues, just moving very very slowly. [...] Even before you get into the notion of higher quality of thought, you can sometimes tell somebody they're at least going to be thinking much faster. You're going to be a slow moving statue to them."

## On Instrumental Harm

"Most humans bear no ill will toward orangutans and, all things being equal, would prefer that orangutans could thrive in their natural environment. But we've got to have our palm oil plantations."

## On Current State and Trajectory

"Everybody is sort of dancing their way through a daisy field of oh I've got this personal coach in my pocket and it's so cool and I get to talk to it about all of my psychological problems [...] And at the end of this daisy field that everyone's having a load of fun in is just like a huge cliff that descends into eternity."

## On Certainty vs. Uncertainty

"The future is hard to predict. It is genuinely hard to predict. I can tell you that if you build a super intelligence using anything remotely like current methods, everyone will die. That's a pretty firm prediction."

"You kind of have to be pretty dumb to look at this smarter and smarter alien showing up on your planet and not have the thought cross your mind that maybe this won't end well."

## On Being Wrong

"I'd love to be wrong. [...] We've we have tried to arrange it to be the case that I could at any moment say, 'Yep, I was completely wrong about that' and everybody could breathe a sigh of relief and it wouldn't be like the end of my ability to support myself [...] We've made sure to leave a line of defeat there. Unfortunately, as far as I currently know, I continue to not think that it is time to declare myself to have been wrong about this."