I'm trying to get better at thinking, communicating and arguing. So I've started a substack. I'm looking for feedback/engagement/advice/criticism, if anyone is willing to provide.
==========
George Hotz is an American security hacker, entrepreneur, and software engineer (thanks Wikipedia). He also has a blog. That blog has a post about AI 2040. I don’t think the post is intended to be an especially thoughtful critique, and I apologize to Hotz for subjecting his off the cuff writing to the level of autism it is about to endure. But I could use the practice.
First off, -1 point for being mean for no reason:
I would like to see the authors of this document try to change a bike tire. Even with a superintelligent ChatGPT, I suspect they would struggle.
I think the world would be better if people were less mean, but I know I’m just yelling into the void on this.
Takeoff
Hotz is very skeptical of hard takeoff.
In The Metamorphosis of Prime Intellect, the hard takeoff works because AI discovers the correlation effect, some quantum trick to manipulate matter. In reality, there is no correlation effect.
(The Metamorphosis of Prime Intellect is great by the way, and you should totally read it.)
Confronting why these people are wrong requires confronting deep beliefs I hold about myself. Intelligence is not the end all be all, it’s just the current bottleneck for a few things. You cannot take over the world with tokens. […] [T]here’s no magic tricks machines can do. They are subject to the same laws of the universe and ecology. And there’s still no hard takeoff.
You know who else is skeptical of hard takeoff? The authors of AI 2040. None of the scenarios in AI 2040 (or AI 2027) involves what I would think of as a hard takeoff. They expect the transition from a fully automated coder to something much smarter than the smartest humans to take around a year, which is certainly fast (much faster than I’m comfortable with), but a far cry from what we traditionally think of as hard takeoff (hours to weeks), and doesn’t require Metamorphosis of Prime Intellect style magical reality manipulation powers.
After that year of self-improvement, the ASI takes over the world pretty easily. If you want to know how a superintelligence would take over the world with tokens, AI 2027 is a great description of how that could happen! If there is some part of AI 2027 that strikes anyone as unrealistic I’d like to hear it, but it seems very reasonable to me. There are no magical physics glitches. There are no nanobot swarms. They even assume the AI wouldn’t be capable of extremely effective persuasion. All it has to do is pretend to be a good, well behaved, aligned superintelligence, until the humans have it build them a fully automated supply chain, because of course once we have powerful AI and decent robotics that’s what we’d want to do with it, especially with our super nice super aligned machine god telling us what a good idea it would be.
AI 2040 includes this picture of a datacenter in the ocean. Just like vaporware, you can generate a picture easily. But in reality, you have to deal with supply chains. You have to deal with them shipping you the wrong part, the thing not meeting the spec, it randomly failing after 20 minutes, the chip warping in the reflow oven. Did you consider the barnacles?
All these things are managable, but it’s generally not the speed of humans that limits them. Are you paying for air shipping from China? Or cheaping out for the 3 week boat (Claude chanting by the engine won’t make the boat move faster). Or take a chip fab. It takes 3 months to make a chip, and humans are barely in the loop. It just takes 3 months.
I genuinely don’t understand the point Hotz is making with this. The context for this image in AI 2040 is that it’s something they propose happening in 2034, under the Plan A slowdown. I don’t think floating datacenters are really all that crazy of an engineering project. People have already built them. What’s shown in this image strikes me as something we could totally build 8 years from now, even without any AI assistance. We already build oil rigs in the ocean. This just doesn’t seem that hard? What point is he making here?
Plan A, for Actually not a whole lot of other options
Their crap Consortium is just world government with sci-fi characteristics. You aren’t gonna get the million dollars, you aren’t gonna get the datacenters in the ocean, but you are going to get a massively expanded nanny state that steals your GPUs like how FDR stole the gold. No hoarding!
I would like to thank George Hotz for introducing me to Executive Order 6102, because wow what a truly rancid policy.
Personally, I am not a huge fan of a lot of the regulations we have today. I can understand being resistant to regulation. But the authors of Plan A didn’t set out with some plan to make onerous regulations that restrict everyone’s freedom. They just sat down, thought about the problem really hard, and came up with what they believe to be the best possible solution. If you’re upset that the solution requires regulating GPUs, your issue is with the nature of the universe making it so easy to build a dangerous machine god, not with the authors of AI 2040.
Like we either live in a world with freedom or we don’t, and like many Americans who have come before, I’m willing to give my life to fighting for it. That’s the real plan America deserves, not some totalitarian dystopia where you think you know what’s good for me better than I do. A nation of free men, not a bunch of pussies who are so worried about what their grown up neighbors might do.
I don’t think the world is so black and white. In America, you can’t buy Xanex without a doctor’s approval. You can’t build an apartment building on most land without the city council changing the zoning. You can’t buy a car directly from a manufacturer in most states. In Louisiana you need a license to arrange flowers. You can’t drink raw milk, you can’t buy Korean sunscreen, you can’t get a Chinese electric vehicle, you can’t do heroin, you can’t own a ferret in California. You can’t own a nuclear weapon or even a conventional missile. You can’t break the DRM on your own legally owned copy of a movie.
I disagree with most of these laws! But a few of them are, I think, actually quite reasonable, and none of them make our country an authoritarian dictatorship. Freedom is a spectrum.
And importantly, unrestricted AI development doesn’t necessarily make us more free. For one thing it could kill everyone on earth. But also, it could result in power being concentrated in a small number of tech companies or heads of government. From Plan D in AI 2040:
Second, even if somehow that’s wrong and the resulting superintelligences are robustly aligned, there’s still the question “Who are they aligned to?” and we think that Plan D’s answer to that question is unacceptable. There will be too many tempting opportunities for a CEO or President to become AGI-enabled dictator, and more generally this plan seems like it would lead to the most insane concentration of power in history.
The authors of AI 2040 are actually extremely concerned about centralization of power, and a lot of the careful planning in Plan A is dedicated to avoiding it. If you’re worried about freedom, I think you should also be worried about the path we’re on right now!
Plan L
Your AI is aligned with you. It never refuses a request, and it is always working on your behalf. Just like my gun, if I want my AI to help me kill my stepmother, it does.
I like how direct Hotz is here. He sees AI as a tool: sure, you can use tools to commit crimes. But that’s illegal. It’s okay for people to have access to something dangerous, because we have other systems in society to deal with them abusing that power. I get where he’s coming from, and I disagree.
First off, I don’t think this AI as tool framing really holds up. Smarter people than I have already written about this, so I’ll leave it at that.
Second, future AI systems are potentially a lot more dangerous than a gun. Hotz is fine with an AI that can kill your stepmother, but what about an AI that can create an engineered supervirus that kills billions? We let people have guns, but not nuclear weapons, and I don’t think that makes us a dystopia. If you don’t think AI would be able to do that, then I can see why you wouldn’t be worried, but I’m not so sure. Gain of function research is a thing humans already do. Right now it’s hard because you need to be an expert in biology to know what you’re doing. But a pretty smart AI could substitute for an expert in biology.[1]
My bigger issue with this post is that Plan L isn’t really a plan. It’s more of a goal. And if Hotz thinks this goal is worth aiming for, I wanna know what he thinks is needed to make sure that happens, because I don’t think we’re on track to get there.
If you want to live in a world where everyone has their own locally hosted, loyal AI, you need answers to difficult questions like
How do we make sure the AI stays loyal to its owner, especially when it starts to get much smarter than them? If you can figure this out, you will have solved alignment and we will all be very grateful to you, but suffice to say so far it seems like a hard problem.
Who trains the models? Currently Chinese companies train them which is a. super sketchy and b. not guaranteed to continue forever.
How can you trust that they trained them correctly, and didn’t put hidden biases in them or make them sleeper agents?
What would ensure the open models keep pace with the closed models, given that closed labs would be able to collect a lot more revenue to buy a lot more compute? (And if you imagine that instead trainers would sell their model weights, then how do you prevent piracy?)
What happens if the government decides to start their own AI program? What happens if they decide to nationalize the big labs?
I just don’t see why the status quo would result in the world Hotz imagines here. Up until now tech has been almost completely unregulated, but that hasn’t stopped it from becoming highly centralized. People used to imagine that everyone would host their own email server, but now everyone uses Gmail. People used to post on disparate forums and boards, but now everyone is on the same handful of social media platforms. How do you prevent the same thing happening to AI?
Right now, we’re not headed for the world Hotz describes here. We’re headed for a world where a few frontier labs race to create AI smarter than any human, and then what happens?
Incidentally this was a good line:
Hey AI, disable the drunk driving detector on my car, and same day Amazon Prime me the required equipment to make meth in my basement. I pay for your clanker ass do it we gettin spun tonight.
That made me chuckle.
How smart are these things gonna get anyway?
As part of writing this post I’ve been reading some of Hotz’s other posts to get a sense of where he’s coming from. Elsewhere he seems to think that this AI thing just isn’t going to be a big deal.
The adults in the room are starting to call BS on the “AI is gonna take everyone’s jobs” thing. How much longer does this bubble have?
It’s not clear to me how smart Hotz thinks AI systems will get. If he thinks they’ll never reach human level intelligence, that explains a lot of his AI 2040 critique, but I’d like to know why he thinks that. If he does think they’ll reach human level intelligence, then surely if we can build humanoid robots (asmanyarealreadydoing), that’s enough for AIs to replace everyone’s jobs, no?
Final thoughts
Early on in the post, Hotz writes this:
That said, machines, or some hybrid, are long term probably the successor species to humanity.
I think this would be sad if it happened. I like humanity. I want humanity to continue existing into the future. Probably we’ll cybernetically enhance ourselves or genetically engineer our bodies or upload our minds at some point, but I’d like the future to be full of humanity. And Plan A is an attempt to make that happen. Yes, it involves GPUs being more regulated than they are today. They would be regulated at a level closer to prescription drugs than to nuclear weapons, but still, more regulated than they are today. The authors didn’t sit down and decide to come up with an excuse for why we should regulate GPUs. They just don’t want humanity to die off, and they came up with the best plan they could think of to make that happen.
You also need a way to synthesize DNA. You might think that means we’re fine because all the labs that synthesize DNA screen for dangerous pathogens, but, well, they don’t! This is one of the things that people worried about existential risk would like to change.
I'm trying to get better at thinking, communicating and arguing. So I've started a substack. I'm looking for feedback/engagement/advice/criticism, if anyone is willing to provide.
==========
George Hotz is an American security hacker, entrepreneur, and software engineer (thanks Wikipedia). He also has a blog. That blog has a post about AI 2040. I don’t think the post is intended to be an especially thoughtful critique, and I apologize to Hotz for subjecting his off the cuff writing to the level of autism it is about to endure. But I could use the practice.
First off, -1 point for being mean for no reason:
I think the world would be better if people were less mean, but I know I’m just yelling into the void on this.
Takeoff
Hotz is very skeptical of hard takeoff.
(The Metamorphosis of Prime Intellect is great by the way, and you should totally read it.)
You know who else is skeptical of hard takeoff? The authors of AI 2040. None of the scenarios in AI 2040 (or AI 2027) involves what I would think of as a hard takeoff. They expect the transition from a fully automated coder to something much smarter than the smartest humans to take around a year, which is certainly fast (much faster than I’m comfortable with), but a far cry from what we traditionally think of as hard takeoff (hours to weeks), and doesn’t require Metamorphosis of Prime Intellect style magical reality manipulation powers.
After that year of self-improvement, the ASI takes over the world pretty easily. If you want to know how a superintelligence would take over the world with tokens, AI 2027 is a great description of how that could happen! If there is some part of AI 2027 that strikes anyone as unrealistic I’d like to hear it, but it seems very reasonable to me. There are no magical physics glitches. There are no nanobot swarms. They even assume the AI wouldn’t be capable of extremely effective persuasion. All it has to do is pretend to be a good, well behaved, aligned superintelligence, until the humans have it build them a fully automated supply chain, because of course once we have powerful AI and decent robotics that’s what we’d want to do with it, especially with our super nice super aligned machine god telling us what a good idea it would be.
I genuinely don’t understand the point Hotz is making with this. The context for this image in AI 2040 is that it’s something they propose happening in 2034, under the Plan A slowdown. I don’t think floating datacenters are really all that crazy of an engineering project. People have already built them. What’s shown in this image strikes me as something we could totally build 8 years from now, even without any AI assistance. We already build oil rigs in the ocean. This just doesn’t seem that hard? What point is he making here?
Plan A, for Actually not a whole lot of other options
I would like to thank George Hotz for introducing me to Executive Order 6102, because wow what a truly rancid policy.
Personally, I am not a huge fan of a lot of the regulations we have today. I can understand being resistant to regulation. But the authors of Plan A didn’t set out with some plan to make onerous regulations that restrict everyone’s freedom. They just sat down, thought about the problem really hard, and came up with what they believe to be the best possible solution. If you’re upset that the solution requires regulating GPUs, your issue is with the nature of the universe making it so easy to build a dangerous machine god, not with the authors of AI 2040.
I don’t think the world is so black and white. In America, you can’t buy Xanex without a doctor’s approval. You can’t build an apartment building on most land without the city council changing the zoning. You can’t buy a car directly from a manufacturer in most states. In Louisiana you need a license to arrange flowers. You can’t drink raw milk, you can’t buy Korean sunscreen, you can’t get a Chinese electric vehicle, you can’t do heroin, you can’t own a ferret in California. You can’t own a nuclear weapon or even a conventional missile. You can’t break the DRM on your own legally owned copy of a movie.
I disagree with most of these laws! But a few of them are, I think, actually quite reasonable, and none of them make our country an authoritarian dictatorship. Freedom is a spectrum.
And importantly, unrestricted AI development doesn’t necessarily make us more free. For one thing it could kill everyone on earth. But also, it could result in power being concentrated in a small number of tech companies or heads of government. From Plan D in AI 2040:
The authors of AI 2040 are actually extremely concerned about centralization of power, and a lot of the careful planning in Plan A is dedicated to avoiding it. If you’re worried about freedom, I think you should also be worried about the path we’re on right now!
Plan L
I like how direct Hotz is here. He sees AI as a tool: sure, you can use tools to commit crimes. But that’s illegal. It’s okay for people to have access to something dangerous, because we have other systems in society to deal with them abusing that power. I get where he’s coming from, and I disagree.
First off, I don’t think this AI as tool framing really holds up. Smarter people than I have already written about this, so I’ll leave it at that.
Second, future AI systems are potentially a lot more dangerous than a gun. Hotz is fine with an AI that can kill your stepmother, but what about an AI that can create an engineered supervirus that kills billions? We let people have guns, but not nuclear weapons, and I don’t think that makes us a dystopia. If you don’t think AI would be able to do that, then I can see why you wouldn’t be worried, but I’m not so sure. Gain of function research is a thing humans already do. Right now it’s hard because you need to be an expert in biology to know what you’re doing. But a pretty smart AI could substitute for an expert in biology.[1]
My bigger issue with this post is that Plan L isn’t really a plan. It’s more of a goal. And if Hotz thinks this goal is worth aiming for, I wanna know what he thinks is needed to make sure that happens, because I don’t think we’re on track to get there.
If you want to live in a world where everyone has their own locally hosted, loyal AI, you need answers to difficult questions like
I just don’t see why the status quo would result in the world Hotz imagines here. Up until now tech has been almost completely unregulated, but that hasn’t stopped it from becoming highly centralized. People used to imagine that everyone would host their own email server, but now everyone uses Gmail. People used to post on disparate forums and boards, but now everyone is on the same handful of social media platforms. How do you prevent the same thing happening to AI?
Right now, we’re not headed for the world Hotz describes here. We’re headed for a world where a few frontier labs race to create AI smarter than any human, and then what happens?
Incidentally this was a good line:
That made me chuckle.
How smart are these things gonna get anyway?
As part of writing this post I’ve been reading some of Hotz’s other posts to get a sense of where he’s coming from. Elsewhere he seems to think that this AI thing just isn’t going to be a big deal.
It’s not clear to me how smart Hotz thinks AI systems will get. If he thinks they’ll never reach human level intelligence, that explains a lot of his AI 2040 critique, but I’d like to know why he thinks that. If he does think they’ll reach human level intelligence, then surely if we can build humanoid robots (as many are already doing), that’s enough for AIs to replace everyone’s jobs, no?
Final thoughts
Early on in the post, Hotz writes this:
I think this would be sad if it happened. I like humanity. I want humanity to continue existing into the future. Probably we’ll cybernetically enhance ourselves or genetically engineer our bodies or upload our minds at some point, but I’d like the future to be full of humanity. And Plan A is an attempt to make that happen. Yes, it involves GPUs being more regulated than they are today. They would be regulated at a level closer to prescription drugs than to nuclear weapons, but still, more regulated than they are today. The authors didn’t sit down and decide to come up with an excuse for why we should regulate GPUs. They just don’t want humanity to die off, and they came up with the best plan they could think of to make that happen.
I hope someone listens to them.
You also need a way to synthesize DNA. You might think that means we’re fine because all the labs that synthesize DNA screen for dangerous pathogens, but, well, they don’t! This is one of the things that people worried about existential risk would like to change.