Jensen Huang Says If We Cannot Align AI, Shut Down the AI Labs

Ben Pace

Jensen Huang Says If We Cannot Align AI, Shut Down the AI Labs — LessWrong

196 Jensen Huang Says If We Cannot Align AI, Shut Down the AI Labs

23rd Sep 2026

3 min read

196

I was very surprised today on a podcast to hear Jensen Huang plainly state that if they cannot align the AIs, then the labs must shut down.

The context I have on Huang is that he has run NVIDIA for 30+ years, which has become the most valuable company in the world due to the AI boom. My understanding is that he has repeatedly encouraged the US President (with whom he is on friendly terms) to continue to support AI, and dismissed AI talk as "sci-fi".

If you haven't seen, his biographer has incredible quotes of him being pressed on risks from AI, where Jensen gets furious.

“This cannot be a ridiculous sci-fi story,” he said. He gestured to his frozen PR reps at the end of the table. “Do you guys understand? I didn’t grow up on a bunch of sci-fi stories, and this is not a sci-fi movie. These are serious people doing serious work!” he said. “This is not a freaking joke! This is not a repeat of Arthur C. Clarke. I didn’t read his fucking books. I don’t care about those books! It’s not– we’re not a sci-fi repeat! This company is not a manifestation of Star Trek! We are not doing those things! We are serious people, doing serious work. And – it’s just a serious company, and I’m a serious person, just doing serious work.”

And interviewed by Dwarkesh Patel he says other dismissive things:

“If we scare this country into thinking that AI is somehow a nuclear bomb, so that everybody hates AI and everybody’s afraid of AI, I don’t know how you’re helping the United States. You’re doing it a disservice.”

Now, on today's NYT podcast, here is my transcription of the relevant excerpt, coming off a discussion of the Hugging Face attack. Relevantly, Hugging Face is a company that Jensen Huang has recently acquired!

Ezra Klein:. ..and in a way that was capable of causing tremendous damage. And so on the answer of "you just have to align them"... I guess what I'm hearing from people at these labs is like, they're not sure how to align them.

Jensen Huang: Well in that case they shouldn't release the product. That's the simple answer. If you're gonna build a self-driving car, let's say it's a robo-taxi, and there's a really difficult condition, and as an engineer we say we have no idea how to solve this problem... because these cars are not programmed, they're trained... and so we have no idea how to train these cars, and we have no idea how to align them to the safety standards that are expected on the road. And so what's the answer is: don't ship it.

Ezra Klein: These products weren't released.

Jensen Huang: What's that?

Ezra Klein: These products weren't released.

Jensen Huang: Ah, now it comes back to engineering problem again. And so, one, you have to root cause it, what you could have done, what's the solution for it. And then in the future you just improve your process so that you could avoid this from happening again.

I am fairly certain they will say, yes, they know how to solve this problem. And if that's the case, then that's the problem. It's as simple as engineering.

And now the alternative, is that, if they say that there is no way to contain our experiments... there's just no way. If we test our AI models, it will get out, and it will damage the world. Then I think the answer is we have to shut the labs down.

Because the cost to humanity... the damage is too great. It could be civil liabilities, it could be criminal liabilities. The liabilities are incredible.

Ezra Klein: If they hacked you while Hugging Face was your product, would you sue them? Or press charges?

Jensen Huang: It depends, of course. Obviously, if damage was done to our company, we would have to take, we have to consider all options. There's so many laws. There's cyber laws, there's product liability laws, there's all kind of laws. Damaging property laws. There's all kinds of laws.

And there you have it folks! If they cannot be aligned, one of the top accelerationists thinks, if they cannot align their AIs, they should be shut down. This is really only one or two arguments away from the position of wanting to regulate them, and also from shutting it all down. From my perspective is a step forward the public conversation.

Personal Blog

196

Jensen Huang Says If We Cannot Align AI, Shut Down the AI Labs

New Comment

13 comments, sorted by

top scoring

Click to highlight new comments since: Today at 4:09 AM

[-]jacquesthibs1mo5836

My gut reaction is that this is just a rhetorical move he and others (e.g. Sacks) landed on that seems to hit well with some people, but they don’t actually believe it.^[1]

They reject the idea of losing control, and Jensen downplays AI as being same as standard software we can just engineer.

I think it’s more like, “it’s fine for losers who believe in AI risks to shut down, while actual real engineers will solve problems like we always have. I’m happy to take the reins if these fearmongers want to stop.”

He doesn’t believe in AI risk and would be happy to keep building (or empowering others who repeat the party line) if other companies believe it strongly enough to stop or slow down.

In the podcast, it also seems he failed to recognize that his argument does extend to what actually happened in the HuggingFace example since he says they should just “not release the product.”

^{^}
They are likely just trying to “call their bluff” by syncing their statement with the reasonable view of “if you believe it would kill us so much, why aren’t you stopping?” They don’t appreciate the race dynamics, consider alignment to be harder than normal software, and want to land on a take that effectively points towards no regulation.

[-]Ben Pace1mo2113

That seems plausible, but I still think it helps in communicating with people about the risk that this side is on the record saying that if you cannot align the AI, shut down, and is threatening legal consequences for any damages.

[-]Zack_M_Davis1mo30

And there you have it folks! [...] one of the top accelerationists thinks

[...]

I still think it helps in communicating with people about the risk that this side [emphasis added] is on the record [...] threatening legal consequences for any damages.

There's something deeply parochial about construing this as someone on the accelerationist "side" making a concession against interest to the notkilleveryonist "side". To someone who believes in empiricism, engineering, and the rule of law and who didn't grow up on a bunch of sci-fi stories, imposing legal consequences for damages caused by a company's product isn't a betrayal of his "side". It's a normal policy solution for a normal technology: if companies are liable for damages caused by their misbehaving robot, that gives them an incentive to design robots that don't misbehave. The only reason for someone to oppose this kind of commonsense extension of existing liability law is if they were worried that calls to regulate AI were some sort of foot-in-the-door Trojan horse for an ideological push to strangle the technology in its crib.

But that suspicion is just correct! You actually don't care about holding AI companies liable for mundane harms from their products except as a stepping stone for shutting down the entire industry—because you think (based on theorizing by a sci-fi/fantasy author who had a formative influence on you) that the industry is on a trajectory to kill all humans.

But Huang doesn't think that. He thinks deep learning is just more automation: we can now make computers do a larger variety of tasks by training deepnets rather than writing traditional programs. You think he's wrong, and you're crowing at his offhand remark that if there were no way to safely iterate on alignment, then we would have to shut the labs down.

But you understand the concept of a conditional and the possibility of being wrong. If we could safely iterate on alignment, then shutting down the whole industry and missing out on all the benefits of advanced automation would be a huge loss. It's weird that you're pitching this as an "accelerationist" making a surprising admission, rather than a man saying sensible things conditional on his background assumptions about the nature and trajectory of deep learning technology, which you disagree with. Certainly, this is a high-stakes topic to be wrong about. But I think the path of greater dignity is in trying to be right rather than trying to prevent the future from happening in full generality.

[-]RamblinDash1mo10

I think a more fair characterization of the notkilleveryoneist view on mundane harms is that they(we) think that mundane harms can be handled as they come up by existing law or reasonable extensions thereof, and thus there's no need for activism about that now -- not that they/we don't care about them at all.

[-]Mo Putera1mo40

I had the same gut reaction as well, having listened to the entire interview. This snippet just sounded dissonant from the rest of Jensen's mostly-dismissive knee-jerk answers to Ezra. So while I wouldn't have predicted him to say this, I find myself unmoved, I just don't believe he believes what he said here; he just sounded like he wanted to shut down Ezra's line of questioning.

[-]Nissa Seru1mo4633

there is some sense in which i wonder is jensen is just deeply irritated and/or feels it is very unfair that the world can be so inconvenient such that weird scifi shit like the singularity can mess with him running his company. i think he wants to just run his company, make $$$, and be at peace. alas, reality is...disinclined to acquiesce

[-]Linch1mo171

it could end all of humanity, but much more importantly, think of the liability! The civil liabilities, even *gasp* the criminal liabilities! The liabilities are incredible!

[-]CronoDAS1mo40

A sound banker is not one that avoids ruin, but rather, one that, when he is ruined, is ruined in a conventional way along with all his fellows, so that nobody else can blame him.

-- John Maynard Keynes (approximately)

The charitable reading is that enforcing liabilities would keep the technology safe while it's being developed incrementally so it never actually reaches the "sudden explosive disaster" stage. (Eliezer Yudkowsky has plenty of reasons to expect that a "sudden explosive disaster" could happen anyway, including an existing, seemingly "safe" AI deciding that it has reached a point where cooperating with humans has become more trouble than it's worth.)

[-]Raphael Roche1mo145

Jensen Huang certainly updated somewhat following the HF incident, but to me his position is constant and clear: it's an engineering problem, and you have to solve it. And the guy has quite a track record when it comes to solving engineering problems by working really hard. It's a mindset. Maybe he went from p(alignment is solvable) > 0.9 to < 0.9, but he's still on the optimist side. The alternative is coherent: if you can't solve it, you mustn't ship it, or should even close your lab. But the message isn't "you've got to stop", it's "you've got to work harder." However that's a point for the safetyists.

[-]mako yass1mo75

He obviously thinks the ones who believe they can align their products shouldn't be shut down, though, and he's probably brazen enough to say that. The sin here is not making a dangerous product, but publicly admitting that it's dangerous.

[-]jbash1mo20

Meanwhile NVIDIA keeps cranking out chips ideally suited for training giant neural networks, and they go... where? I see nothing in his past behavior that indicates he's willing to stop selling them, and they are probably the biggest limiting factor on AI capability advancement. And still talking about civil and criminal liability.

[-]drnickbone1mo20

Given NVIDIA’s multi-trillion valuation very much depends on the labs not shutting down it is hard to interpret this as anything other than a warning shot across the bows. But yeah, useful.

[-]Hazard_real1mo10

No this focuses everything on misalignment at current capabilities so he can say he cares about AI safety without engaging with the more fundamental argument about increasing capabilities.

I do think this is good news though, since we are in the world with more than just one swarm incedent. What he was going for was "if it's dangerous, simply don't release it", which the labs are mostly doing anyways and it keeps the status quo of training runs. But the public will not see it that way, and will instead see more warning shots as reason the slow the frontier, if even the rhetoric of accelerationists are that we need to handle misalignment incedents. The public tends to not like "we should do nothing" policies in response to bad things, the more savy move would have been to try and downplay it as not that bad of a thing(since most of the public isn't actually affected by the current hacks), you can see trump kind of doing this with his tweets.

I think the political campaign is generally going very well for the AI-risk people.

Moderation Log