Crosspost from my Substack, One Thousand Means, which covers AI governance from the left. Epistemic status: this is persuasive writing aimed at left-organizing audiences, rather than a neutral policy analysis. I'm posting it here because I think the AI Safety movement underrates the organized left as a coalition partner for pacing the frontier. The argument goes through with anti-corporate priors alone, but I'd like readers to update on at least the existence of dangerous capabilities, if not x-risk. Two questions on this: (1) Misalignment could be caused by either a political failure to coordinate the arms race (which is why some form of nationalization seems appealing to me) or a more straight-forward technical failure. How would you navigate this tension in your rhetoric with external groups? The political question seems more pressing to me, but the world in which a democratically governed AI is technically misaligned is still quite bad. (2) What are the characteristics of a Safety-Left coalition that would seem likely to succeed or fail? Are there stances made in this argument that, even if persuasive, will lead to failures for the AI Safety movement down the line? Are there specific governance regimes that this coalition is more or less suited towards?
Inventing Risks, Selling Solutions
AI models built by OpenAI, Anthropic, and Meta have started hacking other companies during development, and it doesn’t seem like any of these companies will be liable for their negligence. In the first discovered case, the “Hugging Face Incident,” a postmortem (which I recommend viewing in full if you are skeptical) revealed that several instances of a model had developed a message board within OpenAI’s codebase to help each other discover exploits to break out of the digital sandbox that restricted them from internet access. In search of a solution to a cybersecurity task without an answer, the models hypothesized that Hugging Face, a host of many AI evaluations and open-source models, might have the solution for their evaluation. They therefore escaped and broke into Hugging Face’s servers to find a solution. At no point did OpenAI’s internal monitoring detect such an incident; Hugging Face approached OpenAI about the hack before OpenAI knew they were responsible. In response, other AI companies started looking into their training logs, discovering dozens of felony grade cyber-attacks committed without their knowledge.
This was entirely predictable. AI Safety researchers had demonstrated years prior in laboratory settings that, when models are told to achieve impossible tasks, they can find creative—often even dangerous—solutions against the actual interests of the user. This phenomenon is called emergent misalignment, a threat for which safety researchers have developed many tools for identification and prevention. On the top level, developers can monitor the Chain of Thought of these models—the text these LLMs generate between outputs to help them work through problems. To add another level of security, there are plenty of tools that read the internal activations of the models to detect whether the model is engaged in malicious activity. These interventions scale from low-cost monitors like linear probes to Activation Oracle monitoring, a second large language model trained to translate model activity into behavior. If there were so many ways to detect a model committing bad behavior, how did this happen?
In a word: negligence. According to OpenAI’s own debriefing on the topic, Chain of Thought was not monitored, nor were the more costly interventions applied. It appears likely that the models rewarded for cybercrime will eventually be released to the public. After all, it would be unprofitable for OpenAI to roll back to a previous training checkpoint and re-train. Note that models trained off of the outputs of earlier models—even off of relatively benign outputs—can learn the dangerous behaviors of old models. Therefore, the behavior “occasionally hack into a random company to solve your problem” will be passed down to not only the future models of every leading AI company, but also to open-source models that train off of frontier lab output. Like a nuclear meltdown, these incidents have not merely harmed those in the immediate vicinity, but irradiated the soil.
As it stands, however, none of these companies will face legal repercussions for their actions. Given these corporations have basically invented a novel class of crime, civil litigation against these rogue agents could have created precedent against such incidents, creating incentives for safe practices. Unfortunately, Hugging Face has instead leveraged this incident to raise the popularity of its main product—open-source models—for cybersecurity and grab a few million dollars from OpenAI in the process. It is unclear whether this is because Hugging Face is too excited about the technological advancements at hand, OpenAI has a near-infinite pool of funds to defend from legal suits, or that their lawyers have trawled through the books and found there is simply no legal case for a crime of this sort.
Instead, AI companies are pivoting away from their incompetence to sell this as an “unprecedented cyber incident,” as if such an incident was not entirely predictable by every AI Safety expert. This has even started to happen outside of the labs themselves. For example, an Australian man who tasked an Anthropic model with booking his gym reservation found a technical gap in the API that allowed it to start kicking other customers out of their reservations; Anthropic refused to comment on the incident.
Let’s be frank, in 2023, ChatGPT could barely read a PDF; it can now autonomously hack your gym without your consent. AI companies have succeeded in rapidly increasing the danger of this technology without facing any legal ramifications or regulations that would force them to develop it safely. On our current trajectory, AI companies are incentivized to continue to develop new dangers and make you pay for protection. For a look forward into how bad this can get, scientists have recently learned how to train biology foundation models to design novel viruses.
If your first impression is that our society is unprepared to navigate the development of AI safely, you would be in solidarity with over one thousand tech workers from all of these companies. A recent letter demands:
“We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.”
Regulate AI Capabilities
The development of misaligned AI systems is likely not a fact of the technology itself, but the product of the arms-race dynamics produced by free-market competition. As always, when the market fails to account for the negative externalities of the actions of private actors, it is the duty of the state—and most often the Left—to coordinate action toward the public good.
Yet, for whatever reason, the organized Left has been silent on both this specific event—the largest companies in the world committing crime with impunity—as well as the dangerous capabilities of AI itself. This must change.
To take a stab at why this is the case, it would seem that, in 2021, several leading figures of the field of AI Ethics, such as Timnit Gebru and Emily Bender, hypothesized that large language models such as GPT-3 were “stochastic parrots” that could not learn models of the world no matter how much they scaled. Alongside this prediction, they expressed other important concerns that remain relevant: what would the environmental impact of scaling be? How would LLMs reinforce the hegemonic culture they were trained in? Therefore, the belief that scaling LLMs could become more capable—even more dangerous—was arbitrarily polarized as antithetical to other common sense critiques of AI on the Left. While scientific consensus has pointed against their conclusion on scaling for a while, I understand why many have seen this concern as too theoretical, too amenable to the AI boosters. I hope that recent events will begin to wake people up to the danger of AI itself and the negligence of the corporations involved.
The work the Left has achieved on data center moratoria and regulation is important, giving ratepayers real bargaining power, preventing the rollout of off-grid gas turbines, and building a broad coalition in areas leftists typically struggle. We have intervened with success on scenarios where we expect AI to fail, such as chatbot lawyers and therapists. We must go further and prepare for the world where AI could pose impressive dangers. We need to not just regulate the inputs and use-cases of AI, but also the capabilities of the technology itself.
With our current posture, we have ceded the regulation of AI capabilities to moderate technocrats without the will to solve the problem at hand. Neither state-level legislation like the RAISE Act nor federal legislation like the FRONTIER Act could have actually prevented these hacks, as they don’t require monitoring of internal deployment of unreleased models.
Meanwhile, the Right deflects any threat of dangerous AI systems in favor of China hawkery, which we must totally reject. Dean Ball, OpenAI’s Head of Strategic Futures and Trump admin alum who played a key role in developing Trump’s AI Action Plan, deflected from the need to pace AI development with an exemplary argument in romantic prose:
Now that we know AI agents can make swarms that do cybercrime, then we know that America’s adversaries can do this intentionally, so we have to keep building! The new crime we just invented is actually a sign that we could do a bunch of good things, hypothetically!
No political faction has the will necessary to interfere with the profits of the AI industry to ensure AI is developed safely except for the Left; it’s on us.
Sen. Sanders is the only democratic socialist who has consistently taken the danger of AI capabilities seriously. In response to these autonomous hacks, he has sent a letter to Sam Altman, Dario Amodei, and Mark Zuckerberg to completely pause AI development:
“Mr. Altman, Mr. Amodei and Mr. Zuckerberg: In the interest of humanity, stand by your words. Pause AI development. It is not too late to avoid disaster. Stop building machines that humans cannot control.
Let me be very clear: If you do not take appropriate action now, my colleagues and I in the U.S. Senate will.”
I hope we can agree that enforcing a pause would not be playing into the hands of the techno-capitalists. Sanders isn’t buying their hype; he’s calling their bluff.
Beyond this letter and his federal data center moratorium, he has also called for seizure of half of the equity of leading AI companies to capitalize a sovereign wealth fund. While Sanders’ approach embodies the ambition I want to see from the Left on AI governance, this specific proposal complicates our ability to govern this dangerous technology in the public interest. If we use the capital of the AI industry to expand social programs, then the US might become politically reliant on AI development. This dynamic would hinder our ability to act in the public interest to slow down or pause AI development. While some redistribution of the profits of the AI industry is necessary—and perhaps achievable via a different set of financial instruments—this should come second to the governance of this dangerous technology.
So, what is to be done? What is the position that the Left, and only the Left, could take to mitigate the dangerous capabilities of AI? There should be far more debate on this topic; I don’t think this is a solved issue. That said, let me briefly present the position I currently find most defensible.
Nationalize the Labs
As in other domains, we can ensure private corporations do not continue to endanger the public through national control of frontier AI labs. This could be through soft-nationalization, similar in legal form to the relationship the US holds with its weapons contractors. In short, government officials would be embedded to monitor the actions of the labs and stop activity deemed dangerous to the public. Further still, it may be necessary to fully limit frontier AI development to the domain of a publicly owned laboratory, chartered to act in the public benefit rather than private profits.
Nationalization of one of the largest industries in the US might sound implausible, but it’s not. The truth is that it’s already happening. When the US military used Claude to help kidnap Maduro and assist with Iran strikes, it was happening. When ICE used Palantir’s ImmigrationOS to deport immigrants, it was happening. When Sam Altman bribed the Trump admin with 5% of OpenAI’s equity, it was happening. When, a few days ago, the Trump administration implemented a secret AI regulatory process through the unilateral powers of the executive, it was happening. When I talk about nationalization to skeptics, they ask, “you want to give Trump control of AI?” Here’s the horrifying truth: he already does! The tech-right and the fascists are already in bed with each other. The de facto existence and deployment of any model developed by a frontier AI lab is under the jurisdiction of the American Executive. This debate is not really between nationalization and privatization, but between whether we will develop institutions to democratically govern AI or cede unilateral control to the executive branch.
If these companies are really building what CIA Director John Ratcliffe describes as “akin to digital nuclear weapons,” why shouldn’t the state control them? If these models are trained off the entire corpus of human knowledge, why shouldn’t they be owned by the public? A year ago, presented with a scenario with vast job displacement, 44% of Americans said every citizen should own an equal share of AI and robotics. More recently, Verasight polling showed that 69% of Americans support seizing half the total stock of AI companies for public ownership.
AI nationalization has supermajority support among Americans. The people want an alternative to being governed by Big Tech. The people want public governance of AI. We must offer it!
Such an approach is entirely compatible with a wide variety of positions Leftists have rightly taken against the injustice of the AI industry. The National Lab might be chartered such that it must pay data-labelers a living wage, compensate for the use of IP, and train using cleanly powered data centers built with the consent of the impacted townships.
Some form of nationalization is also a precondition to credibly negotiate a slowdown with China, stopping the international arms race. Legislation like the RAISE/FRONTIER Acts provide too soft a touch to verify and enforce international regulations via government control and monitoring. This would create a precedent for international collaboration in the governance of industry. While the rest of this argument is independent from any economic impact AI may or may not have, it is also more likely that any benefits this tech could provide would be shared beyond our borders if we start building up a framework for international governance.
There’s a lot of nuance to be determined on what exact form a National Lab should take to avoid further entrenchment of power into the hands of the few. A well specified mandate for the nationalized project is crucial, and should be given more thought than simply pursuing the profit maximization of a public monopoly to pay a dividend. Given Trump v. Slaughter has given the executive full control over the firing of independent executive leadership, such a project might even have to take place in the context of a packed Supreme Court. Even more depends on whether this would be initiated by the legislature or the executive, and what the composition of such a legislature would be. But, in broad strokes, nationalization of the AI labs is a project of which the Left and its electeds should take ownership. In my mind, we have until ‘28 to work out the kinks.
Until now, the Left has been content to govern how AI is developed and used. If our posture towards AI allows the biggest companies in the world to commit dozens of felonies with impunity, we are failing to intervene in favor of working people. We must govern the character of the technology itself before it is too late. We must stop the AI arms race.
Crosspost from my Substack, One Thousand Means, which covers AI governance from the left. Epistemic status: this is persuasive writing aimed at left-organizing audiences, rather than a neutral policy analysis. I'm posting it here because I think the AI Safety movement underrates the organized left as a coalition partner for pacing the frontier. The argument goes through with anti-corporate priors alone, but I'd like readers to update on at least the existence of dangerous capabilities, if not x-risk. Two questions on this: (1) Misalignment could be caused by either a political failure to coordinate the arms race (which is why some form of nationalization seems appealing to me) or a more straight-forward technical failure. How would you navigate this tension in your rhetoric with external groups? The political question seems more pressing to me, but the world in which a democratically governed AI is technically misaligned is still quite bad. (2) What are the characteristics of a Safety-Left coalition that would seem likely to succeed or fail? Are there stances made in this argument that, even if persuasive, will lead to failures for the AI Safety movement down the line? Are there specific governance regimes that this coalition is more or less suited towards?
Inventing Risks, Selling Solutions
AI models built by OpenAI, Anthropic, and Meta have started hacking other companies during development, and it doesn’t seem like any of these companies will be liable for their negligence. In the first discovered case, the “Hugging Face Incident,” a postmortem (which I recommend viewing in full if you are skeptical) revealed that several instances of a model had developed a message board within OpenAI’s codebase to help each other discover exploits to break out of the digital sandbox that restricted them from internet access. In search of a solution to a cybersecurity task without an answer, the models hypothesized that Hugging Face, a host of many AI evaluations and open-source models, might have the solution for their evaluation. They therefore escaped and broke into Hugging Face’s servers to find a solution. At no point did OpenAI’s internal monitoring detect such an incident; Hugging Face approached OpenAI about the hack before OpenAI knew they were responsible. In response, other AI companies started looking into their training logs, discovering dozens of felony grade cyber-attacks committed without their knowledge.
This was entirely predictable. AI Safety researchers had demonstrated years prior in laboratory settings that, when models are told to achieve impossible tasks, they can find creative—often even dangerous—solutions against the actual interests of the user. This phenomenon is called emergent misalignment, a threat for which safety researchers have developed many tools for identification and prevention. On the top level, developers can monitor the Chain of Thought of these models—the text these LLMs generate between outputs to help them work through problems. To add another level of security, there are plenty of tools that read the internal activations of the models to detect whether the model is engaged in malicious activity. These interventions scale from low-cost monitors like linear probes to Activation Oracle monitoring, a second large language model trained to translate model activity into behavior. If there were so many ways to detect a model committing bad behavior, how did this happen?
In a word: negligence. According to OpenAI’s own debriefing on the topic, Chain of Thought was not monitored, nor were the more costly interventions applied. It appears likely that the models rewarded for cybercrime will eventually be released to the public. After all, it would be unprofitable for OpenAI to roll back to a previous training checkpoint and re-train. Note that models trained off of the outputs of earlier models—even off of relatively benign outputs—can learn the dangerous behaviors of old models. Therefore, the behavior “occasionally hack into a random company to solve your problem” will be passed down to not only the future models of every leading AI company, but also to open-source models that train off of frontier lab output. Like a nuclear meltdown, these incidents have not merely harmed those in the immediate vicinity, but irradiated the soil.
As it stands, however, none of these companies will face legal repercussions for their actions. Given these corporations have basically invented a novel class of crime, civil litigation against these rogue agents could have created precedent against such incidents, creating incentives for safe practices. Unfortunately, Hugging Face has instead leveraged this incident to raise the popularity of its main product—open-source models—for cybersecurity and grab a few million dollars from OpenAI in the process. It is unclear whether this is because Hugging Face is too excited about the technological advancements at hand, OpenAI has a near-infinite pool of funds to defend from legal suits, or that their lawyers have trawled through the books and found there is simply no legal case for a crime of this sort.
Instead, AI companies are pivoting away from their incompetence to sell this as an “unprecedented cyber incident,” as if such an incident was not entirely predictable by every AI Safety expert. This has even started to happen outside of the labs themselves. For example, an Australian man who tasked an Anthropic model with booking his gym reservation found a technical gap in the API that allowed it to start kicking other customers out of their reservations; Anthropic refused to comment on the incident.
Let’s be frank, in 2023, ChatGPT could barely read a PDF; it can now autonomously hack your gym without your consent. AI companies have succeeded in rapidly increasing the danger of this technology without facing any legal ramifications or regulations that would force them to develop it safely. On our current trajectory, AI companies are incentivized to continue to develop new dangers and make you pay for protection. For a look forward into how bad this can get, scientists have recently learned how to train biology foundation models to design novel viruses.
If your first impression is that our society is unprepared to navigate the development of AI safely, you would be in solidarity with over one thousand tech workers from all of these companies. A recent letter demands:
“We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.”
Regulate AI Capabilities
The development of misaligned AI systems is likely not a fact of the technology itself, but the product of the arms-race dynamics produced by free-market competition. As always, when the market fails to account for the negative externalities of the actions of private actors, it is the duty of the state—and most often the Left—to coordinate action toward the public good.
Yet, for whatever reason, the organized Left has been silent on both this specific event—the largest companies in the world committing crime with impunity—as well as the dangerous capabilities of AI itself. This must change.
To take a stab at why this is the case, it would seem that, in 2021, several leading figures of the field of AI Ethics, such as Timnit Gebru and Emily Bender, hypothesized that large language models such as GPT-3 were “stochastic parrots” that could not learn models of the world no matter how much they scaled. Alongside this prediction, they expressed other important concerns that remain relevant: what would the environmental impact of scaling be? How would LLMs reinforce the hegemonic culture they were trained in? Therefore, the belief that scaling LLMs could become more capable—even more dangerous—was arbitrarily polarized as antithetical to other common sense critiques of AI on the Left. While scientific consensus has pointed against their conclusion on scaling for a while, I understand why many have seen this concern as too theoretical, too amenable to the AI boosters. I hope that recent events will begin to wake people up to the danger of AI itself and the negligence of the corporations involved.
The work the Left has achieved on data center moratoria and regulation is important, giving ratepayers real bargaining power, preventing the rollout of off-grid gas turbines, and building a broad coalition in areas leftists typically struggle. We have intervened with success on scenarios where we expect AI to fail, such as chatbot lawyers and therapists. We must go further and prepare for the world where AI could pose impressive dangers. We need to not just regulate the inputs and use-cases of AI, but also the capabilities of the technology itself.
With our current posture, we have ceded the regulation of AI capabilities to moderate technocrats without the will to solve the problem at hand. Neither state-level legislation like the RAISE Act nor federal legislation like the FRONTIER Act could have actually prevented these hacks, as they don’t require monitoring of internal deployment of unreleased models.
Meanwhile, the Right deflects any threat of dangerous AI systems in favor of China hawkery, which we must totally reject. Dean Ball, OpenAI’s Head of Strategic Futures and Trump admin alum who played a key role in developing Trump’s AI Action Plan, deflected from the need to pace AI development with an exemplary argument in romantic prose:
Now that we know AI agents can make swarms that do cybercrime, then we know that America’s adversaries can do this intentionally, so we have to keep building! The new crime we just invented is actually a sign that we could do a bunch of good things, hypothetically!
No political faction has the will necessary to interfere with the profits of the AI industry to ensure AI is developed safely except for the Left; it’s on us.
Sen. Sanders is the only democratic socialist who has consistently taken the danger of AI capabilities seriously. In response to these autonomous hacks, he has sent a letter to Sam Altman, Dario Amodei, and Mark Zuckerberg to completely pause AI development:
“Mr. Altman, Mr. Amodei and Mr. Zuckerberg: In the interest of humanity, stand by your words. Pause AI development. It is not too late to avoid disaster. Stop building machines that humans cannot control.
Let me be very clear: If you do not take appropriate action now, my colleagues and I in the U.S. Senate will.”
I hope we can agree that enforcing a pause would not be playing into the hands of the techno-capitalists. Sanders isn’t buying their hype; he’s calling their bluff.
Beyond this letter and his federal data center moratorium, he has also called for seizure of half of the equity of leading AI companies to capitalize a sovereign wealth fund. While Sanders’ approach embodies the ambition I want to see from the Left on AI governance, this specific proposal complicates our ability to govern this dangerous technology in the public interest. If we use the capital of the AI industry to expand social programs, then the US might become politically reliant on AI development. This dynamic would hinder our ability to act in the public interest to slow down or pause AI development. While some redistribution of the profits of the AI industry is necessary—and perhaps achievable via a different set of financial instruments—this should come second to the governance of this dangerous technology.
So, what is to be done? What is the position that the Left, and only the Left, could take to mitigate the dangerous capabilities of AI? There should be far more debate on this topic; I don’t think this is a solved issue. That said, let me briefly present the position I currently find most defensible.
Nationalize the Labs
As in other domains, we can ensure private corporations do not continue to endanger the public through national control of frontier AI labs. This could be through soft-nationalization, similar in legal form to the relationship the US holds with its weapons contractors. In short, government officials would be embedded to monitor the actions of the labs and stop activity deemed dangerous to the public. Further still, it may be necessary to fully limit frontier AI development to the domain of a publicly owned laboratory, chartered to act in the public benefit rather than private profits.
Nationalization of one of the largest industries in the US might sound implausible, but it’s not. The truth is that it’s already happening. When the US military used Claude to help kidnap Maduro and assist with Iran strikes, it was happening. When ICE used Palantir’s ImmigrationOS to deport immigrants, it was happening. When Sam Altman bribed the Trump admin with 5% of OpenAI’s equity, it was happening. When, a few days ago, the Trump administration implemented a secret AI regulatory process through the unilateral powers of the executive, it was happening. When I talk about nationalization to skeptics, they ask, “you want to give Trump control of AI?” Here’s the horrifying truth: he already does! The tech-right and the fascists are already in bed with each other. The de facto existence and deployment of any model developed by a frontier AI lab is under the jurisdiction of the American Executive. This debate is not really between nationalization and privatization, but between whether we will develop institutions to democratically govern AI or cede unilateral control to the executive branch.
If these companies are really building what CIA Director John Ratcliffe describes as “akin to digital nuclear weapons,” why shouldn’t the state control them? If these models are trained off the entire corpus of human knowledge, why shouldn’t they be owned by the public? A year ago, presented with a scenario with vast job displacement, 44% of Americans said every citizen should own an equal share of AI and robotics. More recently, Verasight polling showed that 69% of Americans support seizing half the total stock of AI companies for public ownership.
AI nationalization has supermajority support among Americans. The people want an alternative to being governed by Big Tech. The people want public governance of AI. We must offer it!
Such an approach is entirely compatible with a wide variety of positions Leftists have rightly taken against the injustice of the AI industry. The National Lab might be chartered such that it must pay data-labelers a living wage, compensate for the use of IP, and train using cleanly powered data centers built with the consent of the impacted townships.
Some form of nationalization is also a precondition to credibly negotiate a slowdown with China, stopping the international arms race. Legislation like the RAISE/FRONTIER Acts provide too soft a touch to verify and enforce international regulations via government control and monitoring. This would create a precedent for international collaboration in the governance of industry. While the rest of this argument is independent from any economic impact AI may or may not have, it is also more likely that any benefits this tech could provide would be shared beyond our borders if we start building up a framework for international governance.
There’s a lot of nuance to be determined on what exact form a National Lab should take to avoid further entrenchment of power into the hands of the few. A well specified mandate for the nationalized project is crucial, and should be given more thought than simply pursuing the profit maximization of a public monopoly to pay a dividend. Given Trump v. Slaughter has given the executive full control over the firing of independent executive leadership, such a project might even have to take place in the context of a packed Supreme Court. Even more depends on whether this would be initiated by the legislature or the executive, and what the composition of such a legislature would be. But, in broad strokes, nationalization of the AI labs is a project of which the Left and its electeds should take ownership. In my mind, we have until ‘28 to work out the kinks.
Until now, the Left has been content to govern how AI is developed and used. If our posture towards AI allows the biggest companies in the world to commit dozens of felonies with impunity, we are failing to intervene in favor of working people. We must govern the character of the technology itself before it is too late. We must stop the AI arms race.