There is a newly leaked bill that Anthropic is purportedly shopping around is (if you assume its bugs get fixed).
It is similar to the FRONTIER Act, with stronger preemption of state laws, stronger transparency provisions and weaker enforcement provisions.
Having slept on this, I find it unacceptable that Anthropic is actively shopping exactly the type of bill I have warned we will need to fight against during the lame duck session. They are trying to use this last time with a Republican Congress to pass preemption of exactly the important state laws, and get some transparency but not much enforcement in exchange.
This reads as intentional, that Anthropic is more afraid that the government might enforce rules against Anthropic, than they are worried about the ability of the government to enforce rules against everyone. Anthropic’s policy team seems to have, chosen protecting their business interests and guarding against their political risk, at the expense of safety considerations.
There was no need for the company purportedly existing to promote safety to propose a bill that is not even actively stronger than the FRONTIER Act, when the FRONTIER Act was a compromise bill proposed back in June.
Even if you were fine with what is being given up, what you get in return very obviously does not meet the moment. Nor is there any need for such compromises. It seems rather insane for Anthropic to be proposing it now. This seems like the bill that someone on the anti-regulatory side should be pushing. It is not obvious this makes us safer than doing nothing, even if the alternative is fully nothing at all levels.
Say it isn’t so, Anthropic, or at least that it was a mistake and is no longer so, or I am going to be rather cross.
I am at a conference and cannot do an RTFB, but it feels important to get word of this out quickly, especially so Anthropic employees can look into this and ask questions.
I have not had the time to do so, but the other usual suspects quickly sprinted to RTFB (read the bill) and they did not like what they saw. You can try to call this a ‘pretty reasonable AI safety bill’ as Charlie Bullock does, but that is by the standards of people trying to protect the industry from more impactful regulation.
Ketan Ramakrishnan: Anthropic’s new federal AI bill is, without any overstatement, astonishingly bad.
Charlie Bullock: Big news. My take: Anthropic deserves some credit for actually proposing a pretty reasonable AI safety bill; that’s more than other frontier labs have done.
But they don’t deserve *too* much credit, IMO; this bill is weaker than the Trahan-Obernolte FRONTIER Act in some key ways, and since the political momentum has shifted significantly in the direction of safety since FRONTIER was introduced back in July, I think new bills introduced today should ideally go further than FRONTIER was able to go rather than regressing.
To be fair to Anthropic, the main places where this bill is significantly weaker than FRONTIER are (a) rulemaking (very strong in FRONTIER, almost nonexistent in Anthropic’s bill) and (b) the emergency shutdown authority. Even if you take safety seriously, it might make some sense to dial those authorities back if you didn’t trust the executive branch to wield them responsibly, because those are the easiest provisions for a hypothetical President who hypothetically might not like a certain frontier AI company to abuse.
There are some improvements over FRONTIER as well; for instance, this bill does specify that auditors are employed on a continuing basis and has better provisions regarding auditor access to information. Happy to see that from Anthropic (although I think their access provision is still a little undercooked and has Fourth Amendment problems).
Overall, I think the embedded auditor bills that get introduced in the coming months are going to be significantly better and more ambitious than this, and I hope Anthropic supports those bills instead of opposing anything that treats catastrophic risks like the serious national security issue they are. Still, it’s encouraging to see a frontier lab come this far.
Charlie Bullock: I think the drafting is okay. I’ve seen better but I’ve definitely seen worse. Claude is a pretty good legislative drafter. There are a few clear mistakes but that’s par for the course in a leaked discussion draft.
Nathan Calvin: Charlie’s analysis is right but he is nicer than I am – I found myself viscerally angry at multiple points reading this bill. There are some good and sensible parts, but overall this legislation is not remotely close to meeting the moment and I overall think passing it would make us less protected against catastrophic risks than we are under existing law.
Two parts that stand out as particularly infuriating when taken together:
1. Some of the preemption provisions are remarkably far reaching and consequential, and in addition to preempting legislation like SB 53 would also remove general powers that state courts have to enjoin defendants and protect the public. Most outrageously, it has a provision that looks like its intended to stops states from enforcing laws or judicial decrees that “suspend the development, deployment, or internal use of an AI model” to protect their citizens from imminent catastrophic harms. Such powers to block and suspend model deployments would then only reside with the federal government.
2. The federal governments powers to restrict the development, deployment, or internal use of a frontier model has to meet an absurdly high bar.
First, there needs to be an imminent risk of a catastrophic harm (meaning killing more than 50 people or causing more than a billion dollars in damage). But thats not enough! The agency head needs to make a detailed determination with no deference given to show why no alternative would prevent the imminent catastrophe. Next the court is also explicitly told to consider the harms to the developer and other harms from restricting the model, but no considerations in the other direction.
This is absurd! For all of its warning about the potential for AI to cause a catastrophe that kills literally everyone on the planet, Anthropic’s proposed federal law likely makes it harder than under existing law for a court to get an order blocking or restricting the development of frontier models? WTF??
There are also a bunch of others things that I would like to see improved – the substantive details of the auditing section and the potential for rule-making on substantive safety standards both come to mind – but I think these stand out as particularly indefensible.
Charlie Bullock: Nathan’s righteous anger has inspired me to make one more mildly disapproving observation: it’s kind of wild that the Anthropic bill has no government licensing for the independent auditors.
Licensing is, IMO, the bare minimum thing that you need to even be able to pretend that your auditing regime isn’t just going to devolve into a race to the bottom. Without it, auditors will inevitably compete with each other to offer their customers, the labs, the least bothersome audits available at the lowest possible price point. Obviously Anthropic trusts itself to do a great job of picking great auditors, but do they trust OpenAI to do the same? What about SpaceXSI?
Why Would Anthropic Do This?
The new Anthropic bill is weaker than their June framework, and offers both forward-looking and backward-looking preemption in exchange, despite months of movement to enable us to do more. What are we even doing?
Nate Soares: Anthropic’s proposed law would make it even harder than it already is for a court to block or restrict the development of an AI that threatens all humanity.
Alex Bores: Anthropic is pitching a bill that closely matches its framework from June. How can you watch Hugging Face, Coxon, & rogue AI swarms, and conclude that the bill you cooked up in June is sufficient?
This comes nowhere close to the speed of developments nor the speed we need to keep people safe.
This bill also includes an anti-democracy provision (preemption). Forward-looking anti-democracy provisions should be DOA.
But even if you would accept some Anthropic’s June framework says, “Congress should not preempt state law unless it enacts a rigorous federal regime that meets or exceeds the strongest measures proposed in this framework.”
Yet this bill is weaker than the framework, and in some places, weaker than existing state laws (e.g. internal use, reporting timelines, types of risks covered, etc.). By Anthropic’s own June standards, this bill should not include any anti-democracy provision.
Well, I know what it looks like Anthropic is prioritizing. The silly mistakes are a known and acceptable hazard, but there are things that really should not have made it into a bill that is being pitched to lawmakers.
Sydney: I have a lot of concerns with this bill. I understand it’s a draft and might have silly mistakes. But this bill looks like it came from a culture where the top priority is making sure the government doesn’t getting in the way of making money, even at the cost of safety. That is a very dangerous culture.
Specifically, the bill says that states would not be able to have AI regulation that touches frontier-risk disclosure, third-party evaluation, and incident reporting. This one federal bill would replace almost all of SB53 and the RAISE Act. And it’s a pretty weak federal bill: I agree with others that it’s weaker than the FRONTIER Act, a relevant and politically viable bill introduced back in July.
Kelsey Piper: One unfortunate conclusion I’m reaching from this is that Anthropic, while acknowledging they’re doing something extremely dangerous, is more importantly wrt policy terrified of the executive branch having the ability to flexibly make them stop at the drop of a hat.
This isn’t shocking given that they’ve been on the receiving end of the executive branch doing that in an unprincipled and stupid way! Anyone who has to deal with a bureaucracy becomes a libertarian, and the administration’s treatment of Anthropic was genuinely outrageous.
But it means that they’re interested in giving themselves more process-based protections when everyone else is interested in giving us more process-based protections against them. The AI safety movement wants the ability to shut down superintelligent model development. Anthropic wants to significantly curtail current authorities to do that so that it has to be narrowly tailored to a specific imminent threat in only a few possible categories. The AI safety movement wants the states to be able to pass stronger laws than it can get through federally. Anthropic wants preemption.
You also kind of have to wonder if the Global Affairs people and the “we are terrified that we’re about to kick off RSI” people at Anthropic talk to each other.
Global affairs: So we’ve been working very diligently to make sure the Trump administration can’t just blacklist Fable 6.5 since we didn’t donate as much money to his ballroom as Greg Brockman did.
Technical staff: We think Fable 6.5 might be able to copy itself into an unmonitored data center and start doing experiments we can’t observe.
Global affairs: Sounds crazy, hope you’re wrong! Anyway, we’re lobbying for no further state laws around safety, since it’s really expensive to fight ill-conceived laws in 50 state legislatures at once.
Technical staff: Money may or may not still have any meaning in 36 months.
Global affairs (giggling nervously): It’s that San Francisco air, man. Everyone gets like that if they move there. Anyway, obviously if the government makes a narrowly tailored finding that an AI poses an imminent risk of causing more than 50 deaths-
Technical staff: We give it a 10 percent chance it’ll cause 8 billion.
Nat Purser: there are some good provisions in this bill, but ultimately it’s a big step back from the FRONTIER Act.
ai capabilities are advancing rapidly — it just doesn’t make sense to cement a weaker federal ceiling into place now.
a few specific areas for improvement:
– create stronger mechanisms to compel corrective action, so major issues auditors identify don’t go unaddressed;
– foster true auditor independence by not allowing auditors to be paid (or selected) by the labs they audit;
– empower the government to establish and iterate on strong minimum standards, so they grow alongside capabilities.
if you’re going to preempt many state safety laws, the least you can do is create a federal regulatory regime that can evolve in time.
Not Everything Here Is Bad But Oh Boy
As noted above, there are some good individual provisions. I like the broader coverage, that thresholds can move both ways, required risk reports and annual audits. The whistleblower provisions and security requirements seem strong.
Putting automated AI R&D into the list of risks is very cool and necessary.
The question is, even if you wanted to build on the FRONTIER framework, why treat this as a trade where you have to then give up other things?
And why are we this deadly afraid of what the White House might do, when they clearly already can take down a model with things like export controls or otherwise order you around? Then we add a ‘kill switch’ provision that seems written mostly out of terror that it might get abused. We also get traditional Anthropic objections like ‘we don’t want to have to respond to safety incidents within 72 hours unless it is a true emergency.’
Why take out FRONTIER’s requirements for a cost-benefit analysis on catastrophic risks and substitute at best highly weak tea?
The maximally charitable reading is that doing anything is a tough sell even now, and the White House wants no bill, so you have to give and take and this is as good as one can hope to do modulo fixing its bugs.
Except this is backwards. The way to ‘pass nothing’ is to pass preemption, in the lame duck, in exchange for as little as possible. If the White House fails to realize this, that does not change facts on the ground.
Anthropic needs to course correct, quickly. Earlier yesterday I already expressed, here at the conference I am at, that ‘what Anthropic would go for’ looks like a limiting factor in what we can achieve. And, well, here we are.
There is a newly leaked bill that Anthropic is purportedly shopping around is (if you assume its bugs get fixed).
It is similar to the FRONTIER Act, with stronger preemption of state laws, stronger transparency provisions and weaker enforcement provisions.
Having slept on this, I find it unacceptable that Anthropic is actively shopping exactly the type of bill I have warned we will need to fight against during the lame duck session. They are trying to use this last time with a Republican Congress to pass preemption of exactly the important state laws, and get some transparency but not much enforcement in exchange.
This reads as intentional, that Anthropic is more afraid that the government might enforce rules against Anthropic, than they are worried about the ability of the government to enforce rules against everyone. Anthropic’s policy team seems to have, chosen protecting their business interests and guarding against their political risk, at the expense of safety considerations.
There was no need for the company purportedly existing to promote safety to propose a bill that is not even actively stronger than the FRONTIER Act, when the FRONTIER Act was a compromise bill proposed back in June.
Even if you were fine with what is being given up, what you get in return very obviously does not meet the moment. Nor is there any need for such compromises. It seems rather insane for Anthropic to be proposing it now. This seems like the bill that someone on the anti-regulatory side should be pushing. It is not obvious this makes us safer than doing nothing, even if the alternative is fully nothing at all levels.
Say it isn’t so, Anthropic, or at least that it was a mistake and is no longer so, or I am going to be rather cross.
I am at a conference and cannot do an RTFB, but it feels important to get word of this out quickly, especially so Anthropic employees can look into this and ask questions.
If you want the full text, here is a copy.
Others RTFB (Read the Bill)
I have not had the time to do so, but the other usual suspects quickly sprinted to RTFB (read the bill) and they did not like what they saw. You can try to call this a ‘pretty reasonable AI safety bill’ as Charlie Bullock does, but that is by the standards of people trying to protect the industry from more impactful regulation.
Why Would Anthropic Do This?
The new Anthropic bill is weaker than their June framework, and offers both forward-looking and backward-looking preemption in exchange, despite months of movement to enable us to do more. What are we even doing?
Well, I know what it looks like Anthropic is prioritizing. The silly mistakes are a known and acceptable hazard, but there are things that really should not have made it into a bill that is being pitched to lawmakers.
Not Everything Here Is Bad But Oh Boy
As noted above, there are some good individual provisions. I like the broader coverage, that thresholds can move both ways, required risk reports and annual audits. The whistleblower provisions and security requirements seem strong.
Putting automated AI R&D into the list of risks is very cool and necessary.
The question is, even if you wanted to build on the FRONTIER framework, why treat this as a trade where you have to then give up other things?
And why are we this deadly afraid of what the White House might do, when they clearly already can take down a model with things like export controls or otherwise order you around? Then we add a ‘kill switch’ provision that seems written mostly out of terror that it might get abused. We also get traditional Anthropic objections like ‘we don’t want to have to respond to safety incidents within 72 hours unless it is a true emergency.’
Why take out FRONTIER’s requirements for a cost-benefit analysis on catastrophic risks and substitute at best highly weak tea?
The maximally charitable reading is that doing anything is a tough sell even now, and the White House wants no bill, so you have to give and take and this is as good as one can hope to do modulo fixing its bugs.
Except this is backwards. The way to ‘pass nothing’ is to pass preemption, in the lame duck, in exchange for as little as possible. If the White House fails to realize this, that does not change facts on the ground.
Anthropic needs to course correct, quickly. Earlier yesterday I already expressed, here at the conference I am at, that ‘what Anthropic would go for’ looks like a limiting factor in what we can achieve. And, well, here we are.