Mod note: this is an edge case on frontpaging (which mostly goes off of "is this timeless? would someone still care about this in 4-10 years?"). I think probably analysis of this bill will still be useful to read in the future, but, "a particular bill happening this year" is usually not frontpage.
Thanks for making this! For what it's worth, it looks like 22757.12(d) requires developers to be somewhat transparent about risks from internal deployment, which seems quite interesting to me and I'm surprised didn't make the summary.
Today I'm launching SB53.info, a resource to help people understand California's Senate Bill 53. The site shows the text of SB 53 enriched with annotations to explain its political and scientific context.
In a nutshell, SB 53 says seven things. These are the four I consider most important:
And these are the three major provisions that I expect will be less important:
For my most thorough analysis of these key provisions, I encourage you to read SB53.info. Here, I'll give a high-level overview of what each provision does and how it departs from the status quo.
Large AI developers are not currently mandated to adopt safety policies, but under SB 53, they would be. This wouldn't require most frontier developers to do anything qualitatively new, since they already have published safety policies, and they've already made non-enforceable commitments to follow those policies. Anthropic, OpenAI, Google DeepMind, and Meta have all written safety policies that satisfy many of the requirements in § 22757.12.a, and xAI has a draft safety policy that satisfies a few of the requirements. So if SB 53 were to pass, one industry laggard would have to write a safety policy for the first time, other frontier developers would have to make their existing safety policies more robust, and every frontier developer would be legally mandated to follow their own safety policy.
Moreover, they must get an independent auditor to certify that they are following their safety policy, and this explicitly includes certifying that the safety policy is clear enough for it to be determinate whether the developer is complying with it. As far as is publicly known, no major AI developer has ever undergone a safety audit, but such audits are completely routine in other risky industries like aviation. Every airline in the US is required to write a plan explaining what measures they will follow to ensure safety, and they must regularly commission independent audits to confirm that they're following the plan. SB 53 would require companies developing frontier AI systems to do the same.
It is already a widely accepted best practice in the AI industry that when a company releases a new frontier model, they should publish a report called a model card describing the model's capabilities to consumers and to the scientific community. Anthropic, OpenAI, and Google DeepMind have consistently released model cards alongside all of their recent frontier models, and all three companies' model cards likely comply with most of the requirements in SB 53. These cards generally explain how the developer assessed the risks posed by their model, how they intend to mitigate those risks, and whether their model reached any prespecified risk or capability thresholds. If the bill were to pass, the big three AI developers would have to disclose more detailed information about third party assessments run on their models, and developers like xAI that generally don't publish model cards would have to start publishing them.
SB 53 would make large developers civilly liable for breaches of the above rules. No AI company executives will go to jail for failing to publish a safety policy or model card, but their companies can be faced with heavy fines—up to millions of dollars for a knowing violation of the law that causes material catastrophic risk. This is a major change from the status quo. Today, frontier AI developers have no legal obligation to disclose anything about their safety and security protocols to government, let alone to the public. When a company releases a new AI system more powerful than any system before, it is entirely optional under present law for them to tell consumers what dangerous things that system can do. And if a company does choose to adopt a safety policy or publish a model card, there is no force of law to guarantee the safety policy is being implemented or that the model card is accurate. This would all change under SB 53. We'd no longer have to rely on AI developers' good will to share critical safety information with the public.
There is currently no official channel for the California state government to collect reports of safety incidents involving AI. If a frontier AI developer discovered tomorrow that the weights of their leading model had been stolen, the best they could do to alert state authorities would probably be to email the Attorney General's office. If a member of the public witnessed an AI autonomously causing harm in the wild, the fastest way for them to tell the authorities would probably be to tweet about it. SB 53 would replace these slow, informal information channels with an official incident reporting mechanism run by the AG. Just like California has an official website to collect reports of data breaches, there would be another site for reports of critical AI safety incidents.
Existing California law already offers whistleblower protection to AI company employees who report a violation of federal, state, or local law to public officials or to their superiors. Companies may not make rules or enforce contracts that would prevent their employees from blowing the whistle, nor can they retaliate against an employee who becomes a whistleblower. SB 53 expands the scope of these protections in two ways. First, it would grant whistleblower protection to actors who are currently not protected. Independent contractors, freelancers, unpaid advisors, and external groups that help developers to assess and manage catastrophic risk are not protected by existing law if they become whistleblowers, but they would be under SB 53. Second, the bill would protect disclosures of evidence that an AI developer's activities pose a catastrophic risk, whereas existing law only protects disclosures of evidence that a developer is breaking the law. Of course, many ways that a developer could cause a catastrophic risk would also involve breaking the law, but it's conceivable that a developer could do something catastrophically dangerous yet legal. It might also be easier for many would-be whistleblowers to tell whether their employer is causing a catastrophic risk than to tell whether their employer is breaking a specific law.
Finally, SB 53 calls for California to build a publicly owned AI compute cluster called CalCompute. The cluster's purpose would be to support AI research and innovation for the public benefit. Nothing like CalCompute currently exists in California, but similar projects have been announced or are already underway in several other jurisdictions. New York has already built a compute cluster under their Empire AI initiative, the UK has given academics compute access through its AI Research Resource, and the US National Science Foundation's National AI Research Resource aims to provide the same for American researchers. SB 53 does not specify how much funding California will put behind CalCompute, nor how many AI chips it aims to acquire, so it's hard to tell how much this section of the bill will accomplish. If CalCompute is funded generously in the next state budget, it could be a big deal, but if the project only gets a meager budget, it may not achieve much.
I have intentionally kept SB53.info neutral. The purpose of the site is to help readers with low context on AI safety and governance get up to speed and understand SB 53, not to editorialize in favor of the bill or against it. But of course I do have an opinion on SB 53. I think it would probably be good for California and for the world if the bill passes.
Requiring every frontier AI company to follow a safety policy and to publish model cards alongside their strongest models is just about the least we can ask them to do for safety. In the first place, we know it's not too hard for the companies to comply since Anthropic, OpenAI, and GDM already have safety policies and model cards that satisfy most of the requirements in SB 53. It's not asking much for them to be somewhat more transparent and for less responsible companies near the frontier to get serious about safety and transparency.
And if we look beyond the AI industry, it becomes even clearer how light SB 53's safety and disclosure requirements are in absolute terms. We make airlines write detailed safety plans, and we hold them legally accountable if they fail to follow those plans. Why shouldn't we make the largest AI companies do the same? Would it really make sense for companies that are trying to build godlike superintelligence to do less safety planning than companies that fly planes? Again, if a company wants to sell children's crayons, we make them test those crayons for safety hazards and publish a report explaining why they believe it is safe to sell their crayons to consumers. Would it really be sensible for a company to deploy AGI to the public with less safety information than Crayola provides when it sells crayons?[1] My point is that the safety and transparency requirements SB 53 would impose on large AI developers are not out of the ordinary. We already impose them on just about any business that creates even a small risk to the public.
On the other hand, SB 53 is just about the strongest AI safety regulation that we can know right now will be robustly good. In most other risky industries, we pretty much understand what steps businesses need to take to avoid accidents and keep the public safe, so we pass detailed regulations requiring them to take those steps. If you're cooking food, you need to store it at such and such a temperature, protect it from contaminants, and so on. If you're driving a truck, you have to rest for so many hours in each twenty-four hour window, and you have to check regularly that your truck is in good repair. But we are not yet in a position to pass such detailed safety regulations for AI developers. Our best threat modeling is still so janky and the sciences of alignment and control are still so immature that nobody knows precisely what you have to do to guarantee that an advanced AI will be safe. There is no checklist yet.
The best we can do in this situation is to ask large AI developers for more transparency. As the California Report repeatedly stresses, the key benefit of transparency is that it aligns the AI developers' incentives with public welfare. We may not know now exactly what safety measures we'll want developers to be taking in three years,[2] but we'll definitely want to know then what measures they actually are taking. That way, we can assess whether their safety practices are adequate in light of all the evidence we don't have yet but will have gained in the future. We can determine what practices are standard across the AI industry and create a race to the top on safety by applauding leaders and pressuring laggards to improve. And if needed, we can write more detailed, prescriptive regulation from a position of greater knowledge than we have now. None of this is possible without basic transparency.
To their credit, many of our leading AI developers are already quite transparent. They've chosen to share far more information about their safety protocols and their cutting edge models than they have to for purely commercial reasons. This is great, but we shouldn't keep relying on AI companies' good will to keep the public informed of critical safety information. It's time to make transparency non-optional.
One might question whether Crayola should have to test its products and publish safety reports. Maybe the toy industry is over-regulated, and it would be better on balance if we scrapped even the most basic disclosure requirements. For my part, I do not think this is the case, but my higher level response to someone arguing along these lines is that they're objecting to all consumer safety regulation, not to AI regulation in particular. If we want to have that debate, we certainly can, but we should notice that the opponent of AI regulation is now making a much more sweeping claim than they were before.
Why should we be uncertain about what safety measures will be appropriate in three years? What might have happened by then to invalidate our current best guesses? (1) The way we train and deploy frontier AI might have changed dramatically by then. Eg, we might switch from discrete deployments to continuous learning. (2) The background risk environment may have changed. Eg, we may have imposed universal nucleic acid synthesis screening, reducing the need for stringent bio-misuse mitigations. (3) Alignment and control techniques that aren't feasible now might have become feasible, or the inverse. Eg, chain-of-thought monitoring looks like an effective strategy for controlling AIs now, but it could soon become useless if companies train their models to reason less legibly.