AISN #31: A New AI Policy Bill in California Plus, Precedents for AI Governance and The EU AI Office

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

Subscribe here to receive future versions.

Listen to the AI Safety Newsletter for free on Spotify.

This week, we’ll discuss:

A new proposed AI bill in California which requires frontier AI developers to adopt safety and security protocols, and clarifies that developers bear legal liability if their AI systems cause unreasonable risks or critical harms to public safety.
Precedents for AI governance from healthcare and biosecurity.
The EU AI Act and job opportunities at their enforcement agency, the AI Office.

A New Bill on AI Policy in California

Several leading AI companies have public plans for how they’ll invest in safety and security as they develop more dangerous AI systems. A new bill in California’s state legislature would codify this practice as a legal requirement, and clarify the legal liability faced by developers whose AI systems cause unreasonable risks or critical harms to public safety. (Note: The bill was co-sponsored by the Center for AI Safety Action Fund.)

Many of the world’s leading AI developers and cloud compute providers are located in California. This means the state has significant direct control over AI development, and it could serve as a testing grounds for policies that might later be adopted nationally or internationally.

Safety and security protocols for advanced AI models. This bill would require developers of certain leading AI models to adopt safety and security protocols. This requirement only applies to models that may surpass the capabilities of all models previously established as safe. Models that will not exceed the performance of known safe models are exempt from these requirements.

Specific requirements include cybersecurity measures to “prevent theft, misappropriation, malicious use, or inadvertent release or escape of the model weights.” Securing model weights against theft might require multi-year efforts to build air-gapped data centers and insider threat detection programs, argued a recent RAND report. Additionally, developers would need to implement the capacity to fully shut down their models in the event of an emergency. More broadly, they would need to implement best practices as developed by industry, academia, and standard-setting organizations such as NIST.

Developers would not be allowed to release any model that poses an “unreasonable risk… that an individual may be able to use the hazardous capabilities of the model” to cause a “critical harm.” Critical harms are defined to include the creation or use of chemical, biological, radiological, or nuclear (CBRN) weapons and cyberattacks, as well as causing specific economic damages of greater than $500M.

Compliance will be self-certified by AI companies and spot-checked via lawsuits. AI companies would be asked to self-certify that all of these requirements have been implemented faithfully. The bill also creates a new state office within the California Department of Technology called the Frontier Model Division which would gather information about corporate compliance with the law.

If there is evidence that a company has violated the law, the Attorney General of California would be able to sue in civil court. Judges could fine companies for up to 30% of model development costs for violations. Moreover, if there is “an imminent risk or threat to public safety,” companies could be ordered to temporarily shut down or permanently destroy their models.

The bill contains a variety of other provisions, including:

Cloud compute providers would need to conduct Know Your Customer (KYC) screenings for clients seeking large quantities of compute
Companies would be required to report incidents of concerning AI behavior
California would establish a new public computing cluster called CalCompute
Whistleblowers would be protected from retaliation by their employers, though the bill would not create the positive incentives for whistleblowing that exist in other industries

Overall, the bill would serve to codify commitments to safety that some companies have already made. This is important because AI companies face strong commercial incentives to go back on their commitments to public safety. For example, OpenAI transitioned from a non-profit to a for-profit, and Microsoft and Facebook fired their responsible AI teams. The bill has important limitations: Compliance with the law would be self-certified by companies, and it would only apply to AI developers in California. But it represents a significant step towards legal liability for AI developers that cause catastrophic harm.

For more information, check out the bill’s text, ABC News, or this blog post.

Precedents for AI Policy: Healthcare and Biosecurity

AI presents unique challenges, but policymakers can draw lessons for AI governance from other regulatory regimes. Here, we cover two case studies of regulatory systems in the US: the FDA’s regulation of medical devices and the CDC’s regulation of research involving dangerous biological agents.

FDA-style oversight. In December, the Ada Lovelace Institute published the discussion paper Safe Before Sale, which examines the lessons AI regulators can learn from the FDA.

The paper notes the impressive technical expertise of the FDA. The agency employs around 18,000 people with its $8 billion budget, and these staff are responsible for reviewing the design and results of clinical research trials.

Continuous engagement occurs between the developer and the FDA throughout the development process. The FDA has far-reaching powers to solicit and access information from the developers it oversees, both before and after a product has been put on the market.

Risk levels of Class I, II, or III are assigned to each medical device by the FDA, and riskier devices are subject to higher standards of scrutiny. The burden of proof is not on the FDA to demonstrate the risks of a product, but rather on developers to prove their product is both safe and effective. There are also significant concerns about the FDA’s slow and expensive approval processes.

The FDA offers a clear example of a centralized regulatory regime, where new products are reviewed by the government’s technical experts before being sold on the public market. Similar proposals have been made by US Senators Elizabeth Warren, Lindsay Graham, Michael Bennett, and Peter Welch for the governance of AI and other digital platforms.

FSAP-style oversight. Another paper draws lessons for AI regulation from the CDC’s Federal Select Agent Program (FSAP). FSAP regulates work involving dangerous pathogens and toxins. Entities performing such research must obtain licenses from the CDC (for human pathogens) and follow biosecurity and biosafety regulations.

Similarly to the FDA, FSAP regulates even the earliest stages of the use of hazardous biological agents. FSAP employs many technical experts in biosecurity, suggesting that governments can effectively regulate complex technical domains.

The paper points out significant actions taken by FSAP. As of 2016, it had suspended or revoked licenses for at least 10 entities (out of a total of around 300) for violating federal standards. This suggests that risky behavior involving dangerous pathogens would have occurred by default.

One drawback of FSAP’s approach is their reliance on “checklists” and other formulaic regulations. They regulate a specified list of toxins and pathogens already known to be dangerous, but they do not attempt to anticipate new risks from new pathogens. A GAO report found that “the program’s reviews may not target the highest-risk activities, in part because it has not formally assessed which activities pose the highest risk.” The paper recommends that AI regulators go further than FSAP and adopt strong “risk based” and anticipatory regulation.

Enforcing the EU AI Act

The European Union has passed its AI Act. We summarized this landmark piece of legislation here; for a more detailed summary, see this paper.

The EU plans to hire 80 people to work on enforcing the Act at their new AI Office. The office will be responsible for enforcing requirements on high risk AI systems (including general purpose AI systems). Systemic risks from large general-purpose AI systems are among its concerns, although it will also work on a range of other EU AI policies. Further details on its planned activities are available here.

Links

We have a lot of links from the last few weeks, so we’ve organized them by topic.

New Models

Google announced Gemini 1.5. Its context window is 10 million tokens long, meaning that it can read many books or watch dozens of hours of video within a single conversation.
On the same day as the Gemini release, OpenAI released Sora, which can generate videos from text prompts.
OpenAI is reportedly investing in AI agents that would perform tasks through a user’s computer.

News from OpenAI

OpenAI CEO Sam Altman is seeking up to $7 trillion to build semiconductor manufacturing capacity in the United Arab Emirates. This undermines OpenAI’s previously stated safety strategy of spending lots on compute today so that, as AI systems become more capable, there would be no “compute overhang” and growth in AI capabilities would be slowed by limited compute availability.
Although Sam Altman told Congress that he has no equity in OpenAI, he does own OpenAI’s venture capital fund.

Cybersecurity

State actors affiliated with Russia, China, Iran, and North Korea were using OpenAI’s models before OpenAI detected them and terminated their accounts. See here for more detail.
GPT-4 is capable of hacking some public websites without human oversight and assistance.
A UK government report finds that AI will likely increase the volume and impact of cyberattacks.

International Politics

The UK Labour Party would aim to strengthen Conservative PM Rishi Sunak’s voluntary agreements with AI companies into legal requirements.
Ukrainian President Zelensky announces a new branch of the military focused on unmanned systems.

US Politics

The US AI Safety Institute will be led by Elizabeth Kelly, formerly a top economic adviser to the White House.
The US House of Representatives launches a bipartisan task force on AI.
Donald Trump says AI “is maybe the most dangerous thing out there.”
The FTC launches an inquiry into six prominent AI companies.
The White House released a 90-day update on their Executive Order on AI.
NIST begins a collaboration on reducing risks from synthetic biology enabled by AI.
The Center for AI Safety joins other organizations as part of NIST’s Consortium on AI Safety.
The US proposes Know Your Customer (KYC) requirements for cloud compute providers.
The White House’s top science advisor says the US will work together with China on AI.
Republicans push back against the White House Executive Order on AI.
Polling suggests that US voters prefer candidates who support regulation of AI.

Two interesting papers

A new paper provides a detailed discussion of the role of compute in AI governance.
An economics paper models the cost of automating tasks with AI, and finds that the costs of implementing AI within businesses will slow widespread adoption.

Opportunities

The National AI Research Resource, a US public computing cluster with tens of thousands of GPUs, is now accepting applications for compute access.
NIST requests comments on “Dual Use Foundation Artificial Intelligence Models with Widely Available Model Weights.”
The UK AI Safety Institute is hiring across a number of roles.
For mid-career ML researchers and engineers interested in working on AI safety, the non-profit Arkose is offering career advising calls.
YCombinator requests startups working on explainable AI.

Other

Elon Musk says Neuralink has successfully implanted a brain-computer interface in a human.