The California Appropriations Committee passed the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act (also known as SB-1047) on Thursday, the latest move in the saga of regulating Silicon Valley.

The state Assembly and Senate must still approve the bill before it becomes law.

What is SB-1047?

Known colloquially as California’s AI Act, and monitored closely across the country for possibly setting a precedent for state rules around generative AI, SB-1047 sets out several rules for AI developers:

• Create safety and security protocols for covered AI models.
• Ensure such models could be shut down completely.
• Prevent the distribution of models capable of what the act defines as “critical harm.”
• Retain an auditor to ensure compliance with the act.

In short, the bill provides a framework that prevents generative AI models from causing large-scale damage to humanity, such as through nuclear war or bioweapons, or from causing over $500 million in losses through a cybersecurity event.

The act defines “covered models” as those using computing power greater than 10^26 integer or floating-point operations — the cost of which exceeds $100 million, during training.

Recent version of the act includes input from Anthropic

The version of the bill that passed on Thursday included some changes suggested by AI maker Anthropic and accepted by primary bill author Sen. Scott Wiener, D-Calif.

Anthropic successfully asked the state to remove language from the bill, saying companies in breach of the act could see legal action from the state’s attorney general. The newest version removes the need for companies to disclose safety test results under threat of perjury. Instead, the developers will need to submit statements, which do not have the same legal weight.

Other changes include:

• The shift in wording from AI companies having to provide “reasonable assurance” of safety to “reasonable care.”
• An exception in which AI researchers who spend less than $10 million fine-tuning an open-source covered model are not considered developers of that model.

SEE: Anthropic and OpenAI have done their own digging into how generative AI creates content, including biased content.

The bill no longer calls for the creation of a Frontier Model Division, an agency to oversee the AI industry. Instead, a Board of Frontier Models focused on forward-looking safety guidance and audits will sit within the current Government Operations Agency.

While Anthropic contributed to the bill, other large organizations like Google and Meta have expressed disapproval. Andreessen Horowitz, a venture capital firm known as a16z that is behind many AI startups, has loudly opposed SB-1047.

Why is SB-1047 controversial?

Some representatives of industry and Congres say the act will restrict innovation and make it particularly difficult to work with open-source AI models. Among the bill’s critics was Hugging Face co-founder and CEO Clement Delangue, as noted by Fast Company.

An April study by the pro-regulation think tank Artificial Intelligence Policy Institute found that most Californians voted to support the bill as it stood at the time, with 70% agreeing “future powerful AI models may be used for dangerous purposes.”

Researchers Geoffrey Hinton and Yoshua Bengio, known as the “godfathers of AI” for their pioneering work on deep learning, publicly support the bill as well. The act will “protect the public,” Bengio wrote in an op-ed in Fortune on Aug. 15.

Eight of the 52 Congressional members from California signed a letter on Thursday saying the act would “create unnecessary risks for California’s economy with very little public safety benefit.” They argue that it’s too early to create standardized evaluations for AI, as government agencies like NIST are still working on creating those standards.

They suggest the definition of critical harm might be misleading, saying the bill has gone astray by focusing on large-scale disasters, such as nuclear weapons, while “largely ignoring demonstrable AI risks like misinformation, discrimination, nonconsensual deepfakes, environmental impacts, and workforce displacement.”

SB-1047 includes specific protection for AI company whistleblowers under the California Whistleblower Protection Act.

The act brings up the challenges of balancing regulation and innovation

“We can advance both innovation and safety; the two are not mutually exclusive,” Wiener wrote in a public statement on Aug. 15. “While the amendments do not reflect 100% of the changes requested by Anthropic — a world leader on both innovation and safety — we accepted a number of very reasonable amendments proposed, and I believe we’ve addressed the core concerns expressed by Anthropic and many others in the industry.”

He noted that Congress is “gridlocked” on AI regulation, so “California must act to get ahead of the foreseeable risks presented by rapidly advancing AI while also fostering innovation.”

Next, the bill will need to be passed through the Assembly and Senate. If approved, the bill will then be considered by Gov. Gavin Newsom, likely in late August.