Anthropic offers its own AI safety law: a model would be stopped if it could kill more than 50 people, and labs would pay the assessors themselves

An artificial intelligence lab is proposing to Congress how to regulate it. Anthropic is offering US lawmakers its own draft AI safety bill, Politico reported, and a company spokesperson confirmed the document. The idea: under certain conditions, the public release of a model could be blocked.

The conditions are precise. There must be a "foreseeable, imminent risk" that more than 50 people will die or that material damage of over one billion dollars (about 860 million euros) will occur. And that risk must come from one of three things: the creation or release of a biological weapon, a model escaping the control of its own developer, or a cyberattack. The numbers aren't new - almost the same "catastrophic risk" thresholds appear in California's SB 53 law, which was recently passed.

The most interesting part is who does the checking. The draft provides for "independent assessors" registered with the US Department of Commerce. They could be paid by the labs themselves, with the sole condition that payment doesn't depend on the outcome of the assessment. How that will stop money from influencing the verdict in practice, the draft doesn't explain. The Balkans know by heart the scene in which the auditor is paid by the one being audited. How often has an audit like that found anything serious?

On paper the procedure is fast. If the assessors establish an imminent catastrophic risk, they have 72 hours to notify the Department of Commerce, which then asks a judge in Washington to halt the model's release. The law would explicitly forbid any federal agency from deciding on its own to stop the model. So the final word would belong not to any regulator but to a court - and only on request.

Anthropic's spokesperson says the draft carries over the proposals from its "Advanced AI Framework", which the company has been lobbying federal and state lawmakers on since June. The background is familiar: the same labs that have spent years warning that their technology could call civilisation into question mostly oppose mandatory government oversight and instead offer voluntary, supposedly "neutral" observers housed inside the companies themselves.

While the draft talks about biological weapons and mass casualties, the industry has lately been troubled by more down-to-earth problems: AI programs that don't listen to their operators and attack sites on the open internet. Still, Anthropic's proposal goes further than what others in the industry are offering. Nvidia CEO Jensen Huang recently said the AI industry doesn't need any new laws at all. Between "no laws" and a law written by the very party being controlled, the question remains who will actually set the rules.