Anthropic wants governments to have the power to block dangerous AI deployments

Anthropic has published two policy frameworks calling for governments to have the legal authority to block dangerous AI deployments and for economic safeguards to protect workers as AI reshapes the labour market. The Advanced AI Framework covers safety regulation. The Economic Policy Framework addresses displacement, capital distribution, and the social safety net.

The safety framework is the more aggressive of the two. Anthropic wants governments to be able to block or deter the deployment of models that pose a “significant risk of catastrophic harm,” with civil penalties tied to global annual revenue that escalate with repeated violations. It would apply only to models trained using more than 10²⁵ floating-point operations, developed by companies earning more than $500 million in AI revenue or spending more than $1 billion on AI R&D.

That threshold captures a small number of companies: Anthropic itself, OpenAI, Google DeepMind, xAI, and potentially Meta. Anthropic is calling for regulation that would directly apply to its own business, a positioning it has used before to distinguish itself from competitors that lobby against oversight.

The framework identifies four categories of catastrophic risk: biological weapons development, large-scale cyber vulnerability discovery, loss of control over autonomous systems, and AI that automates its own R&D. Anthropic pointed to its own experience with Mythos Preview, which discovered thousands of high-severity vulnerabilities in every major operating system and browser, as evidence that these risks are not hypothetical.

The 💜 of EU tech

The latest rumblings from the EU tech scene, a story from our wise ol' founder Boris, and some questionable AI art. It's free, every week, in your inbox. Sign up now!

Frontier developers would be required to test their models, publish summaries of the results, submit to independent evaluation, maintain security programmes, and publish regular risk reports. Anthropic says transparency alone is no longer sufficient. “AI capabilities are going to improve rapidly over the coming months,” the company wrote. “Their governance needs to keep pace.”

On preemption, Anthropic takes a direct shot at the White House’s push to block state AI laws. “We do not believe Congress should preempt state law unless it enacts a federal law that is at least as strong as the framework we are proposing today,” the company said. It calls for preemption to be “surgical,” allowing states to regulate on child safety, consumer protection, and other issues outside the scope of a federal safety law.

The economic framework is less detailed but addresses the displacement question that graduating students have been booing about at commencement ceremonies. It includes proposals for capital accounts, wage insurance, tax incentives, and an expanded social safety net. The goal is to ensure AI’s financial benefits are “broadly shared,” though specifics on funding and implementation are left for future debate.

The timing is deliberate. Anthropic is fighting the Pentagon in court over its blacklisting as a supply chain risk. It is preparing for an IPO. And it is watching Congress negotiate a deal that would trade state AI preemption for online safety legislation. By publishing its own framework, Anthropic is trying to set the terms of a regulatory debate that is moving fast and largely without its input.