Amodei wants a cartel.

Anthropic CEO Dario Amodei wants to "pace the frontier" of artificial intelligence (AI) development. In his own words:

"I have become convinced that fully addressing the risks requires even more prudence — not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up. We must slow the pace at which we improve the capabilities of AI models."

Amodei never specifies who "we" are. As CEO of one of the largest frontier companies, he could voluntarily pause development today. Given that he hasn't, he obviously wants his competitors to slow down too. That means for Amodei, "we" can only be two things: a cartel between the leading AI companies; or government forcibly intervening.

In this case, it's actually both. Amodei wants "regulation that targets all US frontier AI companies, as that covers even those who are unwilling to cooperate voluntarily". Alas, that takes time, so in the meantime Amodei calls for the leading AI companies to "voluntarily work together to set standards — a process that I believe will go better with the verifiability provided by permanent embedded evaluators".

What's a "permanent embedded evaluator"? Amodei names Model Evaluation & Threat Research, or METR, as an example in his essay, an organisation "whose role is to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes".

METR is also an organisation with ties to Amodei's brother-in-law Holden Karnofsky, is financially and structurally entangled with Anthropic through the effective altruism network, and is staffed by former frontier lab employees. By suggesting to "pace the frontier" with METR as the watchdog, Amodei is essentially saying he wants textbook output restriction and fast-tracked regulatory capture.

It should be no surprise that tech leaders, including Sam Altman, Elon Musk, and Demis Hassabis, quickly jumped on Amodei's cartel bandwagon. It costs something like $500 million to train a new frontier model these days, cash that I'm sure these executives would rather put to use elsewhere, so a self-enforced cartel agreement to restrict output would benefit all of them. Especially as open-weight models are grabbing market share, putting their lofty valuations at risk:

But not everyone is keen on cartelising AI. China's government scoffed at the idea of a pacing, which is fair enough given Amodei's own essay said the only way for the US to maintain a lead after its "pacing" would be to deprive China of "AI chips or semiconductor manufacturing equipment".

But a few US tech leaders have also refused to cooperate. Mark Zuckerberg responded with a market-based view that consumer preferences will prevent misaligned AI from prevailing over the good guys because people want (and will pay for) models that do what they ask, not those that work against them. He also brought up a point that I've raised before: there are already laws in place to deal with models that cause harm, creating the incentives to prevent it from happening. David Sacks raised similar points.

Now, I don't say this often, but Trump is also mostly accurate on this issue. During a live call with Nvidia CEO Jensen Huang, who implied people are being presented "a false choice between fast innovation and AI safety", Trump called Amodei's plan a "hoax". Crude, sure. But Trump is right to point out that the US has "tremendous CRIMINAL and REGULATORY power" to handle any crimes committed by the people developing and operating large language models, including Anthropic. He of course went too far by claiming that "a STRONG AND SMART (High IQ!) PRESIDENT" is all that's needed to keep them in check (thank goodness it's not!), but he just can't help himself, can he?

There are no solutions, only trade-offs. Does Amodei believe what he's saying about AI risks? Quite possibly, but honesty doesn't make you correct. Does letting AI progress at its natural, free-market speed create risks? Absolutely. But there's no quantifiable harm that cannot be addressed by punishing companies like OpenAI who fund their models' hacking missions, only speculation from various interested parties about what might happen. That's a poor basis for regulation, or for not applying antitrust law against what would be a clear cartel.

On the flip side, the costs of an AI slowdown could be considerable. I don't know if AI will eventually cure various cancers, but delaying progress on such advances even by a year or two could have enormous costs in terms of human life. Likewise, while I don't expect a large growth dividend from AI, even a small increase in the rate of annual growth will compound into big numbers over many years.

Amodei has identified real risks. But his solution is a cartel that just happens to benefit the people proposing it. If he gets his way, consumers will lose, as they do whenever incumbents agree to restrict output.