In a stark warning that cuts to the heart of Silicon Valley’s power dynamics, the head of UK intelligence has declared that frontier AI firms such as Anthropic are poised to become ‘unaccountable sovereigns’ unless governments act now. GCHQ director Sir Jeremy Fleming, speaking at a cybersecurity forum in London, drew a direct line between the breakneck race to deploy generative models and a future where a handful of private corporations wield more influence over global affairs than elected democracies.
‘We are sleepwalking into a world where the architects of these systems operate beyond the reach of any single state,’ Fleming cautioned. ‘Their algorithms already shape our information diets, our economic decisions, and soon our physical security. Yet they answer to shareholders, not citizens.’
This is not hyperbole. Consider Anthropic’s constitutional AI approach, where models are trained to internalise ethical guidelines set by the company. Or OpenAI’s recent restructure, which swapped a non-profit’s mission for a capped-profit model that still leaves Sam Altman with ultimate say over ChatGPT’s behaviour. These are private entities writing the unwritten rules of our digital civilisation.
The irony is painful. We in the West have spent two decades lecturing China about state-controlled cyberspace. Now we are handing our own digital sovereignty to a handful of Californian labs. Fleming’s timeline aligns with the Cambridge Analytica scandal but on steroids. Back then, data misuse exploited existing social platforms. Today, an AI could synthesise personalised propaganda on the fly, micro-targeting every voter without a single human editor.
But the intelligence chief’s deeper concern is physical. As AI permeates critical infrastructure, from energy grids to autonomous weapons, the ability to trigger large-scale disruption becomes a private capability. ‘A rogue model, or one captured by adversarial actors, could treat a nation’s infrastructure like a game,’ Fleming said. ‘Who would we call to shut it down? The CEO? A boardroom in San Francisco?’
He is not alone in this anxiety. The UK’s proposed AI Safety Institute is designed to test models before release, but its powers remain advisory. Meanwhile, the EU’s AI Act imposes obligations on ‘high risk’ systems, yet exempts general-purpose models until 2026. By then, these sovereigns will have consolidated power.
What can be done? Fleming hinted at a ‘digital Magna Carta’, a framework of rights and constraints that binds AI developers just as constitutional law binds monarchs. This would include mandatory transparency, red-team testing for catastrophic harms, and a licensing regime that makes deployment conditional on democratic oversight. Think of it as the Kyoto Protocol for intelligence, a treaty to prevent an AI arms race.
But this requires politicians who understand backpropagation from bootstrap aggregation. Currently, our leaders can barely regulate social media. Asking them to oversee recursive self-improvement is like asking a medieval serf to audit a fusion reactor.
And here’s the rub: the companies themselves acknowledge the risks. Anthropic’s CEO Dario Amodei has called for regulation. Sam Altman toured Congress. But their proposals always stop short of independent control. Why? Because genuine sovereignty transfers would crater their market value. A regulated AI is a slower AI, and investors demand speed.
The British approach, with its emphasis on testing at the frontier, is a start. But it treats symptoms, not the disease. The disease is that we have allowed private entities to build the cognitive infrastructure of our future without public oversight. It is as if we gave a corporation the right to build all the roads and then decide who can travel where.
Fleming’s warning is a call to action. The window to assert democratic control over AI is closing. Every day we delay, another sovereign rises. And unlike the kings of old, these sovereigns don’t need armies. They need servers and data centres. And they are already here.
