The British government has escalated its oversight of frontier artificial intelligence after pulling in Dario Amodei, co-founder of the AI firm Anthropic, for an emergency meeting at the Department for Science, Innovation and Technology. The summons follows Amodei’s stark public warning that deploying powerful AI systems without meaningful human oversight risks “catastrophic outcomes” comparable to a nuclear accident.
Sources close to the meeting say Tech Minister Peter Kyle pressed Amodei on whether Anthropic’s own models, including Claude 3 Opus, meet the threshold of safe “human-in-the-loop” control. The concern is not hypothetical. Last month, a major London hospital trust inadvertently used an unvetted large language model to triage patient symptoms, flagging a case of sepsis as “low priority.” The error was caught by a human nurse, but the incident sent a chill through Whitehall.
Amodei’s argument is nuanced. He does not oppose AI development but insists that systems capable of autonomous goal-setting or manipulation must be constrained by what he calls “situational awareness” controls. This means the AI knows it is being monitored and can be overridden at any point. He compared it to a self-driving car’s “driver monitoring” system: the car can steer, but a human must keep hands on the wheel.
The summons signals a shift in London’s approach. Until now, the UK’s AI Safety Summit rhetoric has favoured innovation over restraint. But the rapid deployment of generative AI in public services, from council benefit assessments to police facial recognition, has alarmed civil liberties groups. The Treasury is also worried: a report last week estimated that unstacked AI errors in financial services could cost the economy £12 billion annually by 2027.
Critics argue the meeting is too little, too late. “Summoning one CEO while thousands of poorly tested AI tools are already live is like fumigating a single room while the house burns,” said Dr. Zara Khan, a researcher at the Alan Turing Institute. She points out that many government AI vendors are not even signatories to the new voluntary AI safety pledge.
Inside the room, Amodei reportedly challenged the Minister to accelerate the proposed AI Bill, which has been languishing in pre-legislative scrutiny. Without statutory obligations, he argued, firms have no incentive to build “circuit breakers” into their models. The government is believed to be sympathetic but wary of spooking the investment community, especially after Microsoft’s recent £2.5 billion data centre pledge in South Wales.
The broader context is the global race for AI dominance. The US and China are moving faster, but the UK is positioning itself as the “safe harbour” for responsible AI. This summons tests that brand. If the government can show it will hold even the brightest founders accountable, it might gain trust. If it backs down, the fear of a Black Mirror-style loss of human agency will only deepen.
For now, the message from Whitehall is clear: the era of asking nicely is over. The next stage of AI policy will be defined not by pleadings but by power. And the first test of that power is whether Anthropic’s warnings lead to code changes, not just press releases.
