In a stark address delivered from a London tech summit, Dario Amodei, co-founder of AI safety company Anthropic, issued a grave warning: artificial intelligence must not be allowed to evolve without human oversight. Speaking to a packed auditorium of policymakers, engineers, and ethicists, Amodei framed the moment as a “red line” for civilisation, drawing parallels to the early days of nuclear fission before the advent of non-proliferation treaties.
“We are building machines that will surpass human reasoning within a decade,” he said, his voice measured but urgent. “If we do not embed control, transparency, and accountability now, we will lose the ability to do so later.” Anthropic, which spun out of OpenAI in 2021, has consistently advocated for a “constitutional” approach to AI alignment, where models are trained to follow ethical guardrails from the outset.
Amodei’s intervention comes weeks after the UK government launched its AI Safety Institute, and days after the EU passed the world’s first comprehensive AI Act. Yet he argued that legislation is moving too slowly. “Regulation is not the enemy of innovation. Uncontrolled AI is,” he said. “We need a global watchdog, akin to the IAEA, but for algorithms.”
The speech landed hard on an industry still reeling from the recent ousting and reinstatement of Sam Altman at OpenAI, and the subsequent boardroom battles over safety versus speed. Amodei pointed to that chaos as evidence of the need for structural safeguards: “When the fate of a multi-trillion-dollar technology hinges on a boardroom vote, you have a governance problem.”
Critics argue that calls for regulation are a power grab by incumbents, but Amodei dismissed that. “We are not proposing a moratorium. We are proposing a seatbelt.” He outlined three concrete demands: mandatory audit trails for all models above a certain compute threshold, independent red-teaming for high-risk applications, and a binding international treaty that ties AI development to human rights frameworks.
The reaction was swift. Tech executives in the audience exchanged wary glances. Some whispered that Anthropic’s own Claude model is already being deployed in healthcare and legal settings; why should they be subjected to rules their rivals might evade? Amodei anticipated that criticism. “The race to the bottom helps no one. The future is not a zero-sum game. Either we all become safe, or we all become sorry.”
Outside the venue, protestors held signs reading “Our future is not an algorithm” and “AI: Think before you code”. Inside, the mood was tense but focused. The EU’s Thierry Breton, who attended via video link, praised Amodei’s remarks as “timely and necessary” and pledged to accelerate enforcement mechanisms.
But the clock is ticking. Amodei cited research showing that AI systems are already beginning to exhibit “emergent” capabilities not explicitly programmed, from negotiating in games to creating novel protein structures. “We do not fully understand how these models think,” he admitted. “That is precisely why we need human control.”
He concluded with a warning that resonated beyond the hall: “Every unchecked line of code today is a potential black box of tomorrow. We do not have the luxury of learning from our mistakes after the machine has already made them.” As he stepped down to a mix of applause and silence, the message was clear: the AI genie is already out of the bottle. The only question left is who holds the lamp.
