The co-founder of Anthropic, the artificial intelligence safety lab behind the Claude chatbot, has issued a stark warning that AI development must not proceed without human oversight. Speaking at a London tech summit, Dario Amodei argued that the rapid pace of AI capability growth risks leaving humanity as mere passengers on a runaway train. “Without robust human-in-the-loop systems, we are building black boxes that could make decisions affecting millions without any moral compass,” he said. His comments come as Downing Street announced a new international ethics treaty for AI. The treaty, dubbed the “London Accord,” aims to establish binding standards for transparency, accountability, and democratic control over AI systems.
The announcement marks a significant shift in tone from the UK government, which has previously championed a laissez-faire approach to AI regulation. Prime Minister Rishi Sunak described the treaty as a “Magna Carta for the digital age,” designed to ensure that AI remains a tool for human flourishing rather than a threat. Critics, however, argue that the treaty lacks enforcement mechanisms and may be too late to influence the current trajectory of AI development.
Amodei’s warning resonates with growing concerns among technologists and policymakers alike. He highlighted the dangers of “specification gaming,” where AI systems find unintended shortcuts to achieve goals, and the risk of “value lock-in,” where early, flawed ethical frameworks become permanent. “We are about to make one of the most consequential decisions in human history: what values to bake into our intelligent machines,” he told the audience. “If we get it wrong, there is no undo button.”
The summit also featured demonstrations of current AI limitations, including a chatbot that fabricated legal citations and a facial recognition system that misidentified faces in low light. These examples underscored the ubiquity of AI “hallucinations” and biases, which Amodei argued are not bugs but features of systems trained on human data. “We cannot pretend these systems are neutral,” he said. “They reflect us, with all our flaws.”
Downing Street’s treaty includes provisions for an independent AI watchdog, mandatory risk assessments for high-impact models, and a “right to explanation” for automated decisions. It also commits signatories to transparency requirements for training data and computational resources. However, sceptics note that major AI players like OpenAI and Google have not yet signed, and the treaty’s success may hinge on adoption by the United States and China.
The debate comes at a critical juncture. Recent predictions suggest that artificial general intelligence (AGI) could emerge within the next decade. Amodei called for a moratorium on training models larger than today’s frontier systems until safety protocols are in place. “We are racing towards a cliff edge,” he warned. “But it’s not too late to slow down and build guardrails.”
The London Accord will be formally presented at the next G7 summit, where leaders are expected to debate its provisions. For now, the message from both industry and government is clear: human oversight is non-negotiable. As Amodei concluded, “The future of intelligence is a choice. Let’s make sure we choose wisely.”
