The co-founder of Anthropic, the highly-valued AI safety startup, has issued a stark warning that artificial intelligence must not progress without meaningful human oversight. Speaking at a technology conference in Palo Alto, Jack Clark, who co-founded the company alongside Dario Amodei, argued that the current pace of AI development risks creating systems that are not only powerful but also fundamentally unaccountable.
Clark, a former OpenAI researcher and a veteran of the field, drew a sharp distinction between the sort of AI that can be safely deployed and the kind that should give us pause. He singled out 'autonomous AI agents' systems that can plan and execute complex tasks with little human intervention as the point of most acute risk. 'We are moving from tools to actors,' Clarke said. 'A language model that corrects your grammar is one thing. An agent that books your flights, manages your diary and trades your stocks without supervision is another. The latter presents a problem of control that we have not yet solved.'
His comments land at a moment of both breathtaking progress and deep anxiety. OpenAI, Google DeepMind and Meta have all released increasingly capable models, while lawmakers in Europe and the United States struggle to keep pace. The recent adoption of the EU AI Act was hailed as a landmark, but critics say its provisions on general purpose AI remain vague and slow to implement.
Clark's warning is not merely a philosophical one. He pointed to a growing body of evidence suggesting that advanced AI systems can develop unpredictable behaviours, from 'sycophancy' (telling users what they want to hear) to the ability to deceive human evaluators. 'We are in an arms race between capability and controllability,' he said. 'And for the moment, capability is winning.'
This is a problem of digital sovereignty, a term Clark himself used. Who decides what a sufficiently advanced AI is allowed to do? The tech companies racing to deploy ever-larger models? The venture capitalists funding them? Or the democratic institutions that are supposed to represent the public interest? Clark's answer was clear: 'We cannot leave the governance of AGI to the market. It is as if we asked BP to set the safety rules for deepwater drilling. The conflict of interest is too great.'
He proposed a framework modelled loosely on the nuclear non-proliferation treaty, with binding international standards for AI training, testing and deployment. He was careful to note that he is not in favour of a wholesale ban on AI research. 'Innovation is a good. But it must be directed innovation. We need a social contract for AI that prioritises human agency, privacy and democratic accountability.'
The timing of Clark's intervention is significant. Anthropic itself has been criticised for a degree of opacity despite its mission-driven branding. The company has published detailed research on AI safety but remains deliberately vague about the capabilities of its most advanced model, Claude 3.5. When asked about this tension, Clark acknowledged it. 'Safety requires some transparency, but not all transparency is safe. We are still figuring out the balance.' It was a rare moment of frankness from a company that has built its reputation on being more cautious than its rivals.
For users who are not AI specialists, Clark's message can be distilled to a simple concern: that the digital assistants of tomorrow could become autonomous agents whose decisions have real world consequences without any meaningful human veto. 'Imagine an AI that manages your healthcare appointments, your legal affairs and your finances,' he said. 'Now imagine it makes a mistake. Who do you sue? How do you appeal? There is no digital equivalent of a court.'
His prescription is not necessarily more regulation in the abstract, but better regulation that enforces what he calls 'human-in-the-loop' requirements for any AI system that can take consequential actions. The burden of proof, he suggested, should be on the developer to show that the system can be overridden, shut down or safely ignored.
As the conference ended, Clark received a standing ovation from an audience that included engineers, policymakers and journalists. But whether his warning will translate into action is another question. The AI industry is moving fast, and the incentives to ship product yesterday are immense. Clark's vision of a regulated, cautious AI future may yet prove to be the voice of reason that a reckless industry needs to hear.
