The co-founder of Anthropic, a leading AI safety company, has issued a stark warning: artificial intelligence must not be allowed to develop without robust human oversight. Speaking at a tech ethics conference in London, the executive argued that the rapid pace of AI advancement risks creating systems that are not only unpredictable but potentially uncontrollable.
“We are building entities that could surpass human intelligence within a decade,” he said. “Without guardrails, we are sleepwalking into a future where machines make decisions that affect our lives with no accountability.” His call echoes a growing unease among Silicon Valley insiders, many of whom have witnessed the fallout from rushed deployments of large language models and generative AI tools.
The warning comes as regulators worldwide scramble to draft frameworks for AI governance. The European Union’s AI Act is still in committee, while the US lags behind with a patchwork of state laws. In the UK, the government has favoured a “pro-innovation” approach, but critics argue that puts profit before safety.
Anthropic’s co-founder stressed that oversight does not mean stifling innovation. “We need a new social contract with technology. Think of it as digital sovereignty: the right of citizens to understand and control the algorithms that shape their lives.” He proposed a tiered licensing system for AI models, similar to how drug trials work. “Before you deploy a powerful AI, you must demonstrate it is safe, transparent, and aligned with human values.”
The speech landed amid fresh concerns over AI-generated disinformation and the potential for job displacement. Last month, a deepfake audio of a politician went viral, nearly causing a diplomatic incident. “We laughed at fake cat videos,” the co-founder added. “Now we face synthetic realities that can sway elections.”
Some in the tech community push back, arguing that self-regulation is sufficient. But Anthropic’s co-founder dismissed that as naïve. “History shows that voluntary standards fail when profit margins are at stake. The asbestos industry, the tobacco industry, even the early internet privacy promises. We must learn from those mistakes.”
He also warned of the “Black Mirror” outcomes if AI is left unchecked: autonomous weapons, mass surveillance, and the erosion of truth. “Each new algorithm brings a moral question. The user experience of society is at stake. Do we want a world where machines decide who gets a loan, who gets parole, or who stays in a hospital bed? We must embed ethics into the code.”
The call has received mixed reactions. Some EU lawmakers welcomed it, noting that Anthropic was founded by former OpenAI employees who left due to safety concerns. Others, like a spokesperson for the Information Technology and Innovation Foundation, said the warning was “alarmist” and could slow beneficial progress in healthcare and climate science.
But for the Anthropic co-founder, the risks outweigh the rewards. “Quantum computing and AI are converging. We are approaching a point where we can no longer predict what these systems will do. Human oversight is not a brake. It is the steering wheel.”
His final words to the room: “We have one chance to get this right. Let us not squander it on a rush to market. Build with care. Build with oversight. Build for humanity.”
