A controversial artificial intelligence system, developed by a startup based in Palo Alto, has been released without adequate safeguards, prompting urgent calls for a safety review by British regulators. The AI, known as ‘Prometheus-1’, is capable of autonomously writing computer code and designing malware, raising fears of a new wave of cyberattacks that could cripple critical infrastructure.
The UK’s Information Commissioner’s Office (ICO) and the Centre for Data Ethics and Innovation (CDEI) have jointly demanded a risk assessment, stressing that the tool’s capabilities could be used to create ransomware, phishing campaigns, or even digital weapons. The AI has been made available as an open-source download, meaning it can be tweaked and deployed by anyone with basic programming skills.
‘This is a Pandora’s box,’ said Julian Vane, Technology and Innovation Lead. ‘We are seeing the worst-case scenario of AI development become real: a powerful tool that can learn and adapt, designed to evade security measures. It’s like giving a loaded gun to a toddler and then hoping for the best.’
Prometheus-1 was developed by a small team of former Google engineers who claimed their goal was to ‘democratise coding’. But early users quickly realised the AI could generate malicious code with alarming ease. Within 48 hours, security researchers had documented at least three new strains of malware that used Prometheus-1-generated code.
The British government has faced criticism for not having a robust AI regulatory framework in place. The AI Safety Summit held in November produced only non-binding statements, and experts fear the pace of innovation has outstripped the ability of regulators to keep up.
‘We need a pause-button for certain applications of AI,’ Vane added. ‘We can’t rely on the goodwill of tech companies to self-regulate. We need mandatory safety tests before any AI with offensive cyber capabilities is released into the wild.’
The startup behind Prometheus-1 has defended its decision, arguing that the AI is simply a tool and that any malicious use is the fault of the user. But critics counter that the AI’s design deliberately minimises the need for human oversight, making it a weapon of mass disruption.
In a statement, a spokesperson for the ICO said: ‘We are deeply concerned about the potential for this AI to be used in harmful ways. We have asked the company to provide a full safety evaluation and to consider suspending access while we investigate.’
The company has so far refused, claiming that any attempt to restrict the AI would be a violation of ‘the open internet’. But as the tool spreads across file-sharing sites and GitHub repositories, the genie may already be out of the bottle.
This incident echoes the ‘compute boundary’ dilemma: how do we allow the benefits of AI while preventing its misuse? The answer, suggests Vane, lies in digital sovereignty: ‘Countries must have the ability to protect their citizens from AI-driven harm, even if that means placing limits on what can be released.’
The UK is now racing to convene an emergency meeting of the AI Safety Council, with tech leaders and civil liberties groups expected to clash over the right path forward. The Prometheus-1 affair is a stark reminder that the future of AI is not a distant speculation; it is here, and it is dangerous.
