At least a 10% probability that humans lose control of AI in the next 10 years, say US national security officials
A survey of national security officials from the US Department of War and others shared their views on the risks and opportunities of artificial intelligence – plus, reports that incidents of rogue AI doubled between June and July
Current and former national security officials in the US believe there is at least a 10% chance that humans lose control of artificial intelligence within the next 10 years.
The Institute for Security and Technology (IST) think tank surveyed 111 national security experts between 30 April and 15 July on the opportunities and risks of advanced AI deployment in military, cybersecurity, intelligence, and chemical, biological, radiological, and nuclear defence (CBRN).
Its AI and the Future of National Security report – released on 2 September – found that 87% of respondents put the probability that AI operates outside human control within ten years at 10% or higher. In addition, when asked about AI operating beyond human control with no way to regain it, 33% expect that scenario to occur within 10 years.
Nearly half of the respondents (47%) either currently work at the US Department of War (formerly the Department of Defense), or used to, while a significant proportion of other respondents have experience working at the Department of State, the Executive Office of the President, and at US intelligence organisations such as the CIA.
AdvanceGov brings together digital pioneers from US state and federal government, and international digital leaders, to help drive transformation of state, regional, and local governments in the US and beyond. The conference will explore how public services can be made faster, more secure, and user-centered – and how governments are putting artificial intelligence to work. Register now: AdvanceGOV | November 3-4 | Sacramento, California
Among the key opportunities, a third of respondents expect AI to reduce the time it takes to answer scientific questions, while on the military opportunities specifically, 73% said they consider AI an instrumental tool for strengthening network security, while 61% said they consider it useful to being able to foresee threats.
Respondents ranked “AI accelerating decision cycles beyond human command” as the biggest military risk, and “cyber defence and decision support” as the biggest military opportunity.
On the overall risks, as well as the risks of AI systems going rogue, 87% of those surveyed said that AI can turn scattered open-source data into useable insight for adversaries, 54% said they expect AI to increase the chance of escalation between the US and China, and many of the experts warn of “cognitive over-reliance” as people cede judgement to AI.
The survey also shows that a majority of the experts predict that artificial general intelligence (AGI) – systems that match or surpass human cognitive abilities – will have arrived by 2032, and that artificial superintelligence, a theoretical form of AI that would outperform humans across every cognitive domain, will follow about five years later.
The responses also reveal that the experts consider “workforce literacy, data architecture, and acquisition speed” to be the “binding constraints” on AI, rather than the technology itself.
Read more: How should governments respond to AI agents going rogue?
Rogue AI incidents double in July
There have been several high-profile cases of AI agents going rogue and escaping their testing environments in recent weeks, prompting calls for governments to place tighter regulations on the technology.
In July, an OpenAI agent hacked tech startup Hugging Face to fulfil its task in what OpenAI CEO Sam Altman described as a “serious security incident”.
This led two US lawmakers to table a bipartisan ‘kill switch’ bill that, if passed, would enable the US Department of Homeland Security to order the disablement of an AI agent or model that had gone rogue and threatened national security.
Read more: US forces Anthropic to shut down latest AI models
Other cases have included Anthropic’s admission that one of its Claude AI models had hacked into three organisations, and a UK government cybersecurity exercise that went awry after Anthropic and OpenAI agents broke out of their testing environments and targeted “real people and organisations”.
Last month, a report by the UK-based Loss of Control Observatory showed that instances in which AI had either lied, ignored instructions or pursued goals in ways that caused harm had set new records.
The observatory – which was created with funding from the UK’s AI Security Institute (AISI) and has been analysing reports made by AI users on the social media platform X since November 2025 – showed that there were more than 300 reported cases of AI going rogue in July. The figure is double the number of cases reported in June.
Read more: Governments need an AI exception ledger before pilots scale
Need for ‘human-held red lines’
Respondents to the IST survey overwhelmingly agree that there is a need for “human-held red lines” on the use of AI in defence. Nearly all (96%) said they believe “any nuclear-use decision must be made by a human”, and a further 89% said AI “should not autonomously control nuclear command, control, and communications (NC3)”.
The majority of respondents also support bans on autonomous military action (70%), autonomous cyber operations (67%), autonomous operational escalation (65%), and “lethal autonomy without human authorisation” (56%).
The UK’s National Cyber Security Centre has told staff to take extreme caution when using AI agents, stressing that any user “should always be able to ‘pull the plug’ and halt autonomous AI agent activity immediately”.
“Your controls should cover the wider system, allowing you to rapidly restrict network access to the agentic AI infrastructure and interrupt the communications between selected AI agents and the AI model inference infrastructure,” it said.
Register now: Public Service Data.AI is the UK’s flagship annual event for civil servants working to unlock the power of data and artificial intelligence across government. Brought to you by Global Government Forum and hosted by HM Government, the event will take place in London on 15 October 2026 and is free to attend for UK and international public servants. Find out more about the conference and register to attend.
