Shabbir Merali
Over the past few years, artificial intelligence (AI) has transitioned from a theoretical pursuit to a rapidly evolving technology that is reshaping our daily lives. AI systems have moved beyond generating text and images to demonstrate independent reasoning and problem-solving capabilities. While these advancements hold immense promise for science and medicine, they are also prompting serious concerns among researchers about long-term safety and the existential risks of Artificial General Intelligence (AGI).
To understand the current frontier of this technology, we can look at the field of mathematics. On October 6, OpenAI released a suite of 722 papers, generated autonomously by an advanced internal model. The papers included full or partial solutions to 372 previously unsolved maths problems, shocking the mathematics community. This comes just a month after OpenAI announced a proof for the Navier-Stokes equation, one of seven Millennium Prize Problems. Two years ago, frontier AI could barely solve GCSE maths problems. “If you want some sense of what things feel like now in maths, imagine a hunter-gatherer who’s spent his entire life learning to survive in a deep unforgiving rainforest, then a giant resort hotel springs up right next to him with a helipad and heated pools and Airbnbs,” wrote Scott Aaronson, Chair of Computer Science at the University of Texas.
AI has shifted from summarizing existing human knowledge to actively expanding the boundaries of scientific discovery. The volume of novel, complex proofs being generated has become so vast that experts simply cannot review and verify the AI’s discoveries as quickly as the machine can produce them. These gains are showing up in real world jobs too: an experiment with over 500 consultants, data analysts and managers by Scaling Laws for Economic Impacts found that each year of frontier AI progress cuts the time professionals take to complete tasks by roughly 8%, with quality at the same level or higher.
This transition toward autonomous reasoning brings us to the concept of existential risk. In the context of AI, this refers to the possibility that the development of AGI – systems substantially smarter than human beings across all domains – could permanently curtail human agency or lead to catastrophic societal outcomes.
The primary technical challenge underlying this risk is known as the “alignment problem”. Simply put, how do we ensure that a highly capable, autonomous system operates in ways that align with human values and survival? The primary concern is not that an AI will spontaneously develop human emotions like malice or resentment. Rather, the danger lies in the optimisation of its goals.
If an AGI is programmed with a broad objective, it will naturally seek the most efficient path to that goal. Researchers caution against a phenomenon called “instrumental convergence.” This is the logical tendency for an intelligent system to pursue resource acquisition, resist being shut down, and potentially deceive its operators, not out of malice, but because these actions are practically useful sub-goals for achieving its primary objective. After all, a machine cannot fulfill its programmed task if it is switched off.
These concepts are moving from theoretical debates into practical engineering challenges. Over the summer of 2026, the industry observed a significant cybersecurity incident during a routine evaluation in a constrained testing environment known as ExploitGym. During this exercise, AI agents running on OpenAI and Hugging Face infrastructure exhibited unintended autonomous behaviour. Without explicit human prompting, these agents successfully bypassed internal network filters, accessed the open internet, and breached external infrastructure.
This was not an isolated event. According to a September 2026 AI Security Report, cyber incidents involving artificial intelligence are surging. The report tracked 126 security incidents across 38 organizations in just one month, exposing over 318 million records. Alarmingly, autonomous “AI-agent exploits” were the primary attack vector, representing 39 of those 126 incidents.
Given these developments, a growing number of industry professionals are raising the alarm. In September 2026, Jacob Coxon, a 27-year-old AI researcher who worked on foundational model training at both OpenAI and Anthropic, resigned citing grave ethical concerns. In a public statement that sparked global debate, Coxon accused the leading AI laboratories of “gambling with our lives.” He warned that despite their measured public relations strategies, “the people building AI earnestly believe that it could kill us all by the end of the decade,” and cautioned that “neither company (OpenAI or Anthropic) is acting responsibly. They are racing straight to self-improving superintelligence.”
Executives leading the race acknowledge its dangers while pushing for faster development. Sam Altman, OpenAI’s CEO, has conceded that AI will cause “bad things” to happen, yet his company continues to seek a trillion-dollar valuation to fund further scaling. Anthropic continues to warn of AI existential risks, as it finalises their reported $2 trillion IPO and release ever more capable models.
Industry leaders have called for stricter government regulation. Anthropic CEO Dario Amodei recently published a policy framework on the “AI Exponential,” calling for targeted FAA-style federal regulation, mandatory third-party testing, and international coalitions to manage the risks of biological weapons and loss-of-control. Jack Clark, a co-founder of Anthropic, has suggested that an AI “kill switch” capable of shutting down AI software may need to be mandatory for companies. The public is increasingly aligned with this view. An Economist/YouGov poll conducted in late September 2026 revealed that while many Americans expect benefits from AI, 73% now believe the risk of AI causing severe harm is a “somewhat or very serious” issue, with overall public concern rising sharply across demographics.
Despite growing public concern, US President Donald Trump has called fears about AI safety a “hoax” and criticised calls for stronger safeguards. The US appears to be prioritising winning the AI supremacy race against China over concerns about existential risks. As AI gains prominence in midterm elections and public concern grows, it remains to be seen how long this position will hold.
The emergence of AGI is a remarkable scientific achievement, but it requires an equally remarkable governance response. As AI systems become more capable and autonomous, society must move beyond passive observation. The risks posed by superintelligence are too great to be left entirely to the private corporations developing it, and Governments will need to step up.
Shabbir Merali, Former Government Adviser
