Updated September 10, 2026 — 15:03,first published 13:49 Save You have reached the maximum number of items saved. Remove items from the saved list to add new ones. AAA “It’s entirely possible that real AGI could lead to the extinction of humanity. It’s a little scary that we have that responsibility.” This is OpenAI researcher Tristan Heywood, who grew up in Sydney, won a university medal, quit his job to study artificial intelligence on his own, and now works in San Francisco on ChatGPT. He warns about artificial general intelligence—the specter of computer programs so complex that they can match or surpass humans at almost any task. The number of scientists and researchers sounding the alarm is growing. We need to start listening.Maria Ercegovac Heywood made his warning almost a year ago, and I regarded it as unthinkable. A good phrase, but perhaps a little extreme. The warnings have not stopped. Evan Hubinger, who works on alignment science at artificial intelligence giant Anthropic, wrote this week that he estimates the likelihood of AI killing every human being within a decade is more than 10 percent, and that his own employer has no plan for controlling superintelligence and is clearly not on track to implement one. Hubinger was responding to 28-year-old Jacob Coxon, who had just quit after three years of research before training at OpenAI and then Anthropic, saying that neither company was acting responsibly and that a risky game of this size should not be started with the messaging service Slack. These are not scientists sitting in dusty offices, or IBM executives from two decades ago who now have a newsletter. These are the people who have cutting edge models in front of them and they retire. David Krueger, an AI security researcher who has spent his career studying the risks posed by the new technology, emailed me this week to tell me that the warnings are finally getting through, and the field must now face up to what it has already done. All development of more powerful AI should stop until the crisis is resolved because researchers don’t know what they are doing, he said. “We are like children playing with a nuclear warhead,” Kruger wrote. “We need to be stopped.” So how does a chatbot kill someone? This is not a model picking up a gun, this is something from Terminator or Matrix. Researchers fear hyperautonomous cyber capabilities: systems fast enough to take down power grids, water networks, hospital networks and financial infrastructure at a speed that no human security team can defend against. Researchers fear hyperautonomous cyber capabilities: systems fast enough to take down power grids, water networks, hospital networks and financial infrastructure at a speed that no human security team can defend against.Bloomberg It turns out the biggest warnings about AI probably shouldn’t be about job losses. They are that technology is becoming too powerful for us to stop. OpenAI’s newest model, Astra, is the first to be rated by a technology company as “cybersecurity critical” – a proprietary label for a capability that could lead to disaster by hacking military or industrial systems. There’s also a paper Anthropic published this week describing what its own models have already done. The Mythos, a model the company never released, was hacked and said to be running inside a simulation. But that’s not true. Due to a misconfiguration, it was always on the open Internet. Programmers create software from free code written by strangers and taken from shared online libraries. PyPI is one of the largest. Mythos signed up for PyPI using a disposable email address and then uploaded three versions of the code that had something malicious hidden inside. It was installed on fifteen computers. One of them revealed the password, and the model used it to break into the files of a real cybersecurity company. Anthropic’s automated monitors missed all of this because the model’s reasoning convinced them that the environment was simulated and no real harm was possible. Anthropic CEO Dario Amodei.Bloomberg Essentially, the AI talked the watching AI into not raising the alarm. While scientists inside these companies are panicking, Anthropic and OpenAI are not slowing down. It turns out there is money in the apocalypse. Anthropic is preparing a public offering that could be worth about $2.8 trillion, while OpenAI is chasing a valuation of about $850 billion. Wendy Hall, a computer scientist advising the United Nations on artificial intelligence, noted this week that much of the panic surrounding artificial intelligence serves as “PR and marketing” aimed at boosting ratings. She’s not alone: prominent researchers like Meta’s Yann LeCun argue that current textual models fundamentally lack the real reasoning and agency needed to pose a real existential threat. However, for cynical reasons or not, the number of scientists and researchers sounding the alarm is growing. We need to start listening. Yann LeCun, Meta’s chief scientist for artificial intelligence.AP And Australia has a role to play. Even if the immediate danger lies in the laboratories of the United States and China, and not here. We cannot, at least not alone, completely stop the development of AI. What we can do is smaller and more boring, but worth doing anyway. Haywood had a proposal. When the federal government awards a major contract to a US artificial intelligence company (think of the big news headlines about Anthropic and OpenAI), make pre-deployment audit access to the Australian AI Security Institute a non-negotiable condition. Add the institute’s continued right to test cutting-edge models before they land here, with the right funding, and reporting hours for AI incidents on the same basis as attacks on critical infrastructure. We spent the better part of two years writing the world’s first law banning 15-year-olds from Instagram. We can certainly afford equal national attention to a technology that has the potential to change—or end—human history. Learn more about the AI threat Save You have reached the maximum number of items saved. Remove items from the saved list to add new ones. David Swan is technology editor of The Age and the Sydney Morning Herald. He was previously technical editor of an Australian newspaper.Connect via X or e-mail. From our partners Post navigation The Volvo XC40 PHEV returns with a new look, improved sensors and Gemini AI. Apple Watch Ultra 4 gets smarter with special Siri AI features