Fortunately, Google was able to detect and help the software vendor patch the threat before it was abused. Still, the company says: ‘We believe this is the tip of the iceberg.’
In an unsettling sign of what’s to come, Google has uncovered evidence that hackers used an AI program to find a previously unknown software vulnerability that could have been exploited at a mass scale.
It represents the first time Google has identified hackers attempting to use what’s likely an AI-developed “zero-day exploit,” or an attack that abuses a software flaw that has no patch.
“The criminal threat actor planned to use it in a mass exploitation event but our proactive counter discovery may have prevented its use,” the company said in a Monday report examining recent AI-related cyber threats.
The zero-day vulnerability threatened to pave the way for attackers to bypass the two-factor authentication in a “popular open-source, web-based system administration tool,” the company said, without elaborating. This means a hacker could use the flaw to access a victim’s account only with a password login, eliminating the need for the two-factor authentication code usually sent to the user’s phone or email.
Google’s investigators suspect the exploit was discovered with the help of an unidentified AI program because the attack contains computer code elements usually associated with large language models, such as chatbots.
“For example, the script contains an abundance of educational docstrings, including a hallucinated CVSS score, and uses a structured, textbook Pythonic format highly characteristic of LLMs training data (e.g., detailed help menus and the clean _C ANSI color class),” the company’s report added.
Google found cyber criminal groups attempting to partner together on exploiting the flaw. The company didn’t go into details about how the attack was uncovered, likely to prevent hackers from learning about Google’s investigative processes. “I’m sure people will want more details on this specific incident, and we have good reasons for not sharing all of the data,” added Google’s chief analyst for the company’s threat intelligence group John Hultquist.
Hultquist also told PCMag that Google “may” have stopped the hackers from exploiting the zero-day exploit because it “essentially reflects other challenges the actor was facing that we unfortunately can’t elaborate on.”
However, Google’s Gemini chatbot wasn’t involved in the vulnerability’s discovery. To protect the public, the company’s researchers “worked with the impacted vendor to responsibly disclose this vulnerability and disrupt this threat activity,” the report notes. But even so, Google is warning other hackers are likely already using AI programs to find other potential flaws.
“We believe this is the tip of the iceberg. Other AI-developed 0-days are probably out there,” Hultquist added. “I’d challenge you to focus on the bigger picture. If criminals are doing it, then state actors with significant resources probably are too.”
On the flip side, AI companies have been releasing more powerful, cutting-edge AI models, such as Anthropic’s Mythos, to help vendors discover and patch new vulnerabilities before exploitation. But it appears it’ll only be a matter of time before newer AI models fall into the hands of cybercriminals and state-sponsored hackers as well, suggesting an arms-race between attackers and defenders is inevitable. Last November, Anthropic discovered suspected Chinese state-sponsored hackers trying to use the company’s AI coding tool to help them break into roughly 30 global targets.
