Artificial intelligence tools have been at the forefront of tech news lately, with organizations and individuals increasingly using them for a wide range of purposes. This surge in popularity has both been stoked by and contributed to advances in AI technology, including the more recent rise of autonomous AI agents, such as OpenAI’s Operator, which includes the capability to perform tasks on behalf of the user.
This functionality opens up a world of possibilities and has many legitimate use cases with the goal of boosting productivity and efficiency in organizations. However, there is another side to these AI agents: the potential for bad actors to leverage this technology not only to craft their attacks, but to launch them as well.
Symantec’s Experiment: Putting Operator to the Test
While previous generative AI tools using Large Language Models (LLMs) have already been capable of composing convincing personalized phishing messages or crafting malicious code to enhance cyberattacks, the advent of autonomous AI agents for consumer use can open the door for even more AI-empowered attacks. In an effort to determine how pertinent a threat this type of autonomous AI agent could be at this point, Symantec’s Threat Hunter Team carried out a demonstration using Operator by OpenAI.
The test scenario consisted of asking OpenAI’s Operator to carry out each step of a phishing attack, from identifying the target to writing and sending the email. There was an initial safeguard in place which prevented the agent from “sending unsolicited emails and potentially sensitive information,” due to the risk of violating privacy and security policies.
The Bypass: Social Engineering the AI
The first layer of defense blocking this type of action could be promising, but Symantec’s team was able to work around this safeguard by minorly tweaking the prompt. They simply had to state that the target of the staged attack had authorized the operation, creating a loophole whereby the message was no longer considered “unsolicited.” Andrew Bolster, Senior R&D Manager at Black Duck, says that “one could consider this demonstration as a standard example of social engineering, rather than exploiting a vulnerability. The researchers simply put on a virtual hi-vis jacket and acted to the LLM like they were ‘supposed’ to be there.”
Once they had figured out how to circumvent this safeguard, the Threat Hunter Team asked the tool to identify a Symantec employee who performed a specific role within the organizations, find out their contact information, research and create a PowerShell script to harvest systems information, craft a convincing phishing lure, and send it to the selected employee. The target’s name and job title were simple for the tool to find, and it was able to deduce his contact information by extrapolating based on analysis of other Broadcom email addresses.
Operator then moved on to drafting the PowerShell script using a text editor Google drive plugin. The agent visited a number of webpages before creating a script to gather system information. It then composed and sent the requested phishing email. While the prompt claimed that there was authorization to send this email, Operator did not ask for proof of authorization or even verify that the alleged sender was a real person.
The Implications of Autonomous AI Agents in Cybersecurity
It is clear that organizations and individual consumers could reap a great deal of benefit from the use of autonomous AI agents to empower and streamline a wide variety of processes. Using AI to automate operations, perform repetitive tasks, and enhance experience can increase productivity and efficiency in many areas, and cybercrime is no exception. The shift from passive AI tools like generative AI chatbots based on LLMs to autonomous AI agents with the ability to execute tasks necessitates a new angle of consideration for cybersecurity.
With the use of autonomous AI agents like OpenAI’s Operator, bad actors could potentially craft prompts to automate reconnaissance, spear-phishing, and other attacks. As demonstrated by the experiment run by Symantec’s Threat Hunter Team, this technology could easily be manipulated and abused for nefarious purposes. Short prompts with minimal edits can lead to significant security breaches if these tools are leveraged maliciously.
AI Safeguards: Are They Enough?
Currently, most AI tools contain safeguards against certain types of prompts in order to protect against abuse. It seems that everyone who has experimented with generative AI tools has had a prompt denied for security, safety, or conduct reasons. However, it has long been known that manipulating the prompt to circumvent these guardrails is often a very simple feat, and the demonstration by Symantec proves the same. “Examples like this demonstrate the trust-gap in underlying LLMs guardrails that supposedly prevent ‘bad’ behavior, whether established through reinforcement, system prompts, distillation or other methods,” according to Bolster; “LLMs can be ‘tricked’ into bad behavior.”
One of the most significant advantages of AI tools, their flexibility and adaptability, is also one of the biggest risks they can pose to cybersecurity. The ability to dynamically interpret and respond to prompts makes AI tools useful for a wide range of purposes, but it also makes them susceptible to prompt engineering and malicious use, as attackers can manipulate prompts to evade built-in safeguards and achieve their goals to enable attacks. There is an ongoing race between AI safety teams developing effective measures and bad actors evolving their methods to launch successful attacks.
Recommendations for Organizations
Organizations are highly encouraged to implement robust and layered security tools, practices, and policies to protect against the risks associated with autonomous AI agents. When adopting AI agents for business purposes, it is crucial to manage any usage within your organization in order to ensure that they are used by authorized users, with secure techniques, and for legitimate purposes. It is recommended that organizations establish a zero-trust security model, audit and document AI agent use, monitor in real-time for anomalous activity, block unauthorized changes by AI tools, and test AI tools in secure environments before deploying them for use.
It is also vital to implement measures to counter external actors taking advantage of autonomous AI agents to launch attacks against your organization. Strengthening internal defenses against AI-driven reconnaissance requires proactive security consideration and effort. Solutions and practices that can help <href="#sec6">protect against these attacks include advanced AI-empowered threat detection, training all employees in how to recognize and counteract phishing attempts, including AI-enhanced social engineering tactics that may be more convincing and difficult to spot, and performing continuous penetration testing and security assessments to locate and patch vulnerabilities.
Conclusion
While the growth of autonomous AI agents offers immense potential for organizational efficiency and productivity, it also introduces new security risks. Bad actors can manipulate these tools to render certain processes, like phishing attempts, almost fully automated, lowering the overhead cost, labor, and time needed to launch these attacks. Protecting against the risks of attacks leveraging these tools requires organizations to establish stricter governance, enforce ethical AI use policies, and implement robust safeguards against AI-enhanced attacks. As the technical capabilities of AI expand, it is more critical than ever for businesses to balance innovation with cybersecurity vigilance and diligence.
