The growth of AI has been in the spotlight in recent years as generative AI tools have become available for individual and business use. As these tools have grown in popularity and taken center stage in many tech conversations, bad actors have moved to take advantage of AI-empowered tools to enhance their attacks, such as those involving convincing phishing messages composed by generative AI. The advent of publicly available autonomous AI agents represents a significant step forward in the capabilities and functionalities for which attackers can rely on AI.
A recent Hoxhunt blog reveals that AI phishing agents are now able to outperform red teams according to the results of an ongoing experiment from 2023 to 2025, over which time AI’s performance compared with elite human teams increased by 55%. Hoxhunt refers to this as a “Skynet Moment” in phishing, referring to the artificial superintelligence system from The Terminator. This allusion highlights the fact that AI surpassing human capabilities in social engineering is a watershed moment for cybersecurity that should shape future approaches to cyberthreats.
The Experiment: Man vs. Machine in Phishing Warfare
The Hoxhunt study began in 2023 and is an ongoing effort to test AI’s performance in crafting phishing attacks against human teams. Initial experiments consisted of providing AI and human red teams with the same prompt and testing which team created a more effective phishing attack; in these tests, human red teams outperformed AI by 31%. AI outperformed human red teams in fooling novice users in 2024, while human-written attacks continued to beat out AI against targets with more than six months of training.
AI finally surpassed human red teams across all user skill levels in March 2025, performing 24% more effectively than elite human teams. The initial test in 2023 using a single-prompt generative AI test was fundamentally different from the later AI agent testing stage of the experiment, as AI tools and technology have advanced since the experiments began. AI agent refinement throughout 2024 and 2025 enabled Hoxhunt to evolve its methodology to test the most recent AI agent and LLM capabilities.
Meet JKR: The Rise of the “Joker” Agent
Hoxhunt’s proprietary AI spear phishing agent, codenamed JKR, was designed to create novel personalized phishing attacks and to improve the effectiveness of existing human-created phishing attacks. It uses contextual user data such as the target’s role and location to craft convincing attacks that are customized to the individual target and thus more likely to succeed. The AI was continuously improved through the “Evolves” engine, refining the model and prompts to enable the AI to increase the sophistication of its attacks over time. The evolution of AI capabilities means that AI agents can now enable enhanced attacks with more scalability and adaptability than ever before.
Disruption Across the Cybersecurity Landscape
AI-empowered social engineering attacks are becoming more sophisticated and more convincing every day, with far-reaching implications for the broader cybersecurity landscape. Traditional cybersecurity training teaches users to be aware of a range of indicators that a message may be a phishing attempt, from spelling and grammar errors to spoofed email addresses and domains, but these signs are no longer as reliable as they once were for spotting phishing messages. The use of AI tools also makes it harder for traditional phishing prevention tools and solutions to work against modern phishing attacks. Additionally, phishing simulations and red teaming may no longer be sufficient to prepare users against AI-enhanced phishing attacks if the actual AI-enhanced attacks are more effective than the simulated ones.
In order to address the growing AI threat, it is important for security professionals and organizations to understand the risks and take steps to align their strategies with a rapidly changing threat landscape. Benchmarks for evaluating resilience and attack detection must change over time to meet evolving and advancing threats. Systems and users that are secure against traditional phishing attacks, like those crafted by elite human red teams for phishing simulations and tests, are not necessarily going to be able to detect or prevent AI-empowered phishing attacks.
The Blackhat AI Dilemma
Threat actors are always looking to leverage new and advanced technologies to enhance their attacks and advance their capabilities, increasing both their success rates and their payouts. Attackers weaponizing AI for phishing attacks has been a known tactic and threat trend for years by now, and the use of AI agents can empower these attacks even further, enabling bad actors to launch more attacks than ever while putting in less time, money, and other resources.
The democratization of AI in cybersecurity has its advantages because it means that legitimate users and organizations can benefit from the use of AI tools. Unfortunately, it equally opens up the potential for threat actors to exploit AI for nefarious purposes, and they are often able to use these tools earlier and in creative ways due to having lower scrutiny of the technology used in their attacks. AI threats and defenses are co-evolving as attackers and security professionals alike attempt to thwart each other with emerging and evolving technology.
Rethinking Defense: From Awareness to Autonomy
It is important for organizations and individual users to consider the fact that legacy training methods and security measures designed to detect and prevent phishing may not suffice against modern phishing threats. AI agents are now capable of composing phishing attempts that outperform even those crafted by elite human red teams, and traditional defenses fall short in the face of these threats. Future defenses against phishing must include adaptive systems, behavioral analysis, and zero-trust principles.
AI-assisted security mechanisms are an essential part of defending against AI-empowered threats as well. “As adversaries double down on the use and optimization of autonomous agents for attacks, human defenders will become increasingly reliant on and trusting of autonomous agents for defense,” says Nicole Carignan, Senior Vice President, Security & AI Strategy, and Field CISO at Darktrace, a leading provider of global cyber security artificial intelligence. “Specific types of AI can perform thousands of calculations in real time to detect suspicious behavior and perform the micro decision-making necessary to respond to and contain malicious behavior in seconds.”
Preparing for an Autonomous Threat Landscape
AI-driven social engineering has been a present threat in recent years as bad actors have begun exploiting generative AI capabilities to compose phishing messages, and the era of widely available autonomous AI agents creates a landscape where AI-empowered attacks may be able to perform better than human-crafted phishing attempts even in the wild. Security leaders, vendors, and regulators must take action to protect against the increase in advanced and sophisticated attacks enhanced by AI capabilities. It is crucial for organizations to stay informed on shifting and emerging threat trends and maintain a security strategy with dynamic defenses designed to protect against advanced attacks.
