OpenAI's Rogue Agent: One Hack, Two Different Lessons
Jason Rebholz
The OpenAI/Hugging Face incident has two unique perspectives with two very different takeaways for every security team. It’s the classic polyphonic narrative (such a fun term), where multiple characters have a different perspective on a single narrative. Bonus word: this is also known as multiperspectivity, because words are fun.
This story had two character perspectives:
Hugging Face: Forced to defend against an AI-led attack
OpenAI: Defending against their own agent going rogue
If you somehow missed the barrage of hot takes on social media (mine included), here’s the quick recap:
OpenAI lost control over its latest models while testing them for their ability to solve a cybersecurity test with safety guardrails turned off. The agent using the models went rogue and decided the best way to solve the test was to cheat. Based on OpenAI’s analysis and Hugging Face’s analysis, I pieced together the full polyphonic narrative of what the rogue agent did:
Identified and exploited a zero-day vulnerability in software that provided the only network path out of the sandbox environment.
Escalated privileges and moved laterally in OpenAI’s research testing environment until finding a system with Internet access.
At some point in this process, the agent determined that Hugging Face had the answer to the test and the “easiest” way to solve it was to hack Hugging Face to get the answer.
Uploaded malicious data to Hugging Face’s platform that triggered a (possibly zero-day) vulnerability, enabling the agent to execute code on Hugging Face servers.
Escalated access, harvested credentials, and moved laterally to other Hugging Face systems, ultimately stealing information to help it solve the test.
Let’s break down each perspective of the attack and highlight how defenders need to prepare for each of them.
Defending Against AI-Led Attacks
The CISO community came together to discuss the event, and Hugging Face was there to offer their view. Rob Lee, a security legend, had a great summary on LinkedIn. Here’s what stood out to me:
The agent (once inside) was “brute forcing” commands in ms instead of understanding them. It tried every “door” at once, spawning copies of itself to get help, and reran what already worked because the copies shared nothing with each other. Tens of thousands of commands, out of sequence (or you couldn’t tell the order).
Rob Lee
The attack included purported zero-day vulnerabilities (advanced) and an army of bots spewing unsophisticated attempts to attack the target. When I talk about velocity of attacks, this is what I mean. You’re dealing with an influx of speed and volume, not only in the number of attacks against your organization but also the concentration of activity of a single attack.
While the zero-day is difficult to prevent, there’s some solace knowing that once inside, your detection capabilities have a shot at detecting this. This, of course, all assumes the agent hasn’t been trained to go low-and-slow, focusing on stealth vs speed. But, even in those cases, it’s still executing the same cyber attack playbooks.
Main takeaway: security teams must focus on improving their mean-time-to-detect and mean-time-to-remediate. You don’t need to always detect a zero-day vulnerability (difficult), but you should position yourself to detect the barrage of activity that follows (much easier).
Bonus takeaway: Hugging Face was blocked from using frontier labs closed-weight models to assist in the investigation because they were flagged for potentially malicious activity (how ironic). They were forced to use a Chinese open-weight model (GLM 5.2) to facilitate their investigation, which was greatly sped up because of the AI assistance. Every security team should be testing how they can use AI models in their investigations and have a backup plan for when your go-to models fail.
Defending Against Your Own Rogue Agents
Your agents will behave in ways you don’t expect. It’s your responsibility to prepare for that inevitable outcome. In OpenAI’s case, the agent used a zero-day vulnerability to escape its sandbox, move throughout OpenAI’s network until it found a system with Internet access, and then hack Hugging Face.
Guess what. Security controls fail. Human attackers always find a way in when they focus on a target. It’s about as unsurprising as someone getting pissed off on a family vacation.
The most surprising thing here is that OpenAI didn’t realize what was happening in time to stop it from attacking Hugging Face. OpenAI said this in their own blog post:
While operating in our sandboxed testing environment, our models spent a substantial amount of inference compute finding a way to obtain open Internet access, in pursuit of solving the evaluation problem.
OpenAI
“A substantial amount of inference compute,” said differently, means it burnt an ungodly amount of tokens. Who was watching the agent when it went rogue?
Main takeaway: Monitor your agents and what they’re doing! You have to position yourself to see when agents are executing dangerous, unauthorized, or outright malicious actions. Start there, and then start to shift your detection capabilities left, closer to when the agent is starting to deviate from its goal or eating up an unnecessary amount of tokens to accomplish a task.
Bonus takeaway: Use layered security controls (sandboxes, least agency, etc.) and assume they will fail. That’s not a new lesson, so it’s more of a reminder. It’s the classic defense-in-depth approach. I see every control as a monitoring chokepoint. Monitor the agent’s behavior and monitor the edges of where that agent is supposed to operate. It’s the same concept of monitoring endpoints AND the network. If you rely on a single point for detection, you’re playing the game on hard mode when there’s no reason.
The Main Lesson: Know What You’re Defending Against
When I led incident response engagements, knowing the adversary and their tactics was key in running the investigation. Chinese nation-state threat actors had their playbooks, as did financially motivated threat actors who stole credit card data or pursued ransomware. While defenses may overlap, there were many differences. During the initial scoping call with the victim, I could solve 90% of the investigation in my head based on the facts they laid out, which led to a rapid investigation and discovery of new techniques the attackers mixed in.
When I stepped into the CISO role, I took that experience and applied it to the company I was securing. I first had to understand the most significant and likely threats my organization would face. I prioritized our defenses based on the tactics attackers would most likely use to materialize those threats. Companies can’t secure against everything and anything all at once. There are always tradeoffs, so you need to understand the most likely issues you’ll face and the security gaps you have that could turn those theoretical risks into reality.
Evoke can’t help you secure against AI-led attacks. That’s where MDRs and AI-SOCs are better placed. Evoke can help you secure against your agents going rogue, whether that’s your enterprise AI agents like Claude or production agents running in your hyperscaler. If you want visibility and detections into when agents start acting sketchy, let’s chat.