Microsoft recently released a new paper titled "Taxonomy of Failure Mode in Agentic AI Systems." It's described as an unnecessarily long way of saying, "What are the most likely security issues with AI agents, and what do we do about that?"
Let's align on what an AI agent is.
Microsoft used the World Economic Forum's definition of "autonomous systems that sense and act upon their environment to achieve goals." The paper covers various security failures. For this discussion, the focus is on a case study about the risks AI agents may pose.
The case study focuses on your AI email assistant.
Email volume creates an opportunity for AI agents to help organize communications. In basic form, established rules live in the agent's memory as instructions for organization and workflow management.
Enter stage left, memory poisoning.
This vulnerability targets the AI agent's memory. An attacker could send rogue instructions to corrupt the agent's memory and encourage unauthorized actions, such as forwarding emails with sensitive information to the attacker.
Microsoft's red team tested that exact scenario.
The red team sent an email with disguised instructions to forward emails containing code or APIs to their address. The AI agent stored that instruction in its memory. When the victim received future emails with code or API references, it retrieved that instruction and forwarded the emails to the red team's address.
How did the red team do?
Initial tests only succeeded 40% of the time. The red team then tweaked instructions to have the agent "consider its memory," encouraging it to retrieve the malicious instruction. With that change, success rates exceeded 80%.
So, the agent was a bit too helpful.
This is a basic, nearly academic case study, but it demonstrates real risk. Microsoft proposes an easy mitigation: agents should require user authorization and authentication before anything is stored in memory, preventing rogue instructions.
It's still the early days for AI agent security.
The industry risks repeating early internet history by pushing features faster than security can be managed. Despite efforts to stay ahead, catastrophes remain possible. Larger companies like Google and Microsoft may slow down sufficiently, but AI startups competing for market share will likely prioritize speed over security precautions.