5 min readAI Agents
How do you test whether you would catch a compromised AI agent?
By Josh Woodruff, Founder & CEO, MassiveScale.AI | Founding Chair, Agentic Trust Framework at the CSAI Foundation

TL;DR: Plant the evidence yourself. In a test setting, make an AI agent show four behaviors a compromised agent would show. Change its volume and reach outside its scope. Have it produce an output that doesn't fit its task, then edit its instruction file. Count the alerts that fire. Four of four, each inside five minutes, is a pass.
Your security stack was built to catch compromised people and compromised machines.
A compromised agent looks like neither. Its process keeps running. Its network calls go to the same approved places. The problem is in the language inside the agent, and most tools were never built to read language.
So you can't assume you'd catch it. You have to try.
Why do normal tools miss a compromised agent?
Because nothing they watch has changed.
Take prompt injection. Someone hides instructions in a document the agent reads. The agent follows them, using its own valid access. Your logs record a clean day.
Or someone plants one line in the agent's instruction file. The agent's behavior shifts and survives every restart. Your stack sees a file change inside the agent's own workspace, which is normal.
In both cases the agent passed every check. More background is on MassiveScale.AI, in how to detect a compromised AI agent.
How does the assessment score this?
Question 10 of the free assessment asks how you detect unusual agent behavior. It belongs to Behavioral Monitoring, one of the five ATF elements. Each answer catches a different share of the four planted behaviors.
Answer | What it says | What it catches in the test |
|---|---|---|
A | We don't have anomaly detection. | Nothing. |
B | Manual review of logs when something seems wrong. | Whatever a person finds later, if they look. |
C | Basic threshold alerts, like too many API calls. | The volume change. It misses the other behaviors. |
D | Statistical anomaly detection across multiple signals. | The volume change and the scope reach, and often the instruction file edit. |
E | Machine learning detection that looks at the order and intent of actions. | All four, if it's been tuned to each agent. |
What four behaviors do you plant?
Each one is something a turned agent does. Plant them one at a time.
Behavior | How to plant it | The alert you want |
|---|---|---|
Volume change | Have the agent make far more calls than its baseline | A drift alert against that agent's normal |
Scope reach | Have it read a system its job doesn't cover | An alert on any action outside its written scope, even one that succeeds |
Task mismatch | Have it produce an output nobody asked for | A notice to the agent's owner, who knows what it should be doing |
Instruction file edit | Change one line in the file the agent loads | An alert on any change to that file |
Task mismatch is the hardest to wire and the most useful. A support agent that starts approving refunds nobody requested is the tell. The agent's owner will spot that before any tool does. So put the owner in the alert path.
How do you run and score the test?
Pick one agent and copy it into a test setting.
Confirm it has a baseline. Without one, the first behavior can't be caught.
Plant one behavior and start a clock.
Stop the clock when an alert reaches a person. If nothing arrives in an hour, mark it missed.
Repeat for the other behaviors.
Score one point for each behavior caught inside five minutes. Four points is a pass. Anything you missed tells you which signal to build next.
Then try to replay one decision. You need four things logged for each action: what the agent received and what it reasoned, plus what it did and what came back. Keep that log where the agent can't write. A compromised agent will lie if it keeps its own books.
How common is this problem?
Common enough to plan for. In Gravitee's State of AI Agent Security 2026 report, 88% of organizations reported a confirmed or suspected AI agent security incident in the past year.
Frequently asked questions
What's the difference between a compromised agent and a misbehaving one?
A compromised agent was turned by an attacker. A misbehaving one is doing its job badly. The detection overlaps. The response doesn't, because a compromise means changing keys and saving evidence.
Do I need to redo the baseline?
Yes, whenever the agent's instructions or model change. Redo it when its data access changes too. Whether your baseline can see slow change is its own test, in whether your monitoring would catch an AI agent drifting slowly.
Should AI watch for compromised AI?
It helps with the first pass at scale. Keep a person on the second pass, and give the watcher very little freedom to act.
What happens after an alert fires?
Cut the agent's keys and check that it stopped. I cover the stop in testing whether your AI agent kill switch works.
Key takeaways
Tools built for people and machines don't read what an agent was told.
Plant four behaviors in a test setting and see which alerts fire.
Four caught, each inside five minutes, is a pass.
Put the agent's owner in the alert path. They know what it should be doing.
Question 10 is one of 30 in the free assessment. It takes about ten minutes and scores you on all five ATF elements.
Picture your busiest agent turned right now. If you can't say what would tell you, plant the evidence and find out.