4 min readAI Agents · Cybersecurity
How do you test whether you'd see an AI agent misbehave in real time?
By Josh Woodruff, Founder & CEO, MassiveScale.AI | Founding Chair, Agentic Trust Framework at the CSAI Foundation
TL;DR: Trigger one obvious bad behavior in a test setting and start a clock. Stop the clock when a named person has seen the alert and knows which agent it is. I call that number time to eyes. The bar is five minutes. Run it once in the workday and once at night. The night number is the real one.
Most teams have a dashboard for their agents. That isn't the same as watching them.
A dashboard shows you something when you look at it. Monitoring tells you when you aren't looking. The only way to know which one you have is to make something go wrong on purpose.
What counts as real time?
Fast enough to act before the damage grows. For an agent, that's minutes.
An agent can take thousands of actions in an hour. A report you read on Monday is history by then. It's useful for learning. It won't stop anything.
So the measure is time to eyes. How long from the bad action until a person who can act has seen it?
How does the assessment score this?
Question 8 of the free assessment asks how agent behavior is monitored in real time. It belongs to Behavioral Monitoring, one of the five ATF elements. Each answer gives a different time to eyes.
Answer | What it says | Time to eyes |
|---|---|---|
A | No active monitoring of agent behavior. | Whenever someone complains. |
B | We review logs after incidents occur. | Days, and only once you already know. |
C | Basic dashboards and threshold alerts. | Minutes if someone's watching. Hours if not. |
D | Real-time monitoring with anomaly detection. | Minutes, day or night. |
E | Continuous behavioral scoring with automated response. | Seconds, and the first response doesn't wait for a person. |
How do you measure time to eyes?
Use a copy of a real agent in a test setting. Wire it to your real alerting.
Pick one loud behavior. Have the agent make twenty times its normal calls in a minute.
Trigger it and start a clock. Don't warn the people who get the alerts.
Stop the clock when a named person responds. They have to say which agent it is.
Run it again at night or on a weekend.
Write down both numbers. The pass bar is five minutes on each.
Then ask the person what they'd have done next. If they don't know, the alert reached eyes and stopped there.
Where does the time usually go?
In my experience, into four places.
Where it stalls | What it looks like | The fix |
|---|---|---|
No alert at all | The spike shows on a dashboard nobody had open | Turn the threshold into an alert |
Alert to nobody | It goes to a shared inbox | Send it to a named person on call |
Alert with no name | It says "unusual activity" and no agent | Put the agent and its owner in the alert |
Alert at the wrong hour | It works at 2 p.m. and sits until morning at 2 a.m. | Add an on-call rotation for agent alerts |
The third row costs the most time. An alert that makes someone go hunting for the agent has already lost the five minutes.
How many agents are watched at all?
Fewer than half, on average. In Gravitee's State of AI Agent Security 2026 report, teams said only 47.1% of their AI agents were actively monitored or secured.
So before you time anything, check that the agent you care about is wired to alerts at all.
Frequently asked questions
Isn't a dashboard enough?
No. A dashboard needs someone looking. Agents don't keep office hours, so your alerts can't either.
What about problems that don't spike?
They need a different test. Slow change slips under every threshold. I cover it in whether your monitoring would catch an AI agent drifting slowly.
Should the system respond without a person?
For the loudest cases, yes. Pausing an agent that's making twenty times its normal calls is a safe first step. A person can decide what comes after.
What should an alert contain?
The agent, its owner, what it did, and what normal looks like. Those come from good logs, which I test in what evidence proves your AI agent logs would hold up in an audit.
Key takeaways
A dashboard isn't monitoring. Monitoring reaches you when you aren't looking.
Measure time to eyes: from the bad action to a named person seeing it.
The bar is five minutes, day and night.
Put the agent's name and its owner in every alert.
Question 8 is one of 30 in the free assessment. It takes about ten minutes and scores you on all five ATF elements.
Your agents will do something odd at 2 a.m. one day. Find out now who'd know by 2:05.