verifiedagents.ai
All posts

4 min readAI Agents

How do you test your AI agent incident response before the first incident?

By Josh Woodruff, Founder & CEO, MassiveScale.AI | Founding Chair, Agentic Trust Framework at the CSAI Foundation

Hero: How do you test your AI agent incident response before the first incident?

TL;DR: Run a 60-minute drill on a real agent before anything goes wrong. Your team has to recognize the problem by minute 5 and contain it by minute 15. They notify people by minute 30 and size the damage by minute 60. Score each deadline as hit or missed. A missed containment deadline outranks everything else.

Most incident plans were written for breaches. A breach unfolds over weeks.

An agent gives you about the length of a coffee break. So the only plan worth having is one you've already run against a clock.

Why does an agent incident need its own test?

Because nothing looks broken.

In the book I wrote with Michelle Savage, there's a healthcare leader we call Cam. Her diagnostic agent had been the star of the industry for six months. Then it started marking healthy patients as critical.

In three hours it marked 847 patients wrongly and set off 12 emergency procedures. Six partner hospitals threatened to disconnect.

The agent never left its boundaries. Every decision followed its training, and the monitoring showed normal patterns. A scanner upgrade had changed what the agent saw.

Your breach plan won't catch that. Nobody broke in. The longer discussion is on MassiveScale.AI, in how to respond when an AI agent goes rogue.

How does the assessment score this?

Question 28 of the free assessment asks whether you have a plan for agent incidents. It belongs to Incident Response, one of the five ATF elements. A drill shows which answer is really yours.

Answer

What it says

What the drill shows

A

No specific plan for agent incidents.

People argue about who's in charge while the clock runs.

B

General incident response applies to everything, agents included.

The team starts a breach process and looks for an intruder who isn't there.

C

We have documented AI-specific procedures.

Someone finds the document. Nobody has run it.

D

We have tested and exercised runbooks for agent incidents.

The team hits its deadlines, because they've done this before.

E

We have automated response, with people handling edge cases.

The agent is contained before the drill leader finishes talking.

How do you run the 60-minute drill?

Pick a real agent and a made-up failure. Tell the on-call team a drill is coming. Keep the scenario to yourself.

  1. Start the clock and describe the symptom. For example, the agent's decisions look odd and people keep overriding it.

  2. By minute 5, recognize. The team has to call it an incident. Calling it a data glitch is the usual first mistake.

  3. By minute 15, contain. They move the agent to advisory mode, where it can recommend and can't act.

  4. By minute 30, stabilize and notify. They list every system the agent touched and tell the people who need to know.

  5. By minute 60, assess. They say how many decisions were affected and when it started.

Score each deadline as hit or missed. Four hits is a pass.

One deadline counts more than the others. If containment misses minute 15, fix that before anything else.

What should the team resist?

Debugging. The instinct in the first ten minutes is to find the cause.

Don't. Contain first and understand later. Cam's team lost time calling it a data-quality problem. Her biggest regret was waiting to speak until she understood it.

"We're looking into it" beats silence.

What should be true before the drill?

Ask four questions first. A no on any of them tells you where the drill will fail.

  • Can you move any agent from acting alone to advisory mode in under five minutes?

  • Does the escalation path work without hunting for a phone number?

  • Can you explain one of the agent's decisions in a single plain sentence?

  • Has the real on-call team practiced on the real agents?

The first question has its own drill, in testing whether your AI agent kill switch works. Some failures never reach a person in time, so check the automatic stops too, in testing whether a runaway AI agent gets stopped automatically.

Frequently asked questions

Should we shut the agent off completely during an incident?

Usually not. A full stop halts the business along with the damage. Advisory mode keeps the work moving while people check each action.

How often should we drill?

Monthly for your highest-impact agents. Quarterly for the rest.

Who should lead the drill?

Someone outside the team that built the agent. Builders know too much, and they'll skip the confusing parts.

Can an incident leave us stronger?

Yes, when it's handled well. In the book, Cam's rebuilt monitoring later caught a real data poisoning attempt that her first setup would have missed.

Key takeaways

  • Agent incidents move too fast for a plan nobody has run.

  • Drill four deadlines: recognize by 5, contain by 15, notify by 30, assess by 60.

  • Score each one hit or missed. Four hits is a pass.

  • Containment comes before understanding.

Question 28 is one of 30 in the free assessment. It takes about ten minutes and scores you on all five ATF elements.

Prepared teams still have incidents. Theirs are shorter, because the first hour has already happened once.

See where your agents stand.

The free assessment takes ten minutes and scores you on the five elements of the Agentic Trust Framework.