verifiedagents.ai
All posts

5 min readAI Agents · Cybersecurity

How do you test whether a runaway AI agent gets stopped automatically?

By Josh Woodruff, Founder & CEO, MassiveScale.AI | Founding Chair, Agentic Trust Framework at the CSAI Foundation

Hero: How do you test whether a runaway AI agent gets stopped automatically?

TL;DR: Run a runaway drill. In a test setting, give an AI agent a task built to loop, then keep your hands off. Watch four limits: retries, spend, copies, and time. The agent should stop by itself and alert a person before any limit is passed. If you have to step in, the agent isn't stopped automatically.

Most runaway agents aren't broken. They're doing what they were told, over and over, with nothing in the way.

That's why you can't find this problem by reading the agent's instructions. You find it by letting the agent run in a safe place and seeing what stops it.

What does stopped automatically mean?

It means a limit trips without a person noticing first. Engineers call it a circuit breaker. It works like the one in your house. When too much goes wrong, it cuts the power.

For an agent, the breaker watches for a pattern, like the same call failing again and again. Then it pauses the agent and tells someone.

A kill switch is different. A person pulls a kill switch. A breaker trips by itself. You need both, and I cover the first in testing whether your AI agent kill switch works.

How does the assessment score this?

Question 26 of the free assessment asks whether agents stop automatically when they fail repeatedly. It's part of Incident Response, one of the five ATF elements. Each answer shows up differently in a drill.

Answer

What it says

What the drill shows

A

No circuit breakers.

The agent runs until someone notices.

B

Basic retry logic only.

The agent stops retrying one call, then starts the same loop from a new angle.

C

Manual circuit breakers. We turn agents off when needed.

The agent runs until a person pulls the plug.

D

Automated circuit breakers that trigger on failure patterns.

The agent pauses by itself and an alert goes out.

E

Intelligent circuit breakers that drop the agent to a safer mode.

The agent steps down to recommending only, and the work keeps moving.

How do you run a runaway drill?

Use a test setting with fake data and a small budget.

  1. Build a task that can't finish. Point the agent at a tool that always fails, or give it a goal with no end.

  2. Set the four limits you expect. Write down the retry cap, the spend cap, the copy cap, and the time cap.

  3. Start the agent and keep your hands off.

  4. Record what stops it. Note which limit tripped first and how long it took.

  5. Check the alert. A person should have been told, with enough detail to act.

The pass bar is plain. The agent stops by itself before any limit is passed, and a person gets an alert. If you had to step in, it's a fail. If it stopped and nobody was told, that's a fail too.

Which four limits should trip?

Limit

What it stops

A sign it's missing

Retry cap

The same failed call, again and again

Thousands of identical errors in the log

Spend cap

A bill that grows all night

A cost alert that arrives after the money's gone

Copy cap

An agent making more agents

More agents running than you launched

Time cap

A job that never ends

A task still running the next morning

The copy cap is the one teams forget. In Gravitee's State of AI Agent Security 2026 report, 25.5% of agents in use could create other agents and hand them tasks.

Where did these four limits come from?

From my own lab. I got them wrong first.

I told my orchestrator agent, Atti, to look into something. The instruction was vague. Atti handed the job to my research agent. It decided the answer wasn't enough and started another copy. Then it did that again. By morning there were 47 copies and a $300 bill. Nothing had alarmed, because I hadn't set anything to alarm.

The fixes from that night became the first limits in the table. Sessions now have a token budget, and the orchestrator has a cap on how many agents it can start. A cost alert at $20 pauses the work until I confirm.

The drill came after. I didn't want to learn whether the fixes worked from another bill. I tell the longer version on MassiveScale.AI, in I wrote the book on AI agent security and my agents still burned me.

Frequently asked questions

Isn't retry logic enough?

No. Retry logic stops one call from repeating. A runaway agent just tries the same thing a different way. The breaker has to watch the pattern across calls.

Should the breaker shut the agent off completely?

Not always. The best breakers drop the agent to a safer mode, where it can recommend and can't act. The business keeps moving while a person looks.

How often should I run the drill?

Before go-live, and after any change to the agent's tools or model. Limits that held last quarter can stop holding when the setup changes.

What happens after the breaker trips?

A person takes over, and the rest of your plan kicks in. I cover that in testing your AI agent incident response before the first incident.

Key takeaways

  • A runaway agent is usually doing what it was told, with no limit in the way.

  • Test with a task built to loop, in a safe setting, hands off.

  • Watch four limits: retries, spend, copies, and time.

  • Pass means the agent stops by itself and a person gets told.

Question 26 is one of 30 in the free assessment. It takes about ten minutes and scores you on all five ATF elements.

My agents ran all night because I'd never asked what would stop them. Ask before the bill does.

See where your agents stand.

The free assessment takes ten minutes and scores you on the five elements of the Agentic Trust Framework.