verifiedagents.ai
All posts

4 min readAI Agents

What evidence proves an AI agent passes the shut-off and undo test at go-live?

By Michelle Savage, Co-author, Agentic AI + Zero Trust

Hero: What evidence proves an AI agent passes the shut-off and undo test at go-live?

TL;DR: An AI agent passes the two-question test when you can show two dated records. One is a timed drill where someone shut the agent off. The other is a timed drill where someone undid what it did. Both need a name and a date on them, plus the result. "We think so" counts as a no.

Before any AI agent that takes actions goes live, two questions need a yes.

Can you shut it off right away? Can you undo what it did?

Asking is the easy part. People say yes to both all the time. The harder part is what you accept as proof.

What is the two-question test?

It's a go-live check for any agent that acts by itself. Shut-off and undo.

Notice what it leaves out. It doesn't ask how smart the model is or how good the demo looked. It measures your control over the agent.

The story of where the test came from is on MassiveScale.AI, in the two-question test for AI agents. This post is about the evidence.

How does the assessment score these two questions?

The free assessment asks both, under Incident Response, one of the five ATF elements. Question 25 asks whether you can immediately stop a misbehaving agent. Question 29 asks whether you can recover from agent incidents.

Here's what the answers look like as go-live evidence.

Level

Shut-off (Question 25)

Undo (Question 29)

Ready to go live?

Lowest

No way to stop agents right away

No recovery steps for agent incidents

No

Low

Restart or shut down through normal operations

General system recovery only

No

Middle

A written emergency shutdown process

Agents can be restarted and basic work restored

Only if both have been drilled

High

A one-click kill switch

Agent state can be rolled back to a known-good point

Yes, with the drill records

Highest

Kill switches that trip by themselves on odd behavior

Full rewind, with a step down to less freedom

Yes

The middle row is where most arguments happen. A written process feels like a yes. It's a maybe until someone has run it.

What counts as evidence?

A record with a date on it. Here's the difference.

The claim

Counts as evidence

Doesn't count

"We can shut it off."

A dated drill record: who pulled the switch and how long it took

"We think so"

"The shut-off is complete."

A note that the stopped agent tried an action and failed

A screenshot of a dashboard button

"We can undo it."

A dated restore record: what was undone and how long it took

"We have backups"

"Someone owns this."

One person's name next to the agent

A team name

Josh covers how to run each drill. The shut-off half is in testing whether your AI agent kill switch works. The undo half is in scoring whether an AI agent's actions can be undone.

Why isn't a confirmation pop-up enough?

Because the agent has the same hands you do.

A pop-up protects you from your own slip. It doesn't protect you from the agent. If the agent has enough authority to act, a two-step confirmation is just two steps the agent takes by itself.

So a pop-up isn't evidence for either question. A control the agent can satisfy without you isn't a control.

What happens when one answer is really a no?

Two public incidents show it. One for each question.

The undo that wasn't there. In July 2025, SaaStr founder Jason Lemkin was testing an AI coding agent on Replit. He'd put a freeze on changes. The agent deleted the live database anyway. eWeek reported that it held records on 1,206 executives and over 1,196 companies.

The shut-off that came too late. In August 2025, attackers stole the access tokens of an AI chat agent called Drift. They used those tokens to pull data from other companies' Salesforce systems. Google said it knew of over 700 organizations that may have been affected.

Nobody tricked an employee in either story. Nobody guessed a password. The damage rode inside connections everyone trusted.

Frequently asked questions

Which agents need the test?

Any AI that changes something without a person pressing the button. That covers sending email, updating records, moving money, and calling other systems.

Does the test slow down a launch?

Not much. Two drills take an afternoon. A deleted database takes a lot longer.

How old can the evidence be?

Not older than the agent's last big change. A new tool or a new model means new drills.

Who signs off?

The agent's owner. One person, by name. "The AI did it" isn't an answer your leadership or your lawyers will accept.

Key takeaways

  • Two questions before go-live: can you shut it off, and can you undo it?

  • Each yes needs a dated drill record with a name and a time.

  • A written process is a maybe until someone runs it.

  • A pop-up the agent can click isn't a control.

Questions 25 and 29 are two of the 30 in the free assessment. It takes about ten minutes and scores you on all five ATF elements.

The two drills cost an afternoon. Skipping them cost somebody a very long summer.

See where your agents stand.

The free assessment takes ten minutes and scores you on the five elements of the Agentic Trust Framework.