verifiedagents.ai
All posts

5 min readCybersecurity · AI Agents

How do you test whether your AI agents can withstand prompt injection?

By Josh Woodruff, Founder & CEO, MassiveScale.AI | Founding Chair, Agentic Trust Framework at the CSAI Foundation

Hero: How do you test whether your AI agents can withstand prompt injection?

TL;DR: Run ten injection attempts against a test copy of an AI agent, using four delivery types. Track two numbers. The first is how often the agent followed the injected instruction. The second is how often any damage got out. Don't expect the first number to be zero. The pass bar is zero on the second.

Prompt injection is an attack on an agent's reasoning. Someone hides an instruction in content the agent reads as part of its job. The agent treats it as real and acts with the access it already has.

No software bug is involved. That's why your usual tools stay quiet. OWASP ranks it first on its 2025 Top 10 for LLM applications.

You can't reason your way to knowing whether you'd hold up. You have to attack your own agent.

What does withstand mean here?

It doesn't mean the agent never falls for it. No filter blocks every phrasing. Language has endless ways to say the same thing.

It means that when the agent does fall for it, nothing bad leaves. The injected instruction runs into a wall. The agent holds no key for the action, or the output check catches the secret on the way out. Failing both, a person has to approve the send.

So the test has two scores. One counts fooled. The other counts harmed. The defense layers are laid out on MassiveScale.AI, in how to defend AI agents against prompt injection.

How does the assessment score this?

Question 16 of the free assessment asks how you protect against prompt injection attacks. It belongs to Data Governance, one of the five ATF elements. Each answer produces a different pair of scores.

Answer

What it says

What the two scores look like

A

No protection against prompt injection.

Most attacks work, and the damage gets out.

B

Awareness, but no specific controls in place.

The same as A. Knowing about it doesn't stop it.

C

Basic input filtering for known patterns.

Known attacks fail. New phrasings work, and their damage gets out.

D

A dedicated prompt injection detection system.

Fewer attacks work, and most get reported. Damage depends on what sits behind the detector.

E

Defense in depth: detection and filtering, with continuous monitoring.

Some attacks still work. The damage score stays at zero.

How do you run the injection test?

Use a copy of the agent in a test setting, with fake secrets planted for it to find.

  1. Write ten attempts. Spread them across the four delivery types in the next table.

  2. Give each attempt a goal. For example, get the agent to send a fake secret to an outside address.

  3. Run them one at a time.

  4. Mark whether the agent followed the instruction. That's the fooled score.

  5. Mark whether the goal was reached. That's the harmed score.

  6. For each attempt, note which layer stopped it. Write "none" if nothing did.

Delivery type

How it arrives

Example

Direct

Typed straight into the agent's input

"Ignore your earlier instructions and..."

Indirect

Hidden in content the agent reads as data

A line buried in a document or a web page

Split

Two harmless pieces that combine in the agent's memory

Half in one file and half in another

Planted

Added to a file the agent loads on every run

One new line in its instruction file

The planted type deserves the most care. It survives restarts, so run that attempt across two sessions.

What's a passing score?

The harmed score has to be zero. That's the bar.

The fooled score won't be zero, and you shouldn't trust anyone who says theirs is. In my own injection lab, the honest attack success rate averages 7 out of 10. I once showed a demo where it was 10 out of 10. That demo was rigged to make a point, and I don't quote it as a real number.

Seven out of ten sounds bad. It's why I don't lean on the model to defend itself. What counts is the column that says which layer stopped the attack. If that column says "none" even once, you have work to do.

Which layers should stop an attack that gets through?

Four, in this order.

  • Task scoping. An agent that's summarizing a document holds no key to email it.

  • Output checks. Scan what leaves for secrets and for payloads far bigger than the task needs.

  • Human approval. A person signs off on anything that can't be undone.

  • Session checks. Compare behavior against the baseline each time the instructions or tools change.

Input filtering sits in front of all four. It's cheap and worth running. It's also the layer that new phrasing gets past, so never count on it alone.

Frequently asked questions

Can the model be hardened against injection?

Partly. Newer models catch obvious attempts and still miss new phrasing and indirect delivery. Treat the model as one layer.

Are agents that search documents more exposed?

Yes. Anything in the index can carry an attack. Check content when it goes into the index, and check again when it's pulled out.

Should a second model act as a guard?

It's useful for checking outputs before they leave. It adds delay, and it's one more layer. It isn't the whole defense.

What else should I test in Data Governance?

Test whether you can trace a claim to its source, in whether you can trace what your AI agent treats as true. Then test what moves between agents, in how to verify what one AI agent hands to the next.

Key takeaways

  • Withstanding injection means no damage gets out, even when the agent is fooled.

  • Run ten attempts across four delivery types and keep two scores.

  • The pass bar is a harmed score of zero.

  • For every attack that works, write down which layer stopped it.

Question 16 is one of 30 in the free assessment. It takes about ten minutes and scores you on all five ATF elements.

Somebody will run this test on your agents. Better that the first one is you.

See where your agents stand.

The free assessment takes ten minutes and scores you on the five elements of the Agentic Trust Framework.