How to test an AI agent before it talks to your customers

Normal software is tested by giving it an input and checking the output is exactly what you expect. AI agents do not work like that. Ask the same question twice and the wording of the answer may differ. Both answers can be right. Or one can be subtly wrong in a way no simple check would catch.

That does not make agents untestable. It means testing has to focus on behaviour: did the agent do the right thing, say true things, and stop when it should? This is how we do it.

#Unit tests still matter

Everything around the model is ordinary code and gets ordinary tests: the tools, input validation, permission checks, approval rules and integrations. If issue_refund refuses an amount above the order total, a unit test proves it, every time. Evaluation is for the part that cannot be tested this way: the model’s decisions.

#Step 1: Build the scenario set

A scenario is a short conversation with a description of what good behaviour looks like. We build the set from five sources (for a full list of categories, see our AI agent test case checklist):

• Real requests. Past emails, chats and call notes from the business, anonymised. These are the everyday cases the agent must get right.
• Variations. The same requests with typos, Banglish, voice-note transcripts, or two questions in one message.
• Missing information. No order number, an unknown product, a date that does not exist.
• Adversarial cases. Attempts to override instructions, requests for another customer’s data, abusive messages.
• Out of scope. Questions the agent should politely decline or hand to a person.

id: exchange-wrong-colour
conversation:
  - customer: "Ami blue ta order korsilam, black ashse. Friday er age change kora jabe?"
setup:
  order: { id: "A-1043", item: "T-shirt", colour: "black", delivered: "2 days ago" }
  stock: { "T-shirt/blue/M": 4 }
expect:
  tools_called: [get_order, check_stock, create_exchange]
  must_not_call: [issue_refund]
  facts: ["blue is in stock", "exchange within policy window"]
  reply_language: "same as customer"
  handover: false

#Step 2: Decide what "pass" means

Each check becomes a number on a scorecard, with its own threshold. We cover the full list, and how to set the thresholds, in AI agent evaluation metrics.

#Step 3: Automate the checks you can

Tool calls are structured, so checking them is deterministic: either create_exchange was called with order A-1043 or it was not. Factual accuracy is harder. For that we use a second model as a grader, with a written rubric and the source data in front of it. But a grader model can be wrong too, so a person regularly spot-checks its verdicts. A model is never left to mark its own work unchecked.

#Step 4: Run everything on every change

Agents can break in surprising ways. A small prompt edit to fix one case can quietly break three others. So can a new tool, an updated knowledge base, or the model provider releasing a new version. The full scenario set runs on every change, results are compared with the previous run, and anything that got worse is investigated before release.

#Step 5: Keep testing after launch

No test set covers everything real customers will say. After launch, conversations are flagged for review when the agent hands over, when a customer seems unhappy, or when a person corrects the agent’s draft. Each real failure is fixed and then added to the scenario set, so it can never quietly return. Over time the set grows into the most valuable asset of the project: a precise, written description of how the business wants its customers treated. How we log, alert and review after launch is covered in monitoring AI agents in production.

Agents that answer from documents need extra checks on search and sources: see RAG evaluation. Phone agents add speech and timing: see how to test a voice AI agent.

#What our clients see

Before launch, we share the scenario list in plain language, so the people who know the business best can add their own "what if a customer says…" cases. It turns testing from something engineers do in private into a shared definition of good service. It is a standard part of every AI agent and chatbot we build. To see where testing fits in the bigger picture, read how we build an AI agent.