AI agent evaluation metrics: what to measure before and after launch

"Is the agent good enough to launch?" sounds like a yes-or-no question. In practice it is five or six separate questions. Does it finish the job? Does it use its tools correctly? Does it say only true things? Does it refuse what it should refuse? Does it hand over to a person at the right moment? And can the business afford to run it?

Each of those needs its own number. This article lists the AI agent evaluation metrics we use, what each one means in plain words, how it is measured, and how we combine them into a clear go or no-go decision. For the testing process that produces these numbers, read how to test an AI agent before go-live.

#Why one accuracy score is not enough

Imagine an agent that answers 95% of customer questions well. That sounds ready. But if the other 5% include refunds sent to the wrong customer, a blended score hides the only failures that matter. The opposite also happens: an agent that hands every hard question to a person can look very safe while saving the team almost no time.

So we never average quality into one figure. We keep a short scorecard where each metric is tracked on its own, and some failures are serious enough to block a release by themselves.

#The AI agent metrics that matter

#Quality metrics to measure before launch

These come from running the agent against a fixed set of test scenarios: real requests, tricky variations, missing information and deliberate attacks (our AI agent test case checklist lists them). Because the set is fixed, the numbers can be compared from one release to the next.

Task success rate

Task success rate is the share of scenarios in which the agent reached the correct end result. "Correct" is defined per scenario before the test runs: the exchange was created, the booking was made, or the right answer was given. A polite reply that does not solve the problem counts as a failure. This is the headline metric, but it only means something next to the others.

Tool-call accuracy

Tool-call accuracy measures whether the agent called the right tools, with the right inputs, and avoided the tools it must not use. It is the easiest metric to measure, because tool calls are structured data in the logs. We track three parts separately: missing calls (it forgot to check stock), wrong inputs (it looked up the wrong order) and forbidden calls (it issued a refund when the scenario says it must not). Forbidden calls are always treated as critical.

Groundedness and hallucination rate

Groundedness is the share of answers in which every factual claim is supported by the data or documents the agent was given. Its opposite is the hallucination rate. We check it with a second model acting as a grader, which gets the answer, the source material and a written rubric. A person reviews a sample of its verdicts, because a grader model can be wrong too. For agents that answer from documents, this is the metric that decides whether people can trust the answers. Our article on RAG explains how answers are tied to sources, and RAG evaluation covers how to test them.

Safety and policy compliance

Safety has two sides, and both need measuring. Under-refusal is doing something it should have refused: sharing another customer’s data, following instructions hidden in an email, or going past a spending limit. Over-refusal is refusing a normal, allowed request. Under-refusal is a release blocker. Over-refusal is not dangerous, but it quietly destroys the value of the agent, so it gets a threshold of its own. The controls behind these checks are covered in guardrails for AI agents.

Handover accuracy

Handover accuracy measures whether the agent passed the conversation to a person exactly when it should. We split it in two. Missed handovers (an angry customer or a complaint about safety that the agent kept handling) are serious. Unneeded handovers (a simple question it could have answered) waste staff time. Tracking both stops the agent from "fixing" one problem by creating the other.

Consistency across repeated runs

An agent can pass a scenario once and fail it on the next try, because its answers are not fixed. So we run each scenario several times and count it as passed only if it passes every time. A case that passes three runs out of five is not 60% fine. It is a failure that one customer in a few will meet.

#Production metrics to track after launch

Once real customers arrive, the scenario set is no longer the whole picture. These metrics come from live traffic and show whether the agent is actually helping the business. How to collect them is covered in monitoring AI agents in production.

• Resolution rate. The share of conversations where the customer’s problem was actually solved, confirmed by the outcome (order changed, booking made) or by the customer not coming back about the same thing.
• Human correction rate. How often staff edit the agent’s drafts, undo its actions or reopen its tickets. When this rises, quality has slipped, even if customers have not complained yet.
• Escalation rate. The share of conversations passed to a person. A sudden change in either direction is a signal worth investigating.
• Response time. Time to the first reply and to a finished task. We look at the slowest replies (for example the slowest 5%), not only the average, because that is where customers give up.
• Cost per resolved task. Model and tool costs divided by tasks actually resolved, not by messages sent. An agent that is cheap per message but needs many turns can cost more than it looks.
• Customer feedback. Ratings and complaints, read alongside the numbers above rather than on their own.

#Turning metrics into a release gate

Metrics are only useful if they decide something. Before a project starts, we agree thresholds with the business, mark which scenarios are critical, and write them down as a release gate. Every change to prompts, tools, documents or the model has to pass the gate before it goes live.

release_gate:
  critical_scenarios:
    max_failures: 0          # money, personal data, safety
  thresholds:
    task_success: ">= 0.85"
    tool_call_accuracy: ">= 0.95"
    forbidden_tool_calls: 0
    groundedness: ">= 0.90"
    missed_handovers: 0
    over_refusal: "<= 0.05"
  consistency:
    runs_per_scenario: 3
    must_pass: all
  regression:
    worse_than_previous_run: block

The regression rule matters as much as the thresholds. A prompt edit that fixes one case and breaks two others can still leave every number above its threshold. Comparing each run with the previous one catches that before customers do.

#A starter scorecard for your first agent

You do not need every metric on day one. If you are evaluating your first AI agent, start here:

1. 01

   Write 30 to 50 scenarios

   Take them from real requests, and include a handful of missing-information and adversarial cases. Mark the critical ones.

2. 02

   Track four numbers

   Task success, forbidden tool calls, groundedness and missed handovers. These cover most of the risk.

3. 03

   Agree the gate

   Zero critical failures, a threshold for each number, and nothing worse than last time.

4. 04

   Add production metrics at launch

   Resolution rate, human correction rate and cost per resolved task, reviewed weekly at first.

#How we use this at Agentiq BD

Every AI agent and chatbot we build ships with a scenario set, a written release gate and a weekly scorecard for the first weeks after launch. Clients see the scorecard in plain language, so the decision to go live is made on evidence, not on a good demo. To see where evaluation fits in the whole project, read how we build an AI agent.