How to test a voice AI agent before it answers your phone

A voice AI agent is an AI agent with a phone in front of it. Underneath, it still has to understand the request, use the right tools and know when to hand over, so everything in how to test an AI agent still applies. But voice adds problems that a chat agent never meets: there is no screen to check what was understood, the line may be noisy, and timing matters as much as content.

This article covers what changes when you test a voice agent, and the extra checks we run before one answers a real business line.

#Three layers to test

1. 01

   Speech in

   Speech recognition turns the caller’s voice into text. Did it hear the words that matter correctly?

2. 02

   Logic

   The agent decides what to do and calls its tools. Same checks as any agent: right action, right facts, right handover.

3. 03

   Speech out

   The reply is spoken back. Is it clear, correctly pronounced, short enough, and delivered at the right moment?

Test the logic layer first, with text, because it is the fastest to run and fix. Then test the two speech layers, which is where most voice-specific failures hide.

#Speech in: test the details, not the word count

Speech recognition is often scored by word error rate: the share of words transcribed wrongly. It is a useful engineering number, but it misses what matters to the business. A transcript can get 95 words out of 100 right and still miss the one digit in a phone number that makes the booking useless.

For a voice agent, the key speech metric is entity accuracy: how often it captures names, numbers, dates, times and addresses exactly right. We test these specifically:

• Phone numbers and reference numbers, said in groups, said slowly, said quickly, and with corrections ("zero one seven… no, one nine").
• Names, especially local names and spellings the recogniser may never have seen.
• Dates and times, including "the day after tomorrow", "half past four" and the local way of saying times.
• Addresses and area names, which are often the hardest words on the call.
• Different voices and conditions: older and younger callers, strong accents, a busy street, a car, a speakerphone.

#Logic: the same checks as any agent

Run the agent’s full scenario set as text: bookings, cancellations, questions, missing details, out-of-scope requests and callers who want a person. Our AI agent test case checklist covers the categories. For voice, add the cases that only happen on calls: the caller speaking to someone else in the room, a silent caller, and a caller who hangs up halfway through a booking.

#Speech out: what the caller hears

• Pronunciation of the business name, staff names, product names and local place names.
• Numbers read aloud the way people expect: prices, times and phone numbers in natural groups, not as one long figure.
• Length. A reply that reads well on screen is often too long to listen to. Spoken answers should be short, with one question at a time.
• No visual formatting. Lists, links and symbols that make sense in chat sound strange when read out.

#Timing and turn-taking

Timing is where voice agents feel most robotic, and it is easy to miss in testing because testers are patient. We check:

Response time is measured from the moment the caller stops speaking to the moment the agent starts. Look at the slowest calls, not just the average: a pause that feels small in a chat feels long on the phone, and it is the slow moments that make callers hang up or start talking over the agent.

#Handover and the end of the call

• Transfers work on the real phone system, to the right number, during and outside office hours.
• Call-backs are recorded with the caller’s number, name and reason, and reach the right person.
• The summary is accurate. Compare the summary sent to the CRM or inbox against the actual call.
• Urgent calls always go to a person, whatever else is happening, with no menu to get through first.

#How to run voice tests

Use them together. Automated runs catch regressions on every change; live calls from ordinary mobile phones catch what lab tests miss. We always finish with staff calling the real number from their own phones, in the situations customers will be in.

#A note on Bangla and mixed-language calls

Recognition and voice quality vary by language, accent and phone line. For Bangla and mixed Bangla-English calls, we test with the business’s own call scenarios during discovery and say plainly what works before anyone commits. If names or numbers are unreliable in a language, the agent can confirm them by SMS or hand those calls to a person.

#After launch

Review a sample of real calls every week at first: transcripts next to recordings, so you can see what was misheard. Every misheard detail and awkward pause becomes a new test case. How to set this up is covered in monitoring AI agents in production.

This testing is part of every AI voice agent we build. If your business misses calls while the team is busy, that page explains what a voice agent can take off your hands.