AI voice agents: which calls they can handle today and which they can't

AI voice agents are the easiest demo to sell and the easiest project to crash. In a demo everything works: the caller speaks clearly, asks for one simple thing and never interrupts. In production someone calls from the car with the window down and says "hi, look, it's about my wife's order, the one that came in wrong" and expects the system to know what they mean. The gap between those two calls is the real distance between an AI phone support demo and a service you can leave switched on at nine on a Tuesday morning.

This guide is about that gap. About which kinds of call a voice agent can handle today, why latency outranks every other metric, how to hand over to a human without making the caller repeat everything, and what each call actually costs once you add up all the pieces.

What is an AI voice agent, and how is it different from the old phone menu?

An AI voice agent is a system that answers or places a phone call, understands what the caller says in natural language, queries your systems and replies out loud. Three pieces chained together in real time:

• Speech recognition (ASR). Turns audio into text while the person is still speaking.
• Reasoning. A language model decides what to say and, more importantly, what to read from or write to your systems. Same logic as any AI agent, with the phone as the channel.
• Speech synthesis (TTS). Turns the answer back into audio.

What separates it from a touch-tone menu ("press 1 for billing") is that there's no tree to walk: the caller says what they want in their own words. What separates it from a recorded message is that the agent can act — look up an order, move an appointment, open a ticket.

That's also the trap. An agent that speaks beautifully and touches no system is an expensive answering machine. The value isn't the voice: it's what the agent does while it talks.

If your voice agent can't read or change anything, you haven't automated a call — you've recorded a longer message.

Which calls can a voice agent handle today, and which can't it?

The right question isn't "can AI answer the phone?" but "what share of my calls has a shape an agent can close?". To answer it you have to look at your history: call types, average duration and how much data each one needs.

| Call type | Works today? | Why | |---|---|---| | Booking or moving an appointment | Yes, very well | Single intent, bounded data, queryable calendar | | Order or delivery status | Yes | One identifier and one system lookup | | Opening hours, location, terms | Yes, but cheaper to solve on the website | Pure information | | Logging an incident with structured data | Yes, with validation | The agent captures and classifies; a human resolves | | Outbound reminders and confirmations | Yes, the most profitable of all | Short call, closed script, yes/no answer | | Complaint from an angry customer | No | Needs judgement, concessions and a human tone | | Consultative selling | No | Depends on reading the person, not on fetching a fact | | Clinical or safety emergencies | No, and with a hard cut-off protocol | The cost of being wrong is unacceptable | | Anything requiring an exception | No | The agent has no authority to bend the rules |

A rule of thumb that works well when prioritising: if a brand-new human agent could resolve that call by reading a one-page script, it's a candidate. If it takes two years on the job, it isn't.

Rollout order matters too. Outbound confirmation calls (appointments, deliveries, renewals) are usually the best first case: high volume, closed script, low risk and measurable savings from week one. Inbound comes afterwards.

Why does latency decide the project?

In text, a second of waiting goes unnoticed. On a call, a second of silence is a chasm. The caller assumes the line dropped, repeats themselves, talks over the agent, and the conversation falls apart.

The budget we treat as acceptable is under 800 ms from the moment the caller stops speaking to the moment they start hearing an answer. Above 1.2 s the conversation feels artificial and abandonment climbs. That budget has to be split:

• End-of-turn detection (the system realising you've finished): 150–300 ms.
• Final transcription: 50–150 ms if it's streaming.
• Model reasoning: 200–500 ms, and it spikes if a slow system has to be queried.
• Synthesising the first chunk of audio: 100–300 ms.
• Network and telephony: 50–150 ms.

The architecture decisions that actually move that number:

• Stream everything. Transcribe while they speak, generate while it thinks, synthesise the first chunks before the sentence is finished.
• Fast queries, or deferred ones. If your ERP takes four seconds to answer, the agent can't wait in silence. Either you cache, or the agent says it's checking and fills the gap, or that lookup doesn't belong in the call.
• Barge-in. The caller must be able to cut the agent off mid-sentence and have it stop instantly. Without barge-in, any long answer becomes unbearable.
• Natural filler phrases. A well-placed "one moment, let me check that" buys 1.5 s without anyone feeling awkward. Used bluntly, it sounds like a tic.

Measuring this isn't optional. Latency is measured in production, by percentile (p50 and p95), and by segment. A bad p95 ruins the perception even when the p50 is excellent.

How do you hand over to a human without annoying the caller?

Handover is where you can tell whether the project was done properly. Three rules:

1. Explicit, generous exit criteria. The agent must transfer when: the caller asks (first time, no insisting on helping), it detects anger or vulnerability, the intent isn't one of those covered, ASR fails twice in a row, or the operation crosses a risk threshold (amount, cancellation, sensitive data).

2. Handover with context. Whoever picks up should see on screen who's calling, what they asked for, what the agent did and the transcript. If the customer has to start from scratch, you've made the service worse than the old switchboard. That requires real integration with the CRM or ticketing system — the same work we describe in integrating ERP and CRM.

3. A plan for when nobody's there. Out of hours or with a saturated queue, the agent should capture the details, open a ticket and commit to a specific callback. "I can't put you through to anyone right now" with no alternative is the worst possible experience.

And one metric worth watching from day one: first-call resolution, not containment. Containing calls is easy — just make sure the agent never transfers. What matters is how many are resolved without the person calling back within 48 hours.

What does an AI-handled call really cost?

The per-minute cost of the components is the small part, and the one everyone looks at. An honest breakdown of what you have to budget for:

| Item | Nature | Note | |---|---|---| | Telephony (numbers and minutes) | Recurring, per minute | The most predictable; depends on carrier and destination | | Speech recognition and synthesis | Recurring, per minute | Falls year on year; mainstream languages are cheap | | Language model | Recurring, per conversation | Varies a lot with how much context you pass each turn | | Integration with your systems | Upfront + maintenance | Almost always the biggest line | | Conversation design and scripting | Upfront | Business hours, not engineering hours | | Evaluation and testing with real audio | Upfront + ongoing | What nobody budgets, and what makes the difference | | Monthly supervision and tuning | Recurring | Transcript review, intent adjustment |

Two budget warnings. First: minute costs scale with volume, but integration and supervision don't — so a pilot of 200 calls a month almost never pays for itself in money. It pays for itself in learning. Second: a conversation that goes from 3 to 6 turns doesn't cost twice as much, it costs more, because every turn drags the previous context along.

To compare against alternatives, the useful calculation isn't "cost per AI call vs. cost per human call" but the usual one: baseline, total cost, and hours genuinely freed up. We cover it in measuring AI ROI and in what an AI project costs.

What does the law require, and what does common sense require?

A phone call is personal data processing, and voice is particularly delicate data. The minimum to sort out before switching anything on:

• Say it's a machine. At the start, unambiguously. Beyond being the honest thing to do, it avoids the moment the customer feels deceived and you lose the relationship, not just the call.
• Legal basis and transparency. If you record or transcribe, say so and document what for, for how long and who has access. If the voice provider processes data outside the EU, that needs looking at explicitly: we go into it in GDPR and AI.
• Don't use voice to identify people. Recognising what is said is not the same as recognising who said it. Voice biometrics is a different legal league; don't drift into it by accident.
• A human exit, always available. Beyond compliance, it's the guarantee that makes customers tolerate the system at all.
• Sectors with sensitive data. In healthcare, insurance or banking, the line on what an agent may confirm over the phone is narrower. In AI for clinics we look at that limit in more detail.

How do you build the first case without risking your reputation?

A six-week plan we've seen work, with the caveat that the weeks are indicative and the order isn't:

| Phase | What you do | What has to come out of it | |---|---|---| | Week 1 | Listen to 100 real calls and classify them | Intent map with volume by type | | Week 2 | Pick 1-2 intents and write the script | Script with exit cases defined | | Week 3 | Integrate the required lookup or write | System call working in under 1 s | | Week 4 | Test with real audio (noise, accents, interruptions) | A 50-case battery with measured results | | Week 5 | Pilot in one time slot, with human fallback | Latency, resolution and transfer metrics | | Week 6 | Tune and decide whether to scale or stop | Report with real cost per call |

The step most often skipped, and the most expensive to skip, is week 4. Testing with your own team's voices in a quiet room predicts nothing. You have to test with real recordings: background noise, hands-free, accents, people talking over each other, badly dictated numbers. And you have to test failure too: what happens if the CRM doesn't respond, if the audio cuts out, if the caller says something completely off-script.

The rest is the same as any project that changes how a team works: if the people picking up transferred calls don't understand what the agent does or how to correct it, the system quietly gets switched off within two months. That's change management, not technology.

Frequently asked questions

Can people tell they're talking to an AI?

Synthetic voice is very good now and many people don't notice in the first few seconds. What gives the system away isn't the timbre but the rhythm: long silences, no way to interrupt, and rigid answers to anything unexpected. Either way, you should say so at the start of the call.

Will a voice agent replace my phone support team?

Not in any realistic scenario. It absorbs the repetitive band — appointments, statuses, confirmations — and frees the team for calls that need judgement. Where it shows most is in peaks: the agent absorbs the surge with no queue and no temporary hires.

What happens if someone calls from a noisy street?

Transcription quality drops, and that's the first link in the chain. A well-built system detects the low confidence, asks for confirmation of critical details and transfers after two consecutive failures instead of insisting. That's why testing with noisy audio is mandatory before going live.

Is it better to start with voice or with chat?

Chat, almost always: it tolerates latency, leaves a written trail and is cheaper to correct. If you already have a chatbot with well-tuned intents, voice reuses a good share of the integration and scripting work.

If you're weighing up putting voice into your phone support, the first step isn't choosing a vendor: it's looking at your calls and knowing what share of them can realistically be automated and what integrations that demands. That's exactly what we look at in the audit, and if you'd rather talk it through — on the phone, with a person at the other end — let's talk.