An AI voice agent is software that answers and makes phone calls, understands what the caller says in natural speech, and completes the task inside your business systems. Not a menu tree. Not a recorded message. A conversation, at machine scale.
Most guides on this subject explain how a voice agent works when everything goes right. This one covers what happens when it does not, because that is where enterprise deployments actually succeed or fail. A demo call in a quiet room with a clear speaker tells you very little. A real call comes in from a moving auto-rickshaw, in two languages, with the caller interrupting halfway through your agent's sentence.
This guide is written for the person who owns average handle time, CSAT, deflection and cost per contact, and who has to defend the decision afterwards. It draws on what we see running Helo Voice across Indian enterprise call volumes, and on 25 years of Vivaconnect infrastructure behind banks including Kotak and HDFC, where the calls are regulated, multilingual and unforgiving.
Key takeaways
- A voice agent is judged on completion, not conversation. If it cannot read from and write to your systems mid-call, it takes messages rather than closing cases.
- Seven failure modes decide production success: interruption, end-of-turn detection, accent, background noise, code-mixing, API stalls and handoff. Demos are staged to avoid all seven.
- The response budget is under a second. Industry guidance puts the target near 300ms and the ceiling at 500ms before callers notice. Chat-first platforms with voice bolted on inherit a pipeline never built to that budget.
- Inbound and outbound are different products. Buying for one and discovering you need the other is a common and expensive mistake.
- Indian deployments carry a specific regime — DLT, TCCCPR, DPDP Act 2023, plus RBI and IRDAI. A platform with a perfect global compliance posture and no DLT understanding will still fail your first outbound campaign.
What an AI voice agent is
An AI voice agent handles live phone conversations the way a trained human agent would. It listens, works out what the caller wants, retrieves what it needs to know, acts on your systems, and speaks back in natural language.
The difference from earlier phone automation is not that it sounds better. It is that it can finish the job. A menu system routes. A recorded assistant reads out options. A voice agent looks up the order, changes the delivery date, confirms it aloud and writes the change back to your system before the caller hangs up.
The three layers on every turn
Speech to text (STT), also called ASR. The caller's audio is transcribed in real time. Accuracy here sets the ceiling for everything downstream. If the transcription is wrong, the reasoning is reasoning about the wrong sentence. This is the same speech-to-text layer that contact centres already use for transcription, running live instead of after the call.
The language model. It interprets intent, holds context across the conversation, decides what to say, and decides when to call one of your systems.
Text to speech (TTS). The response is synthesised as audio and streamed back to the caller.
What sits around them
Around those three sit the parts that decide whether the thing works in production: telephony that connects the call, function calling that lets the agent act, a knowledge layer that grounds the answers, and observability that tells you what happened.
AI voice agent vs IVR vs chatbot
These get conflated constantly, including by vendors who should know better. Some explainers still describe menu-driven response systems as a component of how a voice agent works, which is backwards. The agent replaces the menu; it does not sit behind it. If you are still running menus, the prior question is which menu paths to keep, fix or replace — not every path needs an agent.
Feature | AI voice agent | Menu-driven phone system | Chatbot |
|---|---|---|---|
Input | Natural speech, any phrasing | Keypad presses or fixed keywords | Typed text |
Handles the unexpected | Yes, reasons over open-ended requests | No, fails outside the script | Varies by implementation |
Interruptions | Handled mid-sentence | Not supported | Not applicable |
Context across the call | Retained throughout | None between menu levels | Retained within session |
Completes the task | Yes, acts on your systems | Routes only | Depends on integrations |
Latency pressure | Severe, sub-second required | None, caller expects menus | Low, typing indicator covers it |
Best at | Support, sales, collections, bookings | Simple deterministic routing | Web self-service, async support |
The distinction that matters commercially: a menu system measures success by whether the call reached the right queue. A voice agent measures success by whether the caller's problem was solved.
The seven things that break a voice agent on a real call
This is the section to read before you sit through a vendor demo. Every one of these is a failure we have seen in production, with the specific test that exposes it.
Demos are staged to avoid all seven. A quiet room, a clear speaker, a cooperative script. Run these tests instead and you will learn more in ten minutes than in a two-hour deck.
1. Barge-in and interruption
Callers interrupt. They start talking before the agent finishes, especially when the agent is reciting something they already know. A weak system either talks over the caller or, worse, keeps talking while the caller's audio is discarded entirely.
Good interruption handling means the agent stops speaking almost immediately, discards the rest of its planned response, and processes what the caller actually said.
Test it: start talking over the agent mid-sentence. Does it stop within a beat? Does it then respond to what you said, or to what it was already going to say?
2. End-of-turn detection
The system has to decide when you have finished speaking. Set the threshold too short and it cuts off anyone who pauses to think, which disproportionately affects elderly callers, people reading a number off a card, and second-language speakers. Set it too long and every reply feels sluggish.
This is one of the least discussed and most damaging failure modes, because it does not sound like a technical fault. It sounds like the agent is rude.
Test it: read out a long account number with a pause in the middle. Then say "my problem is..." and pause for two seconds before continuing. Does it wait?
3. Accent, dialect and regional speech
ASR trained mostly on American or British English degrades sharply on Indian English, and degrades further across regional accents within India. The gap between benchmark accuracy and real accuracy on your actual customer base is often large enough to change the business case.
Test it: put three colleagues with different regional accents on the same script. Compare transcripts, not just outcomes.
4. Background noise
Real calls come from traffic, factory floors, railway platforms and rooms with a television on. Noise degrades transcription, and cross-talk can trigger false end-of-turn detection, making the agent respond to half a sentence.
Test it: call from a busy street. Then call with a second conversation audible in the background.
5. Code-mixing
Indian callers switch languages mid-sentence, constantly and without warning. A caller starts in Hindi, uses English for the product name and the numbers, and returns to Hindi. Many systems that claim multilingual support handle one language per call and break when the language changes inside a sentence.
This is the single most common gap between what a global vendor demonstrates and what an Indian deployment needs, and it is why multilingual voice AI for Indian businesses is a different engineering problem from adding a language to a dropdown.
Test it: ask a question that mixes two languages in one sentence. Then switch language entirely halfway through the call and see whether context survives.
6. The API stall
The agent needs your system to answer. Your core banking system takes 1.8 seconds under load. The caller hears silence and says "hello?"
Handling this well means the agent covers the wait naturally, the way a human agent says "let me pull that up for you," and has a defined fallback when the system does not respond at all.
Test it: ask for something requiring a live lookup during a busy period. Ask what happens when the backend times out.
7. The handoff gap
Transfer to a human is where most deployments visibly fail. There is a silence, a hold tone, and then a person who asks the caller to explain everything again.
Test it: ask for a human. Time the silence. Then check whether the person who picks up already knows why you called.
These seven are the reason we mix speech engines per language in Helo Voice rather than forcing one engine across all of them, and why interruption handling and turn-taking sit in the core pipeline rather than in a post-processing layer. They are also the seven we would ask you to run against us.
Run the seven tests on us. Dial the demo line, interrupt it, switch language mid-sentence, then ask for a human. Hear it live →
Where the latency goes
A voice response has only a few hundred milliseconds to complete the full pipeline: speech detection, transcription, intent, retrieval, API calls, generation, speech synthesis, and network delivery.
Around 300ms feels responsive; beyond 1 second, callers notice the delay. Unlike chat, voice cannot hide latency behind a typing indicator.
That’s why buyers should ask for stage-by-stage latency on live calls, not a single headline number. If a vendor cannot show where the milliseconds go, they cannot reliably control them.
What enterprises use voice agents for
Use case | What the agent does | What it is measured on |
|---|---|---|
Inbound support | Resolves order status, account queries, policy questions and complaints end to end | Calls closed without a human, not calls answered |
Lead qualification | Answers product questions, captures details, scores intent, routes hot leads to sales | Leads routed while still warm |
Outbound campaigns | Renewals, reminders, offers and surveys, triggered from your systems | Contact rate and completion, not dials made |
Collections | Calls before the due date, explains the amount, guides the customer to payment | Payments completed |
Appointments | Books, reschedules and confirms, synced to your calendar systems | Slots held without human touch |
Verification and onboarding | Guides KYC steps, document checks and first-use setup | Completion rate under full audit trail |
After-hours cover | Answers calls outside working hours instead of routing to voicemail | Calls recovered that were previously lost |
Across all of these, the dividing line is whether the agent can act. An agent that answers from your own documents, policies and catalogues, and then calls your APIs mid-conversation to fetch the order, update the record, raise the ticket or confirm the payment, closes the case on the call. One that cannot only takes a message.
It is worth deciding which of the rows above you need closed rather than logged before you shortlist anyone. If your first target is a single high-volume support queue, an inbound voice bot for support calls is the narrower version of the same architecture and the easier place to start.
Inbound and outbound are not the same product
Most guides treat these as two bullets on one list. They are different systems with different requirements, and buying for one and discovering you need the other is a common and expensive mistake.
Aspect | Inbound | Outbound |
|---|---|---|
Who initiates | The customer, with intent already formed | You, into an unprepared moment |
Latency tolerance | Very low, the caller is waiting | Very low, plus the first three seconds decide everything |
Core constraint | Answer rate and resolution | Consent, connect rate and regulatory compliance |
Regulatory load | Moderate | Heavy, DLT registration and preference scrubbing apply |
Infrastructure needed | Concurrency at peak, queue elimination | Dialer logic, retry rules, caller ID reputation |
Failure mode | Caller abandons the queue | Number gets flagged as spam and your connect rate collapses |
Caller ID reputation deserves particular attention on outbound. Run high volumes from numbers that customers report and your calls stop connecting, regardless of how good the agent is. This is an infrastructure problem rather than an AI problem, which is why it rarely appears in AI-focused buying guides and why carrier relationships matter more than model choice here.
The practical ask is that one platform covers both directions. In Helo Voice the same agent handles inbound while campaign APIs launch thousands of simultaneous outbound calls from your own systems, so the logic and the knowledge stay in one place instead of drifting apart across two deployments.
Compliance for Indian deployments
Global guides handle this in a line: SOC 2, GDPR, done. Deploying voice in India brings a different and considerably more specific regime, and outbound is where it bites.
DLT registration
DLT registration under the TRAI framework governs commercial communication. Sender identities, headers and content templates are registered with the operators, and unregistered traffic does not get delivered. If your platform cannot handle registered templates and approved sender identities, your campaign does not run at all. We run DLT management as a managed service alongside the platform for exactly this reason.
TCCCPR and consent
TCCCPR regulations govern unsolicited commercial communication, including consent capture and scrubbing against customer preference registers before a campaign goes out. Ask how the platform handles scrubbing, and whether the audit trail proves consent per call.
The DPDP Act, 2023
The DPDP Act, 2023 sets the data protection regime: purpose limitation, consent, data-principal rights, breach notification. For voice this reaches into transcript retention, PII inside call recordings, and who in your organisation can play a call back.
Sector regulators
Sector regulators add a layer above all of it. RBI requirements on data localisation and audit reach any BFSI deployment, and IRDAI adds its own for insurance. RBI has also reiterated that customer communication should be issued in Hindi, English and the relevant regional language, which makes multilingual capability a compliance question in banking rather than only a CX preference. In practice these bind your architecture harder than anything on a vendor's trust page.
A platform with an excellent global compliance posture and no understanding of DLT will still fail your first outbound campaign. It is also why carrier flexibility is a compliance feature and not only a cost one: being able to mix providers by region, as Helo Voice does at the channel layer without touching agent logic, is what lets you satisfy a data residency or routing requirement without a rebuild. We document our own controls on the security and compliance pages, and the same regime shapes how we deploy into banking, insurance and utility operations.
The platforms worth knowing
Grouped by what they are actually built for. Trade-offs included, including where a platform beats us.
Two caveats. Published latency and pricing figures vary between sources and change often, so treat anything here as directional and get a quote modelled on your own volume. And verify capability against current vendor documentation before it goes into a business case.
Voice-first platforms
Platform | Strongest at | Watch out for |
|---|---|---|
Helo Voice | Native SIP, WebRTC and PSTN. Any carrier, mixable by region. Live transfer with no audio gap. Indian languages and code-mixing. Per-stage latency on every call. 25 years of BFSI infrastructure | More platform than a self-serve buyer needs |
Retell AI | Call quality and telephony control, SIP trunking, batch outbound, warm transfer, published per-minute pricing | Works best with developer support, lighter on broader CX |
Bland AI | API-first, programmable control, very high concurrency | Pricing harder to predict than headline rate, setup complexity |
Synthflow | Fast no-code launch, voice cloning, white-label for agencies | Cost at higher tiers, support responsiveness |
PolyAI | Ambiguity handling, accent robustness, high containment in large contact centres | Premium pricing, long implementation cycles, limited self-service |
ElevenLabs | Best-in-class voice quality and cloning, broad language coverage | Not a full telephony stack, lighter CRM and helpdesk integration |
Contact centre and CX platforms
Platform | Strongest at | Watch out for |
|---|---|---|
Sierra | Brand-aligned agents, multi-turn memory, governance and policy control | Enterprise pricing only, complex implementation |
Cresta | Real-time agent assist and coaching rather than full automation | Needs scale to pay off, per-seat cost |
Decagon | Helpdesk-native, resolves and closes tickets end to end | Support-only focus, depends on clean helpdesk data |
Replicant | Resolution-first automation for high-volume contact centres | No public pricing, enterprise sales cycle |
Regal | Outbound sales motion, lead prioritisation, multi-channel cadence | Annual contracts, needs clean CRM data |
Where we would not be the obvious pick
If you want a voice agent live this afternoon on a credit card, a self-serve platform will suit you better. If your problem is coaching human agents rather than automating calls, buy an agent-assist product. If your calls are entirely in US English and you have no India footprint, our biggest advantages do not apply to you.
What a voice agent costs
Pricing in this market is quoted three different ways, and the model matters more than the headline number because each one fails differently at scale.
Model | How it is charged | Where it bites |
|---|---|---|
Per minute | A rate per connected minute | Cheap in a pilot, climbs directly with success. Model the cost at your peak month, not your test month |
Per seat or per agent | A monthly licence per configured agent | Predictable, but reintroduces the capacity ceiling you were trying to escape |
Outcome-based | Charged per resolved contact or completed action | Aligns incentives best, but only if you and the vendor agree on what "resolved" means before signing |
Two questions cut through all three. First, what is the total cost per resolved contact, not per minute or per seat — a cheap call that resolves nothing costs you twice, because you pay for it and then pay an agent to handle the repeat. Second, what happens to the rate when volume doubles, and is that in the contract or in a conversation. Build a rough contact centre cost baseline before you take any vendor's ROI model, because without your own number you cannot check theirs.
On build versus buy: building your own stack from ASR, model and TTS components gives you full control and is defensible if you have an engineering team that wants to own a real-time pipeline. Most enterprises underestimate that the hard part is not assembling the three layers, it is the telephony, the observability and the eighteen months of tuning against real audio. Buy the pipeline, own the logic.
How to evaluate a voice agent platform
Nine criteria, in the order they should appear in your RFP.
1. Completion, not conversation
Ask for a live demonstration of the agent reading from and writing to a real system mid-call. Then ask how actions are scoped and approved per topic, because in a regulated queue you constrain what an agent may do, not only what it may say.
2. Telephony architecture
Is telephony native to the platform or bolted on at the edge? Can you connect existing carriers over SIP and keep your numbers? Can you mix carriers by region for cost, latency and compliance? Can you switch provider without rebuilding agent logic?
3. Latency, measured properly
Per-stage figures on live calls, under load, not a marketing number from a quiet test.
4. Language coverage that survives contact with your customers
Which Indian languages for recognition and for synthesis, how code-mixing is handled, and what each language costs in latency.
5. Interruption and turn-taking
Run the tests from the failure-modes section above. This is not a spec-sheet item, it is something you hear.
6. Handoff quality
Live and in-flight, or a callback? Any audio gap? Does the receiving agent get the full conversation or a summary? Our position is that anything less than in-flight transfer with the full conversation attached fails the caller, so it is worth making a pass or fail criterion rather than a scored one.
7. Observability
Full transcripts, stage-by-stage latency, structured logs, post-call webhooks into your warehouse and your dashboards. If it needs third-party tooling, that is a cost and a gap. Nothing about your customer calls should be a black box, least of all in a queue an auditor will eventually ask about.
8. Security and governance
Role-based access control, encrypted credential storage, encryption in transit and at rest, audit logs on every call and action, configurable retention, PII redaction at transcript level. This is the floor a regulated deployment starts on rather than a tier you upgrade into, and it is the list we hold Helo Voice to. Then ask the questions vendors dislike: is our data used for model training, where does it reside, and what is the audit scope behind any certification claimed.
9. Scale and pricing behaviour
Concurrency ceilings, behaviour at peak, and how cost moves as volume grows. Usage pricing is cheap in a pilot and climbs fast. Seat pricing is predictable and reintroduces the ceiling you were escaping.
Measuring a deployment
Metric | What it tells you | The catch |
|---|---|---|
Containment rate | Share handled without a human | The number vendors quote. An agent that traps callers until they hang up contains beautifully |
Resolution rate | Share where the problem was solved | Correlates with everything you care about, harder to measure, so containment gets quoted instead |
Transfer rate | How often humans are needed | Meaningless without transfer quality alongside it |
AHT | Handle time on automated calls | Falls fast; watch that resolution does not fall with it |
Per-stage latency | Where the pause actually is | Turns "feels slow" into "retrieval takes 400ms, here is the fix" |
Answer rate and abandonment | Inbound health | The first number to move, and the easiest to prove |
Connect rate | Outbound reachability | Collapses if caller ID reputation is damaged |
Cost per resolved contact | The real commercial number | Not cost per minute or per seat, those are inputs |
Track resolution and transfer quality alongside containment from the first week. A deployment tuned for containment alone looks excellent on a dashboard while CSAT falls. The same trap applies to menu systems, which is why call abandonment has to be read next to containment rather than under it.
Running a pilot that tells you something
Pick one journey with high volume and well-understood rules. Order status, balance enquiry, appointment confirmation. Resist the temptation to start with your hardest queue.
Define success before you start, in resolution and CSAT, not containment. Cap the volume, run against real traffic rather than a test bench, and listen to calls daily for the first two weeks. Not dashboards, actual audio.
Then check the seven failure modes against real recordings rather than staged ones. Expand only when the numbers hold for a full cycle including a peak day.
When you do expand, the cheapest expansion is the one that reuses the agent you have already tuned. One configuration across phone, web and WhatsApp means a prompt change or a knowledge update lands everywhere at once. In our case Helo Convo owns the WhatsApp journey and Helo Voice owns the call, sharing one spine, so the customer meets the same intelligence whichever way they reach you.
Hear it live
Dial the demo number and try to break the conversation. Interrupt it, switch to Hindi mid-sentence, pause in the middle of a number. Then ask for a human and watch the transfer. That is the whole evaluation, in about four minutes.
Frequently asked questions
What is an AI voice agent?
Software that handles live phone conversations using speech recognition, a language model and speech synthesis. It understands natural speech, holds context across the call, and completes tasks inside your business systems rather than only routing the caller.
How is an AI voice agent different from a phone menu system?
A menu system moves callers through fixed options and can only route. A voice agent lets the caller speak naturally in their own words, handles requests nobody scripted, keeps context across the conversation, and resolves the request end to end.
How fast does a voice agent need to respond?
Sub-second, and ideally closer to the two hundred millisecond gap of human conversation. Common industry guidance is to target under 300 milliseconds and treat 500 milliseconds as the ceiling before callers notice. Past one second the call feels unnatural. Past two, callers start talking over the agent.
Can it work with our existing phone system?
Voice-capable platforms connect over SIP trunking, so you keep your existing numbers and carrier relationships instead of migrating telephony. Confirm this early, because chat-first platforms typically need a separate voice layer added on, which is where latency problems begin.
Does it handle Indian languages and code-mixing?
Coverage varies widely between vendors, and code-mixing is the specific thing to test since Indian callers switch languages mid-sentence. Ask about recognition and synthesis separately, per language. Helo Voice supports multilingual speech in and out, including Indian languages, on the same agent.
What happens when the agent cannot help?
It should transfer to a human live and in flight, passing the full conversation so the customer does not repeat themselves. How cleanly a platform does this is one of the sharpest differences between vendors.
How long does deployment take?
A single well-defined journey on a platform with native connectors can be live in days. A full enterprise rollout with custom integrations, security review and compliance sign-off runs weeks to months, and the security review is usually the long pole.
How much does an AI voice agent cost?
Pricing is quoted per minute, per seat or per resolved outcome, and the model matters more than the headline rate. Per-minute pricing is cheap in a pilot and scales directly with success; per-seat pricing is predictable but caps capacity. Compare on cost per resolved contact against your current cost per resolved call, and model it at peak volume rather than pilot volume.
How do we measure ROI?
Cost per resolved contact against your current cost per resolved call, plus the revenue effects a pure cost model misses: calls answered that previously went unanswered, leads qualified while still warm, collections completed before the due date.
Should we build our own voice agent or buy a platform?
Buy the pipeline, own the logic. Assembling ASR, a language model and TTS is the easy part; the hard parts are telephony, observability and sustained tuning against real audio. Building makes sense if you have an engineering team that wants to own a real-time streaming pipeline as a product. For everyone else the platform is the faster path, provided you keep control of the conversation design and the data.
Is call data secure?
That depends entirely on the platform. Require role-based access control, encryption in transit and at rest, audit logs on every call and action, configurable retention and PII redaction at transcript level. Ask directly whether your data trains models and where it resides.
Will customers accept talking to an AI?
Broadly yes, when the agent solves the problem quickly and hands off cleanly when it cannot. Resistance comes from agents that waste the caller's time or trap them, not from the fact of automation. Disclosure practice varies by jurisdiction and sector, so check your regulator's position.
