Every voice AI demo is rehearsed, so it shows the vendor at its best. If you are still shortlisting, our roundup of the Best AI voice agent platform options covers that stage. This voice AI vendor evaluation checklist starts after the shortlist, with 20 questions you can check rather than trust.
A vendor answer counts only when you can point to its proof. That proof comes from a document, your own test calls, or a signed clause. Each question below names its proof source, so you know when to ask it.
The stakes are rising because vendors now pitch voice agents as a core channel, not an experiment. Gartner predicts that by 2029, agentic AI will resolve 80% of common customer service issues without a human. However, that forecast covers common issues, and your edge cases decide whether a vendor works for you.
What Counts as Proof in a Voice AI Vendor Evaluation Checklist
Proof is evidence you can inspect yourself instead of taking a salesperson’s word. It comes in three grades, and each grade outweighs the one before it. Here is how they differ during a vendor review:
Paper proof: a dated document, such as a SOC 2 report, data map or sample invoice.
Pilot proof: behavior you observed on your own scenarios, phone lines and caller data.
Contract proof: a signed clause that makes the vendor liable if the behavior changes later.
Most buying mistakes start when a team scores a demo like pilot proof. A demo runs on the vendor’s script, audio and data, so it shows what is possible. It cannot tell you what happens when your callers phone in from a moving auto-rickshaw.
A useful voice AI buyer guide should therefore tell you which grade each answer needs. Some questions only need paper, like where recordings are stored. Others, like latency or interruption handling, mean nothing until you hear them on your own calls.
Inside One Voice AI Call: Six Stages Every Vendor Must Own
A voice agent call is a relay race between six systems. A weak handoff at any stage shows up as a pause or a wrong answer. Knowing the stages tells you which claims to test, in this order:
Telephony: the call arrives over a carrier or SIP trunk, usually as narrowband phone audio.
Turn detection: voice activity detection decides when the caller has stopped or is only pausing.
Speech-to-text: the recognizer turns audio into words, including names, amounts and dates.
Reasoning and tools: a large language model picks a reply and may call your CRM or calendar.
Text-to-speech: the reply becomes audio, and streaming lets playback start before the sentence finishes.
Logging and handoff: the platform records each step and passes context to a human on transfer.
The phone line matters more than most buyers expect. Standard telephony under the ITU-T G.711 codec samples audio at 8 kHz and carries roughly 300 to 3,400 Hz. That is far thinner than browser audio, so flawless online recognition can slip on a real call.
Latency also stacks across stages, because each one adds its own delay. A vendor quoting a fast speech model may still be slow from end to end. Use this voice AI vendor evaluation checklist to test the whole relay, never a single stage in isolation.
Write Your Knock-Out Rules Before Any Voice AI Vendor Demo
Decide what disqualifies a vendor before you see any demo, because a charming voice makes teams forgive real gaps. Most advice on how to choose voice AI starts with features, but constraints should come first. Write down three to five knock-out rules that end the conversation, such as these:
Call recordings must stay in a region your regulator or largest client accepts.
The agent must handle every language that carries more than a tenth of your volume.
The platform must write results back to your CRM, not just read from it.
The vendor must sign your data processing agreement without stripping out deletion terms.
Pricing for your expected volume must arrive in writing before the pilot starts.
Keep the list short, because every rule removes vendors from the race. Then put these rules and the 20 questions into your voice agent RFP. Every vendor then answers identical prompts in writing, which gives you paper proof to check against the pilot.
The 20 Questions Mapped to Where Each Gets Proven
The tables below condense this voice AI vendor evaluation checklist into a printable page for vendor meetings. Each row states what you are really asking and what evidence settles it. The sections after the tables explain what strong and weak answers look like.
Questions 1 to 10: How the Voice AI Vendor Handles Calls and Data
# | What you are really asking | What proves it |
1 | Recognition accuracy on your recorded calls, split by language and entity type | Scored transcripts from your own redacted audio |
2 | Response gap at the 95th percentile on a live phone line | Timing logs from pilot calls placed in your region |
3 | Agent behavior when callers interrupt or make filler sounds | Pilot recordings where testers cut in mid-sentence |
4 | Agent behavior when approved knowledge has no answer | Test calls asking for prices it was never given |
5 | Languages tested on phone audio, with call counts for each | Per-language pass rates from your scenario set |
6 | Context a human receives during a transfer | A live warm transfer your own staff receives |
7 | Call handling when your CRM or calendar fails mid-call | A pilot call with the integration switched off |
8 | Carrier choice and phone number ownership, often a knock-out | Written porting and bring-your-own-carrier terms |
9 | Storage region, retention and model training on your calls, often a knock-out | A data map plus the signed processing agreement |
10 | Certification scope covering your product and hosting region | The current audit report, checked for scope and date |
Questions 11 to 20: Compliance, Cost and the Exit Path
# | What you are really asking | What proves it |
11 | Who owns AI disclosure, consent and caller-ID declarations | A responsibility matrix attached to the contract |
12 | Tracing one call from transcript to every tool action | An audit export you pull without vendor help |
13 | Cost lines missing from the advertised per-minute rate | An itemized sample invoice with names removed |
14 | Rounding of part-minutes, minimums and concurrency fees | Billing terms written into the order form |
15 | Cost per resolved call at your current volume | Pilot resolution rate applied to quoted real rates |
16 | Price changes at renewal and at volume breakpoints | A renewal cap and tiered rate table in the contract |
17 | Concurrency limits, uptime history and incident handling | A status page plus two post-incident reports |
18 | Protection against model updates breaking your agent | A change-notice clause and a rerun regression suite |
19 | Ownership of call-flow edits and support response times | Support terms with response targets by severity |
20 | Data, configuration and numbers you keep after exit, often a knock-out | An export you test before signing, plus exit clauses |
Questions 1 to 4: Will a Voice AI Vendor Survive Your Real Calls?
These four questions test conversation quality, the area where demos flatter vendors most. Every one needs pilot proof, because no document shows how an agent sounds on your lines. Weight this group heaviest if callers judge you mainly on the phone experience.
1. Will you measure recognition on our recorded phone audio?
A headline accuracy figure usually comes from clean, wideband test sets that look nothing like your calls. What you need is accuracy on your audio, split by language and entity type. Names, amounts and dates matter most, because one wrong digit can ruin an otherwise perfect call.
A strong vendor runs 50 or more of your redacted recordings and shares the scored transcripts. A weak one sends a single word error rate with no test details attached. Only pilot transcripts settle this question, so do not score it from a slide.
2. What is your 95th-percentile response gap on a live phone line?
Human talk runs on tight timing, so a slow reply feels broken even when it is correct. Conversation research led by Tanya Stivers, published in PNAS in 2009, found speakers in ten languages avoid long silences. A 2015 review by Levinson and Torreira puts the typical gap between turns near 200 milliseconds.
Few voice agents match that, but the 95th percentile shows how bad the slow calls get. Ask for p50 and p95 timings measured on phone calls from your region, not a browser average. Our breakdown of Sub-second AI voice agent latency explains where each millisecond goes.
3. How does the agent react when a caller talks over it?
Real callers interrupt, mutter agreement and pause to find a reference number. Barge-in handling decides whether the agent stops gracefully or keeps talking over them. Poor turn detection also fails the other way, cutting in whenever the caller takes a breath.
Test it by having a tester interrupt mid-sentence, cough, and go silent for four seconds. A good agent stops within a beat, picks up the new point and waits through the silence. Instead of trusting a demo, watch for agents that restart their whole sentence after every small noise.
4. What happens when the answer is not in its knowledge base?
A language model can fill gaps with something plausible unless the vendor restricts it. For a voice agent, that might mean inventing a refund policy, a free slot or an amount due. Hallucination control is therefore a safety question first and a polish question second.
Ask testers to request prices, dates and policies the agent was never given. The right behavior is a clear admission plus a transfer, callback or text with the correct link. For regulated work, one invented figure in twenty calls is enough to fail this test.
Questions 5 to 8: Languages, Handoffs and the Systems Behind the Call
This group checks whether the agent fits your callers and your existing stack. Most answers need pilot proof, though paper settles number ownership. A gap here rarely shows in a demo, yet it decides how much work your team inherits.
5. Which languages have been tested on phone lines, and with how many calls?
Listing 40 supported languages is easy, but tested languages make a much shorter list. Ask how many phone-audio test calls back each language you need. In India, also ask about code-switching, where a caller drops English amounts into a Hindi or Gujarati sentence.
Strong vendors share pass rates per language from structured test sets. Weak vendors point to the speech model’s language list and stop there. Run your own scenario set in each language, because an average can hide the one language that fails.
6. Does a human agent inherit the full context on transfer?
Some calls need a person, and a blind transfer makes the caller repeat everything. A warm transfer should pass the summary, verified details and the reason for escalation. Ideally, your staff sees this in the CRM or agent desktop before saying hello.
Ask what triggers a transfer, such as anger, a repeated request or a policy exception. Then receive a live transfer yourself during the pilot and judge what arrives. If your agent must ask the caller’s name again, the handoff failed.
7. What happens to the call if your CRM or calendar times out?
Third-party systems time out, and the caller should not feel it when they do. Ask whether the agent can both read from and write back to your CRM. Read-only integrations leave staff copying call outcomes by hand, which defeats much of the point.
During the pilot, switch off the calendar connection and place a booking call. A well-built agent explains the delay, captures the request and confirms a callback time. A weak one invents a confirmed slot, which is worse than a dropped call.
8. Can we bring our own carrier and keep our numbers?
Numbers rented through a platform often belong to that platform or its carrier. If you leave, porting them out can be slow or impossible, even though customers saved those numbers. So ask whether you can connect your own SIP trunk and keep numbers in your company’s name.
Number reputation matters too, because Indian operators now screen automated traffic closely. TRAI cited data showing operators’ AI systems flagged about 771 crore calls as suspected spam in April 2026. Ask how the vendor monitors spam labels on your numbers and how fast it reports them.
Questions 9 to 12: Security and Compliance Proof to Demand from a Voice AI Vendor
Security and compliance answers start on paper and end in the contract. Treat questions 9 and 11 as likely knock-outs for regulated industries. Bring your security and legal reviewers in early, since their review often sets the timeline.
9. Where do recordings live, for how long, and do they train your models?
Voice calls carry names, account details and sometimes health or payment information. You need the storage region, the retention period per data type and every subprocessor that touches audio. Also ask in writing whether your calls train the vendor’s systems or any model provider’s.
India’s DPDP Rules, notified in November 2025, phase in full obligations by May 2027. Vendors serving Indian callers should explain how they will support consent records, deletion and breach reporting. Otherwise, a vague promise of compliance is paper with no proof behind it.
10. Which certification covers the exact voice AI product and region we are buying?
A SOC 2 or ISO 27001 logo proves little until you read the report’s scope. Check that the audit includes the voice product, the hosting region and the model providers in your data path. Then check the report date, since an expired report describes the past.
Ask for role-based access controls, PII redaction in transcripts and the incident notification window in hours. For a deeper list of controls, see our guide to AI voice agent security for enterprises. Your security team should sign off before any real customer audio enters the pilot.
11. Who handles AI disclosure, consent and caller-ID registration in each market?
Calling rules differ by country, so the vendor’s role in meeting them belongs in writing. In the US, the FCC ruled in February 2024 that AI-generated voices count as artificial voices under the TCPA. As a result, consent and caller identification duties apply to many outbound campaigns.
In India, TRAI’s 2026 amendment makes automated-call senders declare that use and their numbers to operators. TRAI treats undeclared automated calls as unsolicited commercial communication. Ask who files that declaration, and on whose numbers.
12. Can we audit any single call from transcript to tool action?
When a customer disputes what the agent said, you need the full record within minutes. That means the recording where permitted, the transcript, each tool call and the data the agent saw. Without that trail, your team cannot tell a model error from a CRM error.
During the pilot, pick a random call and try to rebuild it yourself from the dashboard or export. If you need the vendor’s engineers to do it, expect slow dispute handling in production. Score this on what you can do alone, not on what support can do for you.
Questions 13 to 16: What a Voice AI Vendor Costs per Resolved Call
The per-minute rate is the number vendors advertise, but cost per resolved call is the one you pay. These four questions move the conversation from headline price to outcome cost. Answers start on paper and should end as clauses in the order form.
13. Which cost lines does your per-minute rate leave out?
A voice agent minute bundles several costs: speech-to-text, the language model, text-to-speech, telephony and number rental. Some vendors include all of them, while others quote only the platform fee. Ask for an itemized sample invoice so you can see what a real month looked like.
Published rate cards help here, because they let you compare line items before any sales call. For example, OnDial pricing shows how one provider lays out its plans for buyers to review. Whatever rate card you use, match its lines against the vendor’s sample invoice.
14. How do you round part-minutes, and what minimums apply?
Billing increments quietly change your bill on short calls like reminders and confirmations. A 75-second call costs 1.25 minutes under per-second billing but 2 minutes under per-minute rounding. That is a 60% markup on the same conversation.
Also ask about monthly minimums, unused-minute expiry and concurrency fees. These may not appear on a pricing page, yet they can appear on an invoice. Get each one written into the order form before the pilot ends.
15. What would our cost per resolved call be at today’s volume?
This number decides most deals, so calculate it yourself rather than accepting a vendor’s estimate. Take your monthly call volume and average call length, then apply each vendor’s real rate and pilot resolution rate. The hypothetical example below shows why the cheapest rate can lose.
Hypothetical input: 10,000 calls averaging 2 minutes 15 seconds | Vendor A | Vendor B |
Advertised rate per minute | ₹7.00 with telephony included | ₹5.00 with telephony billed on top |
Real rate once every cost line is added | ₹7.00 per minute | ₹6.50 after ₹1.50 telephony |
Billing method and minutes billed per call | Per second, 2.25 minutes | Rounded up, 3 minutes |
Monthly bill for all 10,000 calls | ₹1,57,500 | ₹1,95,000 |
Containment, or calls resolved without a human, in the pilot | 62%, or 6,200 calls | 48%, or 4,800 calls |
Cost per resolved call | ₹25.40 | ₹40.63 |
Vendor B looks about 29% cheaper on its pricing page, yet each resolved call costs 60% more. The gap comes from telephony, rounding and a weaker resolution rate, and no rate card shows these. Run this math with your own pilot numbers before you compare quotes.
16. How can our price change at renewal or at volume breakpoints?
Year-one pricing is often a discount, and renewal is where budgets break. Ask for a cap on renewal increases and the exact volumes where rates drop. Also ask what happens to your rates if the vendor’s model provider raises prices.
Strong vendors put a renewal cap and a tiered rate table in the contract. Weak vendors promise to keep pricing fair and leave it there. Fairness is not a clause, so push for numbers.
Questions 17 to 20: Life After Go-Live and the Way Out
A voice agent keeps changing after launch, because models, prompts and your own policies change. These four questions cover who keeps it working and how you leave if it stops. They are easy to skip during a busy evaluation, yet costly to discover late.
17. What uptime and incident history can the voice AI vendor show?
A promised SLA tells you the goal, while incident history tells you the track record. Ask for a public status page, twelve months of incidents and two written post-incident reports. Also ask the highest number of concurrent calls they have tested with an agent like yours.
Next, find out what happens when a speech or model provider in their stack goes down. Good answers name a fallback provider or a graceful message plus a callback. If live calls simply drop, plan your own fallback before launch.
18. How do you stop a model update from breaking our tested agent?
Vendors swap models, voices and prompts to improve the product, and each change can shift behavior. An agent that passed your pilot in March can fail the same test in June. Silent changes are hard to spot, because calls still sound fine while accuracy drifts.
Ask for advance notice of model changes and the right to pin a tested version. The vendor should also rerun your scenario suite after each change and share the results. Without both, you are buying an untested system every time they ship an update.
19. Who edits the call flow after launch, and how fast does support respond?
Your scripts will need edits once real calls reveal gaps. Find out whether your team can change prompts and flows itself, or whether every edit needs a ticket. Then ask what each change costs if the vendor does it for you.
Support terms should list response targets by severity, with a named contact for outages. Ask for two references in your industry, ideally running your languages. A Hindi collections reference tells you far more than a dental clinic in Ohio.
20. What can we take with us if we leave the voice AI vendor?
Exit terms decide how trapped you become, so test them while you still have leverage. You should be able to export recordings, transcripts, prompts, call flows and analytics in usable formats. Phone numbers and any custom voice should also stay yours if you paid for them.
Run a small export during the pilot and open the files without vendor help. Confirm in writing how the vendor will delete your data after exit and when you get proof. Until you test it, an exit promise is still only paper.
Scoring Each Voice AI Vendor: Knock-Outs First, Weights Second
Score in two passes: remove any vendor that fails a knock-out rule, then weight the rest. A single average lets a beautiful voice hide a deal-breaking data gap. Two passes stop that from happening.
Rate each question from 0 to 5, using your evidence rather than the vendor’s confidence. Unverified paper answers can earn at most 3 points until a test or clause confirms them. Then average each group and apply weights, as in this hypothetical comparison:
Question group and weight | Vendor X | Vendor Y | Vendor Z |
Real-call quality, questions 1 to 4, weighted at 30% | 4.5 | 3.5 | 5.0 |
Languages, handoffs and systems, questions 5 to 8, weighted at 20% | 3.5 | 4.5 | 4.5 |
Security and compliance, questions 9 to 12, weighted at 20% | 4.0 | 4.5 | Failed the data-region knock-out |
Cost per resolved call, questions 13 to 16, weighted at 15% | 3.0 | 4.5 | 4.0 |
Operations and exit, questions 17 to 20, weighted at 15% | 4.0 | 4.0 | 4.5 |
Weighted total out of 5 | 3.90 | 4.13 | Removed before weighting |
Vendor Z had the best-sounding agent, yet it never reached the weighted round. Vendor Y wins overall, and its 3.5 on call quality clears the floor. Set that floor in advance: any group below 3 means a retest before signature.
A voice AI vendor evaluation checklist only works when its weights match your risks. For example, a clinic booking line might raise the systems weight because calendar write-back is the whole job. A lender running collections might instead raise compliance to 30% and lower cost.
How to Evaluate AI Calling Vendor Pilots on Equal Terms
A fair pilot gives every finalist the same scenarios, the same phone lines and the same pass rules. If one vendor gets your easy calls and another gets the hard ones, the scores mean nothing. Follow these steps to keep the comparison clean:
Write 30 to 50 scenarios from real call types, including angry, noisy and off-script callers.
Set pass rules for each scenario in advance, and keep the scenarios hidden from vendors.
Place every test call over real phone lines from the regions your customers call from.
Score transcripts blind where possible, so reviewers do not know which vendor they are judging.
Finish with a live slice of real traffic, capped at a small share of daily calls.
Compare results against your human baseline for resolution, handle time and complaints.
Plan for two to three weeks so the live slice covers busy and quiet days. Keep the pilot to two or three finalists, because each extra vendor multiplies scoring work. Every recorded call then becomes evidence for the pilot rows of your voice AI vendor evaluation checklist.
Clauses That Turn Pilot Results into Promises
A pilot proves behavior today, but only the contract protects it tomorrow. Before signing, convert your most important pilot results into named clauses. These six cover the costliest gaps this checklist flags:
Advance notice of model, voice or prompt changes, plus the right to pin a tested version.
Response targets by severity, with service credits and a named escalation contact.
A renewal price cap and a written rate table for each volume tier.
Data deletion within a set number of days after exit, backed by a written certificate.
Full export of recordings, transcripts and call flows in standard formats on request.
Number porting support and confirmation that numbers bought in your name stay yours.
Attach the vendor’s written answers to the contract wherever they influenced your decision. Otherwise, a promise from the sales stage may carry no legal weight. Your legal team decides the final wording, but these clauses give them a solid starting list.
What Voice AI Agents Still Struggle With, and Why Honest Vendors Say So
Even strong voice agents have limits, and a vendor that admits them is usually easier to trust. Scoring honesty down only pushes vendors to overpromise. Expect many current platforms to struggle with some of the following:
Heavy background noise combined with strong regional accents, especially on weak mobile networks.
Long calls with several unrelated issues, where context can drift after many turns.
Emotionally charged complaints, bereavement or hardship calls that need human judgment.
Policy exceptions, negotiations and any promise your rules do not already cover.
Long strings of digits, such as account or policy numbers read in one breath.
None of these rules out voice AI, but they shape where you deploy it. Route these call types to people, and judge vendors on how cleanly they hand them over. A vendor claiming no limits at all has probably not tested hard enough.
Voice AI may also be the wrong fit if most of your calls are complex disputes. In that case, an assist tool for human agents may pay back faster than full automation. No voice AI vendor evaluation checklist can rescue a use case that should stay human.
Your Next Step After the Scorecard
Picking a vendor gets easier once every answer in your voice AI vendor evaluation checklist has a proof source. Score your finalists, run the pilot, then lock the results into clauses. To get OnDial’s written answers to all 20 questions, Contact sales and request a pilot.
