How to test your AMD accuracy before going live

Why you can’t trust AMD accuracy from the dashboard alone

ViciDial’s AMDSTATUS field records one of three values for every call: HUMAN, MACHINE, or NOTSURE. What it does not record is whether that classification was correct. The dashboard shows you what the AMD engine decided — not whether it was right.

This is the core measurement problem: the only calls you see in your reports are the ones the system flagged one way or another. The calls flagged MACHINE were typically abandoned without connecting to an agent. If any of those were live humans, they got dropped and you got no data on them except a disposition of “AMD” or “Machine” in the agent screen. No one listened. No one complained (they just stopped answering). And your accuracy stats stayed clean because they only count the cases you handled well.

The only way to know your real false-positive rate on live humans is to pull recordings of calls flagged MACHINE and listen to them. It’s tedious, which is why almost nobody does it. It’s also the most important 30 minutes you can spend before you change anything about your AMD configuration.

The 50-call manual audit method

You don’t need a large sample to get a directionally accurate read. Fifty calls flagged MACHINE is enough to spot a problem. Here’s how to run the audit:

1. Pull the call list. In ViciDial reports, filter by AMDSTATUS = MACHINE for the last 3–7 days. Export 50–100 rows. Pick calls spread across campaigns and time-of-day buckets if possible; don’t just grab the first 50 from a single campaign at 9 AM.
2. Locate the recordings. ViciDial logs the recording path in the vicidial_log table (recording_filename). If your server stores recordings, you can listen directly. Otherwise pull from your archive server.
3. Listen and tally. For each recording, mark it as one of: actual voicemail greeting, carrier intercept / IVR, carrier false-answer (call was “answered” but there’s no audio or just a beep), or — the one you’re looking for — a live human saying “Hello?” before the line goes dead.
4. Count your false positives. Any recording where a live human clearly answered is a false positive. Divide that count by your total sample.

What a bad number looks like: more than 5% of your MACHINE-flagged calls being live humans is a serious problem. At typical dialer volumes that’s dozens of real prospects dropped per hour. Below 2% is acceptable for heuristic AMD. Below 0.5% is what acoustic AI AMD should deliver.

If your sample turns up 10–15% false positives — which is not unusual on stock Asterisk AMD with a fast-answering mobile-heavy list — you now have a concrete number to take into a configuration change or a provider evaluation.

The NOTSURE rate as a proxy for accuracy

You don’t always need to listen to recordings. Your NOTSURE rate is a faster proxy. Asterisk returns NOTSURE when the AMD engine can’t commit to a classification within the configured timing window — typically because the audio in the first two seconds didn’t fit a clean voicemail or a clean greeting.

A high NOTSURE rate means your AMD is sitting in the gray zone on a large fraction of calls. Those are exactly the calls most likely to contain live humans who answered fast, said something short, or picked up in silence. Your dialplan decides what to do with NOTSURE — and most default configs either drop the call or treat it as MACHINE. Either way, the humans in that bucket are getting lost. See how NOTSURE routing quietly drops your best leads →

If more than 8–10% of your answered calls are coming back NOTSURE, that’s a red flag. Benchmark it against a test window using a different AMD engine and compare the NOTSURE rate directly. The one that moves more calls out of NOTSURE and into HUMAN (without inflating MACHINE false positives) is the more accurate engine.

Testing carrier false-answers

Carrier false-answers (FAS) are a specific class of misclassification that your MACHINE recordings won’t reveal the usual way. A FAS call is flagged MACHINE or NOTSURE because the call was technically “answered” by the network — but there was no one there. The carrier sent an answer signal without connecting a real destination.

The diagnostic is timing. Pull your MACHINE-flagged calls and look at the classification timestamp. Any call that came back MACHINE in under 400 milliseconds is almost certainly a FAS — no voicemail greeting starts that fast. If you hear dead air or a brief tone when you listen, that confirms it. Full breakdown of carrier FAS and why stock AMD can’t catch it →

Stock Asterisk AMD returns MACHINE on most FAS calls because it hits the silence threshold immediately. AI AMD can classify these as a distinct FAS category, which matters for billing reconciliation — you may be paying your carrier for connections that never happened.

Setting up an A/B comparison on live traffic

Once you have a baseline from the manual audit, the most rigorous test is a live A/B split: send a portion of your traffic through your current AMD and a portion through AMDY, then compare the key metrics side by side.

In ViciDial you can split traffic at the campaign level. Create a duplicate of your active campaign, point its AMD method to AMDY (WebSocket to your AMDY endpoint), and assign 20–30% of your leads to that campaign. Let it run for 24–48 hours, then pull reports on both campaigns and compare the metrics in the table below.

Keep the campaign type, calling hours, and list composition identical. The only variable you want changing is the AMD engine. If you can run both campaigns against the same lead file (even-odd split, or alternating rows), the comparison is cleaner.

Metrics to track — and what good looks like

Agent idle time is an underrated signal. If AMD is dropping a lot of live humans, your agents spend more time in the connected-but-silent gap (the AMD is still deciding or just dropped someone). Agents in better AMD environments typically have noticeably higher talk-time-per-hour and lower idle seconds-per-call. Pull this from your ViciDial agent time reports — it doesn’t require listening to a single recording.

Before you change anything

The 50-call audit takes under an hour and gives you a concrete false-positive percentage to anchor every subsequent decision. Run it before tuning amd.conf, before switching providers, and before claiming your AMD is “working fine.” The number might surprise you.

If you find your false-positive rate is above 5% and tuning hasn’t moved it, that’s the ceiling of heuristic AMD on modern mobile traffic — not a configuration problem you can solve by adjusting silence thresholds. That’s where acoustic AI AMD changes the math.