Two phone validation APIs disagree with each other 27% of the time

Every phone validator claims to be accurate. Almost none publish a number, and the ones that do never say how they measured it.

So we measured ours, wrote the method down first, and published everything, including the run where we came out badly.

The finding that has nothing to do with us

We compared every number against two independent commercial references: Twilio Lookup and Veriphone. On the 513 numbers where all three of us returned a classification:

Two paid APIs disagree with each other about 27% of US numbers. Ask them whether a number is mobile, landline or VoIP and more than one in four times you get different answers.

That reframes every accuracy claim in this category, ours included. There is no oracle. When a vendor tells you they are 99% accurate, the honest question is "against what?", because the two most obvious answers disagree with each other a quarter of the time.

What we found about our own data

Splitting the disagreements three ways tells you what they mean:

Where both references agree on an answer, we match them 98% of the time. The 27% is not our error and not theirs; it is a question the industry cannot currently answer.

The headline numbers

Mobile recall is the number that matters for an SMS campaign: of the numbers a reference calls mobile, we find 96 in every 100.

The run where we came out badly

The first benchmark returned 66% agreement with Twilio, below what we had been claiming. We published the method before running it, so there was no version of this where the result quietly disappeared.

The disagreements were not scattered. 108 of 124 ran in one direction — blocks we called landline that both references called VoIP, and 90% of those sat on a handful of carriers: Onvoy/Inteliquent, Bandwidth.com, Peerless.

The cause turned out to be a category error in the underlying numbering data. It labels line type from the carrier's regulatory class, and maps CLEC to landline. CLEC is a licensing category, not a technology. Several of the largest CLECs are wholesale VoIP providers. They hold competitive-carrier licences and deliver VoIP. Their blocks arrived labelled landline while the numbers on them were VoIP.

The fix, and why it is narrower than it looks

The obvious fix, reclassifying CLEC blocks as VoIP, would have been wrong. We tested it against the data first:

Level 3 and MCImetro are CLECs too, and their blocks really are landline. A category-wide flip would have introduced about 90 new errors while fixing 266. The correction is scoped to carriers with measured evidence, at least 90% agreement and at least 10 samples, and nothing is added to that list on the strength of its name.

Then we measured again, properly

Re-scoring the original sample after the fix gave 93%. That number is not publishable and we are not claiming it: it was measured on the very data that motivated the change.

So we pre-registered a second benchmark, new seed, a fresh draw sharing no area code and no number with the first, and our predictions frozen from the corrected classifier before either reference was queried.

The out-of-sample result matched the in-sample estimate instead of collapsing toward it, which is the outcome that says a fix generalises rather than having been fitted to the data that found it.

Where we are still wrong

Two known defects, both published here because you should know them before you buy, not after.

Coverage gaps: 13% of our "invalid" verdicts

We also tested the other direction: 200 numbers in exchanges our data says were never allocated. 80% were confirmed dead by both references. But 13% were live blocks we do not know about, held by AT&T Wireless, Verizon Wireless and T-Mobile.

We traced it: a structural hole in the 88x exchange range, where our source holds roughly a sixth of what neighbouring ranges hold, plus ordinary staleness against recent allocations. Both are in the upstream data rather than in our code, and fixing them needs a better source rather than a better parser. That work is scoped and not yet done.

Ported numbers

The permanent limitation of any offline validator, and the reason the 27% contested band exists at all. We read the block's original carrier assignment. A number ported to a different carrier keeps its block and changes its real line type, and no amount of local data can see that. For live carrier truth on a segment that matters, run those numbers through a live lookup with your own API key, which the app supports, and which is the right way to spend an API budget: scrub locally first, then pay only for what survives.

Why two references disagreeing is the useful result

The instinct on reading a disagreement figure is to decide one source is wrong. That is the wrong conclusion here, and understanding why changes how you should read every accuracy claim in this category, ours included.

Line type for a North American number is not a fact stored in one place. It is an inference, assembled from block allocation records, porting data and carrier-facing queries, each with its own update cadence and coverage. Two providers building that inference from different inputs will differ, and both can be defensible on the same number.

So a vendor quoting a single accuracy percentage is quoting agreement with whatever they measured against, usually without saying what that was. The number is not meaningless, but it is not comparable across vendors, and it is not the same kind of claim as "this number is in an unallocated block", which is checkable and settled.

The practical reading: treat invalid detection as reliable, treat line type as a strong signal rather than a fact, and be suspicious of any figure presented without its reference.

What we would need to do better

Being specific about the limits of this benchmark, because a benchmark whose weaknesses are unstated is marketing.

• No ground truth. Every figure is agreement with another inference, not with the carrier's own record. Establishing real ground truth would mean confirming line status by a means outside all three systems, at a scale we have not attempted.
• US-only. The NANP is the hardest case, which makes it the right place to test, but it means the numbers here say nothing about the 206 country codes where the plan separates mobile from fixed by range and the problem is far easier.
• A sample is not a list. The sample was constructed to be out-of-sample and pre-registered. A real customer list is not randomly drawn, and its error profile will differ in ways this design cannot predict.
• One point in time. Allocation data and porting records both move. A re-run next year would produce different numbers, and we would report those rather than these.

Method

• 800 US numbers: 600 drawn from allocated blocks across 12 area codes (stratified: 4 dense, 4 middle, 4 sparse by block count), plus 200 from exchanges absent from the numbering plan.
• Predictions frozen and hashed before any lookup.
• Twilio Lookup v2 line_type_intelligence; Veriphone /v3/verify. Caller name was never requested. it returns a real person's name and there is no lawful basis for collecting that to produce a chart.
• Agreement computed only over numbers where both sides returned a classification; the rest are reported as a separate count rather than dropped.
• Wilson 95% intervals throughout. No individual numbers are published.

Sample hashes:
v1 77b54d29040d2d2ebbedd9ee8aa43c7ad75d06e2cc1859ba6b54bcfe5fe883cf
v2 45abedd7fb900e9364cd7b7b05ab247a46a1e41a027ab5f20a25589c819ef2ba

One limitation on the second reference: every Veriphone response carried "mode": "static", indicating cached rather than live data — methodologically closer to our own approach than to Twilio's. Where Veriphone agrees with us against Twilio, some of that may be shared method rather than shared correctness. Our 75% against Twilio is the more conservative figure, and it sits just above what Twilio and Veriphone manage with each other.

The Twilio arm covers 580 of 600 line-type numbers. The remaining 20 will be completed and this page updated either way.

What we take from it

Invalid detection is the part that is deterministic, and it is where most of your savings come from: a number outside an allocated block cannot ring, and removing those costs you nothing per number. Line type is a strong first-pass signal that saves you most of a live-lookup bill, and on a quarter of US numbers, "line type" is a question two paid APIs cannot agree on.

We would rather tell you that than sell you a number we cannot defend.

Common questions

Are phone validators accurate?

They are accurate about different things, and the distinction matters more than any single figure. Detecting a number that sits outside any allocated block is deterministic and reliable. Determining line type and carrier is an inference that can be out of date where a number has been ported. In this benchmark two well-regarded paid APIs agreed with each other only 73% of the time on line type, which is the clearest available evidence that no single source is ground truth.

What does it mean when a validator says a number is invalid?

It should mean the number cannot exist or cannot ring: wrong length for its country, or inside a block that no carrier was ever allocated. That verdict is checkable. It does not mean the number is disconnected, that the person moved, or that the number is a bad lead, and a validator that blurs those together is overstating what its data supports.

Why would two phone validation services give different answers?

Because line type is assembled from different inputs with different update cycles: block allocation records, porting data, and carrier-facing queries. Two providers using different combinations will legitimately differ on the same number. This is why an accuracy percentage without a named reference is not comparable between vendors.

Can I test a phone validator before buying?

Yes, and you should test it on your own data rather than on a sample someone else chose. Take a few thousand rows from a real list, run them, and check the disagreements by hand against whatever source you already trust. A benchmark run on someone else's sample tells you about that sample.

Is phone validation legitimate, or is it a scam?

The service is legitimate and the data behind it is real. What is worth scrutinising is the claim attached to it. Numbering-plan allocation is published data and checks against it are verifiable. A vendor quoting a single accuracy percentage without naming what it was measured against is not lying exactly, but the figure cannot be checked and is not comparable with anyone else's. Ask what the reference was, and treat invalid detection and line type as separate claims with different levels of certainty.