Two phone validation APIs disagree with each other 27% of the time

Every phone validator claims to be accurate. Almost none publish a number, and the ones that do never say how they measured it.

So we measured ours, wrote the method down first, and published everything — including the run where we came out badly.

The finding that has nothing to do with us

We compared every number against two independent commercial references: Twilio Lookup and Veriphone. On the 513 numbers where all three of us returned a classification:

Two paid APIs disagree with each other about 27% of US numbers. Ask them whether a number is mobile, landline or VoIP and more than one in four times you get different answers.

That reframes every accuracy claim in this category, ours included. There is no oracle. When a vendor tells you they are 99% accurate, the honest question is "against what?" — because the two most obvious answers disagree with each other a quarter of the time.

What we found about our own data

Splitting the disagreements three ways tells you what they mean:

Where both references agree on an answer, we match them 98% of the time. The 27% is not our error and not theirs; it is a question the industry cannot currently answer.

The headline numbers

Mobile recall is the number that matters for an SMS campaign: of the numbers a reference calls mobile, we find 96 in every 100.

The run where we came out badly

The first benchmark returned 66% agreement with Twilio, below what we had been claiming. We published the method before running it, so there was no version of this where the result quietly disappeared.

The disagreements were not scattered. 108 of 124 ran in one direction — blocks we called landline that both references called VoIP — and 90% of those sat on a handful of carriers: Onvoy/Inteliquent, Bandwidth.com, Peerless.

The cause turned out to be a category error in the underlying numbering data. It labels line type from the carrier's regulatory class, and maps CLEC to landline. CLEC is a licensing category, not a technology. Several of the largest CLECs are wholesale VoIP providers — they hold competitive-carrier licences and deliver VoIP. Their blocks arrived labelled landline while the numbers on them were VoIP.

The fix, and why it is narrower than it looks

The obvious fix — reclassify CLEC blocks as VoIP — would have been wrong. We tested it against the data first:

Level 3 and MCImetro are CLECs too, and their blocks really are landline. A category-wide flip would have introduced about 90 new errors while fixing 266. The correction is scoped to carriers with measured evidence — at least 90% agreement and at least 10 samples — and nothing is added to that list on the strength of its name.

Then we measured again, properly

Re-scoring the original sample after the fix gave 93%. That number is not publishable and we are not claiming it: it was measured on the very data that motivated the change.

So we pre-registered a second benchmark — new seed, a fresh draw sharing no area code and no number with the first, and our predictions frozen from the corrected classifier before either reference was queried.

The out-of-sample result matched the in-sample estimate instead of collapsing toward it, which is the outcome that says a fix generalises rather than having been fitted to the data that found it.

Where we are still wrong

Two known defects, both published here because you should know them before you buy, not after.

Coverage gaps: 13% of our "invalid" verdicts

We also tested the other direction — 200 numbers in exchanges our data says were never allocated. 80% were confirmed dead by both references. But 13% were live blocks we do not know about, held by AT&T Wireless, Verizon Wireless and T-Mobile.

We traced it: a structural hole in the 88x exchange range, where our source holds roughly a sixth of what neighbouring ranges hold, plus ordinary staleness against recent allocations. Both are in the upstream data rather than in our code, and fixing them needs a better source rather than a better parser. That work is scoped and not yet done.

Ported numbers

The permanent limitation of any offline validator, and the reason the 27% contested band exists at all. We read the block's original carrier assignment. A number ported to a different carrier keeps its block and changes its real line type, and no amount of local data can see that. For live carrier truth on a segment that matters, run those numbers through a live lookup with your own API key — which the app supports, and which is the right way to spend an API budget: scrub locally first, then pay only for what survives.

Method

• 800 US numbers: 600 drawn from allocated blocks across 12 area codes (stratified — 4 dense, 4 middle, 4 sparse by block count), plus 200 from exchanges absent from the numbering plan.
• Predictions frozen and hashed before any lookup.
• Twilio Lookup v2 line_type_intelligence; Veriphone /v3/verify. Caller name was never requested — it returns a real person's name and there is no lawful basis for collecting that to produce a chart.
• Agreement computed only over numbers where both sides returned a classification; the rest are reported as a separate count rather than dropped.
• Wilson 95% intervals throughout. No individual numbers are published.

Sample hashes:
v1 77b54d29040d2d2ebbedd9ee8aa43c7ad75d06e2cc1859ba6b54bcfe5fe883cf
v2 45abedd7fb900e9364cd7b7b05ab247a46a1e41a027ab5f20a25589c819ef2ba

One limitation on the second reference: every Veriphone response carried "mode": "static", indicating cached rather than live data — methodologically closer to our own approach than to Twilio's. Where Veriphone agrees with us against Twilio, some of that may be shared method rather than shared correctness. Our 75% against Twilio is the more conservative figure, and it sits just above what Twilio and Veriphone manage with each other.

The Twilio arm covers 580 of 600 line-type numbers. The remaining 20 will be completed and this page updated either way.

What we take from it

Invalid detection is the part that is deterministic, and it is where most of your savings come from: a number outside an allocated block cannot ring, and removing those costs you nothing per number. Line type is a strong first-pass signal that saves you most of a live-lookup bill — and on a quarter of US numbers, "line type" is a question two paid APIs cannot agree on.

We would rather tell you that than sell you a number we cannot defend.