How our benchmark is built

Hunter published a benchmark of 15 verifiers on 3,000 real business addresses that measured overall accuracy between 31.2% and 70.0%. Hunter sells a competing product, yet its benchmark is cited everywhere. That's because Hunter described what it did.

That is the lesson. A self-interested benchmark with a disclosed method beats an independent-sounding claim with no method at all. The method is what makes it checkable.

So here is ours. The dataset is a list of addresses with known outcomes. We got those by sending to them and recording what came back, not by asking another verifier. Anything else measures agreement between vendors rather than truth.

The dataset has to include the hard cases in their real proportion. A benchmark run only on domains that answer honestly measures the easy 65% of the problem. It yields the same inflated figure everyone else prints.

Coverage and accuracy as a pair, with the false positive rate stated separately. A false positive is an address called valid that bounced, and it is the error that costs you something. A false negative costs you a contact and no reputation.

The breakdown by provider, because an aggregate hides everything interesting. A domain that rejects unknown recipients cleanly and a Yahoo domain that accepts every probe behave nothing alike. Publishing only the average would blur that difference.

The dataset description: how many addresses, which providers, and how each outcome was established. Another team can then build the same kind of file and check the method for themselves.