- Deduplication, defined
- Deduplication removes repeated addresses from a list before verification, and doing it first matters because most verifiers charge per address submitted rather than per unique address.
Lowercase the domain always, since domains are case-insensitive by specification. The local part technically is not, and in practice no mailbox provider treats it as significant.
Gmail adds two more equivalences: dots in the local part are ignored, and anything after a plus sign is a tag. j.smith+news@gmail.com and jsmith@gmail.com reach one inbox.
Applying those rules everywhere is a mistake. Other providers do treat dots as meaningful, so aggressive normalization merges two different people at the same company.
A list exported from several sources commonly carries 5 to 15 percent duplicates. On a hundred thousand addresses that is real money, and the work is one query.
How ZapBounce reports it
Duplicates are collapsed before a run starts and charged once, not once per copy. The report shows how many were removed, so the gap between what you uploaded and what was billed is visible rather than assumed.
Three exports, one list, 53,500 rows
Say you merge a CRM export of 38,000 contacts, a webinar platform's 9,500 registrants and 6,000 trade show badge scans. That's 53,500 rows. An exact string match finds 2,300 duplicates.
Then you clean the strings first and try again. Trimming spaces and lowercasing finds another 1,900. Removing junk that came along for the ride finds 600 more: a mailto: prefix from a copied link, angle brackets and a display name like Dana Reyes <dana@acme.example>, a trailing comma, a non-breaking space pasted from a web page. The badge scans are the worst for this.
You finish with 48,700 unique addresses and 4,800 duplicates, about 9% of the file. More than half of them were invisible to the naive comparison. Invisible characters deserve special suspicion, because two strings that look identical on screen will still compare as different.
Which copy survives the merge
Finding duplicates is the easy half. Deciding what the merged record says is where the damage happens, because the three copies disagree.
Opt-out status has to win over everything. If any copy of the address is unsubscribed, bounced or marked as a complaint, the merged record carries that status, whichever copy was newest. Get this backwards and the merge quietly re-subscribes people, which is the most expensive mistake available here.
For consent, keep the earliest timestamp and its source, since that's your evidence of when and how the person joined. Activity works the other way, so keep the most recent date. With names and job titles, prefer the copy from the system a human maintains over the one from a badge scanner.
Write those rules down before you run the merge, and run it on a copy first. Then pick twenty merged records by hand and check each against its sources. It's dull work, and it's far cheaper than explaining to an unsubscribed contact why they heard from you again.
Deduplication: common questions
Should I deduplicate before uploading?
It is good practice regardless. With us the duplicates are not billed either way, and a smaller file processes faster.
Are Gmail dots significant?
No, Gmail ignores them, so the dotted and undotted forms reach the same inbox. Most other providers do treat them as distinct.
How many duplicates does a typical list have?
Between 5 and 15 percent when the list was merged from several sources.