ZapBounce

ZapBounce and MongoDB

MongoDB's schema flexibility is its strength and it produces a specific problem for any field-level operation.

In a collection grown over several years, the email field can be a string in some documents, an array in others, and nested inside a contact object in the ones written by the version nobody documented.

The same field has several shapes in one collection

A query written against the string form silently skips the documents where the address is an array, and those documents are invisible rather than reported as errors.

So a verification run appears to complete and has left a third of the collection untouched, with nothing recording which third.

Schema validation added later only applies to new writes, so the historic shapes persist until somebody migrates them.

How the data moves

  1. Survey the field shapes first

    An aggregation grouping on the field's type tells you what you are dealing with before you write a job against one shape.

  2. Normalize into a verification collection

    One document per address, extracted from whatever shape it was in. That collection is what the job reads.

  3. Write verdicts to their own collection

    Keyed on the address, with a checked timestamp, rather than mutating documents whose shape varies.

  4. Add schema validation for new writes

    So the shape problem stops growing, even though it does not fix the history.

Setting it up

  1. Run an aggregation grouping by the type of the email field to survey the shapes present.
  2. Write an extraction job that normalizes every shape into a verification collection.
  3. Verify the addresses from that collection as batches.
  4. Store verdicts in their own collection keyed on the address with a timestamp.
  5. Add schema validation on the source collection so new writes use one shape.
  6. Report how many documents each shape accounted for, so the coverage of the run is known.

MongoDB: common questions

Why did my job miss documents?

Almost certainly a field shape your query did not match. Survey the types before writing the job.

Should I migrate the historic shapes?

Eventually. The extraction job works around it; the survey tells you how much work the migration would be.

Where do verdicts live?

Their own collection keyed on the address. Mutating documents whose shape varies invites the same problem again.

Reference

Check a MongoDB export today

100 free checks a month, no card. Unknown results and duplicates are never billed.

Get an API keySee pricing