Menu
Cart

Napkin Folding — algorithm

Searching through distributed datasets: The Mod-Binary Search

Posted by Cameron Davidson-Pilon at

On a not-too-unusual day, one of my Spark jobs failed in production. Typically this means there was a row of bad data that entered into the job. As I’m one to write regression tests, this “type” of bad had likely never been seen before, so I needed to inspect the individual offending row (or rows). Typically debug steps include: Manually inspecting all the recent data, either by hand or on a local machine. The failed job might print the offending...

Read more →