Continuous Pipelines at Google

This presentation will focus on the real life challenges of managing data processing pipelines of any depth and complexity. I’ll cover the frequency continuum between periodic pipelines that run very infrequently up through continuous pipelines that never stop running, discussing the discontinuities along the axis that can produce significant operational problems. A fresh take on the master-slave model is presented as a better alternative to the periodic pipeline for reliable Big Data scaling.

Dan Dennison has been a Site Reliability Engineer at Google for the past eight years, working on various large pipelines in Ads Quality. The Ads Quality team at Google improves, expands, and supports one of the world’s largest machine learning deployments. Ads Quality also crawls and indexes all advertisements, and enforces policies on new and existing ads.