Panel: Data Pipelines—Scaling and Reliability

Data processing pipelines, in some form or another, are the lifeblood of all large systems that aggregate data, sort and structure unordered input, or compute features for machine learning. These kinds of systems have become much more common in recent years, and problems with delayed or incorrect results are becoming more likely to have business and user impact. Designing and running pipelines is quite different from designing and running serving jobs. In this panel, pipeline experts from a range of organisations and applications will discuss their experiences scaling pipelines and dealing with their pitfalls.