Distributed Consensus Algorithms for Extreme Reliability

Distributed consensus algorithms (such as Paxos, ZAB, and Raft) deal with reaching agreement among a group of processes connected by an unreliable communications network. For decades these protocols were largely of interest to academics, but in today's world they are incredibly relevant as a way of building reliable, stateful, multihomed systems that do not require a lot of human effort to run.

This talk will cover:

• Use cases for distributed consensus algorithms (and what can go wrong trying to use adhoc solutions): split brain problems, faulty group-membership, avoiding the need to have humans resolve bad data as a result of problems, etc
• System-level design patterns that have been used successfully to create reliable systems at scale: replicated state machines, replicated datastores, highly available services with a single lelader and failover, distributed queuing and messaging, distributed coordination and locking
• Performance of these protocols in various different operating environments: different geographical deployments
• Scaling these systems: batching, sharding
• Pitfalls and monitoring

Laura Nolan has been a Site Reliability Engineer at Google for the past two years, and has worked with multiple large systems that use distributed consensus to achieve reliable, stable multihoming that really works without human babysitting. Prior to that she kept the e-commerce site gilt.com fast and stable during flash sales, and worked for the Irish software company Curam Software, which is now part of IBM Smarter Cities. She is a keen traveller and scuba diver and an international-level weightlifting referee.