InStream: Large Scale Distribution using BitTorrent, Python, Salt, and Kafka

Wednesday, 30 August, 2017 - 14:3015:00

Harsh Sharma, LinkedIn

Abstract: 

Deploying application/services to all the servers across every datacenter can be painful for any company with a big infrastructure, including LinkedIn.

Our deployment model had some centralized pieces which became bottlenecks at scale. This talk will describe how we built a service in Python, based on Saltstack and Kafka, which can deploy any service to all servers asynchronously with a P2P distribution model, rate limiting and fast rollbacks.

Harsh Sharma, LinkedIn

I've been an SRE at LinkedIn for over a year, working with Platform and Horizontal teams, and as one of the primary owners of InStream, building internal tools and supporting different platform services. I enjoy being an SRE and wish to contribute as much as I can to the global SRE community.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

BibTeX
@conference {205484,
author = {Harsh Sharma},
title = {{InStream}: Large Scale Distribution using {BitTorrent}, Python, Salt, and Kafka},
year = {2017},
address = {Dublin},
publisher = {USENIX Association},
month = aug
}

Presentation Video 

Presentation Audio