Summit Program

When done well, Release Engineering can be a force multiplier that enables software companies to ship software efficiently and reliably, to grow in the marketplace, to grow internally by hiring more developers, and to compete against other above-their-weight companies. To understand this value in all these different contexts, it needs to be clearly explained in non-Release-Engineer language. I’ll describe some approaches that have, and haven't, worked for me. While this session is focused on RelEng, many of the same principles apply to IT, QE and other similar service groups.

John O’Duinn is a software developer, systems architect, and director. Over the last 22 years, in startups and multinationals, he has designed and helped build practical, reliable cross-platform, release engineering infrastructure which is both scalable and efficient. His infrastructure improvements have allowed companies to gain a competitive edge in the marketplace while improving productivity and retention of both developers and release engineers.

John has also developed a culture where distributed teams and individuals work seamlessly across groups in a global workplace. During his 6 1/2 years as Director of Release Engineering at Mozilla, he built a tightly knit team of 18 Release Engineers in 14 cities, in 4 non-adjacent time zones, with one of the highest retention rates across Mozilla. In Jan 2014, John transitioned from Mozilla to a new role at Hortonworks, a “startup” company which is spread across 4 non-adjacent time zones and works closely with the geo-distributed Apache Hadoop open source project.

When moving to a continuous release deployment model it is important that you combine signals from your production and canary monitoring systems into the automation pipeline. With adequate instrumentation and configuration of your release processes you can be confident that only known good (regression free) builds end up in production while minimizing the necessary delays. Your production monitoring infrastructure is a valuable source of signals that should be an intimate part of your releasing (Unit/integration tests are not the whole picture). You don't want to automatically promote a build that has a performance regression any more than you want a fleet-wide upgrade to commence as you remediate a service outage.

Caskey Dickson is a Site Reliability Engineer/Software Engineer at Google where he works writing and maintaining monitoring services that operate at "Google scale." In online service development since 1995, before coming to Google he was a senior developer at Symantec, wrote software for various internet startups such as CitySearch and CarsDirect, ran a consulting company, and even taught undergraduate and graduate computer science at Loyola Marymount University. He has a B.S. in Computer Science, a Masters in Systems Engineering, and an M.B.A from Loyola Marymount.

In this presentation, I will talk about the motivation and work behind getting the Netflix Edge server from being released once every three weeks to a daily cycle. I will discuss the huge role automation plays in getting this done and the testing and metrics we have in place, as well as emergency rollback functionality when things don’t go as planned. Roles and responsibilities in the new vs. old model will also be discussed. Attendees will have a better understanding of why we chose to deliver code daily as well as the tools and techniques we used so they can evaluate whether this is something they should pursue in their development cycle.

Daniel Zapata is a software engineer on the Edge Team at Netflix. He recently lead the successful migration from Perforce to Stash, as well as the migration from ANT to Gradle. Daniel has 15 years of release management, automation, and both white and black box testing experience. Prior to Netflix, he worked at various startups establishing release processes and defining and implementing automation architecture. He holds a BS degree in Computer Science from the University of Southern California.

The overwhelming and at times exclusive focus of release engineering today is on Web-driven applications. The focus of this presentation is on some of the challenges and even dangers faced in other types of deployments when the safety net of Web-oriented releases is taken away. Web releases are strikingly homogeneous and determined; there is one current codebase, sitting on one underlying infrastructure, which (at least on the server side) is under the complete control of the engineers. Three representative deployment situations, taken as an example from what we face at Portware, illustrate how heterogeneous non-Web releases can be much more problematic.

Daniel Cordes is the manager of the DevOps team at Portware LLC, a financial company in downtown Manhattan. He has been in the industry for almost 10 years. He has Masters degrees in Political Theory from Columbia University and the City of New York Graduate Center.

Facebook's Web front-end release process has evolved into a large-scale pseudo-continuous deployment system where we release anywhere from 30 to 300 changes per push, twice a day, coming from around 1000 developers in 4 distributed engineering offices. What lessons did we take from our successful web deployments when we made the shift to a mobile-centric company? We'll discuss how we managed to keep features of our fast-flowing development and release process and apply them to the very different world of mobile release, which is essentially a shift back to packaged software. We'll describe how we maintain a date-based release schedule, how we test and collect mobile metrics, and describe the real-world issues of dealing with package deployment and end-user uptake.

Chuck Rossi leads Release Engineering at Facebook and started as Facebook's first release engineer in 2008. Chuck has worked in release engineering for over 20 years and prior to joining Facebook, he was a Senior Build and Release Engineer at both Google and VMware.

At Basho, we develop and package Riak, a distributed key-value store, and Riak CS, an S3-API compatible object store. Many large companies rely on Riak and Riak CS as a critical part of their infrastructure, which means that any packaging or release mistakes are costly and disruptive. This talk describes how we build reliable software on the Github platform, using both the tools and API that Github provides, as well as home-grown tools we’ve made to augment GitHub for working with multiple repositories that, together, form our releases. Our tools help address areas such as issue tracking, unit and integration testing, and automatic build results triggering automatic branch merging.

In many ways, Basho is unique because we not only ship enterprise software to paying customers, but we also provide most of our code as Apache Licensed open source. We need to please both those customers who pay us for the software and those who are open source users who depend on our openness and quality. We currently have 136 public repositories and 104 private repositories on GitHub, so good tooling is extremely important to keep track of issues and changes going into our software.

One of the trickier problems in software delivery is environment disparity. How does one implement a consistent release and testing process when you deal with platforms as varied as developer laptops, virtual servers and large enterprise production environments? Why does it seem there are never enough environments to really do automated testing? And without automated testing, how do you deliver software that actually works the first time without wasting the time of your QA team or, even worse, affecting production? Depending on your environment, container engines like Docker can solve many of these problems. This talk will give a practical demonstration of how to implement Docker in your continuous integration pipeline and demonstrate the benefits of using container engines in the continuous integration process.

Glenn Brown is a DevOps consultant for Maven Wave Partners. He has over 20 years experience in information technology and has done Release Engineering for over a decade at companies like JP Morgan Chase, The Savo Group and Orbitz. He lives in Chicago with his wife and daughter who are the apples of his eye and his cat who eats him out of house and home.