How do you instill the agility and effectiveness of a startup company within the walls of one of the most storied animation studios in the world? This question guided our design of a new systems organization at Walt Disney Animation Studios. Our goals were simple: break down traditional top-down management silos, empower staff to make autonomous decisions, and remove bureaucracy. We used scientific method experimentation with different structures and ideas, discussing the impact of each change with our staff along the way. We’ll discuss the methods we used to empower sysadmins, and how we’ve evolved into an organization that’s designed for and by technical staff.

Jonathan Geibel is the Systems Technology Director at Walt Disney Animation Studios where he leads a team of 50 technologists that push the boundaries of high-performance computing to advance the art of Animation. A 20-year technology veteran, Jon’s interests focus on designing adaptable organizations to enable engineering teams, high-performance computing, and technology futures.

Ronald Johnson is a 20-year SysAdmin veteran with a passion for communication technologies and organizational structures. He is currently a Systems Manager at Walt Disney Animation Studios.

This talk will discuss the current shortage of skilled system administration professionals, the evolving skill set demanded by the changing global economy, and how we, as practitioners, can move our industry forward. We will look at baseline skill sets, professional development opportunities, attracting the next generation of system administrators, and providing strategic value to the organizations who rely on sysadmins, even when they don't realize it.

Mandi Walls is a technical consultant for Seattle-based Opscode, the makers of Chef. Prior to joining Opscode, Mandi served as a system administrator at Admeld, NHGRI, and AOL, where she ran sites including moviefone.com, games.com, and www.aol.com. She holds a Master's degree in Computer Science from the George Washington University, Washington, D.C., and an M.B.A. from UNC Kenan-Flagler Business School, Chapel Hill, North Carolina, through the global OneMBA program. She is a published author with O'Reilly and speaks at numerous conferences and events.

When running your app "in the cloud" there is a dizzying layer of software controlling your trivially deployed code. In practice, it is your code → language VM → LXC Container → Linux OS → Xen Hypervisor → Linux OS → CPU.

Here we can look at each layer as a "user space," an expressive place that you are empowered to use, and a "kernel," the black box system that imposes strict constraints through an API.

By studying each layer in this way, we see self-similar properties, which offers insight for how to best participate in the ecosystem. We also understand why the cloud is built this way with the huge benefits in power and efficiency this offers to application developers.

Noah Zoschke is a lead engineer at Heroku, a cloud Platform-as-a-Service. He spends his time managing a team of infrastructure and systems engineers on the Heroku Runtime, a distributed code compilation, process management and process execution system responsible for running and scaling millions of applications.

Over the past two years, we have built a diverse software environment of over 200 scientific applications for our research computing platform at Brown University. In this report, we share the policies and best practices we have developed to simplify the configuration and installation of this software environment and to improve its usability and performance. In addition, we present a reference implementation of an environment modules system, called PyModules, that incorporates many of these ideas.

Based on experience testing distributed storage systems in several public clouds, this talk will consist of two parts. The first part will cover approaches for characterizing and measuring storage workloads generally. The second part will cover the additional challenges posed by testing in public clouds. Contrary to popular belief, no two cloud servers are ever alike. Even the same server can exhibit wild and unpredictable performance swings over time, so new ways of analyzing performance are critical in these environments.

Jeff Darcy has worked on network and distributed storage systems for 20 years, including an instrumental role in developing MPFS (a precursor of modern pNFS) while at EMC. He is currently a member of the GlusterFS architecture team at Red Hat and frequently gives talks and tutorials about topics related to cloud storage.

Amy Rich has been a UNIX sysadmin for over 20 years at a variety of companies, has owned her own consulting business, helped organize multiple sysadmin conferences, and written professionally on the topic of UNIX systems administration. She currently works at the Mozilla Corporation and plays the part of both of sysadmin and manager of the Release Engineering Operations IT team, providing the infrastructure which performs automated builds and tests for Firefox, Firefox OS, and Thunderbird across all of the platforms that Mozilla supports. She is a member of USENIX and the LISA SIG, and is a founding member of LOPSA.

Deanna McNeil started out as a mom who had to work part time and discovered a knack for systems administration. Now she pursues technology for the love of always learning.

Amy Forinash was born and raised in Prince George's County, MD, and is a happy product of the public school system there. After studying physics at the University of Maryland at College Park, she went on to become a system administrator in the mid-'90s, a job she still holds and has expanded to meet the IT security needs of her customer. In her personal life she enjoys long distance hiking, dressage with her Lippizan, and consuming adult beverages with other system administrators.

Deirdré Straughan has worked in technology for 25 years, on documentation, training, UI design, marketing, open source community management, event management, social media, video, and more. She tends to operate at the interfaces: between companies and customers, technologists and non-technologists, marketers and engineers, and anywhere else that people need help communicating with each other about technology. Much more about her life and work can be found at beginningwithi.com.

Clouds establish a new division of responsibilities between platform operators and users than have traditionally existed in computing infrastructure. In private clouds, where all participants belong to the same organization, this creates new barriers to effective communication and resource usage. In this paper, we present poncho, a tool that implements APIs that enable communication between cloud operators and their users, for the purposes of minimizing impact of administrative operations and load shedding on highly-utilized private clouds.

Backing up important data is an essential task for system administrators to protect against all kinds of failures. However, traditional tools like rsync exhibit poor performance in the face of today's typical data sizes of hundreds of gigabytes. We address the problem of efficient, periodic, multi-gigabyte state synchronization. In contrast to approaches like rsync which determine changes after the fact, our approach tracks modifications online. Tracking obviates the need for expensive checksum computations to determine changes. We track modification at the block-level which allows us to implement a very efficient delta-synchronization scheme. The block-level modification tracking is implemented as an extension to a recent (3.2.35) Linux kernel.

With our approach, named dsync, we can improve upon existing systems in several key aspects: disk I/O, cache pollution, and CPU utilization. Compared to traditional checksum-based synchronization methods dsync decreases synchronization time by up to two orders of magnitude. Benchmarks with synthetic and real-world workloads demonstrate the effectiveness of dsync.

Brendan Gregg is the lead performance engineer at Joyent, where he analyzes performance and scalability at any level of the software stack. He is the author of Systems Performance (Prentice Hall, 2013), and primary author of DTrace (Prentice Hall). He was previously a kernel engineer at Sun Microsystems where he developed the ZFS L2ARC, and has also developed numerous performance analysis tools. His recent work includes performance visualizations.

The management of virtual machine cluster (VMC) is challenging owing to the reliability requirements, such as non-stop service, failure tolerance, etc. Distributed snapshot of VMC is one promising approach to support system reliability, it allows the system administrators of data centers to recover the system from failure, and resume the execution from a intermediate state rather than the initial state. However, due to the heavyweight nature of virtual machine (VM) technology, applications running in the VMC suffer from long downtime and performance degradation during snapshot. Besides, the discrepancy of snapshot completion times among VMs brings the TCP backoff problem, resulting in network interruption between two communicating VMs. This paper proposes HotSnap, a VMC snapshot approach designed to enable taking hot distributed snapshot with milliseconds system downtime and TCP backoff duration. At the core of HotSnap is transient snapshot that saves the minimum instantaneous state in a short time, and full snapshot which saves the entire VM state during normal operation. We then design the snapshot protocol to coordinate the individual VM snapshots into the global consistent state of VMC. We have implemented HotSnap on QEMU/ KVM, and conduct several experiments to show the effectiveness and efficiency. Compared to the live migration based distributed snapshot technique which brings seconds of system downtime and network interruption, HotSnap only incurs tens of milliseconds.

Live update is a promising solution to bridge the need to frequently update a software system with the pressing demand for high availability in mission-critical environments. While many research solutions have been proposed over the years, systems that allow software to be updated on the fly are still far from reaching widespread adoption in the system administration community. We believe this trend is largely motivated by the lack of tools to automate and validate the live update process. A major obstacle, in particular, is represented by state transfer, which existing live update tools largely delegate to the programmer despite the great effort involved.

This paper presents time-traveling state transfer, a new automated and fault-tolerant live update technique. Our approach isolates different program versions into independent processes and uses a semantics-preserving state transfer transaction—across multiple past, future, and reversed versions—to validate the program state of the updated version. To automate the process, we complement our live update technique with a generic state transfer framework explicitly designed to minimize the overall programming effort. Our time-traveling technique can seamlessly integrate with existing live update tools and automatically recover from arbitrary run-time and memory errors in any part of the state transfer code, regardless of the particular implementation used. Our evaluation confirms that our update techniques can withstand arbitrary failures within our fault model, at the cost of only modest performance and memory overhead.

The networking community is making an increasing amount of noise about software-defined networking (SDN). In an attempt to clarify SDN's increasingly nebulous value proposition, this talk makes the case that software-defined networking simply applies sorely needed, well-known, and standard principles of good software design to the network. So, by asking "what would a programmer do?" to solve network problems, we can derive and make concrete all of SDN's real-world value propositions including improved automation through documented well-structured APIs, higher uptime with automated testing, benefits of refactoring the control/data plane relationship, and increased modularity of network functions. Throughout the talk I will substantiate this claim with real-world examples from my time as a network admin, my work in standards bodies like the Open Networking Foundation, and customer stories from my current day job. Finally, I'll conclude with the point that while network and server admins have historically had disjoint skill sets, SDN presents an opportunity for the two to meet in the middle with a DevOp-style control of both domains.

Rob leads standardization and controller software architecture at Big Switch, where he developed and evangelized the emerging OpenFlow standard and network virtualization. He is the current Chair of the ONF’s Architecture & Framework Working Group and all Northbound API activity and was vice-chair for the ONF Testing & Interoperability Working Group. Rob prototyped the first OpenFlow-based network hypervisor, the “FlowVisor,” allowing production and experimental traffic to safely co-exist on the same physical network, and is involved in various standards efforts and partner and customer engagements. Rob holds a Ph.D. in Computer Science from the University of Maryland, College Park.

Google maintains many servers and employs a file level sync method with applications running in a different partition than the base Linux distribution that boots the machine and interacts with hardware. This experience report first gives insights on how the distribution is setup, and then tackles the problem of doing a difficult upgrade from a Red Hat 7.1 image snapshot with layers of patches to a Debian Testing based distribution built from source. We will look at how this can actually be achieved as a live upgrade and without ending up with a long "flag day" where many machines are running totally different distributions, which would have made testing and debugging of applications disastrous during a long switchover period.

Like a coworker of mine put it, "It was basically akin to upgrading Red Hat 7.1 to Fedora Core 16, a totally unsupported and guaranteed to break upgrade, but also switching from rpm to dpkg in the process, and on live machines."

The end of the paper summarizes how we designed our packaging system for the new distribution, as well as how we build each new full distribution image from scratch in a few minutes.

YinzCam allows sport fans inside NFL/NHL/NBA venues to enjoy replays and live-camera angles from different perspectives, on their smartphones. We describe the evolution of the system infrastructure, starting from the initial installation in 2010 at one venue, to its use across a dozen venues today. We address the challenges of scaling the system through a combination of techniques, including distributed monitoring, remote administration, and automated replay-generation. In particular, we take an in-depth look at our unique automated replay-generation, including the dashboard, the remote management, the remote administration, and the resulting efficiency, using data from a 2013 NBA Playoffs game.

Security has become a first-class engineering requirement. But it is not the only such requirement. In this talk, I'm going to consider various sacred cows in security and ask whether we'll still believe in them in a few years. Does the user model make sense now in a world of app servers? Are biometrics better or worse than passwords? Will DJB become the new NIST? Let's talk about the future of actually delivering security to our users.

Dan Kaminsky has been a noted security researcher for over a decade, and has spent his career advising Fortune 500 companies such as Cisco, Avaya, and Microsoft. Dan spent three years working with Microsoft on their Vista, Server 2008, and Windows 7 releases.

Dan is best known for his work finding a critical flaw in the Internet’s Domain Name System (DNS), and for leading what became the largest synchronized fix to the Internet’s infrastructure of all time. Of the seven Recovery Key Shareholders who possess the ability to restore the DNS root keys, Dan is the American representative. Dan is presently developing systems to reduce the cost and complexity of securing critical infrastructure.

Deploying a large-scale distributed ecosystem such as HBase/Hadoop in the cloud is complicated and error-prone. Multiple layers of largely independently evolving software are deployed across distributed nodes on third party infrastructures. In addition to software incompatibility and typical misconfiguration within each layer, many subtle and hard to diagnose errors happen due to misconfigurations across layers and nodes. These errors are difficult to diagnose because of scattered log management and lack of ecosystem-awareness in many diagnosis tools and processes.

We report on some failure experiences in a real world deployment of HBase/Hadoop and propose some initial ideas for better trouble-shooting during deployment. We identify the following types of subtle errors and the corresponding challenges in trouble-shooting: 1) dealing with inconsistency among distributed logs, 2) distinguishing useful information from noisy logging, and 3) probabilistic determination of root causes.

Google has one of the largest managed fleets of Macintosh computers in the world. With tens of thousands of assets to manage and an ever-changing security landscape, the organization has had to develop many of its own tools to effectively maintain its fleet and keep its end-users safe and productive. Macintosh Operations is the internal team tasked with developing these tools and managing these machines globally.

Clay Caviness has been a Mac and UNIX systems engineer since the early 90s and was thrilled when NeXT bought Apple. He only recently stopped running BSD on a Quadra at home. Clay has worked in advertising and technology companies since 1996 and currently works for Google in New York City.

Edward Eigerman has worked in IT, primarily with Macs, since 1988. He worked as an engineer for Apple for several years and later as an independent consultant, where his clients included The New York Times, Major League Baseball, NASA, and Jon Stewart. He currently works for Google in New York City.

Cloud-based, Infrastructure-as-a-Service (IaaS) platforms present a simple data center resource model to their end-users. The end-user provisions logical resource units of compute, network, and block storage. They then attach a compute unit to one or more network and block storage units. This "attach" and subsequent "detach" of emulated network and storage devices embodies what is referred to as I/O virtualization, the act of scheduling I/O resources independent of compute. For this session we will outline this I/O virtualization concept and then dig deeper into how cloud/IaaS platforms leverage it for both local and global/data-center-wide scheduling.

Dave is a strategist and researcher in EMC's Office of the CTO where he focuses on the costs, scalability, and performance of data center infrastructure, especially in a Cloud setting. During his tenure at EMC, he has provided technical leadership in the areas of Network Virtualization, OpenStack, and converged infrastructure platforms such as OpenCompute. Prior to joining EMC, Dave spent ten years on Wall Street working on various facets of data center infrastructure for companies such as Goldman Sachs and Merrill Lynch.

SSH user keys are ubiquitously used for accessing information systems by automated processes and system administrators. Many large organizations have hundreds of thousands of keys granting access, with many keys providing privileged access without auditing or controls. The talk educates the audience about risks arising from unmanaged access using SSH keys; discusses what is required by compliance mandates; outlines how to establish effective operational processes for provisioning, terminating, and monitoring SSH user key based access; and outlines how to understand and remediate SSH user keys in an existing environment.

Mr. Ylönen invented the Secure Shell (SSH) protocol in 1995 and is founder and CEO of SSH Communications Security. OpenSSH is based on his free version of 1995. He has 29 years of programming and systems management experience and plenty of business management background. He co-authored the IETF guidelines on SSH key management for automated access and is a co-author in upcoming NIST IR series guidelines for managing access using SSH keys. Mr. Ylönen and his company have been deeply involved in several actual SSH key remediation and management projects with some of the leading financial institutions and other enterprises.