Technical Sessions

AOL Technologies has created a scalable object store for web applications. The goal of the object store was to eliminate the creation of a separate storage system for every application we produce while avoiding sending data to external storage services. AOL developers had been devoting a significant amount of time to creating backend storage systems to enable functionality in their applications. These storage systems were all very similar and many relied on difficult-to-scale technologies like network attached file systems. This paper describes our implementation and the operating experience with the storage system. The paper also presents a feature roadmap and our release of an open source version.

Diagnosing performance problems in large distributed systems can be daunting as the copious volume of monitoring information available can obscure the root-cause of the problem. Automated diagnosis tools help narrow down the possible root-causes—however, these tools are not perfect thereby motivating the need for visualization tools that allow users to explore their data and gain insight on the root-cause. In this paper we describe Theia, a visualization tool that analyzes application-level logs in a Hadoop cluster, and generates visual signatures of each job's performance. These visual signatures provide compact representations of task durations, task status, and data consumption by jobs. We demonstrate the utility of Theia on real incidents experienced by users on a production Hadoop cluster.

The OpenStack project’s mission is to produce the ubiquitous open source cloud computing platform that will meet the needs of public and private cloud providers regardless of size. In this talk, I will give a detailed technical overview of the OpenStack Compute software stack in terms of the three major resources provided by it: compute, block storage, and networking services. I will also provide details of the recent Folsom release, as well as provide a sneak peek at some of the features proposed for inclusion in our next release.

Doug Hughes is the manager of all things infrastructure at D. E. Shaw Research, a bio-technology research firm located in Manhattan. He was intimately involved in the specifications, design, and implementation of the company's current built-from-scratch datacenter.

After an initial trial, Trinity College has deployed iPads to its 600 international students. Supporting these devices on the Trinity network has presented challenges relating to configuration management, wireless network capacity, and server load.

We addressed the problem of configuration management with a web application written using the Django framework, which enables students to download and install a customized configuration profile to their iPads. 

This paper describes the requirements and implementation of the configuration management solution, security issues, as well as the experiences and lessons learned from its use.

Modern Web browsers do not provide sufficient protection to prevent users from submitting their online passwords to inappropriate websites. As a result, users may accidentally reveal their passwords for high-security websites to inappropriate low-security websites or even phishing websites. In this paper, we address this limitation of modern browsers by proposing LoginInspector, a profiling-based warning mechanism. The key idea of LoginInspector is to continuously monitor a user’s login actions and securely store hashed domain-specific successful login information to an in-browser database. Later on, whenever the user attempts to log into a website that does not have the corresponding successful login record, LoginInspector will warn and enable the user to make an informed decision on whether to really send this login information to the website. LoginInspector can also report users’ insecure password practices to system administrators so that targeted training and technical assistance can be provided to vulnerable users. We implemented LoginInspector as a Firefox browser extension and evaluated it on 30 popular legitimate websites, 30 sample phishing websites, and one new phishing scam discovered by M86 Security Labs. Our evaluation and analysis indicate that LoginInspector is a secure and useful mechanism that can be easily integrated into modern Web browsers to complement their existing protection mechanisms. Security system administrators in our university commented that such a tool could be very helpful for them to strengthen campus IT security.

As the size and performance requirements of storage systems have increased, file system designers have looked to new architectures to facilitate system scalability. Ceph is a fully open source distributed object store, network block device, and file system designed for reliability, performance, and scalability from terabytes to exabytes. Fault tolerance is a key challenge for both system design and operations. Ceph is designed to be both highly available and elastic. In large clusters, disk, host, and even network failures are the norm rather than the exception, hardware is heterogeneous and incrementally deployed or deprovisioned, and availability must be continuous. This talk will describe the Ceph architecture and the impact it has on system operations, including failure management, monitoring, and provisioning.

Janice Gelb is a Senior Developmental Editor at Oracle Corporation, where she is responsible for editing print and online documentation for numerous software products. She is a certified trainer for their internal SGML/XML authoring tool. Janice also worked as a technical editor at Ashton-Tate and at Scitex Corporation. She has presented papers at several technical communication conferences and was the project lead for the popular editorial style guide Read Me First! A Style Guide for the Computer Industry.

System administrators are often asked to apply their professional expertise in unusual situations, or under tight resource constraints. What happens, though, when the “situation” is a foreign country with only basic technical infrastructure, and the task is to bauild systems which are able to survive and grow in these over-constrained environments?

In this paper we report on our experiences in two very different countries – Cuba and Ethiopia – where we ran a number of ICT projects. In those projects we assisted local universities to upgrade their ICT infrastructure and services. This included skills and process building for local system administrators.

Based on our experiences we formulate a model for sustainable ICT capacity building. We hope this model will be useful for other organizations doing similar projects.

This paper describes training and professional development activities at a mid-sized IT consulting firm over the last roughly 15 years. These activities have successfully engaged many of the consultants and provided significant career bonuses and advantages for the company. We present the types of activities, their effectiveness and success, and the evolution of professional development efforts over time. Many of these activities proved effective and valuable and are still in use, including annual skill reviews and development recommendations, regular organized training and discussion type events, training and materials reimbursements, and escalation support. Challenges and failures with other training and activities are described. Recommendations are made for other organizations’ own professional development programs.

Large enterprises have long needed to deal with the problem of copying or migrating data between sites. This problem becomes more acute when those enterprises try to move work to/from public clouds, to avoid having computation arrive in the cloud with no data on which to work. This talk will cover methods for managing data location, including various tradeoffs of efficiency, consistency, and user-friendliness.

Lightning talks are fast-paced and high-energy. These are back-to-back 5-minute presentations on just about anything. Talk about a recent success, energize people about a pressing issue, ask a question, start a conversation!

Lightning talks are an opportunity to get up and talk about what’s on your mind. You can give several lightning talks if you have more than one topic. 

To submit a lightning talk, complete the form here by December 12, 2012. 

Monitoring is a core function of systems administration, and is primarily a problem of communication – a good monitoring tool communicates with users about problems, and communicates with hosts and software to take remedial action. The better it communicates, the greater the confidence administrators will have in its view of their environment. Nagios has been a leading open-source monitoring solution for over a decade, but in that time, the way it gets data in and out of its scheduling engine hasn’t changed. As applications are written to extend Nagios, each one has to figure out its own way of getting data out of the Nagios core process. This paper explores the use of messaging middleware, in an open-source project called NagMQ, as a way to provide a common interface for Nagios that can be easily utilized by a variety of applications.

The automated identification of network service dependencies remains a challenging problem in the administration of large distributed systems. Advances in developing solutions for this problem have immediate and tangible benefits to operators in the field. When the dependencies of the services in a network are better-understood, planning for and responding to system failures becomes more efficient, minimizing downtime and managing resources more effectively.

This paper introduces three novel techniques to assist in the automatic identification of network service dependencies through passively monitoring and analyzing network traffic, including a logarithm-based ranking scheme aimed at more accurate detection of network service dependencies with lower false positives, an inference technique for identifying the dependencies involving infrequently used network services, and an approach for automated discovery of clusters of network services configured for load balancing or backup purposes. This paper also presents the experimental evaluation of these techniques using real-world traffic collected from a production network. The experimental results demonstrate that these techniques advance the state of the art in automated detection and inference of network service dependencies.

We aim to detect and diagnose code misbehavior that wastes energy, which we call energy bugs. I will describe a method and implementation, called Carat, for performing such diagnosis on mobile devices. Carat takes a collaborative, black-box approach. A non-invasive client app sends intermittent, coarse-grained measurements to a server, which identifies correlations between higher expected energy use and client properties like the running apps, the device model, and the operating system. Carat has been deployed on more than a quarter of a million devices and has detected thousands of app instances exhibiting energy bugs in the wild.

Owen DeLong is an IPv6 Evangelist at Hurricane Electric and a member of the ARIN Advisory Council. Owen brings more than 25 years of industry experience. He is an active member of the systems administration, operations, and IP policy communities. In the past, Owen has worked at Tellme Networks (Senior Network Engineer); Exodus Communications (Senior Backbone Engineer), where he was part of the team that took Exodus from a pre-IPO startup with two datacenters to a major global provider of hosting services; Netcom Online (Network Engineer), where he worked on a team that moved the Internet from an expensive R&E tool to a widely available public access system accessible to anyone with a computer; Sun Microsystems (Senior Systems Administrator); and more. He can be reached as owend at he dot net.

In this paper, the authors discuss their experiences implementing, operating, and optimizing a content delivery network for Canada’s largest news website, that of the Canadian Broadcasting Corporation (CBC). The site receives over one million unique visitors per day. Although CBC uses the Akamai Aqua Platform (formerly EdgeSuite) as its content delivery network of choice, the lessons described here are generally applicable to any infrastructure fronted by a CDN.

In this paper we describe the design and implementation of a system essential to enable the deregulation of the energy market in the Netherlands. The system is used to test and validate secure communications using XML messages through the AS2 standard between the business partners in the market. The tool is comprised of an Enterprise Service Bus component, a service virtualization component, a database with business logic and an user interface added. The version 1.0 of the system was built in less than one month.

Ganeti is a cluster virtual server management software tool built on top of existing virtualization technologies such as Xen or KVM and other Open Source software.  Ganeti takes care of disk creation, migration, OS installation, shutdown, startup, and can be used to preemptively move a virtual machine off a physical machine that is starting to get sick.  It doesn’t require a big expensive SAN, complicated networking, or a lot of money.  The project is used around the world by many organizations; it is sponsored by Google and hosted here.

Diego Zamboni is a computer scientist, consultant, programmer, sysadmin, and overall geek. He works at CFEngine AS as Senior Security Advisor. His role is to advocate the power and usefulness of CFEngine, particularly in the security space, and also to help the CFEngine user community. After finishing his doctoral studies at Purdue University, he worked as researcher in computer security at the IBM Zurich Research Lab and as security consultant at HP Enterprise Services. He is very interested in topics related to computer security, virtualization, configuration management, and system automation. Diego is also the author of the O'Reilly book Learning CFEngine 3 and a frequent speaker in CFEngine webinars. This year he has presented on system administration topics at DevOps Chicago, DevOps Days Mountain View, CampIT, and PICC.

In this paper we describe the network infrastructure we designed, built and operated for the International Mathematics Olympiad in the Netherlands in 2011. The infrastructure was pragmatically designed around OpenVPN tunnels in a star topology between the various venues. VLANs were extensively used to separate functional groups and networks. The actual construction of the event network took about 3 days and was needed for only 2 weeks. The architectural, setup, building and operational aspects of the network are described and we include some lessons learned.

Why do conference and school wireless networks always work so poorly? As IT professionals we are used to the network 'just working' and fixing things by changing configuration files. This mind-set, combined with obvious-but-wrong choices in laying out a wireless network frequently lead to a network that seems to work when it's tested, but that then becomes unusable when placed under load. This is at its worst at technical conferences where there are so many people, each carrying several devices, all trying to use the network at the same time, and in schools where you pack students close together and then try to have them all use their computers at the same time.

Is this a fundamental limitation of wireless? While it is true that there are some issues that cannot be solved, there are a lot of things that the network administrator can do to make the network work better. The key issue is the obvious, but under-appreciated fact that wireless networking is radio communications first. If your radio link doesn't work well, you have no chance of fixing it with your configuration and software. This paper is intended to give you an appreciation of what the issues are, and enough information to know what sorts of things to look out for when planning a high density wireless network.

In the world of information security, it's not a matter of how anymore…it’s a matter of when. With the advent of penetration tools such as Metaspolit, AutoPwn, etc.—plus the day-to-day use of insecure operating systems, applications, and Web sites—reactive systems have become more important than proactive systems. Discovery of penetration by out-of-band processes and being able to determine the when and how to then mitigate the particular attack has become a stronger requirement than active defense. I will discuss the basic precepts of this idea and expand with various types of tools that help resolve the issue. Attendees should be able to walk away from this discussion and apply the knowledge immediately within their environment.

Joe Conway

Joe Conway has been involved with PostgreSQL as a contributor since 2001. He is also the author and maintainer of a PostgreSQL procedural language handler for the R language, PL/R. Joe is President/CEO of credativ USA, which specializes in open source software with its "Open Source Support Center" and a comprehensive range of services, including consulting, architectural and technical advice, software development, training, and personalized support.

Josh Berkus

Josh Berkus is a member of the Core Team of PostgreSQL. He is also president of PostgreSQL Experts Inc., and consults on database performance, scalability architecture, data warehousing, and open source community management. In addition to PostgreSQL, he also uses CouchDB, Redis, Hadoop, Greenplum, and Vertica, and participates in many open source projects. Josh is also a potter and amateur chef.

Selena Deckelmann

Selena Deckelmann is a software developer and specializes in open source software and product management. She's an internationally recognized speaker on open source, PostgreSQL, and developer community. She founded Postgres Open, a conference dedicated to the business of PostgreSQL and disruption of the database industry. She founded and co-chaired Open Source Bridge, a developer conference for open source citizens.