Summit Program

Data breaches are on the rise and placing increased pressure on Enterprise IT to protect the business. With Hackers taking advantage of known vulnerabilities on unpatched or misconfigured systems, Enterprise IT increasingly needs to automate vulnerability management, security management, and compliance checking. OpenSCAP is an open-source tool for automatically verifying the presence of patches, checking system security configuration settings, and examining systems for signs of compromise.

How to put application secrets/credentials securely into the image has baffled quite a few industry experts. The solutions that people employ are insecure, static, and not scalable. Novice users bake secrets into the image. When they publish it to the registry, anyone can pull the image and secrets would be at their disposal.

Advanced users mount the secrets through volumes during the container run. Someone who has access to the machine can mount the same volume and can access secrets for all the images. Other advanced users pass it through ENV variables which is open for snooping for anybody that has access to the machine.

Paranoid users employ public key and elliptic key cryptography to encrypt the secrets in the image using public key. Image is then pushed to the registry. It is safe from snooping. And the secrets can be decrypted only through the private key that is resident on the host machine. But again, this solution is static.

None of these aforementioned solutions truly fit the ephemeral nature of the containers. We should be able to provide secrets to the running container on any machine dynamically during the runtime.

We, at YellowPages, have devised a solution that addresses that concern.

Imran Shaikh breathes DevOps, embodies it and that is what will ooze out if you cut him. He is a Lead Systems Engineer working at YellowPages. He has an industry experience of 10+ year working with Fortune 500 companies. He has worked extensively developing, architecting and managing cloud technologies at YP as well as Yahoo. He has substantial experience running a globally distributed production environment on thousands of systems running hundreds of application in a complex, fast moving and mission critical environment.

Presently, his team is deploying and developing Mesos supported technologies that works at scale. Mesos solutions for centralized logging, metrics, distributed monitoring, distributed storage, application secrets etc. are being worked upon. His team is solving issues which some people think don't even exist.

Tupperware is Facebook's managed deployment system for containers. A scheduler controls how jobs are deployed to machines. An agent running on each machine is responsible for controlling the lifecycle of each process scheduled on it. This talk will focus on the agent, its separation of responsibilities, and how it interacts with the host it runs on. Come learn how we've implemented the system, what challenges we've had in doing so, and what our plans for the future are.

Container technologies are revolutionizing the way we develop, build and deploy applications in large scale production environments. At Yahoo we use containers in our CI build farms and production environments, that are on-demand and dynamic in nature. Applications running in containers often need to connect to various internal/external services that require authentication and authorization. Authenticating client application to a server is a challenge in such dynamic environments because we cannot rely on traditional IP or hostname based checks. IP based authentication no longer works because (1) container IP is dynamic and often repurposed (2) containers often share IPs. Alternate options include the use of TLS client certs and other key based authentication schemes. TLS client certificates provide authentication, but not authorization capabilities by its own and is not easy to configure and operate at scale ­ think about build pipeline spawning hundreds of containers that live only for few minutes!

In this session, we present a novel way of role based identity that provides both authentication and authorization to clients in a fully­automated, easy to configure, scalable fashion. The system comprises of (a) APIs for node and application provisioners to manage and publish public keys (b) Service that provides grouping of public key fingerprints of nodes/applications to form a service role that represents a capability and (c) Attestation service for the nodes to get a signed certificate on demand that asserts the requested node's role membership. The service provider maps the role with service specific capabilities and the requests are validated against the auth certificate placed by the client while making requests to the server. The system is designed from ground up based on our experience with an existing IP based authorization system, keeping practicality, flexibility and security in mind. The implementation makes use of modern security and crypto practices and such as ECDSA, JWT with service delegation capabilities, and works seamlessly with Docker and Chef.

Aditya Mahendrakar is a senior security engineer in the Paranoid Labs team at Yahoo. He has worked on a number of projects including a key management system, static code analysis framework, and input validation libraries. He received his Master's degree at Carnegie Mellon.

Binu Ramakrishnan is a senior security engineer at yahoo with extensive experience in Internet­-scale systems development, anti­abuse and application security. In this role, Binu manages security engagements with Yahoo mail, works with product engineers and leaders to help define and implement security strategy and programs with in Yahoo mail. Prior to this role, Binu worked as a lead engineer with Security and Platforms engineering team, built hosted key management service and managed various shared components that are used across Yahoo.

Linux container technology is seeing rapid adoption in the last few years with its usage and enhancement as platform for building and deploying applications, for example, by Docker. Services are being stood up in public clouds that offer different frameworks for launching and managing user-built containers. At the same time cloud services are beginning to examine container-based deployment as possible alternative to or in conjunction with virtual machines for deploying those services. However, when multiple tenants’ containers get deployed on the same system there is little inherent isolation between tenants in existing containers-as-a-service platforms.

In the Spyre project we focus on developing a resource management framework that allows us to provide performance isolation between multiple tenants deploying containers in the cloud. We introduce the notion of a slice that provides the resource partition for one tenant within a system which com-prises of a defined set of resources—cores, memory, memory bandwidth, network bandwidth, storage, storage bandwidth etc. These are unique to that slice and disjoint from the resources commandeered for any other slice.

In this presentation, we socialize our concept of slices and discuss how they provide the performance isolation important to multi-tenant cloud services that care about tail latencies among other things. We discuss how we think they relate to concepts in existing container platforms and can be adopted for meeting performance-sensitive needs for cloud services. We then introduce our current implementation and invite audience to participate in a broader discussion on the challenges for performance isolation and management for container-based cloud frameworks.

As modern datacenters move more towards a container-centric world, there are a number of open issues to be explored with respect to networking. In this talk I’ll cover many of the approaches being taken or explored at Facebook with respect to container networking. This includes transport security, network virtualization (with and without encapsulation), overlay networking, container-address allocation, utilizing features of IPv6 to simplify management and deployment, address (and job) migration, and resource isolation. One of the current open issues being worked on is how to provide resource isolation for ingress network traffic. That is, resource controls for network traffic arriving at the host supporting a container. While there are a number of approaches for providing resource controls for egress traffic, providing similar controls for ingress traffic has both been less explored and currently has no widely adopted best practices. In this talk I will additionally discuss the topic of resource controls for ingress traffic and about how we are working on improving cgroups and TCP in the Linux kernel at Facebook to address this specific issue. This talk will be interesting to people who are deploying or thinking about deploying large container-based environments.

Blake Matheny is an Engineering Director at Facebook where he is responsible for systems including the Linux kernel as well as Tupperware, the Facebook cluster and job management infrastructure. Blake has been working on large scale distributed systems for more than 10 years, and is currently enamored with C++11, approaches to asynchronous computation, and scheduling algorithms. Although formerly from NYC, Blake currently lives in California with a cat that hates him, his wife, and their collection of books. You can follow Blake at http://fb.me/blake.r.matheny or @bmatheny.