Leonardo Antônio dos Santos, Workday, Inc.
Managing Elasticsearch at tens of petabyte scale requires innovative approaches to overcome the limits of traditional single-cluster designs. In this talk, we introduce a scalable, cost-effective multi-cluster architecture that handles trillions of indexed logs monthly while reducing operational complexity. By shifting to a "Cluster of Clusters" design, we optimize ingestion, search, and cross-cluster search traffic using a centralized management cluster and standardized data clusters.
Key highlights include leveraging a custom cluster health service based on the USE Method for intelligent query routing, implementing real-time auditing for problematic query detection, and automating rate-limiting for high-demand users. Attendees will learn how these strategies cut compute costs by 57%, achieved significant storage savings, and enhanced scalability and migration efficiency.
This session provides practical insights, benchmarks, and real-world examples to help organizations sustainably optimize Elasticsearch while maintaining performance and reducing costs — which is ideal for those overseeing large-scale log data or anticipating Elasticsearch growth.
Leonardo Dos Santos is a Senior Distributed Systems Engineer at Workday, specialized in building, maintaining and scaling large distributed systems. With extensive experience managing systems spanning petabytes and thousands of nodes, Leonardo has led large-scale architecture transformations that have optimized performance at large scale and significantly reduced costs. His work at Workday also includes designing globally distributed CI/CD pipelines and creating customized, eventual-consistent solutions for critical infrastructure. Previously, Leonardo held engineering roles at Amazon, where he led innovative and global projects to enhance AWS Network Active Monitoring. He is an active mentor, interviewer and automation advocate.
author = {Leonardo Ant{\^o}nio dos Santos},
title = {Cattle vs. Pets - A {Cost-Effective} Elasticsearch Architecture to {Scale-Out} Beyond Petabytes},
year = {2025},
address = {Santa Clara, CA},
publisher = {USENIX Association},
month = mar
}