Koon Seng Lim and Sandeep Hooda, DBS
Just as in the well-known myth that the mighty elephant cowers before the tiny mouse; the presence of small files in the Hadoop Distributed File System can literally bring the elephant of big data to its knees! In this talk, we chronicle our journey of managing small files after an unfortunate incident rendered our multi-petabyte cluster inactive for close to 5 days. The aftermath of the incident spawned a flurry of collaborative work between our infrastructure SRE, enterprise SRE, platform SRE and application teams. We discuss the various lessons learned and experience gained from experimenting with and implementing various mitigating measures in the domain of people, process, and technology to combat the scourge of small files; a perennial problem in Hadoop. We show that with the proper mitigating controls and technical capabilities afforded by newer distributed file systems, mice and elephants can coexist happily, just as in real life.
Koon Seng Lim, DBS
Koon Seng received a Masters in Electrical Engineering from Columbia University in 1996 and a Bachelors in Computer Science from NUS in 1991. By day, he is an Executive Director and heads the SRE team of DBS Middle Office. By night, he prowls through code with his trusty pet snake Python 3, hunting for the occasional bug. In his past life, he spent 17 years founding and working for various startups in the US before returning to Singapore in 2010.
Sandeep Hooda, DBS
Sandeep leads the SRE team focussed on learning from the incident, containerization, and responsible for various SDLC tools specializing in DevOps. Prior to this, he was managing cloud infrastructure teams where his responsibilities were usually automation, infrastructure architecture, and working closely with solution architects. He is a passionate user of open source with a strong focus on creating quintessential solutions. He conducts workshops to educate and spread awareness on blameless culture "from tech incidents to biz decisions." He is a Sci-Fi lover, with a keen interest in astronomy, dreams of space exploration, and sailing around the world.
SREcon21 Open Access Sponsored by Indeed
author = {Koon Seng Lim and Sandeep Hooda},
title = {Of Mice \& Elephants},
year = {2021},
publisher = {USENIX Association},
month = oct
}
