Amy Tobey and Shelby Spees, Equinix
Equinix Metal runs two dozen software services deployed across 70+ Kubernetes clusters on six continents. The path from plucky startup to global cloud infrastructure player was rocky, to say the least. There were frequent incidents that lasted hours, with engineers poring over logs and dashboards and often walking away unsatisfied.
Principal Engineer Amy Tobey and SRE Shelby Spees share how the Equinix Metal Engineering team deployed OpenTelemetry tracing for the bare metal provisioning process. After initial efforts from the SRE team to open PRs adding instrumentation for each service, they gained momentum by creating on-ramps for engineers across the org to instrument their own code. The shared effort facilitated knowledge transfer for the globally-distributed, multidisciplinary team, empowering veterans and newbies alike to debug issues more quickly and easily.
Amy and Shelby close with examples of system issues they only identified because of tracing, plus a few major reliability wins.
Amy Tobey, Equinix
Amy Tobey has worked in tech for more than 20 years at companies of every size, working with everything from kernel code to user interfaces. These days she spends her time building an innovative Site Reliability Engineering program at Equinix, where she is a principal engineer. When she's not working, she can be found with her nose in a book, watching anime with her son, making noise with electronics, or doing yoga poses in the sun.
Shelby Spees, Equinix
Shelby Spees is a site reliability engineer who's been making the tech industry more accessible and equitable through better engineering practices since 2015. Shelby joined Equinix Metal in 2021 to implement distributed tracing and service-level objectives for the bare-metal provisioning process. Her goal is to help to build a healthy cloud engineering org without firefighting and burnout. Shelby lives in Los Angeles, CA, where she enjoys drinking iced lattes and making up songs about her rescue pitbull, Nova.
SREcon22 Americas Open Access Sponsored by Blameless
author = {Amy Tobey and Shelby Spees},
title = {Tracing Bare Metal with {OpenTelemetry}},
year = {2022},
address = {San Francisco, CA},
publisher = {USENIX Association},
month = mar
}
