Swing: Short-cutting Rings for Higher Bandwidth Allreduce

Authors: 

Daniele De Sensi, Sapienza University of Rome; Tommaso Bonato, ETH Zurich; David Saam, RWTH Aachen University; Torsten Hoefler, ETH Zurich

Abstract: 

The allreduce collective operation accounts for a significant fraction of the runtime of workloads running on distributed systems. One factor determining its performance is the number of hops between communicating nodes, especially on networks like torus, where a higher number of hops implies multiple messages being forwarded on the same link, thus reducing the allreduce bandwidth. Torus networks are widely used on systems optimized for machine learning workloads (e.g., Google TPUs and Amazon Trainium devices), as well as on some of the Top500 supercomputers. To improve allreduce performance on torus networks we introduce Swing, a new algorithm that reduces the number of hops between communicating nodes by swinging between torus directions. Our analysis and experimental evaluation show that Swing outperforms by up to 3x existing allreduce algorithms for vectors ranging from 32B to 128MiB, on different types of torus and torus-like topologies, regardless of their shape and size.

NSDI '24 Open Access Sponsored by
King Abdullah University of Science and Technology (KAUST)

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

BibTeX
@inproceedings {295653,
author = {Daniele De Sensi and Tommaso Bonato and David Saam and Torsten Hoefler},
title = {Swing: Short-cutting Rings for Higher Bandwidth Allreduce},
booktitle = {21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24)},
year = {2024},
isbn = {978-1-939133-39-7},
address = {Santa Clara, CA},
pages = {1445--1462},
url = {https://www.usenix.org/conference/nsdi24/presentation/de-sensi},
publisher = {USENIX Association},
month = apr
}

Presentation Video