Toward Orchestration of Complex Networking Experiments

Experimentation is an essential tool for developing networked and distributed systems. However, it is inherently complex due to the concurrent, asynchronous, heterogeneous, and prototype-based systems that must be integrated into representative scenarios to conduct valid evaluations. This paper offers a retrospective on the development and use of MAGI, an orchestration tool, that translates an experiment specification into an execution on an emulation-based testbed with high-level directives for message passing, remote process execution, and failure tracking, for conducting large and complex experiments. The MAGI tool has been used for more than seven years in a variety of experiments, including undergraduate education, anonymous communication, cyber-physical systems, and attacker-defender games on the DETER testbed. We hope the insights and takeaways learned from using our tool will aid in developing the next-generation experiment management tools.