eSGD: Communication Efficient Distributed Deep Learning on the Edge

Training machine learning model on IoT device is a natural trend due to the growing computation power and the great ability to collect various data of modern IoT de- vice.In this work, we consider an edge based distributed deep learning framework in which many edge devices collaborate to train a model while using an edge server as the parameter server. However, the high network communication cost of synchronizing gradients and parameters between edge devices and cloud is a bottleneck. We propose a new method called edge Stochastic Gradient Descent (eSGD) for scaling up edge training of convolutional neural networks. eSGD is a family of sparse schemes with both convergence and practical performance guarantees. eSGD includes two mechanisms to improve the first order gradient based optimization of stochastic objective functions in edge scenario. First, eSGD determines which gradient coordinates are important and only transmits important gradient coordinates to cloud for synchronizing. This important update can aggressively reduce the communication cost. Second, momentum residual accumulation is designed for tracking out-of-date residual gradient coordinates to avoid low convergence rate caused by sparse updates. Our experiments show that we reach 91.2%, 86.7%, 81.5% accuracy on MNIST data set with gradient drop ratio 50%, 75%, 87.5% respectively.