Store-Edge RippleStream: Versatile Infrastructure for IoT Data Transfer

Powerful edge compute frameworks address the issue of latency for IoT data processing at the edge. However, continuous application layer WAN streaming to core for consolidated deep analysis and learning consumes excessive bandwidth and becomes the bottleneck for responsiveness at the core. A state-of-the-art storage or hyperconverged system, on the other hand, advertises compelling in-built features like WAN efficient data protection and delta replication, global unified management, space and bandwidth saving through inline data compression and deduplication. Traditional storage semantics and services, however, are built for data at rest while edge analytics prioritizes responsiveness by processing and moving streaming data at an application layer. In this paper, we propose to enable streaming of IoT data transparently through storage replication. Based on this foundation, we further present light-weight storage plugins to reduce IoT data transfer by detecting and translating semantic redundancies to a deduplication friendly form. Our early results demonstrate that (a) leveraging storage to take responsibility of streaming data in an application consistent way results in efficient data transfer (b) real-world IoT time-series datasets exhibit a high degree of similarity which can be detected to reduce data transfer from edge (c) video streams for autonomous cars, the transfer of which cannot be reduced enough using traditional video compression or storage deduplication techniques, have significant semantic redundancy. Collectively, advancing research in this direction paves the way to enhance the versatility of state-of-the-art infrastructure for optimized edge computing.