Tetris: Memory-efficient Serverless Inference through Tensor Sharing

Executing complex, memory-intensive deep learning inference services poses a major challenge for serverless computing frameworks, which would densely deploy and maintain inference models at high throughput. We observe the excessive memory consumption problem in serverless inference systems, due to the large-sized models and high data redundancy.

We present Tetris, a serverless platform catered to inference services with an order of magnitude lower memory footprint. Tetris’s design carefully considers the extensive memory sharing of runtime and tensors. It supports minimizing the runtime redundancy through a combined optimization of batching and concurrent execution and eliminates tensor redundancy across instances from either the same or different functions using a lightweight and safe tensor mapping mechanism. Our comprehensive evaluation demonstrates that Tetris saves up to 93% memory footprint for inference services, and increases the function density by 30× without impairing the latency.