Ajeet Grewal, Jerry Jiang, Gary Lam, Tristan Jung, Lohith Vuddemarri, Quannan Li, and Aaditya Landge, Twitter; Jimmy Lin, University of Waterloo
We present RecService, a distributed real-time graph processing engine that drives billions of recommendations on Twitter. Real-time recommendations are framed in terms of a user's social context and real-time events incident on that social context, generated from ad hoc point queries and long-lived standing queries. Results form the basis of downstream processes that power a variety of recommendation products. A noteworthy aspect of the system's design is a partitioning scheme whereby manipulations of graph adjacency lists are local to a cluster node. This eliminates cross-node network traffic in query execution, enabling horizontal scalability and avoiding "hot spots" caused by vertices with large degrees.
Open Access Media
USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.
author = {Ajeet Grewal and Jerry Jiang and Gary Lam and Tristan Jung and Lohith Vuddemarri and Quannan Li and Aaditya Landge and Jimmy Lin},
title = {{RecService}: Distributed {Real-Time} Graph Processing at Twitter},
booktitle = {10th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 18)},
year = {2018},
address = {Boston, MA},
url = {https://www.usenix.org/conference/hotcloud18/presentation/grewal},
publisher = {USENIX Association},
month = jul
}
