Rode0day: A Year of Bug-Finding Evaluations

Why are some bugs so hard to find? Why are some bug-finding tools more effective than others? How can we improve bug-finding tools? In May 2018, we launched Rode0day, a monthly bug-finding competition designed to answer these questions. In our first year of competitions, we injected thousands of synthetic bugs into more than 50 programs, evaluated 35 bug-finders as they searched for bugs, and collected information on when teams found bugs as well as properties of the bugs themselves. In this talk we will present our analysis of this data and use it identify strengths and weaknesses of tools, discuss what properties of an injected bug make it easy or hard, and suggest ways of improving bug-finders.