Automatic Problem Generation for Capture-the-Flag Competitions

Computer security games, especially capture-the-flag (CTF) competitions, are growing in popularity. A typical CTF contest presents users with a set of hacking challenges, where correct solutions reveal a text “flag” that can be submitted to a scoring server. In traditional CTF architectures, the problem and the flag are the same across the competition.

In this paper we discuss automatic problem generation (APG), where a given challenge is not fixed, but rather can have many different automatically generated problem instances. APG offers players a unique competition experience and can facilitate deliberate practice where problems vary just enough to make sure a user can replicate the solution idea. APG also allows competition administrators the ability to detect when users submit a copied flag from another user to the scoring server. In 2014 we ran a large-scale CTF competition called PicoCTF, where we measured the prevalence of flag sharing. Our results indicate that about 0.8% of flags submitted to AGP problems were copied, with 14% of teams submitting at least one shared flag. In 68% of flag sharing cases, teams went on to eventually solve the problem on their own.