Representativeness in the Benchmark for Vulnerability Analysis Tools (B-VAT)

A variety of tools are used to support software vulnerability analysis processes. However, when analysts want to select the optimal tool for a particular use case, or attempt to compare a new tool against others, no standard method is available to do so. In addition, we have determined that although vulnerabilities can be categorized into vulnerability types, these types are often disproportionately represented in current datasets. Hence, when comparative analyses of tools based upon these datasets are conducted, the results are distorted and unrealistic. To address this problem, we are building a Benchmark for Vulnerability Analysis Tools (B-VAT).

Representativeness is a key element of a good benchmark. In this paper, we use stratified sampling to identify a representative set of vulnerabilities from over 800 CWE’s and 75,000 CVE’s. This set becomes the foundation upon which we will build a purpose-based benchmark for vulnerability analysis tools. By using the correlation between current CWE and CVE data, the proposed B-VAT will assess tools using vulnerabilities in the proportions their types occur in the wild.