USENIX Security '25 Call for Artifacts

Artifact Evaluation Information

Overview

A scientific paper consists of a constellation of artifacts that extend beyond the document itself: software, hardware, evaluation data and documentation, raw survey results, mechanized proofs, models, test suites, benchmarks, and so on. In some cases, the quality of these artifacts is as important as that of the paper itself. To emphasize the importance of such artifacts, the benefits to the authors and the community as a whole, and promote the availability and reproducibility of experimental results, USENIX Security runs an Artifact Evaluation. While research artifacts are expected to be available by default and submitted for availability verification by the above given deadlines, authors of accepted papers can optionally submit their artifacts also for functionality and reproducibility assessments. The AEC will review each submitted artifact and also grant Distinguished Artifact Awards to outstanding artifacts accepted to USENIX Security '25.

Process

Differently from previous AE iterations at USENIX Security, this year, the AE will take place in two submission phases: (1) a mandatory AE for availability after paper acceptance and before the final papers are due; (2) an optional AE for functionality and reproducibility after final papers are due. In both cases, the submitted artifacts will be reviewed by the AEC. Artifacts should be submitted in the same cycle as the accepted paper.

1. Mandatory submission of artifacts for availability verification:
   First, authors of accepted papers (including Accepted on Shepherd Approval and Invited for Major Revision papers) are expected to openly share their research artifacts by default and submit them for the AE for availability verification before the final papers are due. If, for some reasons, artifacts cannot be shared, a detailed justification must be provided. For example, some artifacts may be subject to licensing restrictions, NDA, or ethical concerns (in some cases, e.g., interview transcripts or zero-day exploits). Even if the full artifacts cannot be shared (e.g., proprietary datasets), the rest of the artifacts (e.g., source code) is still expected to be shared and submitted to the AE. Papers that do not openly share their artifacts without a satisfactory reason will be rejected. In case of doubt, contact the PC chairs beforehand. Each submitted artifact will be reviewed by the Artifact Evaluation Committee (AEC).
   
   All papers submitted to USENIX Security '25 are expected to discuss the artifacts the authors will make publicly available; the authors should add this discussion in the one additional page allowed for discussing “compliance with the open science policy.” The text should refer to results from the main body of the paper and clearly enumerate all artifacts that will be made available. The AEC will then verify, during the availability verification phase, that all artifacts that were promised to be made available are publicly available.
   
   Please note that the artifacts need to be made available on a platform that supports permanent access. For this purpose, we recommend Zenodo. Other valid hosting options include institutional and third-party digital repositories such as FigShare, Dryad, and Software Heritage. On the other hand, we do not accept artifacts hosted on personal websites or software development repositories such as GitHub. Please see the artifact submission instructions for more details. In case of any questions, please contact the AEC chairs.
   
   N.B.: Authors of shepherded and major revision papers need to submit their artifacts in the same cycle they submitted their paper in. For example, a shepherded paper in Cycle 1 needs to submit their artifacts to the AE for availability verification in Cycle 1 (note that there is just one day between Shepherding/revision author notification and the submission deadline for availability verification for those papers, so authors should anticipate and plan accordingly).
2. Optional submission of artifacts for functionality and reproducibility assessments:
   Second, authors of accepted papers are encouraged to register their artifacts to also be checked for functionality and reproducibility. Additional information regarding artifact registration deadline (for functionality and reproducibility assessments) and files to submit for review will be provided to the authors of accepted papers.

Authors define the contents of their artifact submission. For example, software, hardware, data sets, survey results, test suites, mechanized (but not paper) proofs, access to special hardware, and so on. Authors choose which badges their artifact should be evaluated towards, i.e., one or two of the following: Artifacts Functional and Results Reproduced. In general, good artifacts are expected to be: consistent with the paper, as complete as possible, well documented, and easy to (re)use. The AEC will read the paper and then judge if the artifact meets the criteria for each of the requested badges.

Each artifact submission will be reviewed by at least two AEC members. The review is single-blind and strictly confidential. All AEC members are instructed to treat the artifact confidentially during and after completing evaluation and to not retain any part of the artifact after evaluation. Artifacts can include models, data files, proprietary binaries, exploits under embargo, etc. Even if authors cannot make their artifacts publicly accessible (e.g., proprietary files), they could still apply for Artifacts Functional and Results Reproduced. Since we anticipate small glitches with installation and use, reviewers may communicate with authors during artifact evaluation to help resolve glitches while preserving reviewer anonymity. Please make sure that at least one of the authors is reachable to answer questions in a timely manner.