Statement Regarding Recent USENIX Security Peer-Review Allegations

Recently, several prominent members of the computer security and privacy research community received emails alleging that a group of reviewers sought to gain an unfair advantage in the peer-review process via omitted conflicts or collusion. These emails include supposed confidential reviewer assignments for conferences including USENIX Security 2022-2024. This information was shared with recipients who were not on the USENIX Security program committees and who could have been conflicted with the papers involved.

The non-conflicted USENIX Security 2022-2024 program committee (PC) chairs, in consultation with USENIX and non-conflicted members of the USENIX Security Steering Committee (SC), have investigated these allegations and have found no evidence to substantiate the allegations. Given the public nature of the accusations, we are taking the atypical step of publicly sharing this conclusion along with some details on the investigative process. We also wish to clarify expectations regarding conflicts of interest, confidentiality, and issue reporting.

Investigative Process and Outcome

USENIX Security chairs—like the program chairs of all USENIX conferences—take allegations of misconduct seriously. Anyone should feel free to contact the program chairs of conferences privately regarding such concerns. To avoid unintended and unfair consequences for the individuals involved, chairs typically treat allegations against PC members or authors, sanctions resulting from allegations, and details of investigations as confidential. In general, the chairs would not necessarily disclose steps, findings, and sanctions to accusers or others. Due to the public accusations and other details of this particular case, we feel that a public statement best serves the community here, but most investigations do not result in public statements.

The specifics of the investigative process depend on the circumstances. For example, chairs may analyze submissions or artifacts of the review process, and they may consult with a variety of parties. Chairs strive to conduct investigations impartially with an open mind.

In this case, the investigative process included checking review patterns, inspecting bids, examining relative review scores, reviewing discussions of submissions, searching for evidence of undeclared conflicts, requesting details from relevant individuals, and consulting with the chairs of other conferences.

Our investigation did not uncover evidence of intentional cheating, conflicted individuals performing reviews, or other misconduct warranting sanctions. All observed clustering of reviewers and authors was reasonably explained by the reviewers' research focus areas and the topics of the submissions.

Conflicts of Interest

Not all connections between individuals are conflicts of interest under the present rules of USENIX Security. For example, co-organizing an event (conference, workshop, summer school) would not create a conflict. For each iteration of USENIX Security, the conference chairs specify the relationships that constitute conflicts of interest in the Call for Papers or other available submission policies. In some cases, authors or PC members reach out to chairs to clarify atypical cases or to highlight suspected errors. Although some fundamental aspects of USENIX Security’s conflict of interest policies have remained stable for many years, these policies evolve, so authors and PC members should carefully review the latest policies.

In the course of investigating the allegations resulting in this statement, we did not discover critical omitted conflicts. Nevertheless, we note that conflict issues pose considerable challenges for chairs. On a routine basis, chairs discover apparent omitted or unexplained conflicts for both submissions and PC members. The origins of these issues are often innocuous, but the implications can be serious. Following submission deadlines, chairs frequently spend substantial time examining conflicts to avoid potential significant issues later. Our experience suggests that our community could be more proactive in ensuring that conflicts are comprehensive and current, in advising students on what constitutes a conflict, and in deviating from the explicit conflicts guidelines only following consultation with chairs.

We also note that some clustering of reviewers and authors tends to occur naturally and is consistent with existing conflict of interest policies. Clustering can be due to specialization. For example, PC members who specialize in fuzzing will tend to review papers by, and have their own papers reviewed by, non-conflicted individuals who also specialize in fuzzing. This helps ensure authors receive expert reviews and, absent evidence of a conflict or improper conduct, is currently permitted.

Confidentiality Expectations

While we have taken the accuser’s allegations seriously, they shared supposed reviewer assignment details with many other individuals, including individuals who were not on the relevant USENIX Security program committees and who may have had conflicts with particular submissions. Reviewer assignments, details of submissions (other than final published papers), reviews, discussion, and other non-public aspects of the review process are generally confidential. Our community trusts PC members not to share these details more broadly. A violation of that trust is unacceptable.

Beyond violating well-established norms in our community, disclosure of confidential details could also unfairly harm, pressure, intimidate, or embarrass members of our community, and it could even constitute harassment. Those with questions regarding confidentiality expectations should not hesitate to reach out to PC chairs.

How to Report Concerns

In most cases, the PC chairs of a conference are the appropriate parties to investigate concerns. As mentioned above, anyone should feel free to contact the chairs privately regarding concerns. Nevertheless, we recognize that individuals may be uncertain how to report good-faith concerns in exceptional cases, such as concerns related to the chairs. Given that, USENIX has documented and released a reporting process for its conferences.

As discussed above, confidentiality is a key aspect of investigations. Reporting concerns via the official USENIX process supports the goal of a fair conclusion based on a thorough investigation. Although individuals raising concerns may not observe an investigation or learn its outcome, our collective experience suggests that chairs and other senior members of our community take concerns seriously. Confidentiality regularly masks considerable investigative action.

Moving Forward

Beyond clarifying expectations regarding confidentiality and providing a clearer process for reporting concerns, USENIX is also exploring steps to reduce the potential scale and scope of future misuse of reviewing data. However, the USENIX Security review model is built heavily on openness and trust. Members of our community implicitly place considerable trust in each other. That foundation and its tremendous past success will factor into future decisions.

Additionally, USENIX is committed to ensuring that conflict of interest policies are more clearly outlined in all USENIX Calls for Papers and to evolving its mechanisms to provide more assistance to authors and program committee members to flag conflicts accurately.