Sponsored by USENIX, the Advanced Computing Systems Association.
All authors of accepted USENIX Security '23 papers (including shepherd approved, but not major revisions) are encouraged to submit artifacts for Artifact Evaluation (AE). Artifacts can be submitted in the same cycle as the accepted paper or in any of the following cycles for 2023. Each submitted artifact will be reviewed by the Artifact Evaluation Committee (AEC). Before submitting your artifact, please check the Artifact Evaluation Information below. Should you have any questions or concerns, you can reach the AEC chairs at sec23aec@usenix.org.
Important Dates
Summer Deadline
- Notification to authors: Friday, September 2, 2022, 11:59 pm AoE
- Artifact Registration deadline: Friday, September 30, 2022
- Final papers due: Tuesday, October 4, 2022
- Artifact submission deadline: Tuesday, October 11, 2022
- Answering AE reviewer questions: Monday, October 17 to Friday, November 11, 2022
- Artifact decisions announced: Tuesday, November 22, 2022
- Final appendix files due: Tuesday, November 29, 2022
Fall Deadline
- Notification to authors: Friday, January 27, 2023, 11:59 pm AoE
- Artifact registration deadline: Friday, February 24, 2023
- Final paper files due: Tuesday, February 28, 2023
- Artifact submission deadline: Tuesday, March 7, 2023
- Answering AE reviewer questions: Monday, March 13 to Friday, April 7, 2023
- Artifact decisions announced: Tuesday, April 18, 2023
- Final appendix files due: Tuesday, April 25, 2023
Winter Deadline
- Notification to authors: Monday, May 8, 2023
- Artifact registration deadline: Friday, June 9, 2023
- Final paper files due: Tuesday, June 13, 2023
- Artifact submission deadline: Wednesday, June 20, 2023
- Answering AE reviewer questions: Monday, June 26 to Friday, July 21, 2023
- Artifact decisions announced: Tuesday, August 1, 2023
- Final appendix files due: Tuesday, August 8, 2023
Artifact Evaluation Committee
Artifact Evaluation Committee Co-Chairs
Cristiano Giuffrida, Vrije Universiteit Amsterdam
Anjo Vahldiek-Oberwagner, Intel Labs
Artifact Evaluation Publication Chair
Alexios Voulimeneas, Delft University of Technology
Artifact Evaluation Committee
Shubham Agarwal, CISPA Helmholtz Center for Information Security
Nikolaos Alexopoulos, Technical University of Darmstadt
Amit Seal Ami, College of William & Mary
Daniel Arp, University College London
Erin Avllazagaj, University of Maryland College Park
Jessy Ayala, University of California, Irvine
Alessandro Baccarini, University at Buffalo
David Balash, The George Washington University
Nils Bars, CISPA Helmholtz Center for Information Security
Raounak Benabidallah, CEA LIST, Université Paris-Saclay
Shay Berkovich, BlackBerry
Lukas Bernhard, Ruhr University Bochum
Yohan Beugin, University of Wisconsin—Madison
Jakob Bleier, Technische Universität Wien
Amel Bourdoucen, Aalto University
Clemens-Alexander Brust, DLR Institute of Data Science
Jiahao Cao, Tsinghua University
Marco Casagrande, EURECOM
Stefanos Chaliasos, Imperial College London
Guangke Chen, ShanghaiTech University
Weiteng Chen, Microsoft Research
Pascal Cotret, ENSTA Bretagne
Daniel De Almeida Braga, Université de Rennes, CNRS, IRISA
Giulio De Pasquale, King's College London
Luca Degani, University of Trento
Nurullah Demir, Karlsruhe Institute of Technology
Omkar Dilip Dhawal, Indian Institute of Technology Madras
Sayanton Dibbo, Dartmouth College
Aneet Kumar Dutta, CISPA Helmholtz Center for Information Security
Alessandro Erba, CISPA Helmholtz Center for Information Security
Yusi Feng, University of Chinese Academy of Science
Christof Ferreira Torres, ETH Zurich
Lucas Franceschino, Inria
Chuanpu Fu, Tsinghua University
Alexandre Gonzalvez, Université de Rennes, CNRS, IRISA
Sindhu Reddy Kalathur Gopal, University of Wyoming
Sanket Goutam, Stony Brook University
Vishal Gupta, EPFL
HyungSeok Han, Georgia Institute of Technology
Muhammad Haseeb, New York University
Ningyu He, Peking University
Tiago Heinrich, Federal University of Paraná
Son Ho, Inria
Hailong Hu, University of Luxembourg
Shengtuo Hu, Meta
Yang Hu, The University of Texas at Austin
Hai Huang, CISPA Helmholtz Center for Information Security
Qiqing Huang, University at Buffalo
Raphael Isemann, Vrije Universiteit Amsterdam
Fabian Ising, FH Münster University of Applied Sciences
Charlie Jacomme, Inria
Jafar Haadi Jafarian, University of Colorado Denver
Yilin Ji, Karlsruhe Institute of Technology
Evan Johnson, University of California, San Diego
Kaushal Kafle, College of William & Mary
Imtiaz Karim, Purdue University
Soheil Khodayari, CISPA Helmholtz Center for Information Security
Hyungsub Kim, Purdue University
Soomin Kim, Korea Advanced Institute of Science and Technology (KAIST)
Rachel King, University of Wisconsin—Madison
John Kressel, University of Manchester
Anunay Kulshrestha, Princeton University
Guilhem Lacombe, CEA LIST
Hieu Le, University of California, Irvine
Hugo Lefeuvre, The University of Manchester
Caihua Li, Yale University
Yuan Li, Zhejiang University
Xu Lin, University of Illinois Chicago
Zhibo Liu, Hong Kong University of Science and Technology
Eleonora Losiouk, University of Padua
Mulong Luo, Cornell University
Ning Luo, Yale University
Charles Babu M, CEA LIST, Université Paris-Saclay
Eman Maali, Imperial College London
Yanmao Man, ByteDance, Inc
Grégoire Menguy, CEA LIST, Université Paris-Saclay
Vladislav Mladenov, Ruhr University Bochum
Shouvick Mondal, Concordia University
Kazi Samin Mubasshir, Purdue University
Abdun Nihaal, Indian Institute of Technology Madras
Yu Nong, Washington State University
Eric Pauley, University of Wisconsin—Madison
Imranur Rahman, North Carolina State University
Mirza Masfiqur Rahman, Purdue University
Dilip Ravindran
Jenni Reuben, Totalförsvarets forskningsinstitut (FOI)
Joe Rowell, Royal Holloway, University of London
Abhinaya S.B., North Carolina State University
Nuno Sabino, Carnegie Mellon University, Portugal
Saiful Islam Salim, University of Arizona
Solmaz Salimi, Sharif University of Technology
Harshad Sathaye, Northeastern University
Tobias Scharnowski, Ruhr University Bochum
Nico Schiller, Ruhr University Bochum
Till Schlüter, CISPA Helmholtz Center for Information Security
Basavesh Ammanaghatta Shivakumar, Max Planck Institute for Security and Privacy (MPI-SP)
Pradyumna Shome, Georgia Institute of Technology
Sudheesh Singanamalla, University of Washington
Sachin Kumar Singh, University of Utah
Johnny So, Stony Brook University
Salwa Souaf, CEA LIST, Université Paris-Saclay
Avinash Sudhodanan, Meta
Xi Tan, University at Buffalo
Zahra Tarkhani, Microsoft and University of Cambridge
Erik Tews, University of Twente
Rodothea Myrsini Tsoupidi, KTH Royal Institute of Technology
Sai Venkata Krishnan V, Indian Institute of Technology Madras
Emanuele Vannacci, Vrije Universiteit Amsterdam
Dawei Wang, SKLOIS, Institute of Information Engineering, Chinese Academy of Sciences
Ningfei Wang, University of California, Irvine
Shu Wang, George Mason University
Wenxi Wang, The University of Texas at Austin
Yuke Wang, University of California, Santa Barbara
Alexander Warnecke, Technische Universität Braunschweig
Feng Wei, University at Buffalo
Shijia Wei, The University of Texas at Austin
Ruoyu Wu, Purdue University
Dongwei Xiao, Hong Kong University of Science and Technology
Jiacen Xu, University of California, Irvine
Peisen Yao, Zhejiang University
Yuanyuan Yuan, The Hong Kong University of Science and Technology
Insu Yun, Korea Advanced Institute of Science and Technology (KAIST)
Samee Zahur, Google
Menghao Zhang, Tsinghua University and Kuaishou Technology
Ruiyi Zhang, CISPA Helmholtz Center for Information Security
Shaohu Zhang, North Carolina State University
Yiming Zhang, The Hong Kong Polytechnic University
Zhiyu Zhang, SKLOIS, Institute of Information Engineering, Chinese Academy of Sciences
Zhiyuan Zhang, University of Adelaide
Xia Zhou, Zhejiang University
Xin'an Zhou, University of California, Riverside
Xugui Zhou, University of Virginia
Huadi Zhu, University of Texas at Arlington
Weidong Zhu, University of Florida
Maximilian Zinkus, Johns Hopkins University
Are you an experienced graduate student, postdoc, or researcher and interested in promoting reusability and reproducibility of security research? Join us! You can apply to be part of the AEC.
Artifact Evaluation Information
Overview
A scientific paper consists of a constellation of artifacts that extend beyond the document itself: software, hardware, evaluation data and documentation, raw survey results, mechanized proofs, models, test suites, benchmarks, and so on. In some cases, the quality of these artifacts is as important as that of the document itself. To emphasize the importance of such artifacts, the benefits to the authors and the community as a whole, and promote the reproducibility of experimental results, USENIX Security will run its fourth (optional) AE this year. The AEC will review each submitted artifact and also grant Distinguished Artifact Awards to outstanding artifacts accepted to USENIX Security '23.
Process
To maintain a wall of separation between paper review and the artifacts, authors will be given the option to submit their artifacts only after their papers have been accepted for publication at USENIX Security. The artifact submission deadline is around five weeks after the paper notification date, but a first stub submission to register the artifact is required around two weeks after the paper notification date. By the artifact submission deadline, authors can submit their artifacts, Artifact Appendix, and other supporting information of their accepted USENIX Security 2023 paper via the submission form using the provided submission instructions.
At artifact submission time, authors can provide artifacts including software, hardware, data sets, survey results, test suites, mechanized (but not paper) proofs, access to special hardware, and so on. In addition, authors can request their artifact to be evaluated towards one, two, or all three of the following badges: Artifacts Available, Artifacts Functional, and Results Reproduced. In general, good artifacts are expected to be: consistent with the paper, as complete as possible, documented well, and easy to (re)use. The AEC will read the paper and then judge if the artifact meets the criteria for each of the requested badges.
Each artifact submission will be reviewed by at least two AEC members. The review is single-blind and strictly confidential. All AEC members will be instructed that they may not publicize any part of your artifact during or after completing evaluation, nor retain any part of it after evaluation. Thus, you are free to include models, data files, proprietary binaries, exploits under embargo, etc. in your artifact. Since we anticipate small glitches with installation and use, reviewers may communicate with authors for a period of up to five weeks after the artifact submission deadline to help resolve glitches while preserving reviewer anonymity.
The AEC will then complete its evaluation and notify the authors of the outcome around six weeks after the artifact submission deadline. Please make sure that at least one of the authors is reachable to answer questions in a timely manner.
As the notification of the AE is after the camera-ready deadline, badges cannot directly appear on the published papers. Nevertheless, authors are given the possibility to add the awarded badges and the Artifact Appendix on the author-version PDF that they can later host on their website or an archive. Moreover, USENIX will publish the badges and the appendices on the conference website. Finally, the badges and the artifact appendices will also be published by USENIX after the conference in dedicated proceedings. The deadline to submit the final artifacts, Artifact Appendix with badges, and other supporting information is around one week after the artifact notification date.
Acknowledgements
The AE process at USENIX Security '23 is a continuation of the AE process at USENIX Security '20–'22 and was inspired by multiple other conferences, such as OSDI, EuroSys, and several other systems conferences. See artifact-eval.org for the origins of the AE process.