Overview
A scientific paper consists of a constellation of artifacts that extend beyond the document itself: software, hardware, evaluation data and documentation, raw survey results, mechanized proofs, models, test suites, benchmarks, and so on. In some cases, the quality of these artifacts is as important as that of the document itself, which is why WOOT organizes an optional artifact evaluation process, inspired by similar efforts in software engineering and security conferences.
Important Dates
All deadlines are at 23:59 AoE (Anywhere on Earth) time:
- Paper acceptance notifications to authors: Thursday, April 10, 2025
- Artifact submission deadline: Monday, May 26, 2025
- Discussion period starts: Wednesday, May 28, 2025
- Badges notifications to authors: Sunday, June 15, 2025
- Final papers due: Tuesday, June 17, 2025
Badges
Authors can choose to have their artifact evaluated against the following badges:
- Artifact Available. The artifact is publicly available.
- Artifacts Functional. The core functionality of the artifact could be confirmed by the AEC.
- Results Reproduced. The results, as reported in the paper, were reproduced by the AEC.
Notice that the three badges can be issued independently.
Benefits
We believe the dissemination of artifacts benefits our science and engineering as a whole, as well as the authors submitting them. Their availability improves replicability and reproducibility and enables authors to build on top of each other's work. It can also help more unambiguously resolve questions about cases not considered by the original authors. The authors receive recognition, leading to higher-impact papers, and also benefit themselves from making code reusable.
Process
Artifact evaluation is a separate process from paper reviews, and authors will be asked to submit their artifacts only after their papers have been (conditionally) accepted for publication at WOOT. After the artifact submission, at least one member of the AEC will download and install the artifact (where relevant) and evaluate it. Since we anticipate minor glitches with installation and usage, reviewers may communicate with authors to help resolve glitches while preserving reviewer anonymity. The AEC will complete its evaluation and notify the authors of the outcome.
We currently aim to complete the initial artifact evaluation process in time for badges to be included in the final papers, but authors are expected to continue to engage with the AEC after badges have been awarded to clarify/resolve any remaining issues/concerns. Where possible, authors are encouraged to provide a stable link to the location that will contain the final version of their artifacts in their final papers; for example, by reserving a DOI on Zenodo ahead of time.
The awarded badges will also be documented separately at https://secartifacts.github.io/, and be announced during the conference itself.
Artifact Details
To avoid excluding some papers, the AEC will try to accept any artifact that authors wish to submit. These can be software, hardware, data sets, survey results, test suites, mechanized proofs, etc. Given the experience in other communities, we decided not to accept paper proofs in the artifact evaluation process. The AEC lacks the time and often the expertise to carefully review paper proofs. Obviously, the better the artifact is packaged, the more likely the AEC can actually work with it during the evaluation process. Thus, we kindly ask authors to submit the artifact alongside setup scripts for reproducible software environments to build and run the artifact (if possible). This could, for instance, be build files for a Docker container or setup scripts for a Vagrant VM.
While we encourage open research, submission of an artifact does not include tacit permission to make its content public. All AEC members will be instructed not to publicize any part of your artifact during or after the evaluation, nor retain any part after evaluation. Thus, you are free to include, e.g., models, data files, or proprietary binaries in your artifact. Also, note that participating in the AEC experiment does not require you to publish your artifacts unless you apply for the Artifact Available badge. Still, we strongly encourage you to do so.
We recognize that some artifacts may attempt to perform malicious operations by design. These cases should be boldly and explicitly flagged in detail in the readme so AEC members can take appropriate precautions before installing and running these artifacts. The evaluation of exploits and similar results might lead to additional hurdles where we still need to collect experience on how to handle this best. Please contact the AE chair if you have concerns, for example, when evaluating bug-finding tools or other types of artifacts that need special requirements.
Submission Instructions
Authors of accepted papers who want to participate in artifact evaluation are expected to submit the following via the artifact submission site.
- A PDF with an abstract for the artifact. The abstract should specify the core idea, the focus of the artifact, and what the evaluation should check.
- A PDF of the most recent version of the accepted paper.
- Documentation about the artifact. Describing what to test and/or how to reproduce the contributions of the paper.
- A link to the artifact. Accessible via a stable reference or DOI. For the purpose of publicly available artifacts, we recommend Zenodo. Since the artifact can evolve during the evaluation to address feedback from the reviewers, another (potentially different) stable reference will be later collected for the final version of the artifact.
Submissions are single-blind, i.e., artifacts do not need to be anonymized. However, authors must ensure that the evaluation process preserves reviewers' anonymity. If possible and applicable, we strongly encourage authors to submit their artifact in combination with setup scripts for reproducible software environments.
We expect submitted artifacts to be:
- consistent with the paper
- as complete as possible
- documented well
- easy to reuse, facilitating further research
For more details, please refer to the Process and Artifact Details above. For any additional questions, please reach out to the artifact evaluation chair via woot25aec@usenix.org.