This page gives the instructions for submitting reproducibility companion papers at the 2019 edition of ACM Multimedia. Such a submission has essentially two parts:
- Companion paper: The companion paper is 2–4 pages in length. It contains a high-level description of the experiments carried out in the original paper and that are implemented in the archive.
- Archive: Contains the artifacts (e.g., code, scripts, data sets, protocols), which are cleanly packaged, ready for download and use to replicate the results from the original paper. It contains detailed readme file(s), examples, and all information needed to successfully carry out the experiments.
These instructions are split into sections that detail:
- The general submission guidelines for 2019,
- The one badge that we consider in 2019, namely the Results Replicated badge,
- What a companion paper typically contains in this case,
- What the archive with the artifacts associated to this companion paper typically contain,
- A few packaging guidelines facilitating the preparation of the archive containing the artifacts,
- The timeline and the 2019 reproducibility committee.
The submission website will open on Feb 15, 2019.
General submission guidelines for 2019
Authors that have a main-conference paper published at ACM Multimedia 2017 and 2018 are invited to submit a short reproducibility companion paper to the new ACM Multimedia Reproducibility track at ACM Multimedia 2019. That companion paper typically focuses on the technical details of what you published at ACM MM 17 or 18.
The companion paper is a short paper that is 2–4 pages long, excluding references. It must follow the standard ACM style format, double column. It has to involve a majority of the authors of the original paper, and provide in the clear their names and affiliations. The original ACM Multimedia 17 or 18 contribution associated to this companion paper must be clearly referenced.
The reproducibility companion paper and the associated artifacts will undergo a reproducibility review, which will result in an accept or a reject decision. Rejected papers are not disclosed.
If accepted, a Results Replicated ACM badge is added to the original paper and to the companion paper, which are both stored in the ACM Digital Library, together with the artifacts.
A badged companion paper will appear in the ACM Multimedia 2019 proceedings and will be presented as a poster in the ACM Multimedia 2019 Reproducibility poster session. The reviewers of the badged companion paper add a section documenting their efforts and become co-authors of the paper.
If, during the evaluation, a serious flaw invalidating the scientific results published in the original contribution is discovered, then the companion paper is rejected, and the authors are encouraged to publish an errata.
Note: If your original paper turns out to be especially challenging to replicate, the committee will recommend that the companion paper be published at ACM Multimedia 2020, instead of ACM Multimedia 2019.
Badge for the 2019 edition
In 2019, we are committed to set the standard high for reproducibility at ACM Multimedia, and the Reproducibility Committee asks authors to target a top quality badge for the companion papers they submit. The target is the Results Replicated badge:
ACM gives the following definition of this Results Replicated badge:
The main results of the paper were replicated by the reviewers and were found to support the central results reported in the paper, using, in part, artifacts provided by the author.ACM DL. Read it here.
Contents of the companion paper
The companion paper must provide means for the committee to download the artifacts that will enable replicating the findings that are in the original ACM Multimedia contribution. The author-created artifacts that are relevant to this paper must have been placed on a publicly accessible archival repository (e.g. github).
The companion paper must describe the procedure allowing reviewers to easily find their way in the artifacts. That procedure might for example explain the reasons for having organized the artifacts in hierarchies of folders. It might specify what items inside the artifacts must be read, and in which order, what are the main elements to fully review first. It might clearly establish relationships between items in the artifacts and the corresponding elements that can be found in the original scientific contribution. It might explain and justify why this or that part of the original paper is not replicable.
Ideally, the companion paper should include text/schemas/illustrations that describe what the artifacts contains and how it should be deployed and then used. The companion paper should also contain notes about parameters that can be set or adjusted and about how to recreate the plots. It has to have examples, with comments. You can of course create documents in the archive that provide additional text/schemas/illustrations. In this case, the companion paper should clearly indicate where that can be found in the artifacts.
Contents of the archive of artifacts
Replicability is grounded in code, scripts, datasets that you provide and that form so called “artifacts”. More formally, ACM defines artifacts as follows:
By “artifact” we mean a digital object that was either created by the authors to be used as part of the study or generated by the experiment itself. For example, artifacts can be software systems, scripts used to run experiments, input datasets, raw data collected in the experiment, or scripts used to analyze results.ACM DL. Read it here.
Artifacts contain digital objects that supplement the companion paper. Artifacts are typically a series of files, possibly organized according to a clear and easy to grasp hierarchy. Artifacts include for example:
- The configuration files and scripts to set up and deploy the environment needed for the reviewers to subsequently run your code,
- The source code if you expose your system as a white box,
- Input Data: Either the process to generate the input data should be made available, or when the data is not generated, the actual data itself or a link to the data should be provided,
- The set of experiments (system configuration and initialization, scripts, workload, measurement protocol, …) used to run the experiments that produce the raw experimental data,
- The scripts needed to transform the raw experimental data into the graphs, tables, plots, …, that can be found in the original submission already published in the proceedings of ACM MM.
All this material should be extensively described, documented, commented, easy to understand, for example in specific files coming together with what they describe.
The central results and claims of the corresponding published paper should be supported by the submitted experiments, meaning we can recreate result data and graphs that demonstrate similar behavior with that shown in that paper. Typically when the results are about response times, we do not expect to get identical results. Instead, we expect to see that the overall behavior matches the conclusions from the paper, e.g., that a given algorithm is significantly faster than another one, or that a given parameter affects negatively or positively the behavior of a system.
Given a system, the authors should provide the complete set of experiments to replicate the results that are in the original paper. Typically, each experiment will consist of the following parts.
- A setup phase where parameters are configured and data is loaded,
- A running phase where a workload is applied and measurements are taken,
- A clean-up phase where the system is prepared to avoid interference with the next round of experiments.
The authors should document (i) how to perform the setup, running and clean-up phases, and (ii) how to check that these phases complete as they should. The authors should document the expected effect of the setup phase (e.g., a cold file cache is enforced) and the different steps of the running phase, e.g., by documenting the combination of command line options used to run a given experiment script.
Each experiment should be automatic, e.g., via a script that takes a range of values for each experiment parameter as arguments, rather than manual, e.g., via a script that must be edited so that a constant takes the value of a given experiment parameter.
We do not expect the authors to perform any additional experiments on top of the ones in their original paper. Any additional experiments submitted will be considered and tested but they are not required.
About Graphs and plots
For each graph/plots in the original paper, the authors should describe how the graph/plot is obtained from the experimental measurements. The submission should contain the scripts (or spreadsheets) that are used to generate the graphs/plots. We strongly encourage authors to provide scripts for all their graphs using a tool such as Gnuplot or Matplotlib. Here are two useful tutorials for Gnuplot: a brief manual and tutorial, and a tutorial with details about creating eps figures and embed them using LaTeX and another two for Matplotlib: examples from SciPy, and a step-by-step tutorial discussing many features.
Similar procedures must be provided by the authors in order to create the tables that are in the original paper.
These packaging guidelines are meant to cover general cases. Please keep in mind that every individual case is slightly different.
Authors should explicitly specify the operating system and tools that should be installed as the environment. Such specification should include dependencies with specific hardware features (e.g., 25 GB of RAM are needed) or dependencies within the environment (e.g., the compiler that should be used must be run with a specific version of the operating system).
System setup is one of the most challenging aspects when replicating experiments. System setup will be easier to conduct if it is automatic rather than manual. Authors should test that the system they distribute can actually be installed in a new environment. The documentation should detail every step in system setup:
- How to obtain the system?
- How to configure the environment if need be (e.g., environment variables, paths)?
- How to compile the system? (existing compilation options should be mentioned)
- How to use the system? (What are the configuration options and parameters to the system?)
- How to make sure that the system is installed correctly?
The above tasks should be achieved by executing a set of scripts provided by the authors that will download needed components (systems, libraries), initialize the environment, check that software and hardware is compatible, and deploy the system.
The committee requests that authors use ReproZip in order to streamline this process. ReproZip can be used to capture the environment, the input files, the expected output files, and the required libraries. A detailed how-to guide (installing, packing experiments, unpacking experiments) can be found in there. ReproZip will help both the authors and the evaluators to seamlessly rerun experiments.
If using ReproZip to capture the experiments proves to be difficult for a particular paper, the committee will work with the authors to find the proper solution based on the specifics of the paper and the environment needed. More tools are available here: https://reproduciblescience.org/reproducibility-directory/
Timeline for 2019
The important dates along the timeline for submitting the companion paper and the associated artifacts are:
- April 1, 2019: deadline for submitting the companion paper
- July 1, 2019: notification of acceptance/rejection,
- August 1, 2019: deadline for preparing the final version of the accepted companion paper,
- End of October 2019: present your reproducibility work at a specific poster session during the 2019 edition of ACM MM in Nice, France.