ACM MM 2019 — Call for Reproducibility papers

This page gives the instructions for submitting reproducibility companion papers at the 2019 edition of ACM Multimedia. Such a submission has essentially two parts:

  • Companion paper: The companion paper is 2–4 pages in length. It contains a high-level description of the experiments carried out in the original paper and that are implemented in the archive.
  • Archive: Contains the artifacts (e.g., code, scripts, data sets, protocols), which are cleanly packaged, ready for download and use to replicate the results from the original paper. It contains detailed readme file(s), examples, and all information needed to successfully carry out the experiments.

These instructions are split into sections that detail:

The site for submission is open: https://easychair.org/journal/ACMMM


General submission guidelines for 2019

Authors that have a main-conference paper published at ACM Multimedia 2017 and 2018 are invited to submit a short reproducibility companion paper to the new ACM Multimedia Reproducibility track at ACM Multimedia 2019. That companion paper typically focuses on the technical details of what you published at ACM MM 17 or 18.

The companion paper is a short paper that is 2–4 pages long, excluding references. It must follow the standard ACM style format, double column. It has to involve a majority of the authors of the original paper, and provide in the clear their names and affiliations. The original ACM Multimedia 17 or 18 contribution associated to this companion paper must be clearly referenced.

The reproducibility companion paper and the associated artifacts will undergo a reproducibility review, which will result in an accept or a reject decision. Rejected papers are not disclosed.
If accepted, a Results Replicated ACM badge is added to the original paper and to the companion paper, which are both stored in the ACM Digital Library, together with the artifacts.
A badged companion paper will appear in the ACM Multimedia 2019 proceedings and will be presented as a poster in the ACM Multimedia 2019 Reproducibility poster session. The reviewers of the badged companion paper add a section documenting their efforts and become co-authors of the paper.

If, during the evaluation, a serious flaw invalidating the scientific results published in the original contribution is discovered, then the companion paper is rejected, and the authors are encouraged to publish an errata.

Note: If your original paper turns out to be especially challenging to replicate, the committee will recommend that the companion paper be published at ACM Multimedia 2020, instead of ACM Multimedia 2019.


Badge for the 2019 edition

In 2019, we are committed to set the standard high for reproducibility at ACM Multimedia, and the Reproducibility Committee asks authors to target a top quality badge for the companion papers they submit. The target is the Results Replicated badge:

ACM gives the following definition of this Results Replicated badge:

The main results of the paper were replicated by the reviewers and were found to support the central results reported in the paper, using, in part, artifacts provided by the author.ACM DL. Read it here.

 


Contents of the companion paper

The companion paper must provide means for the committee to download the artifacts that will enable replicating the findings that are in the original ACM Multimedia contribution. The author-created artifacts that are relevant to this paper must have been placed on a publicly accessible archival repository (e.g. github).

The companion paper must describe the procedure allowing reviewers to easily find their way in the artifacts. That procedure might for example explain the reasons for having organized the artifacts in hierarchies of folders. It might specify what items inside the artifacts must be read, and in which order, what are the main elements to fully review first. It might clearly establish relationships between items in the artifacts and the corresponding elements that can be found in the original scientific contribution. It might explain and justify why this or that part of the original paper is not replicable.

Ideally, the companion paper should include text/schemas/illustrations that describe what the artifacts contains and how it should be deployed and then used. The companion paper should also contain notes about parameters that can be set or adjusted and about how to recreate the plots. It has to have examples, with comments. You can of course create documents in the archive that provide additional text/schemas/illustrations. In this case, the companion paper should clearly indicate where that can be found in the artifacts.


Contents of the archive of artifacts

Replicability is grounded in code, scripts, datasets that you provide and that form so called “artifacts”. More formally, ACM defines artifacts as follows:

By “artifact” we mean a digital object that was either created by the authors to be used as part of the study or generated by the experiment itself. For example, artifacts can be software systems, scripts used to run experiments, input datasets, raw data collected in the experiment, or scripts used to analyze results.ACM DL. Read it here.

 
Artifacts contain digital objects that supplement the companion paper. Artifacts are typically a series of files, possibly organized according to a clear and easy to grasp hierarchy. Artifacts include for example:

  1. The configuration files and scripts to set up and deploy the environment needed for the reviewers to subsequently run your code,
  2. The source code if you expose your system as a white box,
  3. Input Data: Either the process to generate the input data should be made available, or when the data is not generated, the actual data itself or a link to the data should be provided,
  4. The set of experiments (system configuration and initialization, scripts, workload, measurement protocol, …) used to run the experiments that produce the raw experimental data,
  5. The scripts needed to transform the raw experimental data into the graphs, tables, plots, …, that can be found in the original submission already published in the proceedings of ACM MM.

All this material should be extensively described, documented, commented, easy to understand, for example in specific files coming together with what they describe.

About Experiments

The central results and claims of the corresponding published paper should be supported by the submitted experiments, meaning we can recreate result data and graphs that demonstrate similar behavior with that shown in that paper. Typically when the results are about response times, we do not expect to get identical results. Instead, we expect to see that the overall behavior matches the conclusions from the paper, e.g., that a given algorithm is significantly faster than another one, or that a given parameter affects negatively or positively the behavior of a system.

Given a system, the authors should provide the complete set of experiments to replicate the results that are in the original paper. Typically, each experiment will consist of the following parts.

  • A setup phase where parameters are configured and data is loaded,
  • A running phase where a workload is applied and measurements are taken,
  • A clean-up phase where the system is prepared to avoid interference with the next round of experiments.

The authors should document (i) how to perform the setup, running and clean-up phases, and (ii) how to check that these phases complete as they should. The authors should document the expected effect of the setup phase (e.g., a cold file cache is enforced) and the different steps of the running phase, e.g., by documenting the combination of command line options used to run a given experiment script.

Each experiment should be automatic, e.g., via a script that takes a range of values for each experiment parameter as arguments, rather than manual, e.g., via a script that must be edited so that a constant takes the value of a given experiment parameter.

We do not expect the authors to perform any additional experiments on top of the ones in their original paper. Any additional experiments submitted will be considered and tested but they are not required.

About Graphs and plots

For each graph/plots in the original paper, the authors should describe how the graph/plot is obtained from the experimental measurements. The submission should contain the scripts (or spreadsheets) that are used to generate the graphs/plots. We strongly encourage authors to provide scripts for all their graphs using a tool such as Gnuplot or Matplotlib. Here are two useful tutorials for Gnuplot: a brief manual and tutorial, and a tutorial with details about creating eps figures and embed them using LaTeX and another two for Matplotlib: examples from SciPy, and a step-by-step tutorial discussing many features.

Similar procedures must be provided by the authors in order to create the tables that are in the original paper.

Ideal archive of artifacts

Authors are encourage to strive for this ideal submission for truly replicable work…

At a minimum the authors should provide a complete set of scripts to install the system, produce the data, run experiments and produce the resulting graphs along with detailed readme file(s) that describe the process step by step so it can be easily redone by a reviewer.

The ideal submission consists an extremely careful and detailed description of the experiments, their parameters, and of a master script that:

  1. installs all systems needed,
  2. generates or fetches all needed input data,
  3. reruns all experiments and generates all results,
  4. generates all graphs, plots and tables, and finally,
  5. recompiles the sources of the paper

… to produce a brand new PDF for the paper with all the replicated experiments. It’ll allow comparing this replicated material with the one in the original contribution that was accepted at ACM MM.

 


Packaging guidelines

These packaging guidelines are meant to cover general cases. Please keep in mind that every individual case is slightly different.

Environment

Authors should explicitly specify the operating system and tools that should be installed as the environment. Such specification should include dependencies with specific hardware features (e.g., 25 GB of RAM are needed) or dependencies within the environment (e.g., the compiler that should be used must be run with a specific version of the operating system).

System

System setup is one of the most challenging aspects when replicating experiments. System setup will be easier to conduct if it is automatic rather than manual. Authors should test that the system they distribute can actually be installed in a new environment. The documentation should detail every step in system setup:

  • How to obtain the system?
  • How to configure the environment if need be (e.g., environment variables, paths)?
  • How to compile the system? (existing compilation options should be mentioned)
  • How to use the system? (What are the configuration options and parameters to the system?)
  • How to make sure that the system is installed correctly?

The above tasks should be achieved by executing a set of scripts provided by the authors that will download needed components (systems, libraries), initialize the environment, check that software and hardware is compatible, and deploy the system.

Tools

The committee suggests that authors use ReproZip in order to streamline this process. ReproZip can be used to capture the environment, the input files, the expected output files, and the required libraries. A detailed how-to guide (installing, packing experiments, unpacking experiments) can be found in there. ReproZip will help both the authors and the evaluators to seamlessly rerun experiments.

If using ReproZip to capture the experiments proves to be difficult for a particular paper, the committee will work with the authors to find the proper solution based on the specifics of the paper and the environment needed. More tools are available here: https://reproduciblescience.org/reproducibility-directory/


Timeline for 2019

The important dates along the timeline for submitting the companion paper and the associated artifacts are:

  • April 1, 2019: deadline for submitting the abstract of the companion paper
  • April 8, 2019: final deadline for uploading your .pdf and reproducibility archive
  • July 1, 2019: notification of acceptance/rejection,
  • August 1, 2019: deadline for preparing the final version of the accepted companion paper,
  • End of October 2019: present your reproducibility work at a specific poster session during the 2019 edition of ACM MM in Nice, France.

2019 Reproducibility Committee

Chairs:

  • Laurent Amsaleg, CNRS-IRISA, France [email]
  • Björn Þór Jónsson, IT University of Copenhagen, Denmark [email]

Committee members:

  • Martin Aumüller, IT U. of Copenhagen, Denmark
  • Ahmet Iscen, Google, France
  • Xirong Li, RUC, China
  • Naoko Nitta, Osaka U., Japan
  • Michael Riegler, Simula, Norway
  • Stevan Rudinac, UVA, The Netherlands
  • Lucile Sassatelli, U. Nice, France
  • Ujjwal Sharma, UVA, The Netherlands
  • Gwendal Simon, IMT Atlantique, France
  • Francesca De Simone, CWI, The Netherlands
  • Bart Thomee, Google, USA
  • Haoliang Wang, Adobe Research, USA
  • Jan Zahálka, bohem.ai, Czech Republic

About the 2019 edition of ACM MM

ACM Multimedia Reproducibility must be bootstrapped. In 2019, it is a pilot and we will test a compressed version of the full procedure in order to determine if any aspect needs to be adapted. We already anticipate that the deadline for submitting companion papers for future years will be planned for February 1 instead of April 1 as it is the case for this edition.

For the 2019 edition, Laurent Amsaleg and Björn Þór Jónsson chair the committee. In this first year, an ACM MM GC (Laurent) will also serve as a Reproducibility Committee Chair in order to launch the initiative, but in future years the Reproducibility Committee Chairs will not include the GC. For the 2020 edition of ACM MM, Björn Þór will remain and Laurent will be replaced.