Guidelines for Packaging AEC Submissions

Below we outline guidelines on how to package artifacts for submission. We are open to possibilities that we have not considered below; however, in cases where systems ought to fit these options, we suggest authors use them.

In all cases, the artifacts should be accompanied by suitable documentation, to save committee members the burden of reverse-engineering what the authors intended (e.g., a dataset is useless without some explanation on how to browse the data; a tool without a quick tutorial will be very hard to use). In particular, please make concrete what claims you are making of the artifact, if these differ from the expectations set up by the paper. This is a place where you can tell us about difficulties we might encounter in using the artifact, or its maturity relative to the content of the paper. We are still going to evaluate the artifact relative to the paper, but this helps set expectations up front, especially in cases that might frustrate the reviewers without prior notice.

CONFLICTS

If one of the authors is an AEC chair, you may NOT submit an artifact. You can, however, indicate in your paper that you were unable to submit an artifact due to the conflict-of-interest.

If you have a conflict-of-interest with anyone on the committee (including the three chairs), please indicate this in both the Web page and in your submission email.

ARTIFACT SUBMISSION

Irrespective of the nature of the artifacts, authors should eventually create a *single Web page* (whether on their site or a third-party file upload service) that contains *all necessary instructions*.

For artifacts where this would be appropriate, it would be helpful to also provide a self-contained bundle (including instructions) as a single file (.tgz or .zip; please avoid exotic compressors) for convenient offline use: imagine the reviewer who wants to download a single file to expand and work with during a train or bus commute.

The artifact submission thus consists of just the URL and any credentials required to access the files.

CODE ARTIFACTS PACKAGING

Authors should strongly consider one of the following methods to package the software components of their artifacts (though the AEC is open to other reasonable formats as well):

1. A VM (Virtualbox/VMWare) image containing software application already setup in the intended run-time environment (e.g., Mobile phone emulator, Real time OS). This is the preferred option: It avoids installation and compatibility problems, and provides the best guarantees for reproducibility of the results by the committee. Authors using Linux might find the CDE tool to be useful for automatically creating a VM image from existing software without needing to manually re-install all dependencies within a VM (http://www.stanford.edu/~pgbovine/cde.html).

2. A binary installable package. We invite the authors to use CDE (Linux) or MSI Installer (Windows) to package the binary application.

3. A live instance running on the Web.

4. A detailed screen-cast of the tool along with the results, especially if one of the following special cases applies:

4.a) the application needs proprietary/commercial software that is not easily available or cannot be distributed to the committee; 4.b) the application requires significant computation resources (e.g., > 24 hours of execution time to produce the results);

5. An installation or update repository for the tool. (e.g., Eclipse update site or a Debian APT repository). However, we urge the authors to test their repository on standard environments to avoid installation problems by the committee members.

IN ALL CASES, authors should make a genuine effort to not learn the identity of the reviewers. This may mean turning off “call home” features or analytics, or only using systems with high enough usage that AEC accesses will not stand out. In all cases where tracing is unavoidable, the authors should warn the reviewers in advance so the reviewers can try to take adequate safeguards.

NON-CODE ARTIFACTS FORMAT

Non-code artifacts should preferably be in open document formats. For documents and reports, we invite the authors to use PDF, ODF, or RTF. We invite the authors to submit experimental data and results in CSV (preferred option), JSON, or XML file formats. In the special case that authors have to submit non-standard data formats, they should also provide suitable readers.

Original version from AEC @ ECOOP'13