Using the Raspberry Pi and Docker for Replicable Performance Experiments

Replicating software performance experiments is difficult. A common obstacle to replication is that recreating the hardware and software environments is often impractical. As researchers usually run their experiments on the hardware and software that happens to be available to them, recreating the experiments would require obtaining identical hardware, which can lead to high costs. Recreating the software environment is also difficult, as software components such as particular library versions might no longer be available.

Cheap, standardized hardware components like the Raspberry Pi and portable software containers like the ones provided by Docker are a potential solution to meet the challenge of replicability. In this paper, we report on experiences from replicating performance experiments on Raspberry Pi devices with and without Docker and show that good replication results can be achieved for microbenchmarks such as JMH. Replication of macrobenchmarks like SPECjEnterprise 2010 proves to be much more difficult, as they are strongly affected by (non-standardized) peripherals. Inspired by previous microbenchmarking experiments on the Pi platform, we furthermore report on a systematic analysis of response time fluctuations, and present lessons learned on dos and don’ts for replicable performance experiments.

Result files:

• results-jmh.zip: Chapter 4.1/Experiment 1
• results-*-stretch-*-*-*.zip: Chapter 4.2/Experiment 2, mapping in summary.xls/raw-2nd
• results-jpa.zip: Chapter 4.3/Experiment 3
• results-specjenterprise.zip: Chapter 4.3/Experiment 4
• test-spassmeter-*.zip: Chapter 5, mapping/summary in interrupts.xls

Archives/Images

• raspbian-stretch-docker.img.gz: the operating system with docker, please scale to the size of your SD-card
• docker.zip: docker containers or scripts to build them in case that licenses permit sharing the original container (may be made available by the authors on request)
• sources.zip: sources for experiments/benchmarks created by the authors.
• rScripts.zip: contains the R-scripts for analyzing the Moobench log files. timeseries-average.r is the main script to call, the other one is reused by other scripts that are not relevant here. Before running the analysis scripts, you may need to edit lines 7 and 8 of "timeseries-average.r". Line 7 needs to point to the directory where the raw results are stored. Depending on whether the results are from Kieker or from SPASSmeter, you will have to set the flag in line 8 accordingly, i.e., set isSPASS<-TRUE for SPASS-meter logs, isSPASS<-FALSE for Kieker<-logs. The R scripts are necessary for analyzing the moobench results containing the logs our analyses. The results from the paper can be found in different locations in the logs depending on the underlying monitoring solution. For SPASSmeter, the results can be found in the lines marked as "raw all-t (SPASSmeter Javassist)". For Kieker, the results are in the lines marked as "raw all-t (FS logging)". summary.xslx shows which parts we utilized and how we obtained the summary results shown in the paper.