[Artifact] BOLT: Optimizing OpenMP Parallel Regions with User-Level Threads

PACT '19: Artifact of BOLT

This is the artifact of the following paper:

Shintaro Iwasaki, Abdelhalim Amer, Kenjiro Taura, Sangmin Seo, and Pavan Balaji, "BOLT: Optimizing OpenMP Parallel Regions with User-Level Threads," in Proceedings of the 28th International Conference on Parallel Architectures and Compilation Techniques (PACT '19), Sept. 2019

Briefly speaking, the following commands can install, build, and run programs, and plot data. Every step will be explained in the following sections.

## on a compute/login node
sh download.sh

## on Intel Skylake 8180 (or any machine that has AVX 512F)
# need CMake > 3.4.3 and Intel C/C++ Compilers 2017.2.174
sh build.sh

sh run.sh nested_loop
sh run.sh kifmm
# we did the following experiment on KNL, but it works on Skylake as well
sh run.sh qbox_fftw

## on your laptop
python plot/plot_nested_loop.py log/nested_loop/190101_0001_xxx.log
python plot/plot_qbox_fftw.py log/qbox_fftw/190102_0001_yyy.log
python plot/plot_kifmm.py log/kifmm/190103_0001_zzz.log

We note that these scripts do not install any packages; all files created are in this experiment directory.

Software Prerequisite

- Basic Linux utilities
- Including bash/make/gcc/g++/autotools/wget/git/tar

- CMake >= 3.4.3
- Necessary to build Clang/LLVM 7.0

- Intel C/C++ Compiler 2017.2.174 (and MKL)
- This test suite should work with (reasonably) recent Intel C/C++ Compilers EXCEPT KIFMM. KIFMM REQUIRES MKL 2017.2.174 AND INTEL SKYLAKE 8180; at least our current MKL trick does not work on Intel KNL 7210 as far as we checked; maybe the benchmark works with different versions, but we have not confirmed.

- Python2 + matplotlib
- Necessary to plot graphs

Hardware Prerequisite

- Connection to the Internet
- download.sh obtains some files via external websites.

- Intel CPU that has AVX512F
- All benchmarks assume the existence of AVX512F. Please check if avx512f appears in /proc/cpuinfo.

How to download all the necessary files?

# Please check if the compute node (or the login node if the file system is shared) is connected to the Internet
# All files are downloaded to archive/
sh download.sh

If you want to download files on the local machine and then copy them to the remote nodes, please check Q&A “GCC is not properly installed. Why?”.

How to build all the necessary programs?

# Please check the availability of icc, icpc, and cmake > 3.4.3
# Please compile scripts on a compute node (e.g., not a login node)
sh build.sh

This does not install programs locally (e.g., /usr/bin). Consequently, no root permission is required.

How to run experiments?

# All log files are put in log/
sh run.sh nested_loop
sh run.sh qbox_fftw
# Please be sure that you run KIFMM on Skylake 8180 with MKL 2017.2.174; the
# correct behavior is not guaranteed in other environments.
sh run.sh kifmm

How to plot data?

# All outputs (pdfs) are placed in pdfs/
python plot/plot_nested_loop.py log/nested_loop/190101_0001_xxx.log
python plot/plot_qbox_fftw.py log/qbox_fftw/190102_0001_yyy.log
python plot/plot_kifmm.py log/kifmm/190103_0001_zzz.log

Q&A

• Benchmarks do not run because of illegal instructions. Why?
  Although it is not critical to evaluate BOLT itself, all the benchmarks currently require AVX512F. Please check if your machine supports it.

• GCC was not properly installed. Why?
  It might happen if you copy the files to somewhere else. Please see the following issue:
  https://stackoverflow.com/questions/33278928/how-to-overcome-aclocal-1-15-is-missing-on-your-system-warning/33279062

• KIFMM hangs with BOLT. Why?
  KIFMM requires Intel C/C++ + MKL 2017.2.174 and Intel Skylake 8180. In another environment, you might need to rewrite kifmm/fmm/mkl_yield.c properly.

• How to reproduce performance?
  Please check the paper for the detailed experimental environments. The turbo boost feature must be turned off.

• How to uninstall this artifact?
  Please just remove this folder. Our scripts do not specifically create files in /tmp, but some external packages might do so while installation process. Note that this script does not change environmental variables (unless you use source).

• How to customize experiments?
  Unfortunately, we do not have a portable interface. Please see scripts and change the values directly.

Link

Our experiments specifically are based on the following softwares. We thank all researchers and developers who develop and maintain them.

• Compilers/Runtimes
  – Clang/LLVM (https://llvm.org/)
  – GCC (https://gcc.gnu.org/)
  – Intel C/C++ Compilers and MKL (https://software.intel.com/en-us/compilers/)
  – LLVM OpenMP (https://openmp.llvm.org/)
  – MPC (http://mpc.hpcframework.paratools.com/)
  – OMPi/psthreads (http://paragroup.cse.uoi.gr/wpsite/software/ompi/)
  – OmpSs (https://pm.bsc.es/ompss/)
  – Argobots (https://www.argobots.org/)
  – BOLT (https://bolt-omp.org/)

• Applications/Libraries
  – FFTW3 (http://www.fftw.org/)
  – KIFMM (https://github.com/jeewhanchoi/kifmm–hybrid–double-only/)
  – Qbox (https://computation.llnl.gov/projects/qbox-computing-structures-quantum-level/)

For the users’ convenience, this artifact includes some packages by default, but if you are interested in the latest version, please follow a link and download the latest package.

Acknowledgment

Please see the acknowledgment in the paper. We’d also like to thank all the paper and artifact reviewers.

License

Please check each software package.

Primary Contact

Shintaro Iwasaki (siwasaki@anl.gov / iwasaki@eidos.ic.i.u-tokyo.ac.jp)