SC19 AD/AE Appendices of Paper "TriEC: Tripartite Graph Based Erasure Coding NIC Offload"

1. Pre-requisites gcc (tested with 4.8.5) CMake 3.8 or higher CentOS 7.4 OFED-4.5 libevent-2.0 Mellanox ConnectX-5 and up Java 1.8.0 Ant 1.9.2 Maven 3.3.9 2. Environment Setup download tarball from https://drive.google.com/open?id=1FhG5zkqgRi14xJ1LKDKAEKrBQvycpfKW tar xvzf ad.tar.gz cd ad export AD_HOME=$PWD export PATH=$AD_HOME/bin:$PATH export LD_LIBRARY_PATH=$AD_HOME/lib:$LD_LIBRARY_PATH export LIBRARY_PATH=$LD_LIBRARY_PATH cd $AD_HOME/spymemcached ant mvn deploy:deploy-file -Durl=file://${HOME}/local-repo -Dfile=spymemcached-2.12.3.jar -DgroupId=myspy -DartifactId=spymemcached -Dpackaging=jar -Dversion=2.12.3 cd $AD_HOME/YCSB mvn -pl com.yahoo.ycsb:memcached-binding -am clean package 3. Experiments for Table 1: Profiling Numbers on Modern HPC Clusters. Tcomm numbers are taken by ib_send_lat. Two nodes are needed for taking Tcomm numbers, type command ib_send_lat -a -d mlx5_0 on one node of which the IP address is "addr", and type ib_send_lat addr -a -d mlx5_0 on the other node. Other numbers are taken with $AD_HOME/ec_op_benchmark. The command to run is ./ec_op_benchmark. 4. Experiments for Figure 8: Encoding Performance Comparisons with Varied Configurations and Chunk Sizes (Cluster A). To run a benchmark for RS(k,m), the first step is to allocate k+m+1 nodes, in which one is client and the other k+m are servers. Suppose the k+m+1 nodes are labeled as n(0) to n(k+m), then the second step is to execute the following command on n(1) to n(k+m): ./paradigm_encode_and_send_benchmark --benchmark_device=mlx5_0 --benchmark_role=server --benchmark_data_hosts=n(1),n(2),...,n(k) --benchmark_parity_hosts=n(k+1),n(k+2),...,n(k+m). The last step is to execute ./paradigm_encode_and_send_benchmark --benchmark_device=mlx5_0 --benchmark_role=client --benchmark_data_hosts=n(1),n(2),...,n(k) --benchmark_parity_hosts=n(k+1),n(k+2),...,n(k+m) on n(0). 5. Experiments for Figure 9: Performance Comparisons for Recovering m Data Chunks with Varied Configurations and Chunk Sizes (Cluster A). Similar to 4, the fist step is to allocate k+m+1 nodes, in which one is client and the other k+m are servers. Suppose the k+m+1 nodes are labeled as n(0) to n(k+m), then the second step is to execute the following command on n(1) to n(k+m): ./paradigm_recv_and_decode_benchmark --benchmark_device=mlx5_0 --benchmark_role=server --benchmark_data_hosts=n(1),n(2),...,n(k) --benchmark_parity_hosts=n(k+1),n(k+2),...,n(k+m) --benchmark_erasures=E, where E is a comma-separated list of erasures (e.g., 0 or 1,2 or 2,3,4). The last step is to execute ./paradigm_recv_and_decode_benchmark --benchmark_device=mlx5_0 --benchmark_role=client --benchmark_data_hosts=n(1),n(2),...,n(k) --benchmark_parity_hosts=n(k+1),n(k+2),...,n(k+m) --benchmark_erasures=E on n(0). 6. Experiments for Figure 10: Write Latencies with Varied Configurations and Value Sizes(ClusterA) and Figure 11: Latencies of read_with_erasures with Varied Configurations and Value Sizes. The scripts in $AD_HOME/triec are able to submit these experiments to a cluster managed with SLURM. The command to submit a job is sbatch $AD_HOME/triec/triec.sbatch-srun. Note that several variables need to be specified before submission. For instance, if you are going to submit a job for evaluating RS(k,m), then set SBATCH --nodes=k+m+x, where x is the number of nodes running YCSB clients, TRIEC_K=k, TRIEC_M=m, threads=(t), where t is the number of client threads for each YCSB client, in both triec.sbatch-srun and run.sh. The experiments for Figures 10 and 11 are with single client, such that x and t should be set to one. 7. Experiments for Figure 12: Throughput Comparisons for RS (6, 3) (Cluster B). For these experiments, set SBATCH --nodes=17, TRIEC_K=6, TRIEC_M=3, and threads=(64) in both triec.sbatch-srun and run.sh. 8. Experiments for Figure 13: Execution Time to Fully Recover One Node with Varied Number of Keys (Cluster B). The script for submitting experiments is reco.sbatch-srun. Similar to triec.sbatch-srun, the same set of variables needs to be specified before submission. Our configuration is SBATCH --nodes=6, TRIEC_K=3, TRIEC_M=2, and threads=(1) in both reco.sbatch-srun and run.sh. 9. Experiments for Figure 14: Performance Impact of Calculator Cache (Cluster A). The numbers are extracted by profiling numbers for the same kind of experiments in 4 and 5.