Replication Package for "TriggerBench: A Performance Benchmark for Serverless Function Triggers"

This replication package contains the code (`aws-triggers` and `azure-trigger`), data analysis scripts (`data-analysis`), and dataset (`data`) of the TriggerBench cross-provider serverless benchmark.

It also bundles a customized extension of the `serverless-benchmarker` tool to automate and analyze serverless performance experiments.

TriggerBench

The Github repository joe4dev/trigger-bench contains the last version of TriggerBench. This replication package describes the version for the paper "TriggerBench: A Performance Benchmark for Serverless Function Triggers".

TriggerBench currently supports three triggers on AWS and eight triggers on Microsoft Azure.

Dataset

The `data/aws` and `data/azure` directories contain data from benchmark executions from April 2022.

Each execution is a separate directory with a timestamp in the format `yyyy-mm-dd-HH-MM-SS` (e.g., `2022-04-15_21-58-52`) and contains the following files:

• `k6_metrics.csv`: Load generator HTTP client logs in CSV format (see [K6 docs](https://k6.io/docs/results-visualization/csv/))
• `sb_config.yml`: serverless benchmarker execution configuration including experiment label.
• `trigger.csv`: analyzer output CSV per trace.
  • `root_trace_id`: The trace id created by k6 and adopted by the invoker function
  • `child_trace_id`: The trace id newly created by the receiver function if trace propagation is not supported (this is the case for most asynchronous triggers)
  • `t1`-`t4`: Timestamps following the trace model (see paper)
  • `t5`-`t9`: Additional timestamps for measuring timestamping overhead
  • `coldstart_f1=True|False`: coldstart status for invoker (f1) and receiver (f2) functions
  • `trace_ids.txt`: text file with each pair of `root_trace_id` and `child_trace_id` on a new line.
• `traces.json`: raw trace JSON representation as retrieved from the provider tracing service. For AWS, see [X-Ray segment docs](https://docs.aws.amazon.com/xray/latest/devguide/xray-api-segmentdocuments.html). For Azure, see [Application Insights telemetry data model](https://docs.microsoft.com/en-us/azure/azure-monitor/app/data-model).
• `workload_options.json`: [K6 load scenario](https://k6.io/docs/using-k6/scenarios/) configuration.

Replicate Data Analysis

Installation

1. Install [Python](https://www.python.org/downloads/) 3.10+

2. Install Python dependencies `pip install -r requirements.txt`

Create Plots

1. Run `python plots.py` generates the plots and the statistical summaries presented in the paper.

By default, the plots will be saved into a `plots` sub-directory.
An alternative output directory can be configured through the environment variable `PLOTS_PATH`.

> Hint: For interactive development, we recommend the VSCode [Python extension](https://marketplace.visualstudio.com/items?itemName=ms-python.python) in [interactive mode](https://youtu.be/lwN4-W1WR84?t=107).

Replicate Cloud Experiments

The following experiment plan automates benchmarking experiments with different types workloads (constant and bursty).

This generates a new dataset in the same format as described above.

1.  Set up a load generator as vantage point following the description in [LOADGENERATOR](./serverless-benchmarker/docs/LOADGENERATOR.md).
2.  Choose the `PROVIDER` (aws or azure) in the [constant.py](./experiment-plans/constant.py) experiment plan
3.  Run the [constant.py](./experiment-plans/constant.py) experiment plan
   1.  Open tmux
   2.  Activate virtualenv `source sb-env/bin/activate`
   3.  Run `./constant.py 2>&1 | tee -a constant.log`