Replication Package for the paper titled "Starting a new REST API project? A performance benchmark of frameworks and execution environments"
Authors/Creators
- 1. Department of Electrical Engineering and Information Technology Università degli Studi di Napoli Federico II
Description
ABSTRACT
REST APIs have become widely adopted in the software industry, finding extensive usage for business-critical purposes such as data exchange, mobile app development, and microservice-based architectures. Such popularity has led to a proliferation of dedicated frameworks, making it challenging for developers and organizations to choose which one to use for developing resource-efficient solutions. Adding to the complexity is the possibility of adopting also different execution environments, such as GraalVM, that offer advantages such as faster startup times, lower memory footprint, and polyglot capabilities.
Some prior works have investigated the performance of various frameworks for REST APIs. Still, these studies often consider simplistic scenarios with a single endpoint, which fail to capture the complexity and diversity of real-world REST API applications. Furthermore, the impact of different execution environments on performance was often overlooked. Consequently, there remains a significant knowledge gap in comprehensively assessing the combined influence of frameworks and execution environments on the performance of REST APIs. This study aims to move a first step towards bridging that gap, by conducting a thorough performance benchmark that encompasses real-world REST APIs and considers also the effects of different execution environments.
More in detail, the study focuses on two of the most popular programming language and framework combinations for REST APIs, namely JavaScript with the Express framework and Java with the Spring framework. As for the execution environment, we consider both mainstream execution environments for JavaScript and Java (Node and OpenJDK, respectively), and GraalVM, which can execute both Java and JavaScript software. The benchmarking process involves conducting realistic load and stress tests using state-of-the-art tools.
Results reveal significant differences in performance across the considered combinations, providing insights that could support developers and system architects in making more informed decisions on the technologies to use for their REST API projects.
CONTENTS OF THIS REPLICATION PACKAGE
- sportsclub-restapi-express.zip This file contains the source code and docker environments necessary to run the subject REST API implemented using JavaScript and the Express framework.
- sportsclub-restapi-spring.zip This file contains the source code and docker environments necessary to run the subject REST API implemented using Java and the Spring Boot framework.
- Tests This directory contains the performance test scripts we developed. The scripts are implemented using the JMeter tool.
- outputs-and-analysis-scripts.zip This file contains the raw outputs of the performance tests we executed and the R scripts we used to analyze the data and generate the plots and tables that are presented in the paper.
- The /data directory contains the raw results of our experiments. The results of the load and stress testing activities are, respectively, in the /data/load-testing and in the /data/stress-testing directories. The results are also available as an R data frame in the results-load.RDS and results-stress.RDS, which can be directly loaded in an R environment.
- The merge_data.R scripts is responsible for merging the results of all experiments and for creating the provided RDS files containg all experiments and all repetitions.
- The other scripts can be executed to generate plots and to aggregate the results to produce the tables presented in the paper.
- sportclub-db.sql This file contains the commands to ri-initialize the PostgreSQL database required by the REST API applications.
REQUIREMENTS
- The applications were run using Docker 19.03.11 and docker-compose 2.13.0
- The performance tests were executed using JMeter 5.5It is also necessary to install the JMeter Plugin Manager (https://jmeter-plugins.org/wiki/PluginsManager/) and the following plugins: Auto-Stop Listener (https://jmeter-plugins.org/wiki/AutoStop/), and Custom Thread Groups (https://jmeter-plugins.org/?search=jpgc-casutg).
- The data analysis scripts were executed using R 4.3.0.
RUNNING THE SUBJECT REST API IN THE JAVA - SPRING BOOT CONFIGURATION
You can package and run the application using only Docker and Docker Compose, without having to install any other tools.
|
docker compose up --detach |
To run the application with GraalVM, just change the dockerfile argument to Dockerfile.graalvm in docker-compose.yml
RUNNING THE SUBJECT REST API IN THE JAVASCRIPT - EXPRESS CONFIGURATION
To run the application:
|
docker-compose up |
To re-initialize the db :
|
containerid=$(docker ps | grep polisportiva-express-db-1 | sed 's/ / /' | awk '{print $1}') |
|
docker exec -it polisportiva-express-db-1 psql -U admin -d postgres -c "REVOKE CONNECT ON DATABASE polisportiva FROM public;" |
|
docker exec -it polisportiva-express-db-1 psql -U admin -d postgres -c "SELECT pg_terminate_backend(pg_stat_activity.pid) FROM pg_stat_activity WHERE pg_stat_activity.datname = 'polisportiva';" |
|
docker exec -it polisportiva-express-db-1 psql -U admin -d postgres -c "DROP DATABASE polisportiva;" |
|
docker exec -it polisportiva-express-db-1 psql -U admin -d postgres -c "CREATE DATABASE polisportiva;" |
|
docker exec -it polisportiva-express-db-1 psql -U admin -d postgres -c "GRANT CONNECT ON DATABASE polisportiva TO public;" |
|
docker cp /custom_path/polisportivadb.sql $containerid:/polisportivadb.sql |
|
docker exec -it polisportiva-express-db-1 psql -U admin -d polisportiva -f ./polisportivadb.sql |
To run the application with GraalVM, just change the docker file argument of web service in docker-compose.yml
RUNNING THE PERFORMANCE TESTS
In order to run the test from command line and generate a csv file containing the submitted requests:
- Open command prompt.
- Go into JMeter’s bin folder
- Enter the following command to run the test
|
Jmeter -n -t <location of jmeter script> -l <location of results file> |
In addition, it’s possible generate a HTML dashboard with the following command:
|
jmeter -n -t <location of jmeter script> -l <location of result file> -e -o <location of output folder> |