Smelly Bad Practices in Performance System Tests: Detection, Prevalence, and Lifetime: Online Appendix

This repository contains the full replication package for the paper:

“Smelly Bad Practices in Performance System Tests: Detection, Prevalence, and Lifetime”

The package includes the implementation of the detection tool, all intermediate artifacts, and the datasets produced to answer the three research questions (RQ1, RQ2, and RQ3).

The replication package is organized into four main folders:
.
├── pt-linter/
├── RQ1/
├── RQ2/
└── RQ3/

1. pt-linter/
This folder contains the implementation of the detection tool developed in the paper.

1.1 Tool Source Code
The full source code of the tool is provided.
Inside this folder, you will also find a dedicated README explaining:
- How to build the tool
- How to run the tool
- How to extend it with new bad practices
- How to reproduce the analyses

1.2 reports_and_statistics/
This subfolder contains all the automatically generated output files produced by running the tool on the E2EGIT dataset of: 234 performance system tests.
For each analyzed test, the following artifacts are available: JSON and HTML reports.

2. RQ1/ — Taxonomy Construction
This folder contains all artifacts related to Research Question 1, which concerns the identification and definition of bad practices through a grey literature review.

2.1 Grey Literature Review Sources (.xlsx)
An Excel file listing all sources analyzed during the grey literature review.
For each source, the file includes:
- Source metadata
- Classification label: Useful (contributed to the preliminary taxonomy) / Not useful
This file ensures full traceability of the taxonomy construction process.

2.2 Identified Bad Practices (.docx)
A Word document describing the 10 identified bad practices.
For each bad practice, the document includes: A table listing the supporting source references extracted from the grey literature.
This document represents the preliminary taxonomy used to guide tool development and subsequent analyses.

3. RQ2/ — Detection Accuracy and Prevalence 

This folder contains artifacts related to Research Question 2, which evaluates the tool and studies the prevalence of bad practices.

3.1 tool-validation/
This folder contains the material used for manual validation of the detection tool.
It includes: 31 .jmx files
These are the performance test files that were manually inspected for validation.
An additional .xlsx file containing:
- Manually labeled violations
- True positives
- True negatives
- Additional validation annotations
This dataset was used to compute the tool’s validation metrics (e.g., precision).

3.2 history_recap_analysis.xlsx
This Excel file contains the results of the longitudinal analysis.
For each of the 234 tests, the file reports:
The presence of each bad practice
Its prevalence across different versions of the test over time

4. RQ3/ — Lifetime and Commit-Level Analysis

This folder contains artifacts related to Research Question 3, which investigates the lifetime and evolution of bad practices across commit events.

The folder is structured according to the four commit event types defined in the paper:
RQ3/
├── improving/
├── introduction/
├── neutral/
└── pejorative/

Each of these folders corresponds to a specific type of commit event.

Each commit event folder (improving, introduction, neutral, pejorative) contains:
Six subfolders, one for each of the six analyzed bad practices.
Inside each bad practice folder, the following files are available:

4.1 commit_analysis
Contains:
The list of commits classified under the given event type
For example:
In introduction/Excessive Listeners/commit_analysis, you will find all commits that introduced an Excessive Listeners violation.

4.2 owner_author_analysis

Contains analysis results related to:
- The owner tag
As described in the paper.

4.3 project_startup_analysis
Contains analysis results related to:
- The project startup phase
As described in the paper.

4.4 project_status_release_analysis
Contains analysis results concerning:
- Release status
As described in the paper.

4.5 workload_newcomer_analysis
Contains data and results related to:
- Developer workload
- Newcomer status
As described in the paper