Detecting and evaluating order-dependent flaky tests in JavaScript

This repository contains the data and results from the paper "Detecting and evaluating order-dependent flaky tests in JavaScript" - currently under review.

Project Structure

The replication package contains the following directories:

./JS-TOD

JS-TOD (JavaScript - Test Order-dependency Detector) is a tool to extract and detect order-dependent tests in Jest. The tool contains the following scripts:

1. testReorderRunner.js: reorders tests inside describe blocks of a given project for a defined number (reorder) and saves the results in the ___extracted results___ folder inside the project folder. It will then run all the new test suites for rerun number and save the results as a JSON file in the same folder. The results are named starting with the term testOutput, followed by the name of the test suite and the rerun count.

2. testOutputParser.js: parses the JSON files located in the ___extracted results___ folder of a given project, generated after running testReorderRunner.js. It summarizes the results, including the name of the test suite, along with the number of failing and passing tests, in a CSV file. Each CSV file is named after the corresponding test suite followed by the term output. These CSV files are stored in the output parsing folder within the ___extracted results___ directory.

3. describeReorderRunner.js: reorders describe blocks of a given project for a defined number (reorder) and save the results in the ___extracted results describes___ folder inside the project folder. It will then run all new test suites for rerun number and save the results as a JSON file in the same folder. The results are named starting with the term testOutput, followed by the name of the test suite and the rerun count.

4. describeOutputParser.js: parses the JSON files located in the ___extracted results describes___ folder of a given project, generated after running describeReorderRunner.js. It summarizes the results, including the name of the test suite, along with the number of failing and passing tests, in a CSV file. Each CSV file is named after the corresponding test suite followed by the term output. These CSV files are stored in the output parsing folder within the ___extracted results describes___ directory.

5. testSuitesReorderRunner.js: reorders the test suites of a given project for a defined number (reorder), runs them for rerun number, and saves the results in the ___extracted results test files___ folder inside the project folder. The results are named starting with the term testOutput, followed by the number of reordering and rerun counts.

6. testSuitesOutputParser.js: parses the JSON files located in the ___extracted results test files___ folder of a given project, generated after running testSuitesReorderRunner.js. It summarizes the results, including the number of order, along with the number of failing and passing tests, in a CSV file. Each CSV file is named order and order number followed by the term output. These CSV files are stored in the output parsing folder within the ___extracted results test files___ directory.

7. customSequencer.js is the custom sequencer extended from Jest's testSequencer. It is used by testSuitesReorderRunner.js.

./Data

• projectsURL contains the name, version, statistics, URL and description of 81 top-starred GitHub JavaScript projects that using Jest.

• OD-experiment-result contains the result of reordering and rerunning tests in three levels in 81 JavaScript projects. Each contains the following information : - Project Name - Number of tests and test suites before and after reordering the tests, describe blocks and test suites. - Runtime information - Number of flaky tests - A note from each project.

• testLevelResults contains the result of reordering and rerunning tests in 67 JavaScript projects.

• describeBlockLevelResults contains the result of reordering and rerunning describe blocks in 32 JavaScript projects.

• testSuiteLevelResults contains the result of reordering and rerunning test suites in 49 JavaScript projects.

• ./Reordering Tests contains the projects and new test suites after reordering the tests.

• ./Reordering Describe Blocks contains the projects and new test suites after reordering the describe blocks.

• ./Reordering Test Suites contains the projects and the results of running test suites in different orders.

Project Dependencies

This project relies on the following dependencies:

Required modules

• Node.js
• Jest
• npm
• npx
• babel

Specifically, JS-TOD requires the following utilities:

• @babel/parser: A JavaScript parser.
• @babel/traverse: A utility for traversing and manipulating abstract syntax trees (ASTs) generated by Babel.
• @babel/generator: A utility for generating code from ASTs produced by Babel.
• is-utf8: A utility to check if a buffer contains UTF-8 encoded text.

Installation

Before running the scripts, you need to install the necessary dependencies. You can do this by running the following command:

Reordering and Running tests using JS-TOD

1. Change the directory to where JS-TOD is located:

1. Choose the level of reordering(tests, describe blocks or test suites). For example for reordering and running tests of a given project:

• project_path sets the path of the project.

• rerun sets the number of rerunning. The default number of reruns is 10.

• reorder sets the number of reordering. The default number of reorders is 10.