Automatic Requirements Testability Analysis Dataset

This repository contains the data and results from the paper “Requirement testability measurement based on requirement smells”. We announce a public dataset of software requirements along with their quality issues to facilitate the development and evaluation of software requirement quality tools. The Automatic Requirement Testability Analyzer (ARTA) is a web application to measure requirement testability based on the requirement smells. It facilitates the requirement engineering quality assurance process and allows requirement engineers to manage the requirements of their projects.

ARTA dataset is a collection of requirements along with their smells and degree of testability of each requirement. It has been used in the initial evaluation of ARTA testability measurement algorithms. To the best of our knowledge, there is no public dataset for developing and evaluating the requirements smell detector tools.

Dataset Structure

ARTA dataset contains 4752 software requirements extracted from 24 project documentation (in English). The ARTA dataset (release 1.0.0) is available as a collection of Microsoft Excel files. The Datasets directory contains five Excel files, described as follows:

1. DS1.xlsx: This file contains 985 requirements from 6 projects and nine types of smells manually labeled for each requirement.
2. DS2.xlsx: This file contains 1092 requirements from 8 projects and nine types of smells manually labeled for each requirement.
3. DS3.xlsx: This file contains 1522 requirements from 6 projects and nine types of smells manually labeled for each requirement.
4. DS4.xlsx: This file contains 1153 requirements from 4 projects and nine types of smells manually labeled for each requirement.
5. SmellyWordsDictionary.xlsx: This file contains 1000 most frequent words in computer science domains, which is sorted based on their similarity. Four types of requirement smells are detected based on the words in the smelly words dictionary. For each word, the type of smell has been determined manually by experts.

The DS1_Evaluation directory contains the result of different smell detection and testability measurement algorithms, described as follows:

1. 001_dataset1kv1.xlsx (primary dataset): The same DS1 in the Datasets directory. This file contains 985 requirements from 6 projects and nine types of smells manually labeled for each requirement.
2. 002_dataset1kv1_smell_frequency_with_testability.xlsx: This file contains the number of words of each requirement, the number of smelly words, number of smelly words according to the type of smell, degree of cleanness, and testability of each requirement, which has been computed based on “001_dataset1kv1.xlsx”. The requirements are in the same order in “001_dataset1kv1.xlsx”.
3. 003_dataset1kv1_ARTA_result.xlsx: This file contains the result of automatically detected smells by ARTA, our proposed smell detector tool. The requirements are in the same order in “001_dataset1kv1.xlsx”.
4. 004_dataset1kv1_ARTA_result_smell_frequency_with_testability.xlsx: This file contains the number of words of each requirement, the number of smelly words, number of smelly words according to the type of smell, degree of cleanness, and testability of each requirement, which has been computed based on “003_dataset1kv1_ARTA_result.xlsx”. The requirements are in the same order in “001_dataset1kv1.xlsx”.
5. 005_dataset1kv1_Smella_result.xlsx: This file contains the result of automatically detected smells by Smella, another smell detector tool. The requirements are in the same order in “001_dataset1kv1.xlsx”.
6. 006_dataset1kv1_Smella_result_smell_frequency_with_testability.xlsx: This file contains the number of words of each requirement, the number of smelly words, number of smelly words according to the type of smell, degree of cleanness, and testability of each requirement, which has been computed based on “005_dataset1kv1_Smella_result.xlsx”. The requirements are in the same order in “001_dataset1kv1.xlsx”.