Towards a Metadata Schema for Research Software Quality

Quality is important for research software to ensure confidence in the research results it produces and to facilitate reuse. However, it can be difficult for potential users to obtain quality information when evaluating the suitability of software for their research because external quality assessment has limitations. Manual review is very resource-intensive. While some aspects can be checked automatically, others require knowledge of the inner workings of the software, such as executing tests in custom setups. Additionally, actively developed software changes frequently, so quality information can quickly become outdated. On the other hand, developers often already do regular quality assessments as part of their development process, for example, checks and tests in Continuous Integration pipelines, but this information can be difficult to find.

Our aim is to make information on a research software’s quality explicit and visible by providing it as structured and interoperable metadata. We differentiate between Software Quality Definition (SQD) metadata to capture quality requirements, decisions and assurance processes, and Software Quality Reports (SQRs), which contain concrete assessment results for a specific version of the software.

To implement this concept, we developed a schema for the SQD metadata in LinkML. It is based on the ontology of quality model concepts from the ISO/IEC 25002:2024 SQuaRE framework and data quality standards like the Data Quality Vocabulary. We evaluate our schema by using it to describe the quality definition of three diverse research software projects and iteratively refine it based on feedback from the developers.

Preliminary findings indicate that the schema can represent the most relevant quality aspects of these projects. The schema is designed to be flexible and has only a few restrictions, enabling usage across diverse projects. The format of the metadata is YAML, making it easy to read and write for humans. Additionally, the tooling provided by the LinkML framework facilitates providing documentation and validation options.

In the future, we aim to extend the schema with SQR metadata and add automation to allow the creation of these reports using the SQD metadata as a recipe. This approach requires only minimal effort from the developers to publish quality metadata for their software in an interoperable format. We believe this has the potential to encourage developers to follow good quality assurance practices and add value when research software is assessed.