DQ-Kit Web App: Evaluating and Improving Data Quality for Soil and Agricultural Data in the BonaRes Repository

Well-curated research data repositories play a vital role in facilitating the discovery, access, integration, and analysis of scientific data, maximizing research impacts, and ensuring the accuracy and reliability of data-driven technologies. In the field of soil and agricultural sciences, data is obtained from various sources. These sources include agronomic experiments, laboratory analyses, monitoring, economic data, model forecasts, and remote sensing data. The data often includes spatial and temporal information, such as geodata and time series. The diverse range of data adds complexity to the task of realizing FAIR data principles.

Since 2016, the BonaRes Repository has been providing an infrastructure for soil and agricultural research data. It aims to be open, visible, legally sound, and effectively aggregate data in a user-friendly internationally harmonized manner. To further this mission, the repository is currently developing DQ-Kit, an independent web application for automating comprehensive data quality assurance. DQ-Kit is a tool designed for data authors (providers), data reusers, and anyone who wants to evaluate and enhance the quality of their research data. It will offer automated guidance on elements of the data that require review and confirmation.

The checks conducted by DQ-Kit will encompass four main categories. First, formal criteria such as atomization of data, and other issues of structural but also semantic consistency, will be addressed. Second, DQ-Kit will provide a comprehensive and well-structured summary of variables, their properties, and summary statistics. Third, DQ-Kit will allow for exploration of relationships among variables, as well as temporal and spatial patterns, and patterns of missingness. Lastly, we are planning to explore options for data plausibility checks that flag variables containing theoretically "impossible" values and values that seem empirically implausible based on available datasets, scientific literature, and expert knowledge. Initially, this functionality may be limited to soil research data, where our team has the necessary expertise.

However, the responsibility for handling the results and alerts provided by DQ-Kit ultimately lies with the data author, who has the final authority in determining their validity. Likewise, data reusers may have different criteria when selecting data sets for specific purposes. Therefore, we adopt the concept of "fit for use" instead of a binary "acceptable vs. unacceptable" approach. This approach focuses on the suitability of data for specific purposes while acknowledging the efforts of data providers and amplifying their impact.

Ultimately, our plan is to address the challenge of enhancing the metadata of data sets published in the BonaRes Repository with DQ-Kit results. This will enable seamless quality control and facilitate the comparison of different datasets. In summary, these checks ensure the integrity and reliability of scientific data available at the BonaRes Repository, supporting a wide range of research endeavors.