Analysis and Evaluation of Data Management Planning Tools

Data Management Plans (DMPs) are crucial for a structured research data management and often a mandatory part of research proposals. A DMP should contain information about the creation, management, sharing and preservation of research data. The manual creation of DMPs can be very time-consuming, since many researchers have to start from scratch, are unsure about the required content and may run the risk of not meeting the funder requirements. By using tools, DMPs can be effectively developed and managed. There are a variety of tools to support the development of DMPs: from discipline-agnostic tools, which can be used to generate a generic draft DMP, to discipline-specific tools, which support the creation of a DMP in a specific research field, such as psychology, biodiversity, engineering, or the life sciences. The project aims to develop a state-wide DMP service for Saxony, ideally building upon an existing tool. Therefore, the objectives of this work are: (1) the identification of requirement parameters to evaluate existing DMP tools and (2) the evaluation of DMP tools based on the identified parameters. The evaluation results will be used to support the development of a quick and easy to use DMP service for members of Saxon research institutions to generate data management plans. In total, we evaluated 18 DMP tools, 13 of which provide open access. Seven of the tools have a discipline-specific focus, the other eleven follow a discipline-agnostic approach. Eight tools were developed and hosted in Germany. The remaining DMP tools originate from other European countries, the USA and Australia. Based on the findings of 19 expert interviews and a subsequent discussion among the project partners, we defined 32 requirement parameters for the evaluation of existing DMP tools. The parameters were grouped into three main categories: basic functions, technical aspects and user-friendliness. Because the purpose of the evaluation was to identify the tools that could easily be reused as the basis for the DMP service to be developed by the project, we mainly focused on the technical parameters. We aimed at identifying DMP tools, which are easy to host and maintain in order to ensure their adaptability to the specific needs of researchers, institutions, and funders. To further prioritize, a weight factor between zero (not relevant) and three (high priority) was assigned to every parameter. The weight factor was determined individually by each member of the research team, and afterwards the arithmetic mean was calculated. The DMP tools were rated by two different researchers independently according to a fixed rating scheme from zero (poor) to ten (excellent). In a next step, we calculated the arithmetic mean for each requirement parameter. To calculate the final score, the score for each parameter was multiplied by the weight factor. Then, the sum of the rating scores was calculated per main category and for the total score. Furthermore, we wanted to determine, how many of the 32 requirements each of the tools fulfilled. This was done by using a threshold of greater equal five, i.e. a certain requirement was considered to be fulfilled if a minimum score of five was obtained. The evaluated tools satisfied between three and 28 requirement parameters. 11 tools covered at least half of the parameters. When looking at the total scores, it should be kept in mind that the evaluation was carried out against a set of parameters that was specifically developed to fit our purpose. The highest total rating scores were attained by Data Stewardship Wizard (733.5, DSW), DMPTool (645.5) and RDMO NFDI4Ing (579.5). In the main category ’basic functions’, DSW also performed best (269.5) followed by easyDMP (235) and DMPTool (230). The three best performing tools in terms of ’technical aspects’ were DSW (190), DMPTool (190) and RDMO NFDI4Ing (181). The most user-friendly ones were DSW (274), DataPLAN from NFDI4plants (262) and DMPTool (225.5). Experience from RDM consultancy at TU Dresden showed that researchers find pre-fabricated text passages very helpful, which are automatically generated by the DMP tool based on their input. Although such a text might need some refinement by the researchers, it can serve as a first draft of a DMP. Accordingly, the corresponding requirement parameters are of high importance to us. The only tools generating pre-fabricated text passages while also providing the tool’s source code were DSW, DataPLAN from NFDI4plants and the DMP tool from the TU Dresden Service Center Research Data. Moreover, we consider the machine-actionability of the DMP as an important requirement, because it can facilitate data findability, reusability, automated evaluation and monitoring. A machine-actionable DMP is machine- and human-readable and aims to be interoperable, automated and standardized. Seven DMP tools fulfill the requirements of being open source and providing machine-actionable DMPs. In the light of recent developments in the area of DMP tools, this study provides an up-to-date evaluation of 18 DMP tools according to 32 parameters covering basic functions, technical aspects and user-friendliness. Our results show that, evaluated against our set of criteria, DSW, DMPTool and RDMO NFDI4Ing received the highest total scores and can be recommended for researchers and institutions as flexible tools for hosting, which provide numerous functionalities. Our results can support tool developers to identify potential improvements and hosting institutions to select a tool suited to their specific needs. In a next step, we will check the feasibility of adapting each of the three tools according to our needs and estimate the respective workload. The most suitable DMP tool will then be customized for our requirements.