Staying flexible and transparent in a changing world - Agile RDM Service Development

This presentation was held as part of the E-Science Days on 13/03/25 by Petar Hristov and Katja Jansen. 

We present the challenges and solutions of agile RDM service development in a constantly changing world using the RDM platform Coscine as an example. 

In today's changing world, developing robust research data management (RDM) services requires agile practices that adapt to shifting demands and increasing responsibilities. During the development of Coscine — a platform aimed at facilitating the FAIR principles — we are confronted with two major changes: 1) a steadily growing number of stakeholders and 2) constantly evolving RDM concepts and technologies. In our presentation, we show our journey to serve both — our stakeholders and development needs.

At the beginning of Coscine's development as an open-source platform in 2018, the development team worked with a startup mentality and focused on the rapid development of functional prototypes for RDM. The toolchain was featuring simpleArchive, a web based archive tool that allowed metadata management and PID assignment. Funded by the first stakeholder RWTH Aachen University, the team started to work with methods from Scrum and Kanban in GitLab —managing requirements on issue boards. While transferring the simpleArchive to an RDM-tool for the active project phase, two doctoral theses in RDM and computer science were involved in the development of Coscine and led to a high degree of research orientation and innovation.

From 2020 till 2023 Coscine was in pilot phase, which allowed to test various functions and understand the key needs of users - access to storage space in a FAIR environment for “warm” research data. Due to the integrative character of the platform Coscine was able to participate and accommodate in successful projects (e.g. VVMethoden, AIMS, NFDI4ING, NFDI-MatWerk, Coscine.nrw).

To effectively orchestrate the stakeholders, the management of their requirements in a transparent, open, and fair way became more intricate. Thus, we changed our workflow and incorporated roles and elements from the Scaled Agile Framework to improve prioritization and workflow management. Requirements were collected in public epics in GitLab, allowing stakeholders to track progress of their individual project demands. To define the order of epic development, priorities were set by the Weighted Shortest Job First (WSJF) method.

As Coscine entered regular operation in 2023, we were confronted again with a change – balance the adding of new features while maintaining existing services. The growing user base demanded enhanced stability and reliability. Implementing quality checks and high-availability practices became essential, requiring not just new processes, but also a cultural change within the development team. We reevaluated our prioritization methods, moving away from approaches like WSJF (focus on shortest jobs) to user-value, time criticality and risk reduction. This shift acknowledged that long-term topics require sustained focus and that balancing feature development with maintenance tasks is crucial for the platform's longevity.

Throughout our journey with Coscine, we've learned the importance of adaptability and continuous workflow improvement. This demands a great deal of communication and flexibility from the whole team but is rewarded by satisfied stakeholders and an up-to-date RDM service. While we've made significant strides in aligning our agile development practices with the demands of a changing research world, we know that we must keep on adapting.