TAPIS ADVANCED COLLABORATIVE OPEN SOURCE (TACOS) INITIATIVE 2023 Evaluation Report

Project Overview

The National Science Foundation (NSF) awarded The University of Texas at Austin (UT) Texas Advanced Computing Center (TACC) a Phase 1 Pathways to Enable Open-Source Ecosystems (POSE) grant to develop the Tapis Advanced Collaborative Open Source (TACOS) Initiative from September 15, 2022 to September 14, 2023. Tapis is an open-source cyberinfrastructure platform that provides a hosted application programming interface (API) middleware to enable computational research, including collaboration, data management, and code execution on remote systems.

The Tapis team, in collaboration with its partners, has developed an expansive ecosystem of tools including software development kits and libraries in several popular programming languages, public datasets in different application domains, a catalog of exemplar functions and applications, Jupyter notebooks illustrating interactive data analysis on high-performance computing (HPC) and high-throughput computing resources, and various documentation sites and tutorials for working with Tapis. Developing and maintaining these assets often requires knowledge of the Tapis framework as well as subject matter expertise and would benefit from community members' contributions. Therefore, the goal of the TACOS Initiative is to build an open-source ecosystem (OSE) in which the community of cyberinfrastructure researchers, scientists, and professionals who use Tapis can contribute to the ecosystem. Through this Phase 1 grant, the Tapis team sought to identify the building blocks for the OSE and to create a plan for how to execute them.

Evaluation Overview

In Fall 2022, Tapis contracted with TACC’s Expanding Pathways in Computing (EPIC) STEM Evaluation Services to design and conduct an internal evaluation of the TACOS Initiative. EPIC used a mixed methods approach to assess and inform the implementation and outcomes of this Phase 1 initiative. The evaluation objectives are to:

• Assess whether the initiative is making satisfactory progress toward achieving its intended goals to identify the building blocks and create a plan for expanding Tapis into an OSE;
• Identify potential users and stakeholder groups, and assess their needs and interest in engaging in an OSE;
• Recommend reasonable and evidenced-based adjustments to initiative plans; and
• Attest to the integrity of any outcomes and impacts that stem from the initiative.

The evaluation employed an iterative, participatory process in collaboration with the Principal Investigators (PIs) to collect and provide meaningful data on an ongoing basis to inform project strategies and improvements. The evaluators held regular evaluation meetings with the PIs to collect and share data in a formative feedback loop for continuous improvement.

The evaluators used the following data collection methods to address the evaluation objectives.

• Reviews of Documents and Data provided by the PIs to inform development of evaluation instruments, analyses, and reports.
• Tapis User Survey administered in December 2022–March 2023 to identify user and stakeholder groups’ needs and interest in engaging in community activities, serving on a working group and/or steering committee, and contributing to an OSE in some other way. This survey included quantitative and qualitative items and was also used to identify potential interviewees for collecting further information. (See Appendix 1.)
• Tapis User Interviews conducted in June 2023–July 2023 with key users/stakeholders interested in participating in steering committees, working groups, and/or other ecosystem activities to further identify their needs, interests, and engagement, as well as anticipated challenges and recommendations for community building and ecosystem development. (See Appendix 2.)

The evaluators analyzed data from the above evaluation methods by conducting descriptive analyses of quantitative data and content analyses of qualitative data. This report summarizes findings from all evaluation methods and provides recommendations for moving forward with the Tapis OSE development.