D2.4.2: Initial impact assessment of the Research Infrastructure

Good management is one of the key factors that guarantee the healthy status of an organization and its sustainability as well as its development and evolution. At present, the notion of good management for organizations with not-for-profit mission and objectives like PRACE AISBL is widely accepted. It is therefore important that the notion of impact assessment of the actions undertaken by the organization is understood and properly addressed by the governance and management bodies. Of course the non-profit character of PRACE has to be taken into consideration when implementing the necessary processes, tools and procedures for assessing the impact. The organisation drivers have to be different from those considered by commercial and business oriented organizations that focus almost exclusively on how the actions impact revenues and costs reduction. Moreover, impact evaluation is addressed not to improve the operation of the infrastructure or the performance of plans, but to know the Return of Investment of the organisation, the equivalent to the profit made for a profit-oriented organisation.
The objective of this deliverable is to propose a model for assessing the impact of PRACE by means of a selected group of variables whose measures can provide a picture of how well PRACE is doing.
The results of D2.4.1, Monitoring and Reporting Procedures, provided the necessary preliminary identification and analysis of a large set of variables where those related to impact and applicable to the specificity and peculiarity of the PRACE AISBL organization are a subset.
Impact assessment plays an important role in the overall PRACE workflow and evaluation process that provides feedback on how the actual results compare against the planning. We considered three types of evaluation: efficiency, effectiveness and impact.
Based on the PRACE workflow model, the monitoring targets and the derived variables are classified accordingly to four groups:

• Input: what resources PRACE uses in order to create and run its services such as monetary funding and budget, personnel resources and equipment;
• Delivery: the services provided by PRACE;
• Output: the concrete results obtained with allocated compute resources;
• Environment: the market environment in which PRACE delivers its services.

The decision on the types of impact to be considered for PRACE AISBL and how to elaborate on them is the first crucial task. The impact of PRACE AISBL can be divided in four types:

• Scientific: By allowing users to solve important queries in their scientific fields and introducing HPC to new fields of science, PRACE AISBL directly contributes to improving scientific knowledge in all fields of science for which supercomputing is essential;
• Economic: by contributing to the increase of scientific knowledge PRACE AISBL directly or indirectly contributes to the economic development of Europe;
• Social: by increasing scientific knowledge in research fields such as climate, life sciences, etc. PRACE AISBL contributes to improvement of quality of life of European population and European social development;
• Environmental: PRACE AISBL can have important contributions in the fields of:
  • Climate;
  • Energy;
  • Water management.

Due to the characteristics of HPC, impact assessment can be:

• Quantitative: based on defined quantitative targets or compared to quantitative baselines that need to be properly identified and need to be analysed on a regular basis to decide if they are still valid or need to be revised;
• Qualitative: based on case studies or success stories that are properly identified and analysed. The Scientific Steering Committee of PRACE AISBL can have an important role for identifying and validating case studies and success stories.

Although qualitative assessment could prove cumbersome given its nature based not on measurement but rather on some subjective evaluation, it is felt that it could not be neglected because of its relevance when addressing economic, social and environmental impact. The deliverable recommends the following to be implemented by PRACE AISBL:

• The development of quantitative monitoring on the strategic level via Balanced Scorecard that combines between 6 and 12 strategic indicators for steering purpose. They are meant to represent the four dimensions “Input”, “Delivery”, “Output” and “Environment” highlighted by D2.4.1 outcome.
• Regular peer review of the PRACE organisation in order to reflect the state of the organisation at strategic level. The standard tool for this process in the academic realm is the peer review that should carried out by a set of senior experts outside the inner circle of PRACE stakeholders.

In deliverable D2.4.1 the monitoring variables were classified and described in detail. In the present deliverable the focus goes into impact and its assessment. Impact assessment has been investigated from the theoretical perspective to gain a deep understanding on the concept, and best practices have been analysed from different publicly available documents, interviews with managers and directors of non-profit organisations, and experience from the different centers of PRACE members.