Evaluation FWF Special Research Programmes (SFB)

Executive Summary

This final report has been prepared by AIT – Austrian Institute of Technology and the Katholieke Universiteit Leuven (KU Leuven, sub-contractor) for the programme evaluation of the Special Research Programmes (SFB) of the Austrian Science Fund (FWF). The evaluation was carried out between January 2019 and February 2020 and critically reviews the SFB programme based on projects granted in the period 2004-2018. The aim of this evaluation is to assess the SFB programme’s outputs, outcomes, and impacts against its stated objectives and to provide recommendations for the future implementation of the programme. The SFB programme is expected to fulfil the following three objectives:

• Support outstanding research in Austria by creating long-term, generally multidisciplinary/interdisciplinary and (by international standards) exceptionally productive research networks (SFBs) which should reach “critical mass”
• Expand human resources in science and research and improve gender-balanced orientation of research and education
• Achieve broader effects on the Austrian science and research landscape by supporting universities and other research institutions in the autonomous creation of focus areas in their research profiles and improve public awareness of top-quality research by supporting science communication, knowledge transfer and appropriate dissemination strategies

Based on these objectives, the terms of reference and this study identified five evaluation dimensions based on the overarching objectives of the programme. The first three relate directly to the specific SFB objectives and expected impacts of the programme, while the fourth dimension concerns the adequacy of FWF processes for selection, management, and support of SFB throughout its lifecycle, and the fifth dimension relates to the correspondence of the SFB programme to FWF objectives and the Austrian research and innovation system:

• Support for outstanding research in Austria
• Enhancement of human resources in science and research and improved gender balance
• Achievement of broader effects on the Austrian science and research system
• Effectiveness and efficiency of programme implementation
• Coherence of the SFB programme with overarching objectives

In order to address the evaluation dimensions outlined above, a theory-based evaluation and contribution analysis mixed-method evaluation design was chosen. Evidence was collected by means of a literature review, portfolio analysis, bibliometric analysis, control groups (FWF Stand-alone projects and rejected SFB), counter-factual analysis, a large interview programme of individual and focus group interviews, and two online surveys.

This Executive Summary presents the key findings and conclusions along the five evaluations dimensions with relevant recommendations for the future orientation and implementation of the SFB programme embedded in the corresponding sections.

Key findings and recommendations

Coherence and contextualization of the SFB programme

The SFB programme is still the only basic research oriented and network-based research programme in Austria and has therefore a unique selling proposition with a strong relevance for Austria’s research and innovation system and an excellent reputation among researchers and university leaders.

The programme is strictly bottom-up oriented and exclusively focused on research excellence. This constitutes a unique feature in international comparison, where the use of thematic priority setting is wide-spread, and the strategic orientation of basic research networks has changed.

Two major changes in the configuration of the programme occurred since the 2004 evaluation: Firstly, the overall acceptance rate decreased from 54% to 14%. Secondly, the share of SFB funding in the total FWF funding decreased from 14.7% in the year 2014 to 5.6% in 2018. The FWF did not follow the recommendation of the 2004 evaluation which indicated that provisions are certainly needed to keep up or even enlarge the share of budget that goes to the networks.

The programme exhibits a strong dominance of the Natural Sciences (57%) and Human Medicine & Health Sciences (26%), a concentration that is higher than that found in the Stand-alone projects. The distribution does not quite reflect the overall intention of the programme, which explicitly addresses researchers “from all disciplines”. The skewed disciplinary distribution is to some extent due to structural differences in Social Sciences and Humanities and the Natural Sciences (e.g., small size of Social Sciences and Humanities research in Austria, a lack of natural networks, lack of incentives, approach to research). However, the eligibility requirement of concentration of at least 50% of sub-projects at one location seems to be a barrier to researchers in Social Sciences and Humanities to participate in SFB to some extent. The existing disciplinary distribution and the high rejection rates of the programme also constitute an evident barrier to application.

Compared with the last evaluation, the programme has lost momentum and transformed to an extremely competitive niche rather than a systemic oriented flagship programme that is capable to shape profile building of research organisations. The following factors are responsible for this transformation: 1) A low level of absolute annual funding, 2) a negative trend in the share of funding provided for by the FWF, 3) a tight concentration on a limited number of disciplines, and 4) acceptance rates of just about 14%. This is worrying, as all system level analyses confirm an increasing relevance of building new and sustaining critical mass, establishing better career prospects for young high-potential principal investigators, and sharpening of research profiles at Austrian R&I institutions.

Against these findings, the following recommendations are given:

Recommendation 1: FWF’s funding share allocated to network programmes should aim to reach a minimum level of 25%.

Given the level of ambition of the programme, such as achieving broader effects on the Austrian research landscape, the resources do not suffice. For pursuing the overarching objectives set, a better endowment with resources from the FWF budget is needed for network programmes.

Recommendation 2: FWF should elaborate measures that allow stronger participation of more disciplines and support the emergence of new fields of excellence.

Measures could comprise the introduction of calls for certain disciplines as well as measures focussing on structural impact for sharpening research profiles at Austrian R&I institutions. Recent system level analyses confirm an increasing relevance of building new and sustaining critical mass and establishing better career prospects for young high-potential principal investigators. It therefore could also be considered to incorporate aspects of relevance and impact in the funding decision process.

Support for outstanding research

The analyses performed clearly show that SFB succeeded in supporting exceptional research as evidenced by the outstanding publication and citation record of the funded projects. The publication profile of the SFB program and its projects in the period 2004–2017 displays a strong continuity in the growth and development of the scientific activity. The publication analysis shows that SFB researchers receive a high citation impact and that their work reflects a very high scientific standard, and SFB PIs in general publish in high impact journals and receive more citations than expected for these journals. SFB projects also exceed national averages and outperform Stand-alone projects along all metrics of citation impact.

The excellence in scientific outcomes is not paralleled by achievements in terms of interdisciplinarity and international collaboration. SFB projects involve fewer internationally co-authored publications compared to the Austrian average, and average interdisciplinarity scores of the SFB program lies below the average of Stand-alone projects and the Austrian average. Interdisciplinarity is typically a result of very closely related (sub-) disciplines working in a sub-project. It is mainly found in fields of science with clearly overlapping content such as biology and medical theoretical sciences and clinical medicine. In this respect, interdisciplinary co-operation in SFBs resembles to a large extent the cross-disciplinary structure of Stand-alone projects.

In somewhat stark contrast is SFB participants’ own assessment of interdisciplinarity in the programme. Interdisciplinarity is seen as one of the key attributes that makes the SFB attractive and unique and SFB researchers perceive their projects as highly interdisciplinary. On the other hand, interviewees stressed that interdisciplinarity is merely a means to an end. The evidence on this apparent contradiction in opinions on interdisciplinarity shows that researchers have different understandings of the differentiation between multi- and interdisciplinarity; a lack of intrinsic motivation for interdisciplinary integration; and that the peer review process is perceived as detrimental to interdisciplinary projects integrating non-closely related fields of science, thus causing strategic choices of disciplinary distribution in SFB proposals to maximize success rates.

Against these findings, the following recommendations are given:

Recommendation 3: FWF should keep the overall programme structure (network size, funding provided, duration etc.) and principles of a two-stage peer review process.

The two-stage peer review selection system is in principle capable of identifying research networks that can deliver outstanding results in terms of publication impact. For funded SFBs, framework conditions (network size, amount of funding per projects, etc.) which allow delivery of outstanding research results are provided. Hence, there is no need to change programme fundamentals such as the two-stage peer review process and structural characteristics in terms of network size and funding per project.

Recommendation 4: FWF should incorporate measures that strengthen the performance of multi-/interdisciplinary research.

Having the ambition to fund highly multi-/interdisciplinary research networks is sensible. However, the ambition is not being met by the programme. Against the background of the objectives set, FWF should define and integrate review criteria for inter-/multidisciplinarity in the review process. Weighting scores for interdisciplinarity and an interdisciplinary panel design could help to better adhere to the own objectives set.

Enhancement of human resources and gender mainstreaming

The SFB programme is seen as among the most prestigious research grants on national and European level. As such, participation in an SFB has large positive impacts on the internal and external visibility of involved researchers, their reputation and recognition in the scientific community, as well as on their national and international collaboration networks.

SFBs are also particularly well-suited for improved training of PhD students due to their collaborative nature, large network, and scientific excellence and the SFB programme supports a large proportion of early-career researchers.

A consideration of key researchers shows that women are on average slightly younger than male PIs, but at the same time heavily underrepresented. An increasing age of the key researcher as well as an increasing age of the SFB coordinator significantly decrease the probability of success; the size of the effect is albeit low. It is career attainment, proxied by carrying the title of professor, which increases the probability of acceptance in both the first and second stage. Overall, SFB decision processes do not discriminate against younger PIs. However, rejected sub-project leads in granted SFB are observably younger than their successful counterparts. Qualitative analyses support this finding, where SFB participants reported their perception that rejected sub-projects were disproportionately led by younger key researchers. In general, the possibility to exclude individual sub-projects from funding was heavily criticised by participants as this also displays a certain disregard for the scientific and interpersonal contribution of individual sub-projects to the whole network entity that in sum lead to higher impacts.

The statistical analysis furthermore highlights two aspects of the application process: 2) women are less likely to make it through the first stage of the process, 3) the second stage does not favour male or female researchers. Especially the first stage of the application process favours established researchers (professors) and success rates increase with age.

The SFB programme does not live up to its own expectation of funding research excellence while “boosting gender mainstreaming and gender-balanced orientation of research and education” in its activities and results. Specific, structured measures for gender mainstreaming and diversity in SFB networks are absent in most networks. While the challenge is acknowledged, efforts and development of measures at network level are missing. Except in Medicine and Social Sciences, the share of female PhD students employed in SFBs, is considerably lower than the share of PhD graduates in the respective field of science.

Against these findings, the following recommendations are given:

Recommendation 5: FWF should take stronger consideration of the network level in funding decisions and limit interventions into the network composition of SFB

The configuration of a network should be the choice of the applicants, who may best consider the required balance of networks in terms of its key researchers. While interventions into the structural composition of networks should not be prohibited, they should be limited and consequences of removing sub-projects should be considered.

Recommendation 6: FWF should incorporate measures that strengthen gender mainstreaming at a network level

Existence and appropriateness of gender mainstreaming measures applied are neither considered in the funding decision processes nor in the activities of most networks. As SFB research networks represent the elite of all Austrian research networks, specific measures need to be set which guarantee that gender mainstreaming measures are being effectively applied at the network level and capable to develop synergies with organisational strategies.

Broader effects on the Austrian science and research landscape

Long-term collaboration with researchers from other institutions in Austria, increasing research productivity through tighter integration of research and establishing or expanding a research group at the host institution are the core motivations to apply for an SFB. These motivations are very much in line with the programme ambition to support profile building.

The programme is very attractive for the participating researchers because it offers the chance to focus on ambitious research questions with new horizons while tackling new challenges and developing/establishing new concepts and paradigms in their research fields. The main mentioned impact mechanisms are the secure funding for long-term research, its significant investment and collaboration at a network level, and the knowledge circulation facilitated through the collaboration. Projects share a joint vision, have a high level of trust among each other, and the intensity of collaboration is high. The conditions for achieving impact are favourable.

The actual effect on the research profiles of the host institutions was questioned as the programme rather funds existing spots of excellence and reinforces existing strengths. Nevertheless, the results of third-party funding are used by participating organisations to indicate their research priorities and SFBs rank in the top-league for this purpose.

The SFB host organisations provide structural support (top-up funding and additional individual measures for e.g. infrastructure provision) which goes beyond the level of support for other research projects and reflects the structural relevance and prestige of the programme at an organisational level. However, SFB funding has no impact on offerings for new permanent positions and the sustainability of the research groups still requires new third-party funding.

Against these findings, Recommendation 1 and Recommendation 2 of this evaluation study could contribute to better achieving broader effects on the Austrian science and research landscape.

Programme implementation and management

The two-stage application procedure for the SFB is uniformly well regarded and in line with selection procedures internationally. However, the turnaround time between concept proposal and funding decision of minimum 14 months is seen as too long, while the time for drafting the full proposal is too short for its length and complexity.

The overall availability and quality of information is good. There is broad agreement that the response time and professionalism of FWF is very good overall. However, there is some dissatisfaction with the service-orientation of the FWF and its inflexible processes involving aspects such as fixed starting dates of projects and demands placed on the language of application and evaluation.

The explicitly stated aim of the SFB programme is to promote tightly interconnected interdisciplinary research networks, with the expectation that the collaboration produces outcomes that are greater than the sum of their parts. Contrary to this aim, the evaluation process and criteria is discipline-based and places strong emphasis on the scientific quality and reputation of individual project leads as evidenced by their publication track records. The possibility to exclude individual sub-projects from funding also displays a certain disregard for the scientific and interpersonal contribution of individual sub-projects that in sum lead to higher impacts. Lastly, there is no evaluation dimension or criterion assessing the degree and quality of interdisciplinary cooperation.

The evaluation process and criteria could be better used to facilitate the achievement of programme goals in enhancing human resources, boosting gender mainstreaming, and promoting young/early career researchers.

There is dissatisfaction regarding the communication and transparency of the decision-making process (provision of written reviews, instruction of reviewers, balancing review quality), with clear room for improvement in these aspects to more effectively utilize the evaluation process, including the hearing.

The FWF currently does not provide support for project implementation going beyond funding and administrative dimensions and requirements. However, the programme’s goals and interest from researchers indicates a need for action to introduce additional measures related to knowledge transfer, public awareness and dissemination.

Against these findings, the following recommendations are given:

Recommendation 7: FWF should simplify and harmonize application and reporting instructions, forms, and templates.

There is room for improvement concerning the length and complexity of the administrative efforts involved in applying for and implementing an SFB grant, which in turn could improve satisfaction with FWF’s administrative processes.

Recommendation 8: FWF should speed up the communication of reviewer assessments to SFB applicants.

Reviewer assessments should be communicated in advance to the hearing, as this would likely contribute to improving the project selection process and enable better assessment of an application’s scientific quality.

Recommendation 9: FWF should provide additional support mechanisms for promoting knowledge transfer and dissemination beyond the scientific community.

The current programme objectives state that support mechanisms for promoting knowledge transfer and dissemination beyond the scientific community should be pursued. If this target is being taken seriously, FWF needs to build-up specific support strategies for fostering outreach of the networks.