Evaluation of the Arts-based Research Programme of the Austrian Science Fund (PEEK)

EXECUTIVE SUMMARY

The subject of this evaluation is the arts-based research programme (PEEK) of the Austrian Science Fund (FWF). With the introduction of this programme in 2009, the FWF has reacted to the equating of arts universities with other universities as postulated in the amendment to the 2002 Higher Education Act, which was anchored in the Research and Technology Act of 2007 with a corresponding amendment. As an equivalent for “science”, the work area of “developing and opening up the arts” (i.e. “Entwicklung und Erschließung der Künste”) was adopted for the arts universities. The inclusion of “developing and opening up the arts” in the Research and Technology Act was also meant to signalise the upgrading of art colleges to arts universities and their equal treatment with other public universities. To ensure the opportunity to provide adequate research approaches that correspond to the character and scope of the arts universities, arts-based research was identified as a promising approach.

The FWF does not offer a comprehensive definition, but does provide clear indications of what is to be understood as arts-based research. The term “arts-based research” helps clarify that the relationship to artistic practice is the decisive element in the research process. Gaining knowledge and developing methods (also) takes place by means of aesthetic and artistic approaches and practices as opposed to purely scientific knowledge processes (FWF, 2021; PEEK application guidelines).

The choice of arts-based research as a research programme of its own is well justified, as this approach has been gaining importance for the past 30 years as a field of action for both arts universities and research-funding agencies. There are competing terms, such as “artistic research” and “practice-based research”, which come from different traditions and environments and differ from each other in nuances. These differences are outlined in Chapter 5. In understanding arts-based research, it is helpful to understand what it is not (e.g., research about art; research as a tool for preparing artistic production; creative dissemination; etc.; see Section 5.1). In the course of this evaluation, we concluded that a basic understanding of arts-based research is present in the group targeted by the programme. It provides sufficient ontological guidance for research in this particular programmatic corridor of action.

The institutionalisation of research funding for arts-based research in Europe is more diffuse. First, there are only a few cases of explicit research funding for arts-based research (at least at the national level) and these are very differently designed. They range from mainstreamed approaches, as in Switzerland within the Swiss National Science Fund (SNSF), to sophisticated programme design, as in Norway (the Norwegian Artistic Research Programme of the Norwegian Agency for International Cooperation and Quality Enhancement in Higher Education). The latter has used several instruments other than just research funding to build an arts-based research community. In contrast, by introducing a special programme for arts-based research, the FWF has concentrated on actual research funding, which in turn has been handled very professionally. This has undoubtedly provided crucial impulses for the professionalization of arts-based research in Austria, but has also led to a strong institutional concentration due to a lack of accompanying measures. This can be assessed both positively and negatively, which brings us to the core area of this evaluation.

By means of an evaluation, the FWF intended

• to critically review the PEEK funding programme to identify strengths and weaknesses, to quantify and qualify output, outcome and impact;
• to obtain evidence-based recommendations for itself and its supervisory bodies on whether and how the PEEK programme should be continued, improved or restructured; and
• to explore how its overall funding strategy for artistic research (year 2022 and beyond) should further be developed.

The evaluation addressed the following three sets of questions:

First, were the programme objectives achieved? Here, a distinction is made between goal achievement in the sense of

A. support for high-quality and innovative arts-based research in Austria;

B. building research capacities at an international level;

C. increasing public as well as academic awareness of arts-based research; and

D. elevating the profile of arts-based research at universities and other research institutions.

Second, were programme implementation and management adequate and efficient? A distinction should be made here between different phases in the research funding cycle:

E. call preparation and support;

F. project evaluation management;

G. project support; and

H. regulations

Third, what is the status of PEEK in the FWF's programme portfolio and in the Austrian research landscape?

In order to answer this catalogue of questions, the ZSI team, together with Professor Felix Stalder from the Zurich University of the Arts, used a multi-method evaluation design. It consisted of document analyses and database research, content analyses of documents and expert interviews, artefact-based interviews with grantees, i.e. principal investigators (PIs) as well as team members, an online survey with granted and non-granted applicants, and a focus group with representatives of arts universities and other research institutions. The results from these different methods were reflected on and triangulated with each other several times in the course of the evaluation process.

As an overall result, it can be stated in one sentence that the FWF did many things very right with PEEK.

The following findings can be summarised for the first block of evaluation questions (achievement of programme objectives):

A. The introduction of a competitive research programme – with international peer reviews and a dedicated board that quality-assured the review process in a research area that is itself only partially institutionalised – was undoubtedly a significant contribution to the emergence of high-quality and innovative arts-based research in Austria.

Compared to the FWF's Stand-Alone project funding, arts-based research is characterised by a higher degree of interdisciplinarity and transdisciplinarity. This manifests itself in a significantly more diverse output, which - according to our samples - has led to slightly fewer peer-reviewed publications compared to Stand-Alone projects. However, PEEK is characterised by significantly more innovative and diverse dissemination formats.

Almost 100% of the respondent group surveyed, consisting of funded and non-funded PEEK applicants, affirmed that PEEK has contributed to the institutionalisation of arts-based research in Austria, is vital for research activities at arts universities and has improved the international reputation of Austrian arts-based research. A large proportion (89%) also perceived an improvement in the reputation of arts-based research within the research community. About 80% confirmed that PEEK has contributed to increasing the variety of arts-based research output and that the programme has helped to achieve greater diversity in methods and approaches to artistic production. The non- funded respondents were slightly more cautious in their responses to the questions about the impact of PEEK on quality and innovation. They were noticeably more critical only of PEEK's contribution to improving the status of arts-based research within arts communities. The danger of arts-based research becoming too remote from arts and being mainstreamed towards “normal” science/scholary production appropriated by arts universities was noted several times in this context.

B. With regard to the increase in training and career opportunities, PEEK was also seen to make a high contribution among those involved (average agreement of 93%). In the case of PEEK Principal Investigators (PIs), we mostly ascertained effects on securing their status in the academic field. In turn, the younger and less established PEEK project team members showed significantly more career- related effects. This was confirmed by both the survey and career tracking. From the point of view of the PEEK PIs interviewed, both they and the team members have benefited equally from PEEK in terms of competence building, gaining visibility, international cooperation, personal career advancement, acquiring new  qualifications and  enhanced academic reputation. However,  it was repeatedly criticised that PEEK projects are not sufficiently incorporated into teaching and are understood as an extracurricular activity of their own.

As far as identity is concerned, it can be stated that by far the largest proportion of the individuals studied feel they belong to both the art world and the academic world. PEEK users create art and publish scientific articles. However, rarely do those individuals describe themselves as arts-based researchers, even though the funded PEEK project is prominently featured in self-portrayals. The identity of a full-time arts-based researcher is basically inexistent.

C. As far as the value of PEEK in academia is concerned, we found that for some non-arts universities, the existence of PEEK is perceived as another option for the application of alternative methods in research. Cases from architecture and digitalisation (especially artificial intelligence) are particularly common in this respect. Yet for the arts universities, arts-based research is fundamentally central, albeit to varying degrees. Six out of seven PEEK applicants (both funded and non-funded) perceived that PEEK has raised the standing of arts-based research in the research community and contributed to its increased public perception. Within its own guild, PEEK has undoubtedly boosted the international recognition of arts-based research from Austria. Interestingly, however, there are information deficits with regard to other PEEK projects that were not carried out by oneself, which was considered a pity on the part of the funded researchers.

D PEEK has given considerable impact to contour arts-based research as a central research approach in several arts universities. This applies especially to the University of Applied Arts Vienna, the Academy of Fine Arts and the University of Performing Arts and Music in Graz. Two thirds of all FWF-funded projects at the first two universities are attributable to PEEK. Conversely, this also implies extreme dependence on PEEK. The FWF project portfolio of those arts universities that also focus on music is broader in this respect, although it can be said that competitively funded basic research at the arts universities is overall limited. In recent years, however, research offices or more or less institutionalised support services have been established at almost all arts universities. At the arts universities that already make heavy use of PEEK, emphasis is placed on expanding the research programme portfolio, and at those that implement only a few PEEK projects, attempts are made to promote PEEK more strongly within their own institutions. Still, PEEK is not a programme for the arts universities alone. Although their share was more than 50% in each year (both in terms of applications and funded projects) and exceeded or even reached the 80% mark in funded projects in five years between 2009 and 2021, the technical universities and a few specialised institutions, such as the Research Institute for Art and Technology, Ars Electronica and the Association of Visual Artists, have also participated in PEEK. In turn, the medical universities are almost not at all represented. Ensuring that PEEK is an open programme that can be used inter- and transdisciplinarily, and is not understood as a programme exclusively for the arts universities, seems central to in order to guarantee exciting projects with innovative approaches and theories in the future.

A shorter statement can be given with regard to the second major evaluation question requested by the FWF, namely whether programme implementation has been adequate and programme administration has worked efficiently.

E. From the perspective of both funded and non-funded PEEK applicants and the vice rectors and research services who participated in the focus group, the FWF has managed PEEK well and professionally. The services offered by the FWF during the application phase were considered at least satisfactory by an average of 80% of the respondents, although non-funded applicants were consistently more sceptical. This assessment corresponds to the results of most evaluations of FWF programmes (e.g., Doctoral Programmes, Schrödinger Fellowship, START Programme Wittgenstein Award) in recent years. It also applies to the comprehensibility of the application documents. The focus group participants criticised above all that there is only one submission deadline for PEEK per year.

F. The review and evaluation process was perceived as critical to an (exceedingly) high degree. This concerns the quality of the individual reviews as well as the role of the PEEK Board in quality assurance. The evaluation criteria appeared appropriate to only 52% of the respondents and only 53% agreed that the review and evaluation process is transparent. This critical perspective by no means came only from the non-funded PEEK applicants. There is a need for action here.

G. In the course of implementing PEEK projects, the FWF was perceived as being accessible at all times, reacting quickly and flexibly, and providing clear and high-quality responses. However, it was also critically noted that the FWF does little to build an arts-based research community.

H. The FWF's high audit effort is problematic. PEEK projects are complex in their production and knowledge generation conditions and processes, which leads not least to a high number of invoices as well as contracts with third parties. However, the PEEK regulations correspond to the complex production and knowledge generation conditions and processes of arts-based research, which is why other ways of containing the auditing burden should be pursued instead of changing the regulations.

Finally, the FWF wanted an assessment of how PEEK is positioned within the FWF funding portfolio and the Austrian research funding landscape. Our findings are clear. PEEK is a programme with a high unique selling position (USP) and, at the same time, it is also different because it represents an alternative research paradigm that is not based on hypothesis-driven testing of assumptions or empirical findings or observations. Instead, PEEK fundamentally works exploratively with artistic methods and often incorporates interdisciplinary and transdisciplinary perspectives. If PEEK had not been implemented, arts-based research would not exist in Austria at this level and breadth. Only 17% and 7% of the respondents stated that funding from the Austrian Research Promotion Agency (FFG) and the Austria Wirtschaftsservice (aws), respectively, would also be an option for them. Otherwise, arts stipends, Stand-Alone project funding from the FWF, funding from the federal provinces (e.g., from the Vienna Science and Technology Fund WWTF), the EU “Creative Europe” programme and occasionally Horizon 2020 (incl. the European Research Council ERC), private sponsors and internal university funding were mentioned as additional possibilities for conducting arts-based research. The fact that PEEK fills a gap in the funding portfolio despite these isolated alternatives that are suitable only in part can also be deduced from the fact that PEEK funding recipients are merely sporadically active in other FWF programmes. PEEK has thus mobilised a large number of arts-based researchers who have not yet been able to take advantage of FWF funding.

It is therefore hardly surprising that we recommend that PEEK be continued for the next five to ten years.

In order to maintain the diversity of potential applicants, however, more attention should be paid to public relations and community-building. We are aware that, in contrast to public relations work, community-building is not considered a task of the FWF and would also be structurally far too demanding for the Fund, which is why other actors should also be involved in community-building – in coordination and division of labour with the FWF. These are primarily the arts universities themselves, but also the Federal Ministry of Education, Science and Research (BMBWF). Finally, PEEK was created with the dual goal of enabling arts-based research in Austria in order to strengthen research at institutions that are responsible for developing and opening up the arts, first and foremost the arts universities. We consider an opening of PEEK in the direction of the art universities, which have so far participated little in PEEK - but also in the direction of other universities and research institutions, museums as well as major art events such as the Salzburg Festival, the Styriarte or the Vienna Festival - to be beneficial in order to prevent an institutional concentration and a possible accompanying narrowing of potential research orientations. In order to make better use of PEEK and to broaden research in arts-based research, we recommend that arts universities in particular integrate PEEK projects into their teaching. There is still abundant room for improvement in this respect. At the same time, we suggest flanking measures to facilitate the transfer of arts-based research to other FWF programmes. This includes first recognising arts-based research as a paradigm of a different, alternative research approach. In the long term, the exclusion of arts-based research in other FWF programmes, such as the career mobility or international programmes, cannot be justified. However, the PEEK regulations must then also apply to the other programmes if the respective applications are arts-based research (tick-box). The review and evaluation process for such applications should continue to be quality-assured by the PEEK Board.

We also recommend making the application process more flexible by allowing for two annual submission deadlines.

The role and mandate of the PEEK Board with regard to its influence on overruling external reviewer opinions should be made transparent and better communicated. The quality assurance function of the Board in the review and evaluation process should be maintained in any case, and even more value should be placed on external reviewers with well-argued and clearly articulated assessments.

The approval rates for PEEK are below the average FWF approval rate and significantly below Stand- Alone project funding. While a fixed budget was justified when the programme was introduced because the applications often did not have the desired quality, the FWF should now work together with the BMBWF to align PEEK budgeting with the more flexible practice of most other FWF programmes.

Finally, in order to reduce the FWF's auditing costs, which are significantly higher for PEEK projects than for other FWF projects, we recommend establishing contracts with the arts universities with the highest numbers of applications for a division of labour to review the costs incurred in terms of justification of content and formal correctness. The division of labour could go along cost types or as random samples across the invoices and contracts incurred.