Published May 20, 2024 | Version 2
Project deliverable Open

OAE Evaluation Framework & Toolkit: For OAE NAECs, Centers & Nodes

  • 1. ROR icon Cardiff University

Description

This Framework and Toolkit has been designed to assist astronomy educators within the Office of Astronomy Education (OAE) network (NAEC, Centers and Nodes) in systematically evaluating their educational programmes and developing evidence-based practices that can be applied across various contexts. Employing a ‘realist approach’ to evaluation, the resource delves into what works, for whom, and under what circumstances. This approach accounts for the intricate nuances and complexities of different environments, including cultural and practical components that vary across local geographies, and in diverse school settings.

The document guides NAECs, Centers and Nodes through evaluating their programmes from start to finish, through stages including: setting objectives; creating a theory of change; deciding what type of data to collect, when to collect it, and from whom; ethical considerations; and which data collection tools to use.

The Toolkit component comprises a set of data collection tools designed to capture valuable information and feedback from the programme’s audience. Each tool is accompanied by a comprehensive description of how to conduct an evaluation with it. This description includes information on the required resources, such as equipment and space, guidance on implementation with the audience, suggested questions and prompts for participants, a summary of the tool’s advantages and disadvantages, and an overview of analysis procedures to use once the data has been collected.

Files

OAE Evaluation Toolkit v4.pdf

Files (964.2 kB)

Name Size Download all
md5:ba0483921767dc9a1d85444dd09cff09
964.2 kB Preview Download

Additional details

Funding

Klaus Tschira Foundation

Dates

Available
2024-05-20

References

  • Pawson R. and Tilley N. 1997. Realistic Evaluation. London: Sage
  • Greenhalgh J. and Manzano A. 2022. Understanding 'context' in realist evaluation and synthesis. International Journal of Social Research Methodology 25(5), pp. 583-595. DOI: 10.1080/ 3645579.2021.1918484
  • Rameses II Project. 2017. What realists mean by context. [Online] Available at: https://www.ramesesproject.org/
  • Rameses II Project. 2017. What is a mechanism? [Online] Available at: https://www.ramesesproject.org/
  • Better Evaluation. 2021. Realist Evaluation. [Online] Available at: https://www.betterevaluation.org/methods-approaches/approaches/realist-evaluation
  • Bartlett S, Fitzgerald MT, McKinnon DH, Danaia L, and Lazendic-Galloway J. 2018. Astronomy and Science Student Attitudes (ASSA): a short review and validation of a new instrument. Journal of Astronomy & Earth Sciences Education (JAESE) 5(1). DOI: 10.19030/jaese.v5i1.10190
  • Freed R, McKinnon D, Fitzgerald M, and Norris CM. 2022. Development and validation of an astronomy self-efficacy instrument for understanding and doing. Physical Review Physics Education Research 18. DOI: 10.1103/PhysRevPhysEducRes.18.010117
  • Braun V and Clarke V. 2006. Using thematic analysis in psychology. Qualitative Research Psychology 3, 77–101. DOI:10.1191/1478088706qp063oa
  • Cohen L, Manion L, and Morrison K. 2018. Research Methods in Education. 8th Edition. Oxfordshire: Routledge