Report on the methodological framework at macro level

ENVIEVAL has placed the macro-level evaluation of the impact on the environment of rural development programmes and measures firmly in the context of the necessity to combine micro-level and macro-level evaluations using a robust counterfactual approach to assessing net-impacts. This report describes the macro-level logic model and the results of the case studies, which have tested the evaluation methodologies and the logic model.

Data availability and quality are the over-riding constraints for the development of a robust quantitative macro-level evaluation based on a sound counterfactual. The disparity between data for environmental monitoring and policy uptake creates inconsistencies between micro and macro-level assessments which have an adverse effect on the net-impact assessment. However, it also affects an evaluator’s ability to create comparison groups that can support the application of an elaborate statistics-based evaluation.

A robust assessment of comparison groups using elaborate statistics-based methods has been demonstrated to be difficult at a macro level. In most case studies, only a naïve group comparison of with / without and (or) before / after situations could be done relying on a simple mean for the difference-in-difference. Whilst this is mainly caused by data issues, the need to have a large number of different regions to allow for robust econometric assessments of comparison groups at a regional level is also an issue. Macro-level modelling approaches (e.g. applied in the context of the public good soil quality) were able to fill some of the data gaps but relied on established causal links between agricultural practices and the relevant public good. Using elaborate statistics-based methods for the assessment of counterfactuals with comparison groups is more applicable at a micro level.

Two ways of conducting macro-level evaluations of environmental impacts emerge from the experiences of the case-study testing. First, upscaling of micro-level results was possible in most of the macro-level case studies, although statistical representativeness could not always be achieved due to data gaps. However, using GIS-based spatial evaluation approaches potentially provides a solution for consistent aggregation and upscaling of micro-level results to a macro level. Plausibility checks of the results are suggested as a means of reviewing the extent to which the occurrence of indirect effects can be interpreted.

Second, solutions for macro-level evaluations based on specific macro-level modelling approaches were tested successfully. Modelling methods provide different opportunities to fill the gap created by the lack of observational data availability or quality and provide solutions for the evaluation of measures which are implemented area-wide (i.e. not sufficient nonparticipants exist). However modelling methods often require particular skills and expertise and are not without their data issues. For example, both of the climate change case studies that estimate the impact on GHG emissions are data demanding in their own right. Another approach to deal with data gaps is the use of geostatistical methods to assist in aggregating data, such as the FADN data for the HNV case study in Italy.

Overall, the macro-level logic model has proven to be useful in highlighting key considerations at critical moments during an evaluation, which support managing authorities and evaluators to gradually improve the quality of the evaluation towards a more quantitative assessment.