RISIS Research Seminar: Reliability of innovation measures

The Community Innovation Surveys are the main data source for measuring innovation in Europe. The aggregated CIS data tables cover the basic information of the enterprise, product and process innovation, innovation activity and expenditure, effects of innovation, innovation cooperation, public funding of innovation, source of information for innovation patents, etc. Similarly, national statistical institutions collect information on the R&D activities in their respective countries.

We examine the reliability of innovation measures by comparing the responses company representatives have given to the Community Innovation Surveys and national R&D surveys starting from the year 2000. We examine the stability and relatedness of the answers over time as well as their correlation to the company characteristics, in particular to the share of R&D and ICT personnel in the company.

We examine four general classes of reliability estimates, each of which estimates reliability in a different way, that is, Inter-rater reliability assesses the degree to which different raters/observers give consistent estimates of the same phenomenon; test-retest reliability assesses the consistency of a measure from one time to another; parallel-forms reliability assessing the consistency of the results of two tests constructed in the same way from the same content domain; and internal consistency reliability assesses the consistency of results across items within a test.

We wish that our study will improve the applicability of the innovation data by noting the survey data that seem to be reliable and the ones where caution might be needed.

Carter Bloch, University of Aarhus has been involved as discussant.