Trade-offs between reliability, validity and utility in the development of human factors methods

We describe our experiences in developing human factors methods for a variety of purposes over the last couple of decades. In particular, we focus on recent discussions centred on the Fitts List (e.g., de Winter and Dodou, 2011) and accident investigation methods (e.g., the degree to which they are practice-focused, theoretically robust, valid and reliable). We draw on earlier work by Karl Weick (1979) concerning the trade-offs involved in developing theory within the social sciences and apply these to method development within human factors and ergonomics. The final section of the paper outline a set of issues for future work (e.g., satisficing trade-offs, using methods in combination, ways of improving the research-practice feedback loops).