Simulation for Robotics Test Automation: Developer Perspectives - Survey Material

Robotics simulation plays an important role in the design, development, and verification and validation of robotics systems. Simulation represents a potentially cheaper, safer, and more reliable alternative to the widely used practice of manual field testing, and introduces valuable opportunities for extensive test automation. The goal of this paper is to develop a principled understanding of the ways robotics developers use simulation in their testing processes and the challenges they face in doing so. This understanding can guide the improvement of simulators and testing techniques for modern robotics development.

To that end, we conduct a survey of 82 robotics developers from a diversity of backgrounds, addressing the current capabilities and limits of simulation in practice. We find that simulation is used by 84% of our participants for testing, and that many participants want to use simulation as part of their test automation. Using qualitative and quantitative research methods, we identify 10 high-level challenges that impede developers from using simulation for manual and automated testing and in general. These challenges include the gap between simulation and reality, a lack of reproducibility, and considerable resource costs associated with simulation. Finally, we outline ways in which simulators can be improved for use as a means of verification and validation and ways that the software engineering community can contribute to these improvements.