An Experimentalist Approach to Software Testing

Most builders of research software believe they should write tests for their tools, yet many struggle to do so efficiently and effectively. Formal strategies like Test-Driven Design may seem intimidating or inaccessible, while developers relying on an ad-hoc approach can suffer from “blank page syndrome” and struggle to provide efficient, complete coverage of essential functionality. Sometimes this leads to tests which actually spend more time testing mathematical properties or theories rather than the implementation, wasting both compute time and developer time; in other cases, developers fall back to manual tests, leading to developer anxiety and questionable software reliability.

In this workshop, we introduce a scientist-friendly perspective on software tests: tests are controlled experiments run on the implementation code. We will demonstrate a minimal setup for automated testing and the features to expect from a good testing system, and cover examples of common failures in test design. Participants will have the opportunity to apply this approach by revising practical examples—both of flawed tests, and of implementation code that creates friction when writing tests—before we reconvene as a full group to discuss solutions.

A computer will be needed for full participation, though attendees without one will still benefit from the discussion. Exercises will be provided in Python and C++.