A Practical Method for API Testing in the Context of Continuous Delivery and Behavior Driven Development

Abstract—Enterprises are increasingly adopting an API-first approach to connect and expose software services. Saxo Bank is no exception to this. 

Crafting test suites for such APIs can seem straight forward due to the headless nature, but our experience shows that test suites often have two problems. The first problem is that execution of tests tends to fail and pass in seemingly nondeterministic ways (tests are flaky). The second problem is that functional coverage is not clearly documented.

We have found that both problems stem from a lack of explicit focus on initial context (IC), a concept from behavior driven development. When a test is flaky it is often because actual IC in the test environment is not as required by the test. When functional coverage is not clear, it is most often because a systematic analysis involving IC was not performed.

We propose a method for test analysis in which we include IC in the input space when analyzing functional coverage for an API, thereby including anything which can influence the outcome of test cases.

Establishing IC is in general a hard problem. We have found that focus on the bounded context, a concept from domain driven design, of the system under test is a practical way to establish relevant IC.

Experience with Saxo Bank's Open API shows that this method allows testers and developers to cooperate continuously, producing test plan documents which include the reasoning behind functional coverage. Explicit focus on IC in automated test case implementations turns flaky tests into tests which report on required IC in a test environment. The method easily generalizes to all levels of API tests.