Exposing Bugs in JavaScript Engines through Test Transplantation and Differential Testing

Abstract Context. JavaScript is a popular programming language today
with several implementations competing for market dominance. Although a
specification document and a conformance test suite exist to guide engine
development, bugs occur and have important practical consequences. Imple-
menting correct engines is challenging because the spec is intentionally incom-
plete and evolves frequently. Objective. This paper investigates the use of
test transplantation and differential testing for revealing functional bugs in
JavaScript engines. The former technique runs the regression test suite of a
given engine on another engine. The latter technique fuzzes existing inputs
and then compares the output produced by different engines with a differen-
tial oracle. Method. We conducted experiments with engines from five major
players–Apple, Facebook, Google, Microsoft, and Mozilla–to assess the effec-
tiveness of test transplantation and differential testing. Results. Our results
indicate that both techniques revealed several bugs, many of which confirmed
by developers. We reported 35 bugs with test transplantation (23 of these bugs
confirmed and 19 fixed) and reported 24 bugs with differential testing (17 of
these confirmed and 10 fixed). Results indicate that most of these bugs af-
fected two engines–Apple’s JSC and Microsoft’s ChakraCore (24 and 26 bugs,
respectively). To summarize, our results show that test transplantation and
differential testing are easy to apply and very effective in finding bugs in com-
plex software, such as JavaScript engines.