Grammar Mutation for Testing Input Parsers (Registered Report)

Grammar-based fuzzing is an effective method for testing programs that consume structured inputs, and for testing input parsers in particular. A prerequisite of this method is to have a specification of the input format in the form of a grammar. Consequently, the success of a grammar-based fuzzing campaign is highly dependent on the available grammar. If the grammar does not accurately represent the input format, or if the system under test (SUT) does not conform strictly to the grammar for a standard input format, there may be an impedance mismatch between inputs generated via grammar-based fuzzing and inputs accepted by the SUT. Even if the SUT has been designed to strictly conform to the grammar, the SUT parser may exhibit vulnerabilities that would only be triggered by slightly invalid inputs. Grammar-based fuzzing by construction will not yield such edge case inputs. In an effort to identify and overcome these limitations of grammar-based fuzzing, we present a new tool, Gmutator, that performs mutations on an input grammar and leverages the Grammarinator fuzzer to produce inputs that (a) do not conform to the original grammar but are accepted by an SUT, and (b) achieve higher SUT code coverage compared with the standard approach. We present preliminary results applying this technique to two JSON parsing libraries, where we are able to identify a few such inconsistencies and observe an increase in covered code. We present a plan for a full experimental evaluation over four different input formats---JSON, XML, URL and Lua---and eight SUTs (two per input format).