Siloso: Finding Logic Bugs in RDBMS via Dialect-Adaptable Reference Engine Construction

Relational DBMSs (RDBMSs) are ubiquitous, so any bugs or inconsistencies within RDBMSs are highly consequential. Particularly, logic bugs, which can cause an incorrect result to be returned for a given query evaluation, are critical because they are likely to be unnoticed by users. In our work, we conducted a motivating study to investigate important features in bug-inducing test cases and real-world SQL queries, and how they differ across RDBMSs. Based on this, we conceptualized key correctness variabilities across a diverse set of RDBMSs and proposed an approach for detecting logic bugs and inconsistencies in RDBMSs via the design and implementation of an extensible reference engine, which we term Siloso, that can adapt the behavior of a query execution depending on the specified dialect. To reduce the implementation effort needed to adopt Siloso for a new RDBMS, Siloso delegates the evaluation of unimplemented functions and operators to a concrete RDBMS. Moreover, Siloso models ambiguous query execution results at each step of the query evaluation. We evaluated Siloso extensively as a differential test oracle, finding 27 bugs across six RDBMSs, and demonstrated several classes of bugs detected by Siloso that are overlooked by existing state-of-the-art approaches.