DialTest: Automated Testing for Recurrent-Neural-Network-Driven Dialogue Systems

We design and implement DialTest, a RNN-driven dialogue system testing tool. DialTest employs a series of transformation operators to make realistic changes on seed data while preserving their oracle information properly. To improve the efficiency of detecting faults, DialTest further adopts Gini impurity to guide the test generation process.

To validate DialTest, we experiment it on two fundamental tasks, i.e., intent detection and slot filling, of natural language understanding. The experiment results show that DialTest can effectively detect hundreds of erroneous behaviors for different RNN-driven natural language understanding (NLU) module of dialogue systems and improve their accuracy via retraining with the generated data.