SetDroid: Detecting User-configurable Setting Issues of Android Apps via Metamorphic Fuzzing

Android, the most popular mobile system, offers a number of user-configurable system settings (e.g., network, location, and permission) for controlling devices and apps. Even popular, well-tested apps may fail to properly adapt their behaviors to diverse setting changes, thus frustrating their users. However, there exists no effort to systematically investigate such defects. To this end, we conduct the first empirical study to understand the characteristics of these setting-related defects (in short as "setting defects"), which reside in apps and are triggered by system setting changes. We devote substantial manual effort (over three person-months) to analyze 1,074 setting defects from 180 popular apps on GitHub. We investigate their impact, root causes, and consequences. We find that setting defects have a wide, diverse impact on apps' correctness, and the majority of them (=70.7%) are non-crash (logic) bugs, thus could not be automatically detected by existing app testing techniques due to the lack of test oracles.

Motivated and guided by our study, we introduce setting-wise metamorphic fuzzing, the first automated testing approach to effectively detecting setting defects without explicit oracles. Our key insight is that an app's behavior should, in most cases, remain consistent if a given setting is changed and later properly restored, or exhibit expected differences if not restored. We realize our approach in SetDroid, an automated, end-to-end GUI testing tool, for detecting both crash and non-crash setting defects. SetDroid has been evaluated on 26 popular, open-source apps and detected 42 unique, previously unknown setting defects in 24 apps. To date, 33 have been confirmed and 21 fixed. We also apply SetDroid on five highly popular industrial apps, namely WeChat, QQMail, TikTok, CapCut, and AlipayHK, all of which have billions of monthly active users. SetDroid successfully detects 17 previously unknown setting defects in these apps' latest releases --- all defects have been confirmed and fixed by the app vendors. The majority of SetDroid-detected defects (49 out of 59) are non-crash bugs, which could not be detected by existing testing tools (as our evaluation confirms). These results clearly demonstrate SetDroid's strong effectiveness and practicality.

To facilitate reviewing and replication of our work, we have open-sourced SetDroid and provided all the datasets (fully anonymized) at https://github.com/setting-defect-fuzzing/home.