Large Language Models vs. Stack Overflow in Addressing Android Permission Related Challenges

The Android permission system regulates access to sensitive mobile
device resources and data, such as camera and location. To access
these resources, third-party application (app) developers need to
request permissions. However, the Android permission system is
complex and fast-evolving, presenting developers with numerous
challenges surrounding compatibility issues, misuse of permissions,
and vulnerabilities related to permissions. Our study aims to
explore whether Large Language Models (LLMs) can serve as a
reliable tool to assist developers in using Android permissions
correctly and securely, thereby reducing the risks of misuse and
security vulnerabilities in apps. In our study, we analyzed 1,008
Stack Overflow questions related to Android permissions and
their accepted answers. In parallel, we generate answers to these
questions using a popular LLM tool, ChatGPT. In our evaluation,
we focus on how well the AI-generated responses align with
the accepted answers provided by developers on Stack Overflow.
Our findings show that above 50% of ChatGPT-provided answers
align with Stack Overflow's accepted answers. ChatGPT offers
better-aligned responses for challenges related to Documentation
and Conceptual Understanding, while it provides less aligned
answers for Debugging-related issues. In addition, we found that
ChatGPT provides more consistent answers for Documentation and
Conceptual Understanding, while Implementation-related answers
are the least consistent. Our study demonstrates the potential for
using LLMs such as ChatGPT as a supporting tool to help developers
navigate Android permission-related problems.