AccessDroid: Detecting Screen Reader Accessibility Issues in Android Applications via Semantics Trees

# AccessDroid

> AccessDroid is a lightweight dynamic analyzer for detecting screen reader accessibility issues in Android app pages. It runs on Android devices or emulators, generates accessibility reports, and visualizes semantics trees to help developers debug apps from the perspective of screen reader users.

## Contents
- [Features](#features)
- [Environment Requirements](#environment-requirements)
- [Compilation and Installation](#compilation-and-installation)
- [Usages](#usages)
- [Output Format](#output-format)
- [A Quick Example](#a-quick-example)
- [Trouble Shooting](#trouble-shooting)
- [PoseCheck](#posecheck)
- [Comparison Tools](#comparison-tools)
  - [Accessibility Scanner](#accessibility-scanner)
  - [LLM-based Approach](#llm-based-approach)

## Features
- Detects screen reader accessibility issues in runtime app pages.
- Generates detailed reports, UI trees, and semantics trees of an app page.
- Visualizes semantics trees on the device in real time.
- Supports simulated touch interactions (tap, swipe) for automated testing.
- Cross-platform support on Windows, macOS, and Linux.
- Simple command workflow through the provided `ad` script.

## Environment Requirements
AccessDroid is a lightweight tool designed for Android developers. If you already have a standard Android development environment (Android Studio, ADB, and an Android emulator), AccessDroid can be set up and run in a stable manner with minimal effort.

In practice, installing Android Studio alone is sufficient, as it can automatically configure most required Android development components. Therefore, Docker is not required for running AccessDroid.

AccessDroid has been tested on macOS and Windows.

> **Note**: The estimated setup time below is for reference only and may vary depending on your network environment.

1. **Android Studio** (For compilation and android debug bridge) [~**10** minutes]
   - Used to build the AccessDroid APK from source (a prebuilt APK is also provided in the `<artifact_path>/AccessDroid/AccessDroid.apk`).
   - Official download and installation instructions are available at [this](https://developer.android.com/studio).
2. **Android Emulator** or Android device [~**10** minutes]
   - Android Studio offers built-in tools for downloading and managing emulator images. 
   - Running AccessDroid on a physical Android device is also supported, but requires:
     - USB connection to the host machine
     - Developer Mode and USB debugging enabled on the device
   - AccessDroid supports Android 12 and higher versions. (An Android 12 image without Google Play is recommended for a clean system environment.)
3. **ADB** (Android Debug Bridge) [~**2** minutes]
   - ADB is installed automatically as part of the Android SDK when using Android Studio.
   - Ensure that `adb` is available in your system PATH. (e.g. `export PATH="$PATH:$ANDROID_HOME/platform-tools"`)
   - Run `adb help` in the terminal to verify that ADB has been installed successfully.
4. **A shell environment** is required to run the `ad` script. We provide two versions of the `ad` script: one written in `Zsh` syntax and the other in `PowerShell` syntax (`ad.ps1`). [~**2** minutes]
   - For macOS: [Zsh](https://github.com/ohmyzsh/ohmyzsh/wiki/Installing-ZSH) is the default shell.
   - For Linux: `Zsh` can be installed easily via package managers. (e.g. `apt install zsh; chsh -s $(which zsh)`)
   - For Windows: `Powershell` is the default shell and fully supported.
5. Hardware Requirements
   - AccessDroid is designed with a lightweight architecture and has minimal resource consumption.
   - The only practical requirement is that the host machine can run an Android Emulator smoothly.
   - AccessDroid itself has no special hardware dependencies and is suitable for a wide range of development and testing environments.

## Compilation and Installation
> This section describes how to build and install AccessDroid. You may either compile the project from source or use the prebuilt APK provided in the artifacts.

1. Download the source code (`<artifact_path>/AccessDroid`).
2. Build the APK (Optional)
   - Open the project in Android Studio and build the project.
   - A **precompiled APK** is also available at `<artifact_path>/AccessDroid/AccessDroid.apk`. This APK can be used directly without rebuilding the project.
3. Install APKs on the emulator/device
   - Install AccessDroid.apk and the target application you want to analyze on the Android emulator or physical device.
     - Both APKs can be installed by dragging and dropping them into the emulator window.
   - For convenience, we provide 50 sample target applications in `<artifact_path>/PoseCheck`
4. Add tools to PATH
   - Ensure that both `adb` and the `ad` script are accessible from your terminal.
     - For macOS/Linux: `export PATH="$PATH:$ANDROID_HOME/platform-tools:<artifact_path>/AccessDroid/scripts`.
     - For Windows: Find platform-tools folder in Android SDK and `<artifact_path>/AccessDroid/scripts` to the system environment variables. [Help](https://www.solveyourtech.com/how-to-set-environment-variable-in-windows-11-a-step-by-step-guide/)
   - The `ad` script does not require any modification before execution and can be run directly once it is available in PATH.

## Usages

All interactions with AccessDroid are performed via the `ad` script, which wraps ADB commands for convenience.

| Command        |  Description                                                                                                                                   |
|----------------|------------------------------------------------------------------------------------------------------------------------------------------------|
| ad enable      | Grants Accessibility permission for AccessDroid. When successful, the main interface shows service on.                                         |
| ad disable     | Revokes Accessibility permission and stops AccessDroid service.                                                                                |
| ad scan        | (For Jetpack Compose Apps) Analyzes the currently displayed page on the device. Red bounding boxes highlight screen reader focusable elements. |
| ad vscan       | (For Other Apps) Analyzes the currently displayed page on the device. Red bounding boxes highlight screen reader focusable elements.           |
| ad pull        | Retrieves the latest accessibility report, including the UI tree and semantics tree, to the current directory.                                 |
| ad tap x y     | Simulates a tap at screen coordinates (x, y).                                                                                                  |
| ad swipe_right | Simulates a swipe gesture to the right.                                                                                                        |
| ad clear       | Removes red bounding boxes from the screen.                                                                                                    |
| ad debug       | Outputs the UI tree and semantics tree to the file.                                                                                            |
| ad screen      | Save the current screenshot                                                                                                                    |

**Notes**
- Coordinate-based commands such as ad tap depend on the current screen resolution and orientation.
- Output files generated by `ad pull` and `ad debug` are saved to the current working directory.

## Output Format
> AccessDroid generates both visual annotations on the device screen and text-based analysis results that can be inspected locally.

**Visual Output**

- After executing `ad scan`, AccessDroid highlights detected semantics nodes on the device screen using red bounding boxes.
- For each analysis run, a corresponding annotated screenshot is saved at: `<artifact_path>/Results/<app_name>/<id>.png`.
- Red bounding boxes are visual overlays only; they do not interfere with app interaction.

**Exported Files**

- The `ad pull` command exports the following files to the local machine:
  - Accessibility report (`<artifact_path>/Results/<app_name>/<id>_issues.txt`).
  - UI Tree (`<artifact_path>/Results/<app_name>/<id>.txt`).
  - Semantics Tree (`<artifact_path>/Results/<app_name>/<id>_semanticsTree.txt`).

All exported files are plain text and can be opened with any standard text editor.

**Report Example**

Below is an example snippet from the accessibility report:

```txt
[OnClick Label Omission]: (8)View(View)
[Semantics Separation]:
    (15)TextView(TextView);AccessDroid
    (14)View(Button)
```

**Report Interpretation**

- The content inside square brackets indicates the type of accessibility issue detected.
- Each issue is followed by one or more affected node entries, which include:
  - A node ID (e.g., 8, 15, 14).
  - The node type (e.g., View, TextView, Button).
  - Optional semantic information such as text content
- Node IDs are internally assigned by AccessDroid and are not related to Android system IDs.
- These node IDs are consistent across all outputs, including: reports, UI tree, semantics tree, and red bounding boxes displayed on the screen. This consistency allows users to easily correlate reported issues with their corresponding visual elements on the device.

## A Quick Example
> The following example demonstrates a typical end-to-end workflow for analyzing an Android application with AccessDroid.
> [A Video Demonstration](https://www.youtube.com/watch?v=tDU0Cs4yfLY)

1. Install AccessDroid.apk on the emulator, along with the target application to be analyzed.
2. Grant Accessibility permissions:
    ```shell
    ad enable
    ```
3. Launch the target application and navigate to the specific screen you want to analyze.
4. Run accessibility analysis.
   - For Jetpack Compose–based applications:
       ```shell
       ad scan
       ```
   - For applications built with other UI frameworks
       ```shell
       ad vscan
       ```
5. (Optional) Dump UI trees and semantics trees
    ```shell
   ad debug
    ```
6. Pull the analysis reports, the UI tree, and the semantics tree. All files will be pulled to the current working directory.
    ```shell
    ad pull
    ```
7. To analyze another page, navigate to the desired screen in the app and rerun the appropriate scan command.
8. Open the report in a text editor to inspect accessibility issues.
9. Remove the red bounding boxes displayed on the screen.
    ```shell
    ad clear
    ```
10. When analysis is complete, it is recommended to stop the Accessibility service `ad disable`.
    ```shell
    ad disable
    ```

## Trouble Shooting
- Choosing between `ad scan` and `ad vscan`: For apps other than PoseCheck, use `ad scan` for Jetpack Compose–based applications and `ad vscan` for apps built with other UI frameworks (e.g., system apps); if you are unsure which framework an app uses, simply try both, successful execution is indicated by red bounding boxes appearing on the screen.
- No red bounding boxes appear after running a scan.
  - This situation should not normally occur. Open the AccessDroid app and check the main page. If it indicates Service Off, the AccessDroid service was not enabled correctly. 
  - Ensure that adb is correctly added to the system PATH.
  - Try both `ad scan` and `ad vscan`.
- ADB cannot find a connected device.
  - Ensure that only one Android emulator or physical device is connected to the host machine. Running `adb devices` should list exactly one device.

## PoseCheck
- We collect 50 real-world applications with active maintenance as our benchmark from sources such as F-Droid and GitHub.
- The list of applications included in the benchmark can be found in `<artifact_path>/PoseCheck`. This directory contains a `PoseCheck.xlsx` file, which records the name, version, and source of each of the 50 applications.
- We manually extracted 361 distinct pages from these 50 applications. For ease of inspection, screenshots of the 361 pages are organized under `<artifact_path>/Results/<app_name>/<id>.png`. Note that AccessDroid does not rely on screenshot information; the screenshots are provided solely for debugging purposes.
- The ground truth is stored at location `<artifact_path>/Results/<app_name>/<id>_issues_label.txt`.

## Comparison Tools

### Accessibility Scanner

- We integrate the Scanner algorithm into AccessDroid, using the [Accessibility Testing Framework for Android](https://github.com/google/Accessibility-Test-Framework-for-Android).
- To run the scanner, execute the following command:
    ```shell
    ad scanner
    ```
- This command performs an accessibility scan by Accessibility Scanner on the current page and generates the results on the device.
- After the scan is completed, the results can be exported to the host machine using:
    ```shell
    ad pull
    ```
- The exported result file is saved as `<artifact_path>/Results/<app_name>/<id>_scannerResults.txt`.
- Alternatively, we also provide a standalone Accessibility Scanner APK in artifacts (`<artifact_path>/AccessDroid/AccessibilityScanner.apk`). Running the APK directly on the device produces results that are consistent with those generated by the `ad scanner` command.

### LLM-based Approach

- For each page under evaluation, the prompt used by the LLM is stored in artifacts (`<artifact_path>/Results/<app_name>/<id>_prompt.txt`).
- During evaluation, the page information and the corresponding prompt are provided as input to the LLM.
- We conduct our experiments using the [DeepSeek-R1](https://www.deepseek.com) model.
- The LLM output results can be found at `<artifact_path>/Results/LLM_output.txt`.