Replication package for the FRAME paper (ICSME 2026). FRAME turns a mobile UI screenshot into a single embedding by fusing three signals:

• Visual — CLIP ViT-B/32 on the full screen and on each detected UI element.
• Textual — on-screen text via Tesseract OCR, embedded with BERT (bert-base-uncased).
• Structural — a NetworkX graph of UI elements (edges by on-screen distance) with embedding propagation over 1-/2-hop neighborhoods.

The fused embedding is evaluated on screen retrieval (MRR / Hits@K).

config.py                 # ← single place to set all paths & parameters
utils.py                  # shared helpers
UI_Embedding_main.py      # core pipeline; makeEmbedding() + graph propagation
UIEDComp.py               # UIED element detection wrapper
imageEmbedding.py         # CLIP embeddings (full-screen + per-element)
textEmbedding.py          # Tesseract OCR -> BERT text embeddings
graphCreation.py          # build element graph, extract edges
embeddingConsolidation.py # Rips-complex / centroid consolidation
makeCSV.py                # batch: dataset index -> per-screen embeddings CSV
embeddingsimilarity.py    # retrieval evaluation (MRR / Hits@K)
CaseStudyCreate.py        # aggregate metrics by screen type
t_test.py                 # paired significance testing
DimensionReduction.py     # PCA / LLE experiments
pca_transformation_matrix.npy  # precomputed 140x1792 PCA matrix
rips_*.csv                # precomputed embedding dumps (see DATA.md)
screenshots/              # 3 sample screenshots for the smoke test
detectors/Visual/UIED-master/  # vendored UIED detector (third-party, own LICENSE)
run_all.sh                # end-to-end smoke test on the bundled screenshots
Dockerfile                # CPU-only, self-contained reproducible image

See DATA.md for the embedding dumps' schema and dataset details.

Fully self-contained; no manual dependency setup. Requires only Docker.

docker build -t frame .
docker run --rm frame          # runs the smoke test (run_all.sh)
docker run --rm -it frame bash # interactive shell to run individual scripts

The image bundles the code, data, and pre-downloaded CLIP + BERT weights, so it runs offline. It targets linux/amd64 (some pinned dependencies have no arm64 wheels for Python 3.9); on Apple Silicon it builds and runs under emulation.

• Python 3.9.13 (pip 23.3.2)
• Tesseract OCR system binary — required by pytesseract:
  • macOS: brew install tesseract
  • Debian/Ubuntu: sudo apt-get install tesseract-ocr
• git — the CLIP dependency is installed from source (see requirements.txt).
• Network access on first run: clip.load("ViT-B/32") and bert-base-uncased download model weights (~1 GB total) and cache them.

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

All paths and parameters live in config.py — edit it once (or override any value via an environment variable of the same name, e.g. EMBEDDINGS_CSV=/data/x.csv python3 embeddingsimilarity.py). Defaults point at the bundled sample data so the smoke test runs with zero edits.

./run_all.sh

Embeds each screenshot in screenshots/ end-to-end and prints the embedding dimensionality. This does notrequire the labeled dataset.

The retrieval results run as a pipeline. Important: the committed rips_*.csv dumps reference the original dataset screenshots by absolute path, and the evaluation re-opens those images — so you must supply the dataset (or regenerate the dumps). See the caveat in DATA.md.

1. Generate embeddings from a dataset index CSV (config.EMBEDDING_INDEX_CSV → config.EMBEDDINGS_OUTPUT_CSV):

   python3 makeCSV.py

    

   (Or skip this step and use a bundled rips_*.csv dump directly.)

2. Run retrieval evaluation (config.EMBEDDINGS_CSV → config.RESULTS_OUTPUT_CSV); prints MRR and Hits@{1,5,10}:

   python3 embeddingsimilarity.py

    

3. Aggregate metrics by screen type (config.METRICS_INPUT_CSV → config.METRICS_OUTPUT_CSV):

   python3 CaseStudyCreate.py

    

4. Significance testing between two result CSVs (paired t-test over MRR / Hits@K):

   python3 t_test.py

    

The four precomputed embedding dumps (RICO + Avgust) and their schema are documented in DATA.md. RICO: https://interactionmining.org/rico.

• Code: MIT (see LICENSE). The vendored UIED detector under detectors/Visual/UIED-master/ keeps its own third-party license.
• Please cite the paper — see CITATION.cff.