A Study on the Quality of LLM-Generated Authorization Tests for Microservices

Overview

This replication package accompanies the paper "AI-Powered Authorization Testing for Microservices" submitted to AITest 2025. It provides all artifacts required to fully reproduce the experimental evaluation: source code, pre-indexed microservice data, experimental results, and step-by-step execution instructions.

The research addresses the authorization blindspot in microservice testing — the inability of traditional automated tools to detect policy inconsistencies between upstream and downstream services. The framework uses policy-enriched intermediate representations to guide LLM test generation, achieving 100% semantic validity and 92–100% authorization verification, significantly outperforming EvoMaster and EvoSuite baselines.

Package Contents

Directory Structure

cimet-extract-lib-extended/

The CIMET static analysis tool. Run via Maven to index a target microservice repository.

cimet-extract-lib-extended/
├── config.json                  # Target system configuration (repo URL, branch, role hierarchy)
├── pom.xml                      # Maven build file
├── src/main/java/.../
│   ├── common/                  # Shared utilities and models
│   ├── delta/                   # Change impact analysis
│   └── intermediate/
│       ├── create/              # IR extraction logic
│       └── index/               # Endpoint and component indexing (IndexerRunner entry point)
└── output/
    ├── endpoints.json           # All REST endpoints with authorization metadata
    └── components.json          # All components with control flow graphs and remote call resolution

aitest-scripts/

The main pipeline. Three shell scripts drive the full workflow sequentially.

aitest-scripts/
├── 01_run_auth_vector_generator.sh   # Step 1: Generate role-based permission matrices
├── 02_run_scenario_generator.sh      # Step 2: Generate LLM-ready test scenarios
├── 03_run_auth_test_generator.sh     # Step 3: Generate and execute tests via LLM
├── requirements.txt                  # Python dependencies
├── .env.example                      # Environment template (API keys, base URL, credentials)
├── input/
│   ├── components.json               # Copied from CIMET output
│   └── endpoints.json                # Copied from CIMET output
├── output/
│   ├── auth_vectors/
│   │   └── all_vectors.json          # Role-permission matrices per endpoint
│   └── llm-scenarios/
│       └── scenarios_llm_ready.json  # Structured scenarios ready for LLM prompting
├── auth-role-vector-generator/       # Module: call chain analysis and matrix generation
├── test-scenario-generator/
│   ├── scenario_generator.py         # Generates scenarios from auth vectors
│   └── prompt_generator.py           # Builds LLM prompt context per scenario
└── auth-test-generator/
    ├── src/index.ts                  # CLI entry point
    ├── src/generator/                # LLM test generation logic
    ├── src/executor/                 # Test execution against live services
    ├── src/llm/                      # LLM provider integration (OpenAI / Anthropic)
    └── output/
        ├── generated-tests/          # LLM-generated HTTP request specifications
        ├── execution-results/        # Actual execution results (status, body, pass/fail)
        ├── cost-reports/             # Token usage and cost per scenario
        ├── curl-commands/            # Equivalent curl scripts for manual debugging
        └── llm-debug/               # Full LLM prompt and response per scenario

microservice-index/

Pre-computed pipeline output. Use this to inspect results without re-running the full pipeline.

microservice-index/
├── train-ticket-aitest/
│   ├── master/                              # Baseline (correct policies)
│   │   ├── components.json                  # CIMET component metadata
│   │   ├── endpoints.json                   # CIMET endpoint index
│   │   ├── all_vectors.json                 # Role permission matrices
│   │   ├── scenarios_llm_ready.json         # LLM-ready test scenarios
│   │   └── output/
│   │       ├── generated-tests/             # scn_<service>_<id>.json per scenario
│   │       ├── execution-results/           # scn_<service>_<id>.json per scenario
│   │       ├── cost-reports/                # Token usage and cost per scenario
│   │       ├── curl-commands/               # scn_<service>_<id>.sh per scenario
│   │       ├── llm-debug/                   # Full prompt/response debug per scenario
│   │       └── metrics-summary.json         # Aggregated pass rate, detection rate, costs
│   ├── first-inconsistency-injection/       # Mutation #1 (same structure as master)
│   └── second-inconsistency-injection/      # Mutation #2 (same structure as master)
└── yas/
    ├── main/                                # Baseline (same structure as train-ticket master)
    ├── first-injection/                     # Mutation #1
    └── second-injection/                    # Mutation #2

Methodology

The framework is a four-phase pipeline from microservice source code to executable authorization tests:

1. Static Analysis (CIMET): Extracts REST endpoints, authorization metadata, and control flow graphs from Java Spring Boot microservices.
2. Auth Vector Generation: Computes role-based permission matrices across service call chains, surfacing cross-service authorization inconsistencies.
3. Scenario Generation: Applies four prompt template strategies to produce LLM-ready test scenarios with role expectations, sensitivity classification, and data awareness requirements.
4. LLM Test Generation and Execution: Submits enriched scenarios to GPT-4 or Claude Sonnet to generate executable HTTP tests, then runs them against a live microservice deployment.

Subject Systems

Prerequisites

• CIMET Indexer: Java 24+, Maven 3.6+
• Pipeline Scripts: Python 3.9+, Node.js 18+
• LLM Access: OpenAI API key (GPT-4) or Anthropic API key (Claude Sonnet)
• Live Service: Running instance of the target microservice system (e.g., Train-Ticket on localhost:8888)

Researchers who wish to inspect results without re-running the full pipeline can use the pre-computed data in microservice-index.zip.

For detailed step-by-step instructions on running each phase of the pipeline — including build commands, configuration parameters, available CLI options, and troubleshooting — refer to the included README.md file.

Key Findings

A notable finding is the 401 vs. 403 distinction: LLMs correctly determine the authorization outcome (allow/deny) but exact status code matching is 32–42% due to implementation-specific server behaviour. Range-based matching for the authorization decision reaches 86–92%.

License

Provided for research and reproducibility purposes. Benchmark systems (Train-Ticket, YAS) retain their original open-source licenses. See individual component licenses for specific terms.