AI-Powered Authorization Testing for Microservices: Replication Package

Overview

This replication package contains all source code, tools, experimental data, and instructions to reproduce our research on LLM-powered authorization testing in microservice architectures.

Key Results:

• 80-100% AuthZ scenario coverage

• 90-97.4% runnable AuthZ verifications (tests that execute without errors)

• 100% semantically valid authorization policy tests

• 100% precise natural language explanations

• Test generation in under 20 seconds per scenario

• 95.9% static analysis resolution rate for remote call sites

The methodology consists of four main phases:

1. STATIC ANALYSIS (CIMET): Extract API endpoints, authorization policies, and control flow graphs from microservice source code

2. AUTH VECTOR GENERATION: Analyze call chains and generate role-based permission matrices

3. SCENARIO GENERATION: Create LLM-ready test scenarios based on authorization patterns and policy inconsistencies

4. TEST GENERATION & EXECUTION: Use LLMs to generate executable test cases and run them against live services

Package Contents

Prerequisites

For CIMET Indexer:

• Java 24+ (recommended)

• Maven 3.6+

• Git

For AI Test Scripts:

• Python 3.9+

• Node.js 18+ and npm

For LLM Integration:

• OpenAI API Key (for GPT-5.1-Codex-Mini, as used in the paper) OR Anthropic API Key (for Claude)

For Test Execution:

• Running instance of the target microservice system

Replication Steps (Summary)

Step 1: Run CIMET Indexer

mvn exec:java -Dexec.mainClass="edu.university.ecs.lab.intermediate.index.IndexerRunner"

Output: endpoints.json, components.json

Step 2: Generate Auth Vectors

./01_run_auth_vector_generator.sh

Output: output/auth_vectors/all_vectors.json

Step 3: Generate Test Scenarios

./02_run_scenario_generator.sh

Output: output/llm-scenarios/scenarios_llm_ready.json

Step 4: Generate and Execute Tests

./03_run_auth_test_generator.sh run --scenario-id "scn_001" --provider openai

Output: generated-tests/, execution-results/, cost-reports/

See README.md for detailed step-by-step instructions.

Directory Structure

Pre-indexed Data (microservice-index)

Each benchmark version contains the complete pipeline output:

microservice-index/
├── train-ticket-aitest/
│   ├── master/                      # Baseline version
│   │   ├── components.json          # CIMET: Component metadata
│   │   ├── endpoints.json           # CIMET: REST APIs with auth
│   │   ├── all_vectors.json         # Permission matrices
│   │   ├── scenarios_llm_ready.json # LLM-ready scenarios
│   │   └── output/
│   │       ├── generated-tests/     # LLM-generated test specs
│   │       ├── execution-results/   # Test execution results
│   │       ├── cost-reports/        # LLM API costs
│   │       ├── curl-commands/       # Debug curl scripts
│   │       └── llm-debug/           # Full prompts/responses
│   ├── first-inconsistency-injection/
│   └── second-inconsistency-injection/
└── yas/
    ├── main/
    ├── first-injection/
    └── second-injection/

Prompt Templates

LLM prompt templates are included in the source code:

Template Types:

• entrypoint_status_matrix - Validates status codes per role

• entrypoint_public_sensitive - Tests sensitive endpoints exposed as PUBLIC

• downstream_consistent_path - Verifies consistent access along call chain

• downstream_inconsistency_cot - Chain-of-Thought reasoning for policy divergence

To view prompts without API calls:

./03_run_auth_test_generator.sh generate --scenario-id "scn_001" --dry-run

Authorization Configurations

Role Priority (config.json)

{
  "rolePriority": {
    "SUPER-ADMIN": 1,
    "ADMIN": 2,
    "CUSTOMER": 10
  },
  "defaultRolePriority": 50
}

Lower number = higher privilege

Test Credentials (.env)

BASE_URL=http://localhost:8888
AUTH_ENDPOINT=http://localhost:8888/api/v1/users/login
AUTH_ADMIN_USERNAME=admin
AUTH_ADMIN_PASSWORD=222222
AUTH_USER_USERNAME=fdse_microservice
AUTH_USER_PASSWORD=111111

Replication Runs and Statistical Analysis

Due to LLM non-determinism, proper replication requires multiple runs.

Recommended Protocol:

• Minimum 10 independent runs per configuration

• LLM temperature set to 0.4 (use 0.0 for more determinism)

• Separate output directories per run

Running Multiple Replications:

for run in {1..10}; do
  mkdir -p output/run_$run
  ./03_run_auth_test_generator.sh batch-run --provider openai --output-dir output/run_$run
done

Statistical Metrics:

• Detection Rate: Mean, Std Dev, 95% CI

• False Positive Rate: Mean, Std Dev

• Test Pass Rate: Mean, Std Dev

• LLM Cost: Mean, Range

Subject Systems

Train-Ticket Benchmark (20-service version):

• master - Baseline with correct authorization policies (0% modified)

• first-inconsistency-injection - 60% random policy modified

• second-inconsistency-injection - 80% random policy modified

• Statistics: 259 paths, 23.5% with downstream interactions, max depth 17

YAS Benchmark (Yet-Another-Shop):

• main - Baseline version (0% modified)

• first-injection - 60% random policy modified

• second-injection - 80% random policy modified

• Statistics: 194 paths, 37.1% with remote calls, max depth 6

Key Output Files

Contact

For questions about this replication package, please refer to the associated paper or contact the authors.