# AutoSpec

AutoSpec automatically refines safety rules for LLM agents through counterexample-guided
inductive synthesis (CEGIS) guided by inductive logic programming (ILP). This is the
replication package accompanying the AutoSpec paper.

## What's included

This repository contains everything needed to use and reproduce AutoSpec:

- **Tool / implementation** — the AutoSpec refinement engine (`src/rule_evolution/`:
  evaluator, counterexample miner, CEGIS, ILP-guided revision, active learning) and the
  `AgentSpec` safety-rule DSL with its ANTLR grammar and parser (`src/spec_lang/`).
- **Benchmarks / datasets** — labeled execution-trace benchmarks for two agent domains,
  code execution (`data/eval_dataset/`) and embodied agents (`data/eval_dataset_embodied/`),
  each split into seed-labeled / held-out test / unlabeled pool, plus the unified trace
  schema (`schemas/trace_schema.json`).
- **Experiment scripts** — the exact scripts used to produce every result in the paper:
  RQ1 effectiveness, RQ2 search efficiency, RQ3 generalization & interpretability, the
  ablation study (Table V), and the B2–B4 baselines (`src/eval_*.py`, `src/baselines/`,
  `src/plot_*.py`, `src/compute_kappa.py`).
- **User study materials** — the interpretability questionnaires completed by 10
  participants (`user_study/`).
- **Demos** — runnable, per-stage examples of the full pipeline (`examples/`).
- **Rule sets** — manual, LLM-generated, and Apollo-driving rules used in the evaluation
  (`src/rules/`).

This repository does not ship precomputed experimental results. Running the scripts in
[Running the experiments](#running-the-experiments) produces them under `data/`
(`data/phase4_eval/`, `data/rq2_results/`, `data/rq3_results/`,
`data/ablation_results.json`, `data/baseline_eval/`).

## Setup

```bash
conda create -n autospec python=3.10 -y
conda activate autospec
pip install -r requirement.txt
pip install pandas numpy matplotlib        # used by the analysis/plotting scripts

export PYTHONPATH=$(pwd)/src
export OPENAI_API_KEY="sk-..."             # required for LLM-based steps
export ILASP_PATH="$HOME/.local/bin/ILASP" # required for ILP (RQ1–RQ3, ablation)
```

Install ILASP from <https://ilasp.com> (free academic license).

Quick check that the environment works (no API key / ILASP needed):

```bash
PYTHONPATH=src python examples/phase0_runtime_demo.py
```

## Datasets

- `data/eval_dataset/` — code execution domain (`seed_labeled` / `eval_test` / `unlabeled_pool`)
- `data/eval_dataset_embodied/` — embodied agent domain
- Trace schema: `schemas/trace_schema.json`

## Running the experiments

> Most scripts run from the repo root. The ablation and the `plot_*.py` / `compute_kappa.py`
> scripts use `../data/...` paths, so run those from `src/` (commands below already do this).

### RQ1: Effectiveness

```bash
# code domain
PYTHONPATH=src python src/eval_phase4.py --domain code --multi-iter

# embodied domain
PYTHONPATH=src python src/eval_phase4.py \
  --domain embodied --dataset-dir data/eval_dataset_embodied \
  --predicate-domain embodied --multi-iter
```

Output: `data/phase4_eval/{domain}_phase4_report.json`

### RQ2: Search efficiency

```bash
PYTHONPATH=src python src/eval_rq2.py --domains code,embodied
cd src && PYTHONPATH=. python plot_rq2_convergence.py && cd ..
```

Output: `data/rq2_results/rq2_results.json`

### RQ3: Generalization & interpretability

```bash
PYTHONPATH=src python src/eval_rq3_generalization_v2.py --domains code,embodied
cd src && PYTHONPATH=. python plot_rq3_generalization.py && cd ..

# interpretability user study (materials in user_study/)
cd src && PYTHONPATH=. python plot_rq3_interpretability.py && cd ..
```

Output: `data/rq3_results/rq3_results.json`

### Ablation (Table V)

```bash
cd src
PYTHONPATH=. python eval_ablation.py
PYTHONPATH=. python fill_ablation_table.py
cd ..
```

Output: `data/ablation_results.json`. Variants: Full System, No ILP, No Failure-Driven, No Multi-Step.

### Baselines

```bash
PYTHONPATH=src python src/baselines/run_baselines.py --domain code --baselines b2,b3,b4
```

B2 = LLM classifier (needs `OPENAI_API_KEY`), B3 = CEGIS without ILP, B4 = random search.
Output: `data/baseline_eval/`

## Component demos

Each stage of the pipeline has a standalone demo in `examples/`:

```bash
PYTHONPATH=src python examples/complete_phase0_to_phase4_demo.py   # full pipeline
PYTHONPATH=src python examples/phase3_cegis_demo.py                # CEGIS
PYTHONPATH=src python examples/phase4_ilp_revision_demo.py         # ILP-guided revision
```

## Repository layout

- `src/rule_evolution/` — the AutoSpec engine (evaluator, counterexample miner, CEGIS, ILP revision)
- `src/spec_lang/` — the AgentSpec rule DSL (ANTLR grammar + parser)
- `src/rules/` — rule sets (manual / LLM-generated / Apollo)
- `src/eval_*.py`, `src/baselines/` — experiment scripts
- `examples/` — per-stage demos
- `user_study/` — interpretability questionnaires (10 participants)
