API Security Benchmark: Reproducibility Artifact for "Benchmarking API Security Testing Tools Against the OWASP API Security Top 10 (2023)"

This artifact provides a fully reproducible, Docker-based benchmark for evaluating security testing tools against the OWASP API Security Top 10 (2023).

Overview: The artifact accompanies the paper "Benchmarking API Security Testing Tools Against the OWASP API Security Top 10 (2023): Coverage, Gaps, and Reproducibility". It evaluates four security testing tools (Burp Suite Professional, OWASP ZAP, JWT Tool, Nessus Professional) against two complementary testbeds: a custom Node.js API and OWASP crAPI.

Contents:

• Custom Node.js API with 32 planted vulnerabilities covering all 10 OWASP API categories
• Docker Compose configuration for one-command deployment (API + MongoDB)
• Tool configurations for Burp Suite, OWASP ZAP, JWT Tool, and Nessus
• Ground truth inventory with CVSS v3.1 scores and validation log
• Automation scripts for running all four tools
• 6 Jupyter notebooks reproducing all paper analyses
• Statistical analysis code (McNemar's test, bootstrap CIs)
• Figure generation scripts (radar chart, bar chart, heatmap)
• Raw results and computed metrics

Requirements: Docker Desktop, 4 GB RAM, 10 GB disk space, Python 3.10+ (for analysis).

Quick Start:

docker-compose up -d
./scripts/run-all.sh
jupyter lab notebooks/

Reproducibility: All results in the paper can be reproduced in approximately 1 hour. Run notebooks/06_master_integration.ipynb for a complete verification.

Key Findings:

• OWASP ZAP achieves the highest severity-weighted Coverage Score (0.70)
• Burp Suite excels in detection rate (90%) but has high false positives (F1 = 0.53)
• JWT Tool provides perfect precision within its narrow domain
• API10 (Unsafe Consumption of APIs) remains unaddressed by all tools

License: MIT (code), CC BY 4.0 (documentation), CC0 1.0 (data)