MCP Parameter-Description Ablation: Token Savings Without Accuracy Loss

Supplementary dataset for a pre-registered ablation experiment testing whether moving parameter-level detail out of an MCP tool's description string and into inputSchema.properties[*].description costs parameter-filling accuracy, especially for smaller models. Both experiments were blind-scored: raw transcripts were scored with model identity and cell assignment masked, and all primary tests were pre-registered before data collection. Headline finding: parameter detail can live in the input schema at no measurable accuracy cost, cutting tool-description tokens by roughly 45% — the change adopted in production by aptu-coder PR #593.

Experiment 1: exp1-analyze-symbol

A 320-call run across two models (claude-haiku-4-5, claude-sonnet-5) in a 2×2 cell design: rich vs. lean tool description; parameter detail in the tool description vs. the input schema. Cell A reconstructs the tool-description text predating aptu-coder PR #593; cell C is the current, post-PR-#593 production text (~45% tool-description token reduction).

• Accuracy (pre-registered primary): Mann-Whitney U, param-fill score, cell C vs. cell A, two-tailed α=0.05, n=40/cell — no detectable difference for either model (haiku U=720, p=0.372, r=0.10; sonnet U=820, p=0.569, r=−0.025).
• Accuracy (secondary): fractional check-level composite score — confirms the result at a finer scoring resolution (haiku p=0.504; sonnet p=0.569).
• Token cost: input tokens per call significantly lower for the lean description for both models (p<0.001).

Experiment 2: exp2-response-format

A collapse A/B of response-shaping parameters (summary+fields, mode+impl_only) into a single response_format enum: 2 arms × 2 target tools × 2 models, 160 calls.

• Accuracy (pre-registered primary): no detectable difference (haiku U=880, p=0.335; sonnet U=740, p=0.242, n=40/cell).
• Token cost: small but significant input-token saving for the collapsed arm (p<1.2e-14 both models).
• Outcome: NO-GO — collapse rejected because the token saving did not justify the added indirection.

Contents

• recipe/ — harness, blind scorer, and analysis scripts
• experiments/exp1-analyze-symbol/ and experiments/exp2-response-format/ — per-experiment pre-registered protocols, rubrics, scorer prompts, fixtures, label maps, raw call transcripts, and analysis outputs
• docs/architecture.md — harness architecture

Conclusion: moving parameter-level detail from tool-description strings into inputSchema.properties[*].description preserves parameter-filling accuracy at two scoring resolutions for both a small and a large model, while significantly reducing serialized token cost — evidence for schema-level placement of parameter detail in MCP tool design.