Structure Is Cheap, Grounding Is Not: Three measurement faults and a negative result in tool calling at 103M-135M parameters
Authors/Creators
Description
A tool calling model must both emit schema-valid calls and bind arguments to contextual facts. These abilities are evaluated at the 103M parameter scale. Quick SLM is a 103M parameter LLaMA-style model, pretrained on 10B tokens and fine-tuned on 14.7M tokens of synthetic tool calling data. The fine-tuning corpus isolates state grounding by separating server-injected state and session memory, using counterfactual pairs to detect when a model pattern-matches rather than reads the state. The fine-tuned checkpoint emits well-formed calls for 99.9% of inputs (vs. 2.4% for the base model) and selects the correct tool 77.8% of the time, acquiring call structure entirely during fine-tuning. However, it fails at state grounding, grounding both branches of only 4 of 44 counterfactual pairs (exact match). A control experiment with SmolLM2-135M similarly achieved 0 grounded pairs, suggesting this failure stems from the corpus design or sub-135M scale rather than the pretraining run. Detailed error analysis shows grounding failures are largely not state blindness, but divide between tool selection errors and arithmetic failures on read values. Methodological contributions are additionally highlighted: documenting three measurement faults that corrupted initial results, their corresponding cross-checks, and exposing a 0.74-point self-preference bias caused by using the same Gemma model for both corpus generation and evaluation.
Files
Structure Is Cheap, Grounding Is Not.pdf
Files
(419.9 kB)
| Name | Size | Download all |
|---|---|---|
|
md5:a37b2120222108764d5c2092d27efea6
|
419.9 kB | Preview Download |
Additional details
Software
- Repository URL
- https://github.com/techquicks-ai/quick-slm
- Programming language
- Python
- Development Status
- Active