There is a newer version of the record available.

Published July 31, 2026 | Version 1.0.0

Structure Is Cheap, Grounding Is Not: Three measurement faults and a negative result in tool calling at 103M-135M parameters

Description

A tool calling model must both emit schema-valid calls and bind arguments to contextual facts. These abilities are evaluated at the 103M parameter scale. Quick SLM is a 103M parameter LLaMA-style model, pretrained on 10B tokens and fine-tuned on 14.7M tokens of synthetic tool calling data. The fine-tuning corpus isolates state grounding by separating server-injected state and session memory, using counterfactual pairs to detect when a model pattern-matches rather than reads the state. The fine-tuned checkpoint emits well-formed calls for 99.9% of inputs (vs. 2.4% for the base model) and selects the correct tool 77.8% of the time, acquiring call structure entirely during fine-tuning. However, it fails at state grounding, grounding both branches of only 4 of 44 counterfactual pairs (exact match). A control experiment with SmolLM2-135M similarly achieved 0 grounded pairs, suggesting this failure stems from the corpus design or sub-135M scale rather than the pretraining run. Detailed error analysis shows grounding failures are largely not state blindness, but divide between tool selection errors and arithmetic failures on read values. Methodological contributions are additionally highlighted: documenting three measurement faults that corrupted initial results, their corresponding cross-checks, and exposing a 0.74-point self-preference bias caused by using the same Gemma model for both corpus generation and evaluation.

Files

Structure Is Cheap, Grounding Is Not.pdf

Files (419.9 kB)

Name Size Download all
md5:a37b2120222108764d5c2092d27efea6
419.9 kB Preview Download

Additional details

Software

Repository URL
https://github.com/techquicks-ai/quick-slm
Programming language
Python
Development Status
Active