Measuring Local SLM Agents as Systems: Protocol Effects on Quality and Resource Use

Local small-language-model (SLM) agents are usually evaluated as a model-selection problem, although the deployed system also includes the host, artifact, runtime, instruction, exposed and ordered tool schemas, output contract, evaluator, and server lifecycle. We use tool-call generation on five fully imported BFCL-v3 single-turn category files to measure how those protocol components alter conclusions commonly assigned to a model name. In a hash-pinned 2,480-case M4 control, changing the declared instruction and requested JSON shape moves exact match by up to 20 points on non-irrelevance tasks and reverses which of two Qwen artifacts is better when abstention must be decided rather than disclosed. An earlier positive schema-order pattern is then tested under a prospectively frozen 6,400-attempt parallel-category design; only one of sixteen contrasts survives familywise correction, while the remaining intervals do not establish equivalence. Finally, one pinned binary, artifact set, runner, and case set produce byte-identical outputs for all 2,480 pairs on an M4 Max and an A18 Pro host, while mean request latency on the A18 Pro is 4.41 to 9.33 times that on the M4 Max. Supporting controls show that schema exposure changes process RSS and that server lifecycle changes resident-memory and end-to-end measurements without changing paired outputs. Quality and resource claims therefore belong to a declared local-agent protocol, not a model label alone. We provide a protocol design matrix, minimum reporting record, prospective transfer tests, and independently recomputable evidence.

This record contains the public manuscript (version 1.1.0). The independently recomputable evidence and reproducibility package is available at https://doi.org/10.5281/zenodo.21863599.