Peer-to-peer language-model inference has so far been pursued by splitting the model: transformer layers or tensors are distributed across machines, and intermediate activations traverse the public internet on every generated token. This places the design squarely against a bandwidth gap of roughly five orders of magnitude and a latency gap of four to five, between datacenter interconnect and consumer last-mile links.

This paper specifies Swarmbly, a protocol that distributes the problem instead. A client-side orchestrator, itself a small language model, decomposes a request into a directed acyclic graph of semantic micro-tasks; each micro-task is dispatched once, asynchronously, to a volunteer node running a complete small model (1–8B parameters); returned fragments — contigs, in the genome-assembly vocabulary the design borrows — are verified, selected and spliced locally. Network traversal occurs once per fragment per session rather than once per layer per token.

Swarmbly AI