LLMs on Edge: Network Traffic Characteristics of Distributed Inference under the Loupe

Large Language Models (LLMs) have revolutionized Natural Language Processing and now find their ways into various deployments such as end customer appliances or industrial settings. Their deployment at the edge, however, presents unique challenges, particularly regarding network infrastructure and resource constraints. While existing research has focused on LLM distribution in cloud environments, there is a lack of studies addressing the specific requirements and characteristics of edge computing scenarios. Accordingly, there is a rise of distributed LLM frameworks that aim to optimize the deployment of LLMs in edge environments. Due to their work in progress nature, these frameworks lack comprehensive measurements in a real testbed w.r.t. networking. This paper presents a comprehensive analysis of distributed LLM frameworks in edge computing environments, focusing on their networking behavior and deployment requirements. Our measurement results reveal non-obvious behaviors from performance degradation by adding compute nodes to significant traffic pattern complexities.