ComBench: A Benchmarking Framework for Publish/Subscribe Communication Protocols under Network Limitations

Efficient and dependable communication is a highly relevant aspect for IoT systems in which tiny sensors, actuators, wearables, or other smart devices exchange messages with each other or with cloud services. Various application layer protocols of the ISO OSI model are especially suitable in the IoT context to cope with the communication behavior of IoT systems. Many of them follow the publish/subscribe communication pattern. The selection process of a suitable protocol should take the communication behavior of the application, the resource requirements on the end device, and the network environment into account. The decision requires knowledge about the performance and the robustness of the protocols in various settings, especially in wireless networks as often used in IoT environments. Benchmarking is a common approach to evaluate and compare systems, considering the performance and aspects like dependability or security. In this paper, we present our IoT communication benchmarking framework ComBench for publish/subscribe protocols in the IoT context focusing on constrained networks. The benchmarking framework supports system designers, software engineers, and application developers to select and investigate communication protocols' behavior. In particular, our benchmarking framework contributes to (i) show the impact of fluctuating network quality on communication, (ii) compare multiple protocols, protocol features, and protocol implementations, and (iii) analyze scalability, robustness, and dependability of clients, networks, and brokers in different scenarios. Our case study demonstrates our benchmarking framework's applicability to support the decision for the best-suited protocol in various scenarios.