ECBA-MLI: Edge Computing Benchmark Architecture for Machine Learning Inference

Recent developments in Artificial Intelligence (AI) research enable new strategies for running Machine Learning (ML) models. Evaluating application data on a remote server used to be common practice. However, novel AI accelerators herald a paradigm shift by moving the inference step from the cloud closer to the application in the edge. This approach increases service availability while significantly reducing latency. Nevertheless, choosing the right target platform and model for inference is a challenge that depends
on the use case and its non-functional requirements. In this work, we present an Edge Computing Benchmark Architecture for Machine Learning Inference (ECBA-MLI) which provides a universal, reproducible, and comparable solution for evaluating non-functional criteria such as latency and energy consumption for edge deployment scenarios. It further evaluates results for six state-of-art object detection models deployed on twenty-one different platform configurations.