PEPPER: Profiling-based Edge Placement and Partitioning for Deep Learning Execution

Unlocking the full potential of AI at the edge requires overcoming the fundamental challenge of running complex models efficiently on devices with limited computational power. In this work, the challenge of optimizing the deployment of deep learning models in resource-constrained environments is addressed. A novel pipeline is proposed for profiling and partitioning ONNX models, to enhance inference efficiency across heterogeneous hardware platforms. Optimal split points within the deep learning models are identified through the application of Tarjan’s Bridge-Finding Algorithm, and the inference times of the models are predicted per device based on the respective characteristics and CPU load. For the prediction of inference times, the XGBoost algorithm is employed. The effectiveness of the proposed approach is validated through experiments conducted on real-world edge devices, demonstrating that highly efficient and adaptable deployment of complex deep learning models can be achieved in such environments.