Gesture-Driven Remote Robot Mission Activation through Smart Glasses: An XR Interaction Pipeline for Industrial Assistance

This paper presents the design and early validation of an end-to-end interaction pipeline enabling industrial operators to trigger robotic missions through hand gestures captured by wearable extended reality smart glasses. Developed within the MOTIVATE XR project, the proposed approach integrates on-device visual sensing, embedded machine learning inference, semantic gesture mapping within a Unity application, and secure remote command transmission toward a mobile industrial robot executing real-world tasks. The system was experimentally evaluated in an industrial pilot environment where gestures performed by the operator were detected through the smart glasses’ cameras, interpreted locally, and translated into mission commands transmitted to a robotic platform. The study investigates recognition reliability, end-to-end latency propagation, execution consistency, and robustness across distributed system layers. To assess portability beyond the primary deployment, the gesture interaction layer was additionally integrated into an OpenXR-based XR application running on a standalone headset platform. This secondary scenario evaluates usability for interface navigation and workflow interaction, providing insight into cross-platform applicability of the gesture abstraction layer. Results provide initial evidence of the feasibility of gesture-driven control pipelines in distributed XR-robotics environments, while highlighting design trade-offs, integration constraints, and lessons learned relevant to industrial XR adoption.