VISIONBOT VISualizIng and cONtrolling a roBOT via mixed reality interface

VISIONBOT is a Mixed Reality (MR) robotic teleoperation and monitoring system developed for the Meta Quest 3 headset and integrated with the Franka Emika Panda robot through ROS 2.

The system enables bidirectional communication between a virtual robot and its physical counterpart, supporting both teleoperation and real-time mirroring functionalities. Users can control the robot through immersive MR interactions using controllers and hand tracking, while robot state information is continuously synchronized between the physical and virtual environments.

The current release includes the following features:

• Meta Quest 3 Mixed Reality application for robot teleoperation;
• Bidirectional communication between the MR application and ROS 2 through a Python bridge;
• Teleoperation of the Franka Emika Panda robot using the Quest 3 controller;
• Robot state synchronization and real-time virtual robot mirroring;
• Support for hand-tracking-based interaction;
• WebSocket-based communication architecture;
• Setup and deployment instructions for reproducing teleoperation experiments, provided through the project README.

Future Work

The following functionality was partially implemented but not fully validated:

• Franka gripper integration.