An easy Hand Gesture Recognition System for XR-based collaborative purposes

Abstract:

In the last few years, collaborative tools have increased dramatically, partly to compensate for the distance between people due to the pandemic and partly to allow activities (work, entertainment, free time, etc.) to be carried out among people without having to worry about geographical distances. In this scenario, it was necessary to overcome the classic remote meeting tools e.g. video, audio, chat), which have a reduced sense of presence. Extended Reality (XR) represents a Computer Graphics (CG) based innovative technology particularly suited to this purpose. Indeed, XR aims to develop Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (XR) solutions that can transform how people interact by increasing the sense of presence. This last aspect depends not only on the virtual/real objects and scene visualization but also on their interaction. In this regard, human-computer interaction (HCI) techniques represent a possible solution. However, these techniques depend on specific devices, such as Head Mounted Display (HMD), Smart Glasses, Depth and Tracking Cameras, etc., whose costs make access difficult. For this reason, we propose a Hand Gesture Recognition (HGR) system that can be used in XR applications using a simple RGB camera. Our is a deep learning system based on MediaPipe, the state-of-the-art (SOTA) for hand tracking through simple RGB images [1], [2].