How to Realize an immersive Jedi Council Meeting - A Capturing Story of Holograms and Their Hyperdrive Delivery

Holographic communication has been a common trope in sci-fi for the last couple of decades. While true holograms will take a while longer, the pursuit of comparable outcomes has manifested in the form of volumetric videos rendered in virtual reality environments. In these volumetric videos, real-life objects are represented in 3D by either a mesh or a point cloud. In stark contrast to conventional 2D videos, these volumetric counterparts allow for both positional and rotational movement which enable users to interact with each other and walk around in the scene. Consequently, this medium exhibits an immense promise for enabling or enhancing several forms of virtual communication, such as conferencing or digital concerts Nonetheless, this enhanced immersive experience comes at a cost. Raw volumetric videos have significant bandwidth requirements, making them unfeasible to use in a real-time interactive scenario. Compression has been used to reduce these requirements at the cost of introducing additional latency, thereby constraining the potential for seamless real-time interactions among participants. In addition, current volumetric video implementations utilize streaming technologies, such as low-latency DASH, which do not allow for real-time interaction due to their latency. In light of these challenges, we propose a point cloud-based end-to-end pipeline for one-to-many streaming scenarios. These point clouds are encoded using the Google Draco codec, effectively reducing the bandwidth to levels which are supported by current network infrastructures. In addition, we implemented a layered encoding strategy which samples the point cloud into three distinct layers, each sampled at a different rate. These layers allow us to perform quality adaptation for each user based on their position and a bandwidth estimate provided by the Google Congestion Control algorithm. Finally, we transmit the frame by utilizing WebRTC to ensure a significantly lower transport latency compared to current state-of-the-art point cloud streaming implementations.