Production and Delivery of Video for Multi-device Synchronized Playout

In the contemporary living room, the audience's attention is often divided between TVs, second screens and, increasingly, head mounted displays. To address this reality, ImmersiaTV is a H2020 European project which is redefining the end-to-end broadcast chain: production, distribution and delivery. It is built on two ideas: multi-platform synchronous content playout, and orchestrated videos rendered in the head-mounted display as interactive inserts, which allow introducing basic interactive storytelling techniques (scene selection, forking paths, etc.) as well as classical audio-visual language that is not possible to render with 360 videos (close-ups, slow motion, shot-countershot, etc). We demonstrate our pipeline for offline production, distribution and synchronized playout.


INTRODUCTION
The majority of TV consumers now watch TV programs in a multi-display environment [2]. Second screens -most often smartphones -are generally used to check information not directly related to the events in the TV content being watched. Broadcasters have tried to orchestrate these different platforms, and there is reason to believe this contributes to user engagement [3]. However, their success has been limited. This might be caused, at least in part, by the different formats of content being used: mobile apps show graphics and text similar to web content, while TV renders a continuous stream of audiovisual content. The arrival of virtual reality displays to the living room further increases the need for consistent experiences across displays. To address this fact, we have designed and implemented an end-to-end production, delivery and rendering pipeline for offline content production specifically designed to create experiences which give freedom to the end-user to engage with content in one or another device, and are still adapted to the specificities of each device, both in terms of content format and interactive input (or lack of interaction, for the case of the TV). We demonstrate such an approach with an example documentary showing the day life of a kid attending the Porto Football School.

CAPTURE
The creation of content that is both omnidirectional and traditional requires shooting simultaneously in both content formats. The solution found was to use two BlackMagic Micro Studio Camera 4k micro-cameras for the traditional shooting, which could be hidden or, if visible, removed in post-production with a reasonably small amount of effort.

EDITION
We are not aware of an editing tool targeting synchronous rendering across devices. To address this fact, we have designed and implemented a plugin for Adobe Premiere Pro. The ImmersiaTV plugin shown in figure 3 allows defining the inserts that are placed within an omnidirectional video. It also allows selecting which tracks should be rendered in each of 3 possible devices (TV, tablet or HMD). It works both with Mac and Windows, and we have validated that, after going through a tutorial, can use it to create multi-platform content.

DELIVERY
The media encoding uses H.264 and AAC encoding, and adaptive bitrate streaming based on MPEG-DASH (ISO/IEC 23009-1:2014). Encoding is implemented as a cloud service, running on a Linux server using the Dockers virtualization tool as well as MP4Box from Gpac's MP4Box for MPEG-DASH multiresolution encoding. Video decoding uses the Gstreamer multimedia framework. The additional metadata required for playout, which relates audiovisual streams with devices (i.e., allows selecting different streams for TVs and tablets), as well as to define interaction and media orchestration requirements, follows closely the format of MPEG-DASH manifests, and its XML specification is publicly available. Content publication is performed through a transcoding and content publishing server controlled with a custom built web interface. It allows triggering media conversion, as well as monitoring progress on media encoding and publishing a list of content, which is then parsed by the player to access the media assets.

INTER DEVICE PLAYOUT
To play synchronized content [1,4,5] We have also embraced the use of omnidirectional video for HMDs and smartphones, in order to allow the user to visualize the scene in different directions, thanks to a tool integrating Unity3D and GStreamer, publicly available at: https://www.assetstore.unity3d.com/en/#!/content/59897 https://github.com/ua-i2cat/gst-unity-bridge