Automatically Adapting and Publishing TV Content for Increased Effectiveness and Efficiency

By automatically adapting TV content for publication on secondary channels, we aim to increase user engagement and at the same time reduce the manual effort for broadcasters. We present a system architecture for automatic content adaptation and two distinct use cases built on top of it.


INTRODUCTION
TV broadcasters produce content at great cost. Given the migration of younger generations to secondary on-demand publication channels like YouTube, Twitter and their broadcaster websites [3], content producers need to distribute their content effectively and efficiently.
Users expect the content to match the visual language of the channel they are on; adapting the content needs to be part of the publication workflow.
We demonstrate two scenarios that automatically adapt and publish TV content onto secondary publication channels. Those scenarios use the same underlying metadata and system architecture. Both scenarios aim to recommend relevant programs or sections of a program to viewers. After the initial setup and configuration, neither of the scenarios requires manual work from the content owner, increasing the efficiency of the publication process. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s

SCENARIOS
The 4u2 Chatbot interacts with the user through messaging apps, whereas the Content sWitch inserts content into a potentially timeshifted OTT TV stream. Each scenario has a distinct frontend and business logic, but the content adaptation service is largely identical.

4u2 Chatbot
Interviews with TV viewers [2] have shown, that a significant share is interested in having content sent to them periodically through a chatbot.
The messenger app provides the frontend for our Chatbot (e.g. Telegram or Facebook Messenger). The chatbot middleware is the module that parses the inputs from the user and responds appropriately. In order to abstract from the intricacies of the different messenger platforms and to profit from community efforts, we decided to build our middleware using the Rasa 1 framework. In Rasa, we define so-called stories, which are dialogues the user navigates through. The video delivery story (see Figure 1) is the most relevant one, but there are also stories for registration and changing preferences. We solve the cold-start problem by asking the user to choose from a set of pre-defined content categories during the registration story. We also ask users how often they would like to receive content. The selected preferences, as well as the desired schedule, are stored in a database. A second component of the middleware regularly checks this database to see if a user's desired time of delivery is close.
At the time of desired delivery, the middleware loads the user preferences and calls the content adaptation service, asking for a personalised video. Once the content adaptation service returns this video, the middleware sends it to the user as a video message.
The chatbot is currently being tested with real users for ReTV content partners RBB 2 and NISV 3 . The results of those tests will be made available in Deliverable 6.2 4 of the ReTV project.

Content sWitch
The linearity of TV is both a feature and a bug: It creates a sense of belonging for viewers of a live program, but it strips the broadcaster of the possibility to A/B test different content and to personalise the user experience. By personalising only short segments of an OTT TV stream, we believe that we can deliver the best of both worlds.
Our technical setup is identical to how Dynamic Ad Insertion works, of which many examples exist 5 . However, instead of replacing only ads, we replace all types of TV content and integrate this into a holistic framework that adapts content over multiple distribution channels.
The first version of the Content sWitch focuses on trailers. We continuously track the position of a viewer in the TV stream. If the electronic program guide (EPG) data indicates that a trailer is imminent, we send the user profile to the content adaptation service. The content adaptation service then returns a trailer which we splice into the video stream at the appropriate time. Video examples are available online 6 .

Content adaptation service
A classic video recommendation service takes user preferences as input and returns content that best matches those preferences. Our content adaptation service not only recommends videos but also automatically adapts them. We developed a concise domain-specific language based on JSON to describe adaptations. We currently support two types of adaptations: Shortening and merging.
We use shortening in the Content sWitch to make the replacement trailer fit into the slot of the original trailer.
Merging is useful if more than one show covers a topic that is of interest to a user. Our content adaptation service then selects multiple videos that fit user preferences. The videos are passed to the summarisation service, which merges them, ideally keeping only the relevant parts of each input video. The complete content adaptation service is described in detail in the forthcoming Deliverable 3.2 7 .
We use a Field-aware Factorisation Machines (FFM) model [4] to find candidate videos for a given viewer. The advantage of FFM compared to other methods is that it allows us to exploit a large number of features efficiently. This extendability is a valuable property, as we expand the number of content features we extract from the video content. It also models the interactions between the individual feature values, which allows us to discover viewing patterns (e.g. fans of tennis are also interested in golf but not swimming).
Currently, our content features revolve around genres. The source of such features can be an EPG provider; however, we can categorise any content on our own. Our approach is an unsupervised categorisation, based on speech-to-text transcripts and multilingual word embeddings [1] 8 . The categories can be freely chosen, as we have done in the case of RBB and NISV. We categorise each word in a sentence independently, and the final categorisation is a majority voting result. A single program can belong to multiple categories with different weights. The automatic approach has an advantage over standard EPG as it can categorise individual parts of a program. This is particularly important for thematically heterogeneous content, where only a subset of the program might be of relevance to a specific viewer.
While the resulting videos for the 4u2 Chatbot and the Content sWitch are vastly different, we see this as the application of the same two-step process: (1) Find the best videos for a user under a set of business constraints. (2) Adapt those videos to fit the context. Sample business constraints that we can encode are a maximum and minimum length. More advanced business constraints could be to bias the recommendation towards an editorial focus.
We can train our recommendation model on both types of input: implicit user profile, i.e. behavioural data, and explicitly provided user preferences.

CONCLUSION AND FUTURE WORK
We will demonstrate both scenarios with dummy viewer types, where the content adaptation service will produce different adaptations of our content partners' programming according to the implicit or explicit user preferences.
The focus of our future work lies on better algorithms and richer metadata in order to improve the results for the users. In general, we would like to: a) allow for adaptations that are semantically richer, e.g. faster cuts for younger viewers, b) improve the frontends of both the 4u2 Chatbot and the Content sWitch to be more user-friendly, c) improve the model retraining by collecting more detailed user feedback. For the Content sWitch, for example, we would like to track if a viewer watched the show for which we inserted a trailer.