5GCroCo Barcelona Trial Site for Cross-border Anticipated Cooperative Collision Avoidance

Cooperative, connected and automated mobility (CCAM) services along different countries require cross-border solutions to support seamless delivery of services in a multioperator, multi-telco-vendor, and multi-car-manufacturer scenario. The H2020 5GCroCo project will trial 5G technologies in the European cross-border corridor along France, Germany and Luxembourg, as well as in five small-scale trial sites. 5GCroCo analyses three cross-border use cases: tele-operated driving, high- definition map generation and distribution for autonomous vehicles, and anticipated cooperative collision avoidance (ACCA). This paper presents the infrastructure, control architecture, backend software, and end-to-end service orchestration of the cross-border ACCA use case deployed in the Barcelona small-scale trial site.


I. INTRODUCTION
The smooth deployment of cooperative, connected and automated mobility (CCAM) services along different countries is very innovative and challenging from a business and technical perspective. On the technical side, vehicles driving across national borders require cross-border solutions to support seamless delivery of services in such a multi-operator, multi-telcovendor, and multi-car-manufacturer environment [1]. Normally crossing a border also means changing the Mobile Network Operator (MNO) and entering or resuming roaming state. For networks and application servers operating on different physical network infrastructure it is difficult to maintain end-toend QoS. Neutral hosting platforms can help overcoming this challenge by providing and controlling physical infrastructure form a single entity.
In Europe, the H2020 5GCroCo project [2] aims at defining successful path towards the provision of CCAM services along cross-border scenarios and reduce the uncertainties of a real 5G cross-border deployment. In this sense, 5GCroCo will trial 5G technologies in the cross-border corridor connecting the cities of Metz-Merzig-Luxemburg along France, Germany and Luxembourg. Additionally, 5GCroCo also deploys small-scale trial sites which aim at testing and trialing specific parts of the defined use cases and architecture before setting up all on the large scale trial. The five small-scale trial sites developed in 5GCroCo are Barcelona (Spain), Montlhéry (France), Munich (Germany), Motorway A9 (Germany), and AstaZero (Sweden). 5GCroCo analyses three use cases: tele-operated driving (ToD), high-definition (HD) map generation and distribution for autonomous vehicles, and anticipated cooperative collision avoidance (ACCA) [3]. The ACCA use case relates to the possibility to anticipate certain potentially critical events in order to reduce the probability of collisions in situations when typical sensors will have no visibility or a short detection range (a few 100m). The aim is to induce smoother and more homogeneous vehicle reactions by facilitating the anticipated detection and localization of temporarily static events such as traffic jams, high deceleration, emergency braking or unexpected manoeuvres of vehicles in front, etc. This paper presents the ACCA cross-border deployment in the Barcelona small-scale trial site. The paper is organized as follows. Section II presents the infrastructure of the Barcelona small-scale trial site and Section III the deployed vehicles. Section IV depicts the ACCA backend deployed in the MEC and public cloud, and section V describes the considered crossborder network function virtualizaton (NFV) management and network orchestration (MANO) to dynamically deploy end-toend ACCA services. Finally, section VI concludes the paper.

INFRASTRUCTURE
The Barcelona small-scale trial site is composed of a 5G neutral hosting platform deployed in Barcelona (in the 22@ district), and an emulated cross-border Internet Exchange Point (IXP) platform deployed in Castelldefels (at CTTC premises), 25 km south from Barcelona, as shown in Fig. 1. Both platforms are integrated within the 5GBarcelona infrastructure [4] through a dedicated Ethernet/optical transport network.
The 5G neutral hosting platform can deploy multiple virtual mobile network operators (vMNO) emulating a typical cross-border scenario with each vMNO serving one country. Each vMNO provides LTE connectivity to vehicles and MEC services to host the edge part of ACCA backend software (i.e., Edge Geoservice). The cross-border IXP platform provides the physical infrastructure through which multiple vMNOs can exchange data traffic. It is also equipped with public cloud servers for the centrally hosted part of the ACCA backend software (i.e, Central Geoservice). In the deployed ACCA use case, the vehicle information is fused with the real time traffic information from city and road infrastructures.

A. 5G neutral hosting platform
The 5G neutral hosting platform is composed of three lampposts connected to a street cabinet, deployed along the "Ciutat de Granada" street. The lampposts are equipped with LTE small cells at 3.5 GHz used to provide connectivity to the vehicles. The street cabinet is equipped with a MEC platform allowing to deploy the EdgeGeoservice software (i.e., edge part of the ACCA backend) and the virtualized EPC (vEPC) for each vMNO. The street cabinet is connected to the crossborder IXP platform through the optical/Ethernet network.
The 5G neutral hosting platform facilitates the sharing of 5G network infrastructure among different network service providers (telecom operators and/or vertical industries) by leveraging new network virtualization solutions and dynamic configuration enabled by 5G technologies. It is based on ETSI NFV [5] and Network Slicing [6] and empowered by the intelligent combination of underlying controllers such as Open-SourceMano, OpenStack, a proprietary radio access network (RAN) controller, and more. It aims to help 5G infrastructure owners -which in this case could be the municipality of Barcelona or a "telecom tower company"-make their (shared) resources available for on-demand network creation.
The 5G neutral hosting platform offers northbound interfaces that can be used to manage the registration of resources (compute nodes, physical networks, and radio access points), as well as the partitioning ("chunking") of those resources and the creation of slices as collections of such chunks with NFV network services running on top of them. For example, the "launching" of a minimal slice for a vMNO in the 5GCroCo ACCA use case would involve the usage of the platform to create some compute chunks on the MEC servers of the street cabinet and some RAN chunks on the lamppost Small Cells, as well as the creation of a slice by compiling those chunks together and deploying on them a network service that is composed of a set of virtual network functions (VNFs) for the vEPC and the Edge Geoservice applications. Further, the platform seamlessly triggers configuration actions against the LTE Small Cells that are deployed in the 22@ district of Barcelona, which are required so that the created slices fulfill their requirements.
The RAN infrastructure is handled by i2CAT's own developed centralized RAN Controller, which is able to remotely manage Wi-Fi nodes (custom solution) and Accelleran's Small Cells. The controller applies a YANG model management, using NETCONF protocol [7] to remotely configure the nodes and deploy the required RAN slices. In the case of the LTE Small Cells, it is able to dynamically instantiate up to five different slices per Small Cell using custom Public Land Mobile Network Identifiers (PLMN-IDs) and their associated vEPCs, which are based on Open5Gs open source solution 1 . Regarding data paths, they are managed using OpenDayLight 2 open platform, using Layer-2 VLANs controlled by Open vSwitch 3 (OvS) to differentiate the traffic of the different slices through the network.

B. Cross-border Internet Exchange Point Platform
The cross-border IXP platform provides the physical infrastructure through which multiple vMNOs can exchange data traffic. It is also offering public cloud services, deploying both virtual machines and containers, for the Central Geoservice applications, representing the centrally hosted part of the ACCA backend. This part includes services that do not need to be hosted at the edge and enables communication between edge servers in cases when they cannot directly communicate, as can be initially assumed for MECs operated by different MNOs.
The cross-border IXP platform relies on the infrastructure from the ADRENALINE testbed [8]. The interconnection among the different vMNO operators, and the cloud servers is done through a software defined networking (SDN)-enabled switch. It is deployed on commercial off-the-shelf (COTS) hardware and Open vSwitch technology. It is controlled by an OpenDaylight SDN controller and provides connectivity to the 5GBarcelona transport network, the core-datacenter (DC) and two container-based servers. The container-based servers are based on Kubernetes (K8s), and each server has its own K8s controller. The core-DC is composed of a high performance computing (HPC) cluster where virtual machines (VMs) can be deployed. The intra-DC packet network of the core-DC is composed of OpenFlow switches controlled by an OpenDaylight SDN controller responsible for the Intra-DC network connectivity. The HPC servers are controlled using an Openstack controller.
At the transport level, an overarching SDN orchestrator is deployed on top of the two OpenDaylight SDN controllers. It provides end-to-end connectivity services using the transport application program interface (TAPI) [9]. On top of the SDN orchestrator, we deploy an NFV service platform. It interfaces with the SDN controller, the OpenStack controller of the core-DC, and the two K8s controllers. It is based on SONATA [10] Service Platform (SP) which was developed in the H2020 5GPPP 5GTANGO project [11]. It is in charge to manage the life-cycle of NFV network services and network slices. An NFV network service is composed of chained VNFs or cloud-native network functions (CNFs) deployed on VM and/or containers. A network slice is composed of an NFV network service that deploys a set of VNFs and/or CNFs with the CentralGeoservice applications. This NFV network service defines network service end-points associated to VLANs in the transport network. The vehicles used in the Barcelona trial site are equipped with a car PC system which can be used as Communications Control Unit (CCU) in any vehicle, and it can also be used as vehicle emulator. If the car PC is configured as vehicle emulator, it emulates the environment usually provided by a real vehicle (e.g., sensor data coming from the CAN bus) and obtains the position of the emulated vehicle from a trajectory file that contains a sequence of geographical coordinates with associated time-stamps.
The components of the car PC system are shown in Fig.  2. The core of the car PC is a laptop that uses Linux (Ubuntu 18.04) and a software application based on OpenC2X [12], [13], which is an open source software that supports most of the aspects of the ETSI Intelligent transportation system (ITS) protocol stack defined in [14]. The car PC communicates with the backend over 4G LTE by means of an AirPrime EM7565 modem from Sierra Wireless. The car PC exchanges two types of ETSI-ITS standard-compliant messages with the backend: periodical Cooperative Awareness Messages (CAMs) and event-triggered Decentralized Environmental Notification Messages (DENMs). Two different protocols are considered for communication of vehicles with the backend: UDP (User Datagram Protocol) carrying ETSI-ITS encoded messages including Besic Transport Protocol (BTP) and Geo Networking headers, and MQTT (Message Queuing Telemetry Transport) either also with ETSI-ITS payload as for UDP or with JSON encoded messages. In MQTT, the car PC software supports JSON encoded messages. The car PC software implements both protocols, and it facilitates testing the interoperability between vehicles using UDP and MQTT protocols. The laptop of the car PC is connected to a GNSS (Global Navigation Satellite System) receiver in order to obtain the real-time position of the vehicle and to the CAN bus of the vehicle by means of an On-Board Diagnostic (OBD) interface adapter or a CAN to USB adapter. Through the CAN bus, the vehicle sends speed measurements and events to trigger the transmission of DENMs.
The graphical user interface (GUI) of the car PC software allows the interaction of the driver with the car PC. Through the GUI, the driver is able to initiate the registration, authentication and subscription processes with the backend, and emulate the detection of a hazard by triggering the transmission of a DENM. In addition, the GUI shows the current position of the ego vehicle and the location of road hazards on a Local Dynamic Map (LDM) based on OpenStreetMap 4 . The GUI displays a notification message every time a new hazard notification is received.
In the Barcelona trial site, one car PC system is integrated as CCU in a car provided by Peugeot Citroen Automobiles (PSA) group for the trials of the Barcelona site. Another is used as either vehicle emulator or integrated in a regular car.
IV. DEPLOYED CROSS-BORDER ACCA ARCHITECTURE Fig. 3 shows the proposed cross-border ACCA architecture deployed in the Barcelona trial site. Among the multiples options defined in the 5GCroCo project, in this paper we consider MQTT and ETSI-ITS protocols. At the bottom, each vehicle is connected, through a certain vMNO, to the Edge Geoservice applications. The Edge Geoservice is composed Fig. 2 Fig. 3: Cross-border ACCA architecture of an Edge Geoserver and an MQTT broker, depending on the type of Vehicle-to-everything (V2X) protocols available at the vehicle. Deployed Edge Geoservers are able to exchange MQTT messages with the Central Geoservice applications hosted in a centralized cloud. The Central Geoservice is composed of a Traffic Management System (TMS) with an MQTT broker, that is responsible for cross-border message exchange, and a Central Geoserver. It is worth mentioning that an inQueue facilitates to send raw messages generated by vehicles towards the TMS; an OutQueue to disseminate processed information by the TMS towards the vehicles and other connected entities.
Discovery and changing of serving Edge servers (i.e., Edge Geoserver and MQTT broker) are out of the scope of this paper. Selected protocol choices, for the moment, are mostly based on reusing existing implementations and do not necessarily represent a choice for a future wide scale commercial deployment. For the implementation it was essential to allow a proof of concept and realistic performance evaluation. In the meantime standardization on solutions allowing multiple protocol choices between vehicles and backend has progressed and e.g. a Basic Interface was defined for this by the C-Roads Platform [15].

A. Edge MQTT broker
As stated, one of the messaging protocols used is MQTT [16]. It is a lightweight, simple messaging protocol based on a publish/subscribe model, designed for constrained devices and low overhead. It has been used extensively in "machineto-machine" (M2M) or "Internet of Things" (IoT) contexts and in applications with CPU, memory and battery constraints. The protocol supports plain TCP or SSL secured connections. MQTT has been chosen as the messaging protocol used by the different entities (GeoServers, traffic management systems) and optionally the vehicles (which may use UDP).
The Edge MQTT broker is the functional entity that receives and sends messages to and from multiple clients based on how such clients send (publish) and and wish to receive (subscribe) to topics. Its functionality can be embedded in an application or deployed stand-alone, for example, using the Eclipse Mosquitto [17] implementation, an open source (EPL/EDL licensed) message broker that implements the MQTT protocol versions 5.0, 3.1.1 and 3.1. In particular, messages are "published" by MQTT clients with a given "topic", a string that is hierarchical and that can contain wildcard commands. Multiple clients can receive such message based on the topics to which they are subscribed. A careful definition and usage of topics determines which entities receive with messages: for example a given client may publish in the topic "to-master" and subscribe to topic "from-master" resulting in a hub/spoke topology. The broker can store the data in the form of retained messages so new clients may retrieve past messages and it can manage persistent sessions, so the broker manages and tracks all client connection states, including security credentials and certificates.
In our context, MQTT messages correspond to JSONencoded CAM and DENM messages, translating from a ASN.1 encoding to JSON in a 1:1 mapping. The MQTT topics identify if the message originated in a vehicle or an entity and its different levels identify a geographic location. In general, different higher-layer protocols/message formats can be used.

B. Central/Edge Geoserver
The Geoserver is in charge of providing low latency ACCA Geoservices. Accordingly, Geoservers are located primarily in 5G Edges. However, a Centralgeoserver remains located in the cloud to provide Geoservices to locations either not including 5G Edge or for vehicles not supporting 5G Edge technologies. The main task of the Geoserver is to gather awareness, perception and sensor information from vehicles and other IoT devices, consolidate them and evaluate potential hazards. To receive hazard-related information, vehicles and devices must subscribe to the specific Geoservices based on the multi-level tiling previously described. The Geoserver connects to the central MQTT broker to transmit and receive wider scope hazard-related data. It also connects to the Edge MQTT broker to synchronize hazard related data between different types of services (MQTT or UDP). Finally, the Geoserver includes an ETSI ITS module to support ETSI ITS messages (CAM, DENM, CPM) encaspulated in IP packets. By supporting MQTT or ETSI ITS, by being located both in 5G Edges or in 5G Clouds, and being interconnected to a wider-scope TSM, the Geoserver is able to obtain a complete view of all actors involved in potential hazards and can provide low latency Geoservices tailored to various situations (stationary vehicles, accidents, traffic jams, etc..), and this irrespectively to the 5G hosting platforms or operators. Vehicles using UDP interface can perform an initial HTTPS authentication and subscription, getting raw UDP messages with active and valid (past) events. Such UDP messages convey CAM or DENM messages that are of relevance to the vehicle.
As depicted on Fig. 4, the Geoserver consists of five modules. The Subscription Manager module is in charge of handling the authentication and subscriptions of vehicles to the Geoservices. The Data processing module receives,

C. Traffic Management System and Central MQTT Broker
A traffic management system (TMS) is a complex distributed application that enables the monitoring, management and operation of large infrastructures such as a road or a city. It receives heterogeneous inputs from multiple sensor systems deployed in the infrastructure and enables their operators to make decisions on its operation. Traffic flow is usually monitored by camera systems, coils, magnetic vehicle counters or RF beacon sniffers that take advantage of signaling packets produced by the moving vehicles. Roads are also equipped with numerous environmental monitoring systems, including weather stations, animal crossing detection systems. A TMS is also connected to emergency services and other service providers that may be needed in the operation of the infrastructure. For example police or private security personnel and other emergency services are localized in the TMS platform so operators can react to contingencies in a quick manner. The Barcelona trial site integrates the C-V2X infrastructure to the oneMind platform. oneMind is an advanced TMS systems that provides simple and flexible integration of heterogeneous data flows. Its micro-services based architecture enables the constructions of simple data connectors that allows the interconnection with heterogeneous systems. It also integrates different components and databases and offers a central MQTT broker architecture that is able to handle incoming packets from the different Geoservices deployed either at the cloud (central) or in the MEC of the different MNOs and orchestrate such information so that interested parties receive it almost immediately. A packet orchestrator has the responsibility to relay received messages to those Geoservices that are interested in receiving events for a certain region of interest. The Central Geoservice receive it all acting as a global manager in case of failures or unavailability of the Edge Geoservices. As V2X messages are geopositioned, regions of interest can be determined and messages relayed to the corresponding MQTT topics that represent that regions. A tile-based topic naming has been defined so interested parties can subscribe to only those topics that affect a particular region. The multi-level As depicted in Fig. 3, the ACCA end-to-end service is a distributed application composed of functional elements that must be instantiated across the different segments of the network (i.e. in the public cloud managed by the IXP Platform and in distributed edge locations managed by the 5G Neutral Hosting Platform) to guarantee a proper service coverage with respect to the target areas. In the Barcelona trial site, where a cross-border scenario is emulated, the ACCA Geoservice application is deployed across three different domains: a central cloud and two edge locations managed by vMNOs Fig. 5.
The need of running different components of the endto-end service into different administrative and technological domains implies a functional split of the application into nested services to be instantiated and managed by the specific domain platform, while on an upper layer, the end-to-end orchestration of the ACCA service, is handled by the Service Orchestrator (SO) and the Multi-domain Orchestrator (MDO), two different components that form the End-to-End service orchestration Platform. In particular, for realizing the crossborder service orchestration, the SO and MDO components rely on the networks slicing concept and implement a network slice (NS) data-model based on 3GPP TS 28.541 v16.1.0 specification [18], where each network slice sub-net (NSS), composing the end-to-end NS, hosts a nested service running in a single domain.
Specifically, the SO, based on the Vertical Slicer prototype [19] [20], manages the decomposition of the ACCA end-to-end service into service components that must be deployed in single domains. The requirements and the structure of the endto-end NS are identified by the SO according to an applicationdependent service logic, customized for the ACCA service. Starting from the identified requirements, the SO builds a "composite" Network Slice Template (NST), as a composition of NSS to be instantiated in specific locations for covering all the required geographical areas. The MDO, coordinates the provisioning of the composite NS across the edge and core locations, controlled by the Neutral Hosting and IXP platforms that expose different types of interfaces II. The MDO, based on the TIMEO framework [21] [22], provides functionalities for adapting and on-boarding NSTs that correspond to the NSSes to be instantiated in each underlying domain. Moreover, the MDO coordinates the life-cycle management of the end-to-end NS across the underlying domains' platforms (including the coordination of the involved MEC applications and services).
When a request for instantiating the ACCA end-to-end service is performed, the SO, taking into account the specific service logic and requirements, builds the proper NST representing the end-to-end service. Then, the instantiation request is forwarded to the MDO, which takes care of coordinating and provisioning the end-to-end network slice. The MDO processes the NST determining in which domain each NSS has to be deployed and activates the proper driver. In particular, the MDO provides drivers for both the Neutral Hosting and the IXP platforms. With reference to 5, in the case of the Crossborder NSS, the IXP driver takes care of i) translating the nested NST (referred by the NSS) into the specific format expected at the IXP North-bound Interface (i.e. SONATA) ii) on-boarding the resulting template iii) instantiating the NSS. Similarly, the Neutral Hosting driver handles the provisioning of the two NSS comprising the vEPC and the ACCA Edge Geoservice component; the NSSes are created as collections of already available partitioned resources (i.e. chunks) reserved to the two vMNOs. On top of these reserved resources, the service components are instatiated through the NFVO (i.e. OpenSourceMANO). Once all the NSS composing the endto-end Network Slice are successfully deployed, the MDO handles the Life Cycle Management of the end-to-end service.

VI. CONCLUSION
This paper has presented the ACCA cross-border deployment in the Barcelona small-scale trial site of the 5GCroCo project, depicting the deployed infrastructure, the architecture of ACCA use case, the deployed user story, and the proposed cross-border orchestration architecture for end-to-end services. The considered ACCA use case contemplates a dangerous situation across the border of two countries. Therefore, the Barcelona trial site emulates a cross-border scenario, with different virtual MNOs which operate MECs and LTE small cells, with an IXP and public cloud in between them.

ACKNOWLEDGMENT
This work is part of the 5GCroCo project that has received funding from the European Union's Horizon 2020 research and innovation program under grant agreement No825050. It has also received support from RTI2018-099178-B-I00. EU-RECOM acknowledges the support of its industrial members, namely, Orange, SAP, BMW Group, IABG, and Symantec.