Field Trial of Programmable L3 VPN Service Deployment Using SDN-Based Multi-domain Service Provisioning over IP/Optical Networks

Network operators cannot change their foot-print to upgrade the entire network to support SDN from scratch. Hence, network operators adapted the original SDN concept into a hybrid SDN approach to have a pragmatic, evolutionary, and economically viable solution. This article tests an SDN architecture based on a hierarchical structure of a Software-Defined Transport Networking (SDTN) controller, which deals with the end-to-end service provision and topology collection on top of a set of SDN controllers for each technological domain. The authors demonstrate L3/L2 VPN service deployment using an underlay multi-domain IP/Optical network in a field trial. The approach allows a service provider to migrate brownfield scenarios into SDN domains using programmatic interfaces.


IntroductIon
Software-Defined Networking (SDN) has emerged as the new reference paradigm to promote network automation and programmability. It has promoted the idea of a radical transformation in all aspects of service delivery, network, and traffic management, because end-to-end automatic service provision, automated monitoring, fast issue detection, and event-based decision taking are mandatory functionalities to offer a high-quality customer experience [1].
Conceptually, SDN allows a full decoupling between the control and forwarding planes on Physical Network Functions (PNFs). This concept allows the centralization of complex processing tasks and enables integrating white boxes (i.e., smaller equipment, high port density, low processing capacity, made with generic hardware, and lower production cost) in the network access layers. However, this promise is still not a reality on the service provider networks. Although the term SDN seems new, it has already been more than twelve (12) years since a Stanford team defined OpenFlow and founded NICIRA (NICIRA was the first company to develop a commercial SDN controller (NOX)). Nowadays, despite millionaire investments and several SDN controller solutions available [2,3] on the market, almost no service provider has a full operative SDN network controller deployed. Some of the main barriers found by service providers until now are: • There are still many dependencies on manually executing tasks. • Network control tasks cannot be fully centralized. • The stack of protocols deployed in the network is very complicated. The knowledge that network operators require to solve problems continues to be very specific. • Confidence in automation solutions is not very high. • Many networks have grown via company acquisitions. In many cases internally they operate as independent carriers. • Standardization is not complete, generating a lack of full interoperability between vendors. Several new approaches have been gradually released in recent years to adopt SDN as a hybrid solution in brownfield scenarios. Some of the most promising methods were defined using agile methodologies, as software development and IT operations (DevOps) solutions [4].
These agile methodologies usually take a small network task and solve it using a programmatic approach. It allows the simple and more frequent integration of new functionalities. Network provisioning is one of the recurrent tasks selected for agile automation, because it is a diary job that includes the usage of well-defined templates to deploy MPLS-based services on the network.
The Layer 3 Virtual Private Network (VPN) service defined in RFC 4364 [5] provides a multi-point, routed service to a customer over an IP/MPLS core. L3VPNs are widely used in carrier-grade networks to deploy 3G/4G, fixed, and enterprise services. Traffic policies can be applied in these services to reuse the same transport network, and it also makes it feasible to combine access technologies over an MPLS core. On the other hand, L2 services belong to the class of IP virtual leased line (VLL) services or virtual private LAN (VPLS) services [6], which are a fundamental part of the service portfolio offered by service providers.
This work proves the viability of the implementations of programmable network interfaces for the deployment of L2/L3 services in IP over optical networks using common standard models and protocols. A fi eld trial including multiple network controllers (two for IP/MPLS and two for DWDM) and a real-commercial-multivendor-operative network was installed by Telefonica in their premises in Colombia to validate the concept.
The article is structured as follows. The following section provides a review of the state of the art on the provisioning of automated L2/L3 VPN services. We then describe the SDN reference architecture and detail the principles of network programmability, including the concepts of YANG and used protocols. Then, we detail the test architecture, including commercial network controllers and the results obtained in this implementation. Finally, conclusions and perspectives are drawn.

stAte of the Art
The IP service models are YANG modules defi ned to support the creation, deletion, and retrieval of L3/L2VPN services and the IP/MPLS network topology collection. The models used in this work are abstract models that technologically describe the network requirements and are RFCs or working-group-drafts inside the IETF.

l3vpn servIce model (l3sm)
Multi-Protocol Label Switching (MPLS) Virtual Private Networks (VPNs) have seen an unparalleled increasing adoption in recent years. Although their flexibility as transport technology and their effectiveness for traffic engineering are well recognized, VPNs are hard to set up, operate, and manage, due to the complexity of the network, the protocols involved, and possible multi-vendor parameter discrepancies.
Some service models have been defi ned and standardized until now to support the L3VPN creation. L3SM [7] is a Customer Service Model that describes an L3VPN service's configuration parameters based on customer requirements. The L3SM model keeps some commercial customer information as the "customer-site-location." This model provides an abstracted view of the Layer 3 IP VPN service confi guration components. However, it is mainly focused on the provider edge (PE) VPNs and does not have specifi c parameters to configure the different network elements in multi-vendor scenarios nor has the possibility to bind the services with specific transport options inside the MPLS network.
L3NM [8] is a complementary network model of the L3SM. It differs from the L3SM because it is entirely network-centric and groups all the provider edge routers (PE) configurable parameters. Network controllers can expose the L3NM to manage and control the VPN service confi guration in multi-domain scenarios. It contains valuable information to operate the service, such as each network element's identifier in the IP/MPLS domain (NE-ID) and the interface identifier of each customer access. It includes resources such as Route Targets (RTs) and Route Distinguishers (RDs).
L3NM is populated based on the services request (f.i. L3SM query) and an SDNc includes network-specific information in the message sent to the domain controllers like transport LSP binding, routing profiles, or encryption profile. It assigns logical resources such as route targets or route distinguishers to keep the data synchronized between domains. Figure 1 shows the structure of the L3NM YANG data model, where the main container (VPN service) is used to group the information of the VPN-node (VRFs) and the VPN-network accesses (Interfaces).

l2vpn servIce model (l2sm)
Similarly, the IETF has standardized YANG models for the L3VPNs; there is already a standard for the L2 services. The model is named L2SM [9].
The L2SM is a customer-centric model; it has two main sets of parameters, the VPN service and the site information. Figure 1 depicts the main components of the L3NM and the L2SM as a tree structure. The VPN service contains all the technological parameters of the services that are going to be deployed, for example, service type or service topology. The "site" includes all the customer information, like the "customer-location" and the access parameters between the Customer-Edge (CE) and the Provider-Edge (PE).
This model lacks some specifi c network parameters; those need to be stored or derived by the network controller to deploy the network device's confi guration. For futures implementations, additional work to complement the current standard can be proposed.
sdn ArchItecture iFUSION is a reference model architecture defined by Telefonica to reinforce the network automation and programmability in a service provider environment. iFUSION is hierarchical, with specific domain controllers per technological domain (IP/MPLS, microwave, optical) and a hierarchical controller to provide real-time control of multi-layer and multi-domain transport network resources. The iFUSION main principles include the use of: • Standard interfaces based on RESTCONF/ YANG [10] for the communication between controllers and NETCONF/YANG [11] to confi gure the network elements. • YANG data models based on the latest developments in the standards-development organizations (SDOs): IETF, ONF and Open-Confi g.

FIGURE .
L2NM and L3 Data models structures. The hierarchy represents a container in the YANG model. The lines represents cross references between the containers.
The main elements of the SDN iFUSIONarchitecture are the following.
DN Domain: This is a set of network elements under the supervision of the same SDN controller. There are several possible levels in the decoupling of the control and data planes. The preferred level of decoupling depends on the network technology. For example, network element runs the distributed protocols (e.g., IS-IS TE, RSVP-TE) in MPLS, and the controller only needs to confi gure it.
SDN Domain Controller (SDNc): This element is in charge of a set of network elements. It has standard southbound interfaces that depend on the technology, but not in the equipment vendor, to communicate with the network elements. It also has a northbound interface to communicate with the SDTN controller. For some use cases, like performance management, the authors initially identifi ed the necessity to connect the OSS layer with the SDNc directly.
Software Defi ned Transport Network (SDTN) Controller: In case several SDN domains are needed, the SDTN controller is in charge of providing/composing services through several domains.
The iFUSIONarchitecture was designed as a hierarchical model where a dedicated SDN Domain controller controls each network segment. Due to its complexity, the transport network is divided into three main technology domains: IP, optical for transmission, and microwave. The proposed architecture can be shown in terms of components and relationships among them, as depicted in Fig. 2.
The SDTN controller is responsible for orchestrating the respective SDNcs within the trans-port segment (IP, optical and MW) through the domain controllers' NBI, providing an end-to-end transport network vision. The SDTN controller aggregates demands from the management and services layer exposing a unified NBI which should provide resource confi guration abstraction and technology agnostic service defi nition.

Ip domAIn
Traditionally IP networks are deployed following a regional model mixing equipment from diff erent vendors. The IP boxes are interoperable at both the data and control plane levels (e.g., routing protocols such as IS-IS, OSPF, or BGP). Due to scalability and resiliency reasons, the IP administrators divide the whole network into IP domains, so the routing and control protocols are the same in an administrative area.
The goal SDN solution for the IP segment is composed of a single IP SDNc, whose goal is to configure all the IP network elements. The target SBI for vendor-agnostic device configuration shall be compliant with the NETCONF standard protocol. OpenConfi g started as a collaborative work between Google, some vendors and service providers, and now it is an industrial reference for device-specifi c confi guration purposes. The philosophy behind OpenConfi g is to add functionality to the device models based on the operator's needs to avoid complex models which can not be implemented in real networks. Due to its maturity level, the set of device confi guration data models defi ned and used in iFU-SION are OpenConfi g.
It is expected that the IP SDNc will assume the control/management of: [12] architecture has two control layers. The fi rst one having the SDTN controller with the end-to-end view of the network. Second control layer includes the SDN Domain controllers. Those controllers interact with the network devices (Fusion Network). The set of use cases Tested in the POC are listed.
The IP SDNc will be the main entry point to the network elements, to avoid overloading the elements with external requests and providing a full and coherent network view. The NBI of the controller in iFUSION follows IETF YANG models and they are implemented on RESTCONF with JSON encoding. The IETF YANG for the NBI was done based on the support of the supplier to the models. The IETF adoption is wider than any other alternative for the NBI.

optIcAl domAIn
Optical transport networks from different system vendors are deployed on a regional basis, either for technology redundancy, due to different optical performance requirements (metro vs. longhaul), or simply for commercial reasons.
Without line-side interoperability of the different optical transceivers and reconfigurable optical add-drop multiplexers (ROADMs), there is not a competitive advantage on a uniform configuration interface of the optical devices, since they cannot be mixed in a multi-vendor scenario, due to the fact that both line systems and transceivers must be from the same vendor.
With this in mind, in the short term, optical SDNcs are expected to provide network programmability and interoperability toward the upper layers (multi-layer) and between vendors (multi-domain, multi-vendor) through the support of standard NBIs (i.e., coordination will be provided by upper layer hierarchical SDTN controllers). This short term approach will enable the setup and tear down of connections in optical channels (OCh and ODU layers), the discovery of the network resources to compose a layered uniform view based on the OTN hierarchy, and the monitoring of the optical network in a vendor agnostic fashion thanks to the utilization of ONF Transport API (T-API) 2.1 [13], having been tested in several proof of concepts [14]. Please note that in this transport domain, T-API + RESTCONF is also proposed to be the NBI of the hierarchical SDTN controller toward service B/OSS layers.
In the medium and long term, the direct programmability of the components can have interest in point-to-point, metro and regional scenarios, where disaggregation of optical transceivers and line side components can play an important role. In this line, OpenROADM [15] and OpenConfig projects have already defined device configuration models for transponders and open line systems.

IntegrAtIon of sdtn controller In the overAll operAtor's systems ArchItecture
One of the main reasons for deploying an SDTN controller is service automation. It facilitates that manual services, and network configurations become automated and available through its NBIs, enabling the network automation progressively. As the main design principle, the abstraction level provided by the SDTN controller can be different based on the needs of consumers.
Thus, the information exported through the NBI toward OSS and other platforms will cover several functional areas with several levels of abstraction: • Network topology • Service provisioning • Performance management • Network planning and design • Fault management.
Network topology and service provisioning are tested in this article, with the full set of use cases tested as described in Fig. 2. Progressively, the SDTN controller will include the rest of the functionalities (marked as planned). On its SBI, the SDTN controller will integrate with the domain controllers. Each SDNc shall expose vendor-agnostic network-level programmability and resource discovery functionalities, so the SDTN controller will be able to perform the correct data integration and functionalities exposure.

multI-domAIn Ip l3vpn provIsIonIng
The L3VPNs services are not exclusive of single domain implementations; multi-domain IP L3VPN is a common requirement in the service providers. Multi-domain services include: multiple-AS, multiple-IGPs, or multiple-vendor segmentations. Based on this, a set of interactions with more than one IP SDNc may be required to accomplish the service provision process.
The scope of this work includes two IP domains connected by a common core; the IP domains were part of a different IGP process, so each network has its own IP SDNc. Each of the controllers has implemented the IETF L3NM model described earlier to support the service creation requirements. We have proved the SDTN controller as a network orchestrator to create a multi-domain L3VPN, delegating the required provision parameters to each domain controller and after exposing a unified view of it.
The parameter delegation done by the SDTN controller had the following four steps per domain.
Create Site: Indicates the place where the services end-points are located. It includes magement information and service descriptions.
Create Bearer: Indicates the NE and the port assigned to the service.
Create VPN-Node: Indicates the VRF deployment and all its attributes on the PE. It includes the import/export policies.
Create Site Network Access: Refers to the CE-PE connectivity attributes, SSuch as routing protocol exchanged, IP addressing or ethernet encapsulation. Figure 3 depicts the corresponding workflow to create the L3VPN. Each step in the figure matches the sequence previously described. A similar workflow was used for all the RESTCONF operations like Create, Read, Update and Delete.

fIeld trIAl envIronment for IfusIon sdtn demonstrAtIon
In this work, we have deployed the iFUSIONarchitecture in a Telefonica Colombia field trial environment. The field trial includes not just the iFUSIONSDN control layer, but also the network elements described later.

sdn control lAyer
The SDN control layer architecture was built upon the reference design guidelines described above.
The key elements of the control layer were: • Infinera Trascend Maestro, acting as SDTN controller • 2  IP SDNcs one for each IP domain • 2  Optical SDNcs one for each optical domain. SDNcs communicate with NEs via NETCONF/ YANG and RESTCONF/YANG with the SDTN controller using the data communication network (DCN). The optical and IP SDNcs are commercial products from the network element vendors.

netWorK elements
The set-up for this field trial uses a full network with all the hierarchical levels (HL) that compose a service provider real environment. In our notation and architecture, the IP/MPLS-based network is comprised of five (5) HLs with the following responsibilities.
HL1: Core PE-Routers acts as toll gates for the service provider's interconnection with other carriers and using eBGP logical structure for publishing public IPv4/IPv6 prefi xes.
HL2: The Core P-Router is responsible for the transportation of traffic between main cities and metropolitan areas, sending/receiving traffi c to HL1 interconnections from/to the international Internet. Two (2) vendor B routers were used as HL2. The core routers run in an isolated IGP domain.
HL3: PE-Routers are responsible for the aggregation of traffic from metropolitan and regional areas for both fixed and mobile services. Moreover, some HL3s provide connectivity to 4G/5G platforms (EPC, packet core, and so on). Four (4) routers were used as HL3s, distributed in two IGP domains (two for each vendor).
HL4: HL4s collect traffic from fixed access networks (DSLAM/CMTS/OLT) in metropolitan areas and high capacity corporate services. Five (5) routers were used as HL4 devices: three (3) for vendor A and (2) for vendor B. Each PE has connectivity to both HL3 routers of its island.
HL5: Cell site routers connect corporations, enterprises, small businesses and mobile terminal nodes (BTS, NodeB, eNodeB) in remote areas. Formerly known as cell site routers in mobile service providers, they evolved and converged to serve multiple fixed plus mobile segments. One HL5 was used in these tests from vendor A.
The IP/MPLS network was built using seamless MPLS option-C. Clusters and rings organized the network-IP cluster group devices of a specific vendor. As a seamless MPLS, underlay signaling requires that every origin PE-Router (HL4) from a cluster can establish an end-to-end path to the destination PE-Router (HL4) even if the destination belongs to a diff erent cluster.
Thus, the HL3 routers from each region establish an eBGP session against the Core-Routers (HL2). This session exports the Router-ID plus label information of all the routers in the region using BGP label unicast (LU); there is another eBGP session between the HL3 of the region and the core router-refl ectors to export the VPNv4 routes from each VPN service. This eBGP session requires a mandatory Next-Hop-Unchanged confi guration to avoid network loops or misconfi gured paths.
Two-vendor WDM infrastructures transported the IP/MPLS links of vendor B HL2s and vendor A HL3s. The optical transport infrastructure was built as two independent metropolitan optical networks. We used four (4) nodes in each optical network with 100Gb and 10Gb lambdas for these experiments.

multI-domAIn Ip l3vpn provIsIonIng
This section presents the test results for hierarchical SDTN controller integration with the controllers in the IP/optical domains. Two types of tests have been done to demonstrate orchestration functionalities in the multi-layer/multi-domain/ multi-vendor network environment, as depicted in Fig. 4. In a fi rst stage, each SDNc was validated individually. RESTCONF-based queries were sent toward each of the SDNCs to change and retrieve the network information and validate each implementation's compliance. This initial certifi cation aimed to make functional tests over each solution and speed-up the entire solution's fi nal integration. The scalability and effi ciency of the SDTN Controller solution is limited to the SDNc performance.  Once each domain controller was independently certified during the individual validation process, the end-to-end integration was done using multi-domain functional tests. In that case, the Transcend Meastro GUI was used to orchestrate the entire service lifecycle. For example, to create a service between two domains, the SDTN controller provided a graphical template to fill the service creation parameters. The parameters depicted in Fig. 5 include the user-defined configuration requirements and the auto-assigned parameters by the SDTN controller. In both cases, the SDTN controller stored all the service resources, and it continuously monitors the network state to keep the set of values synchronized. The workfl ow used by the SDTN is depicted in Fig. 3. Each time a user created a service, a set of request calls was instantiated between the SDTN controller and the corresponding SDNc. Finally, once the SDNc confi rms the service creation to the SDTN controller, the Transcend Maestro provided three visualization options: service details, geographic view, and layered view, as depicted in Fig. 5: • The VPN service details including the service name, the topology and end-points. • The service route in the topology view. This view includes the full path including the IP and optical devices comprised in the service. • Layered view of the service. This view splits the service connections between layers, so the IP links connection is in the top. The Ethernet connections between routers are in the second layer, and physical plus optical layers are decoupled in this hierarchical structure. The confi guration of the IP L3VPN service in the network elements was verifi ed by using their command line interface as well as the IPSDNc GUI. To validate the data plane, a traffi c generator was used in site to introduce traffi c on both ends of the network and tested the multi-domain L3VPN service. Figure 6 shows the traffi c statistics as seen on the command-line interface of the PE routers. In this fi gure, the two interfaces connected to the VPN services are selected, and their traffi c counters are showed. Figure 6 shows the bandwidth utilization of 93.96 percent and 88 percent on each end of the VPN service.

conclusIons And future WorK
It is fundamental for SDN adoption in service provider networks to defi ne the data models and protocols used across the components. Historically, integration using proprietary interfaces delay the introduction of programmability and automation and have a high economic cost. In this work, the RESTCONF with IETF YANG models was tested for the integration between the SDTN controller and the IP SDNcs. This work shows, for the first time, a standardized L3NM API to provision and retrieve services to a multi-vendor underlay network. Moreover, this article presents an infrastructure using network elements in the production network of Telefonica Colombia.
The tests included the provision of L3VPN using IETF YANG data models and the topology collection. The SDTN controller orchestrated the service creation assigning the resources to each domain controller. During service retrieval, the SDTN controller composed the services based on the information exposed by the domain controllers. For additional tasks, there are still gaps to cover all the IP/MPLS network operation requirements. Following is a brief summary of the issues faced during the integration, as reference points to future work: • Connectivity, latency and internal processing times between the HCO and some of the SDNcs can impact the integration and result in miscommunications creating the timeout of SDN transport protocols, ie. RESTCONF and NETCONF. • Ghost objects which would not be completely deleted in the controllers can lead to misunderstand in the topology construction. • The unsolicited data retrieved by a lack of standardization or a bias in the implementation of the standards can lead to uncompleted transactions or loops in execution tasks. • Absence of data in the SDN domain controllers for an automatic inter-domain link discovery. • Differences in the RESTCONF/YANG implementations on the SDNc. Even if the YANG models were the same, the parameter translation between NBI and SBI can restrict some configurable parameters (i.e max length size of a description field) and may generate implementation differences. This would result in possible errors during the execution of the creation process. • Diff erences in RESTCONF/YANG error handling. A set of well defined error codes is mandatory in the hierarchical architecture. The APIs designed in this work demonstrate the viability of the iFUSIONarchitecture with an SDTN controller orchestrating an underlay multi-vendor, multi-layer, and multi-domain network environment. The iFUSIONAPIs allow any service provider to migrate brownfi eld scenarios into SDN-ready domains using programmatic interfaces.

future WorK
As future work, the hybrid SDN deployment done until now must be complemented with an integration between the NBI exposed by the SDTN controller and the OSS applications ecosystem. The OSS ecosystem can include, for example, strategic and tactical planning applications, able to support year by year demand management and planning tasks done within the organization. A common interface defined and available for these tasks would allow the OSS system providers to focus on the quality of the applications developed, forgetting the complexity of the network management. Economically, it will generate direct reductions in the applications integration time.

AcKnoWledgments
This work has been supported by Telefonica I+D as part of the Fusion, iFUSION, and OpenFusion projects, and partially funded by the EC 5GPPP TeraFlow (101015857) and MINECO AURORAS (RTI2018-099178-B-I00  oScar gonzález de dioS received his M.S. degree in telecommunications engineering and Ph.D. degree (Hons.) from the University of Valladolid, Spain. He has 19 years of experience at Telefonica I+D, where he has been involved in a number of European research and development projects (recently, STRON-GEST, ONE, IDEALIST, and Metro-Haul). He has coauthored over 100 research papers and 10 IETF RFCs. He is currently the head of SDN Deployments for Transport Networks, Telefonica Global CTIO. His main research interests include photonic networks, flexi-grid, interdomain routing, PCE, automatic network configuration, end-to-end MPLS, performance of transport protocols, and SDN. edward echeVerry is Head of Transport (IP and optical network) at Telefonica Colombia. He is an electronic engineer and telecommunications specialist with 15+ years of experience in the field. He has high technical skills and advanced experience in the design, planning and implementation of 3G/4G mobile, IP/MPLS and new generation optical networks, as well as in the management and deployment of projects and architectures.
Juan pedro Fernández-palacioS giménez received the M.S. in telecommunications engineering from Polytechnic University of Valencia in 2000. In September of 2000 he joined Telefonica I+D where his research activities where focused on the design of new data and control plane architectures for IP over optical networks. He is the author of six patents filed in Europe and the U.S. and more than 70 publications in conferences and journals. He was the coordinator of two European research projects on optical transport networks (MAINS and IDEALIST). Since June 2017 he has been leading the Integrated Transport Centre, a global organization in Telefonica in charge of defining the strategic network planning and technology for IP, DWDM, MW and satellite networks.
Janne KarVonen received a M.Sc.Tech. degree in 1993 from Helsinki University of Technology. He is a senior software architect at Infinera Corporation. He works in the Systems Architecture Group, focusing on software defined networking technologies and IP/MPLS network management systems. He has over 30 years of experience in software engineering and more than 20 years of experience in telecommunications network management systems, covering both optical and packet based technologies. His research interests include SDN technologies for multi-layer, multi-domain and multi-vendor networks, with a special focus on SDN API technologies, multi-layer path computation algorithms and utilization of Machine Learning in SDN networks.
Jutta Kemppainen received her Master of Science (Tech.) in 1999 from Helsinki University of Technology (now part of Aalto University). She works as senior principal product manager at Infinera, managing Infinera multi-layer, multi-domain and multi-vendor transport network automation solutions. She has over 20 years of experience in telecommunications software automation products and has been concentrating on Software-Defined Networking (SDN)-based solutions for the last 7+ years. During this time Kemppainen has been co-operating with 70+ network providers, including many of the largest and technically most advanced in the industry, in designing and defining requirements for practical transport network automation solutions. natalia iSaBel maya perFetti is an electronics and telecommunications engineer. She graduated from the Universidad del Cauca, Colombia in 2018, and since then she has been working as a network planning engineer in Infinera Colombia, where she supports different tasks related to DWDM network planning. For the last year she has also been responsible for the testing of the SDTN solution in the field trial environment.