Overarching Control of Flexi Grid Optical Networks: Interworking of GMPLS and OpenFlow Domains

Optical transport networks provide transport, multiplexing, routing, management, supervision, and survivability of optical channels. Within a flexible dense wavelength division multiplexing grid, the optical spectrum can be allocated in multiples of a width granularity, depending on the client signal rate and modulation format. A control plane (CP) can be used for efficient and dynamic provisioning and recovery of flexi-grid connections. Two main CP architectures coexist, with common functions like addressing, automatic topology discovery, network abstraction, path computation, and connection provisioning: a distributed generalized multiprotocol label switching CP (with optional path computation element, PCE path computation and instantiation/modification) and a CP based on software-defined networking, with a logically centralized controller and an open protocol, such as the OpenFlow protocol. Both architectures have their own strengths and weaknesses, and are being extended to address the new requirements associated with the aforementioned emerging optical technologies, such as flexible spectrum allocation, efficient corouted connection setup, and configuration of related optical parameters. However, new use cases such as remote data center interconnection highlight the need for multidomain service provisioning, and heterogeneous CP interworking, potentially requiring an overarching control. Different alternatives, with varying degrees of integration and flexibility, are available: straightforward approaches characterized by the adaptation of one control model to the other or more advanced interworking requiring the definition of common models (e.g., a subset of attributes for network elements) and of coordination and orchestration functions. This paper discusses the main relevant interworking architectures and presents a selected set of use cases and proof-of-concepts.


I. INTRODUCTION
A CCORDING to [1], optical transport networks allow transport, multiplexing, routing, management, supervision and survivability of optical channels. In the flexible dense wavelength division multiplexing (DWDM) grid [2], nominal central frequencies are defined using a spacing of 6.25 GHz, and the required optical spectrum or frequency slot can be dynamically and adaptively allocated, in multiples of a frequency slot width granularity (12.5 GHz), determined by the signal data rate and modulation format. A media channel is defined as a media association that represents both the topology (i.e., path through the media) and the resource (frequency slot) that it occupies. As a topological construct, it represents a (effective) frequency slot supported by a concatenation of media elements.
To enable a dynamic and intelligent flexi-grid optical network, a control plane (CP) is deployed for the dynamic provisioning and recovery of media channels (i.e., optical connections). Moreover, emerging optical technologies bring new requirements such as flexible spectrum allocation, efficient co-routed connection setup and configuration of related optical parameters, which must be fulfilled by CP architectures. Two main CP architectures coexist, enabling common functions like addressing, automatic topology discovery, network abstraction, path computation and connection provisioning. One is based on the distributed generalized multi protocol label switching (GMPLS) CP (with optional path computation element (PCE) [3] path computation, by means of a Path Computation Element communications Protocol or PCEP), the other one relies on software defined networking (SDN) principles, with a logically centralized controller and an open protocol, such as OpenFlow (OF) [4]. New use cases such as data center interconnection highlight the need for multi-domain (defined e.g., administratively, by topology visibility constraints or by vendor boundaries) service provisioning, and heterogeneous CP interworking, requiring an overarching control.
In light of this, in this paper, we present interworking architectures of GMPLS and OF domains, validating their feasibility and efficiency through experimental evaluation and proof-ofconcept development. The rest of this paper is organized as follows: in Sections II and III, we summarize the main GMPLS and OF support for flexi-grid networks respectively. The interworking of GMPLS and OF is detailed in Section IV. We report the proof-of-concepts and experimental validation in Section V and conclude this paper by summarizing our contributions in Section VI.
II. GMPLS SUPPORT FOR FLEXI-GRID NETWORKS GMPLS [5] uses established procedures and protocols for topology dissemination and distributed signalling to set up and release optical connections or label switched paths (LSPs), with labels locally representing the media channel and its associated switched frequency slot. From a standardization perspective, support for flexi-grid networks is still in early stages in the Internet Engineering Task Force (IETF), scoped to the control of network media channels transports a single Optical Tributary Signal [6]. The Open Shortest Path First with Traffic Engineering extensions (OSPF-TE) routing protocol [7] is currently being extended to disseminate, via link state advertisements (LSAs), node and link attributes such as the optical spectrum availability, which are collected by CP entities such as GMPLS routing controllers or PCEs in their traffic engineering database (TED). Additionally, the Border Gateway Protocol extensions for Traffic Engineering (BGP-LS) [8] can also be used to support a mechanism by which links state and traffic engineering information can be collected from networks and shared with external components. The resource reservation protocol (RSVP-TE) is extended with a new switching type and new types for both the sender descriptor traffic specification and for the flow descriptor objects, conveying the requested and allocated slot widths. A GMPLS CP can be augmented with an active stateful PCE (AS-PCE) [9], in which the PCE manages the database of active connections and may, autonomously, suggest their re-routing and eventually instantiate connections.

III. OF SUPPORT FOR FLEXI-GRID NETWORKS
A logically centralized CP is attracting attention, given its potential integration with operators' operations and business support systems and software customization. The (now historical) OpenFlow protocol (OFP) v1.0 circuit switching addendum [10] had basic support for circuit-switched fixed-grid optical networks, and OFP 1.4 [4] has improved it. Recently, a significant number of extensions are addressing flexi-grid optical networks to dynamically create connections while enabling the direct control of bandwidth-variable transceivers. The first extended OFbased CP [11] focused on feasibility validation. Further studies have investigated the control of flexible transmitters [12] or integration with optical performance monitors [13]. From a standardization point of view, the optical transport working group of the Open Networking Foundation is working on the definition of the SDN architecture and requirements, and OFP extensions are still a work in progress. Arguably, one of the reasons behind the success of OF is the logical switch abstraction, hiding vendor-specific hardware details, and mapping high level instructions of the protocol, which mitigates inter-operability issues commonly found in multi-vendor deployments.

IV. INTERWORKING OF GMPLS AND OF
The problem of interworking between GMPLS and OF domains can be addressed by a wide range of architectures and protocols. Such alternatives may rely on combining existing functional entities and/or on mapping and translating different protocols and information objects. Widely speaking, the different alternatives show varying degrees of integration and flexibility. Straightforward interworking approaches are commonly characterized by the adaptation of one control model into the other, whereas more advanced interworking solutions require or rely on the definition of common element and network models (e.g. a subset of attributes) and the definition and use of coordination and orchestration functions, term that refers to the automated configuration, coordination and management of complex systems, commonly with transactional semantics with rollback capabilities.
Scalable solutions are expected to be based on hybrid architectures (combining centralized and distributed elements) with synchronization mechanisms enabling cooperative routing and path provisioning and on the concept of network (e.g. topology) abstraction, which refers to the selection of an entity relevant characteristics, based on targeted functionality and scalability, such as selected network topological information. Abstraction thus refers to the synthesizing of reported TE attribute information for a set of elements or networks, and provides a reachability and connectivity matrix that can be used by external entities. The abstraction may also be subject to policy and management choices. The simplest form of abstraction corresponds to the case where a given network (sub)domain is represented as a node, assuming complete connectivity.
In the following subsections, we discuss the selected adaptation and orchestration models, and in Section V we detail the selected implemented use cases and proof-of-concepts.

1) GMPLS CP with OFP as CCI:
Although not an interworking model in itself, it is possible to use OFP as the connection controller interface (CCI) within the automatically switched optical networks (ASON) architecture [14], considering the OFP exclusively as a protocol/interface (regardless of the network control architecture in which it was defined). The CCI defines the interface between the ASON signaling element, i.e. optical connection controller and the transport network element, and enables such CP entity to program the switching and forwarding (i.e., cross-connection) associated to a connection (cf., Fig. 1). The direct advantage of this is that an OF enabled transport node can thus be controlled by a third party GMPLS controller, relying on an open and standard protocol.
The applicability of GMPLS for ASON RFC [15] defines a transport node (i.e., a network element in the data plane) as a logical network device that is capable of originating and/or terminating of a data flow and/or switching it on the route to its destination. Likewise, a controller is defined as a logical entity that models all CP intelligence (routing, traffic engineering, signaling, path computation, etc.). It is noted that a single controller can manage one or more transport nodes and that separate functions may be hosted at distinct sites and hence could be considered as separate logical entities. Such entities are referred to, for example, as the routing controller, the signaling controller, etc.
It is thus possible to draw a limited analogy between a SDN controller and a GMPLS controller that is controlling multiple transport nodes, in the sense that the GMPLS controller has basic SDN capabilities (notably the control and data plane separation) with limited application layer and programming interfaces, driven by signalling.

2) GMPLS CP With OF Island:
A second simple adaptation model is based on integrating an OF controlled domain (i.e., an OF island with a unique controller) to become part of a wider GMPLS network by requiring that such an OF controller implements (a subset of) the GMPLS protocols. In particular, the OF controller is responsible for abstracting its controlled OF island (e.g., a single, abstract transport node).
If the OF controller implements only the RSVP-TE signaling protocol, the interface between external GMPLS controllers and the OF controller can be assimilated to an ASON user-tonetwork interface (UNI), allowing the establishment and release of connections. Adding the OSPF-TE routing protocol allows the OF controller a finer control and flexibility in the dissemination of its abstracted topology and attributes to neighboring controllers. Multiple LSAs can be generated by the OF controller, eventually including all the transport nodes (OF datapath ids). The mapping of the TE attributes is managed by the OF controller, defined by local policy and configuration.
From a signaling perspective, the OF controller needs to map RSVP-TE objects within, notably, RSVP-TE Path/Resv messages, and translate incoming and outgoing TE interfaces into OF connection endpoints, and to provision the supporting connections. The basic architecture is shown in Fig. 2, and the proof-of-concept for flexi-grid GMPLS UNI over an OF controlled optical network detailed in Section V-B. Other use cases supported by this model relate to, for example, support progressive migration of e.g. vendor domain islands.

3) Virtual OF Node Abstracting a GMPLS Domain:
The last considered adaptation model is based on the concept that a GM-PLS domain can be abstracted as a logical OF node [16], with PF ports corresponding to GMPLS client interfaces, introducing  From an implementation perspective, an OF agent is responsible for the abstraction, and under the control of an OF controller. The agent acts as a proxy to the underlying GMPLS domain and maps a request for a virtual cross-connect (circuit flow addition) into the establishment of a GMPLS LSP. The agent uses the underlying GMPLS provisioning interface (usually vendor dependant) or, more recently, delegated to an AS-PCE with instantiation capabilities [9], [17], [18].
From a macroscopic perspective, the concept of a virtual OF node can be generalized as follows: a GMPLS domain can be abstracted, and augmented with an adaptation layer which provides a single interfacing point to upper layers (i.e., a so called northbound interface), in view of the integration of the GMPLS domain in a wider SDN architecture, such as one defined using a hierarchy of controllers, as we will detail later.

B. Orchestration and Overarching Control
The adaptation models are limited by definition, either in terms of flexibility or scalability, and can be constrained by the involved CP protocols and interfaces. We are generically concerned with the orchestration of multiple technology domains or islands, focusing on architectures addressing such shortcomings  allowing an over-arching control, embracing heterogeneous domains. Key aspects of the architectures must enable a flexible degree of abstraction, uniform addressing to unambiguously identify resources (such as nodes, links, ports), and the adapting generic CP functions to the specific procedures and protocols of the supporting technology (see Fig. 4).

1) Single SDN Controller Model:
A single (logically centralized) controller with full visibility of the OF/GMPLS regions operates as a single control domain (see Fig. 5), while locally separating the technology domains for provisioning purposes. It uses dedicated provisioning interfaces at defined demarcation points, either directly programming cross-connects; requesting the establishment of segments to GMPLS boundary nodes using a provisioning interface, or delegating the task to an AS-PCE. This model is a straightforward deployment with e.g. an Open-DayLight (ODL) controller [19], which provides in its distribution implementations of OF, PCEP and BGP-LS, and is suitable for small domains.
A single controller may not scale or present confidentiality issues, requiring multiple-controllers and inter-controller protocols. Such architectures apply both to heterogeneous control domains (as in GMPLS/OF interworking) and to homogeneous multi-domain networks (i.e., with multiple GMPLS or OF domains).
2) Multiple SDN Controllers in Mesh: A set of controllers, interconnected in an arbitrary mesh, cooperate to provision endto-end services. Commonly, the mesh is implicit by the actual  (sub)domains connectivity. The controllers hide the internal control technology and synchronize state using the so-called East/West interfaces (see Fig. 6). The controllers manage detailed information of their own, local topology and connection databases, as well as abstracted views of the external domains. The East/West interfaces should support functions such as network topology abstraction, control adaptation, path computation and segment provisioning and their formal definition is still a topic of research, requiring and in-depth analysis of requirements and use cases, and could be based on separate instances of existing protocols (e.g. OSPF-TE, RSVP-TE, PCEP) or new ones.
It is worth noting that, abstracting from specific control protocols, this model corresponds to the known concept a hierarchical routing similar to the ASON routing or the private network-tonetwork interface routing protocol used in asynchronous transfer mode networks, inducing a hierarchy of aggregated topologies.

3) Multiple SDN Controllers in a Hierarchical Setting:
A logically centralized controller of controllers or orchestrator (also referred to as parent controller) handles the automation of connectivity provisioning at a higher, abstracted level, covering inter-domain aspects (cf. Fig. 7). Specific per-domain (child) controllers map the abstracted CP functions into the underlying CP technology by means of a multi domain northbound interface (MD-NBI), which generically refers to functions such as network topology abstraction, control adaptation, path computation and segment provisioning to support end-to-end services, as defined in Fig. 4. The generic hierarchical model can be refined showing the main CP functions related to the architecture and concrete protocols and interfaces. In Fig. 8, an orchestration controller orchestrates GMPLS and OF domains. The GMPLS may or may not deploy an AS-PCE (the advantage of doing so is having a single interfacing point and an open and standard protocol that supports part computation and network service provisioning and instantiation). Domain controllers export their abstracted topologies either via RESTful APIs [19], dedicated Interior Gateway Protocol (IGP) instances such as OSPF-TE, or using link state extensions to BGP [8]. Path computation combines local computations within domains with domain selections based on aggregated information, and each controller exports dedicated provisioning interfaces. Let us note that the figure corresponds to a functional decomposition and admits variants, such as the fact that the path computation function and the provisioning function may be integrated in a single AS-PCE, or that topology notifications are embedded into PCEP notification messages.
A concrete implementation of the architecture is presented in Section V-D, using a hierarchical AS-PCE.

V. PROOF-OF-CONCEPTS AND EXPERIMENTAL VALIDATION
This section aims at providing an overview of actual implementations of selected models, focusing on demonstrating the feasibility and obtaining basic performance indicators. It also shows that some models can be realized by combining or extending existing functions, protocols and interfaces.

A. OF CP With Virtual Node Abstraction
The implementation of the OF CP with virtual node abstraction is straightforward with an AS-PCE that abstracts the GM-PLS domain. The proof-of-concept extends the PCE with an OF agent that exposes a dedicated north bound interface, relying on OF extensions for flexi-grid optical networks (referred to as OF-enabled AS-PCE or OF-PCE). When both the OF-PCE and the GMPLS CP start to operate, the OF-PCE obtains its TED by inspection of the OSPF-TE IGP protocol.
At that point, the AS-PCE proceeds to set up an OF connection to its pre-configured controller IPv4 address (i.e., no automated discovery of the controller). After the TCP/SSL socket is established and both entities (OF controller and OF agent in the AS-PCE) get the event that the socket is connected, they start the OF handshake by sending an OFPT_HELLO message. If the respectively announced versions match, the controller will send an OFP_FEATURES_REQUEST message to the OF-PCE.
The OF-PCE is responsible for the virtual node abstraction, identifying from its TED the TE links that are exported to the OF CP (e.g. customer facing links). The virtualization is implemented by filling the CPORT (circuit port) structures of the OFP_FEATURES_REPLY message. Such data structures convey port identifiers and capabilities along with neighboring node identifier or datapath id. The OF-PCE starts numbering the ports and keeping track of the remote port identifiers as assigned by the peer datapath ids.
A request from the controller to provision a cross-connect to the virtual node is conveyed in an OFP_CFLOW_MOD message, which is received by the OF agent in the OF-PCE. From the OFP_CFLOW_MOD message the OF-PCE obtains the virtual port identifiers and the frequency slot parameters (in our case, the nominal central frequency n and the frequency slot width m). The virtual points are then mapped to the PCEP ENDPOINTS of the underlying GMPLS domain and the m parameter directly maps into the sender TSPEC that is included in the forward component of the PCEP BANDWIDTH object. The OF-PCE proceeds to set up the GMPLS LSP as usual and once the report message (PCEP PCRpt) is received, an OF message such as OFP_PORT_STATUS confirming the virtual crossconnect is sent to the controller (see Fig. 10).
In this scenario, an OF controller is responsible for a domain of three nodes, with datapath ids aa-bb-cc-dd-ee-ff-00-01..-03. An OF-PCE acts as a virtual OF node, with datapath id -02. It is a stateful PCE controlling a GMPLS domain of two ROADMs, 10.0.50.1 and 10.0.50.2. The OF-PCE maps the GMPLS topology and exports four ports, as seen in Fig. 9(b).
A connection is requested between OF nodes -00-01 and -00-03, the PCE receives the CFLOW_MOD message and proceeds to set up the LSP. Since the OF controller parallelizes the messages, the setup delay is mainly determined by the GMPLS CP. For illustrative purposes, the setup is done in less than 50 ms in the local LAN testbed (without considering optical hardware configuration delays).

B. UNI Support for OF Islands
A second scenario is related to the use of an OF controller that implements the RSVP-TE signaling protocol, to support GMPLS UNI interfaces for an OF controlled network. In this scenario, a testbed with six nodes has been deployed. Two nodes are regular GMPLS nodes with emulated hardware ( Path Resv messages, the connection attributes, detailed as follows: a) from the SESSION, explicit route object (ERO) and RSVP_HOP objects, it obtains the involved OF endpoints for the supporting connection, and b) from the sender descriptor TSPEC and GENERALIZED_LABEL objects, it knows the requested frequency slot width and the assigned frequency slot. It then proceeds to set up the cross-connect using OFOFP messages (OFPT_CFLOW_MOD and OFPT_CPORT_STATUS) (see Fig. 11(a) and (b). Note that, if all GMPLS controllers represent OF subdomains, the architecture matches the multiple SDN controllers model with GMPLS RSVP-TE (and optional OSPF-TE) protocol as East/West interfaces, enabling a twolevel hierarchy with network abstraction.

C. ODL as a Single SDN Controller
This scenario corresponds to the aforementioned single controller model, and involves deploying an ODL controller, which controls both OF and GMPLS subdomains. ODL presents an internal architecture organized in layers, where, notably, the northbound interface offers network control resources to external applications through a http-based RESTful API. The service abstraction layer (SAL) separates the modules and services of ODL from the southbound plugins to allow services and external applications to be agnostic to the protocol used.
For the proof-of-concept, in which a GMPLS network provides back-haul transport between remote OF domains (see Fig. 12), an ODL controller is responsible for orchestrating the provisioning of connectivity between remote host endpoints, by means of a new PCEP Service that has been implemented, which allows ODL orchestrating applications to request the establishment of lightpaths within the GMPLS controlled domain, via an active stateful AS PCE.
The proposed architecture has been validated with a scenario composed by two packet switching domains with five OF switches each, that have been deployed using standard Custom Off The Shelf (COTS) hardware and run OpenVSwitch (OVS) as OpenFlow Switches (OFS). The two border switches, connecting packet and optical networks, have been implemented using COTS hardware, including 10 Gb/s XFP tunable transponders and OVS software. The GMPLS-controlled optical network is composed of an all-optical wavelength switched network, with two ROADMs and two OXCs providing reconfigurable (in space and in frequency) end-to-end lightpaths, deploying a total of 610 km of G.652 and G.655 optical fiber, with six DWDM wavelengths per optical link. The switches are configured through OFPT_FLOW_MOD messages sent from the ODL controller. The flow-rules establish the forwarding action only for the traffic from the source to destination host MAC addresses.
In short, a new northbound interface exposes a RESTful API to handle LSPs (Create, Delete, List, Get), and the PCEP speaker Service, coupled into the SAL, translates REST calls into the southbound plugin YANG-modeled [20] remote procedure calls. The PCEP speaker plugin is responsible for driving the establishment of the PCEP session with the external AS-PCE and to setup or delete a LSP, using the PCEP protocol implementation available in the BGPCEP ODL subproject. The logic and  workflow are implemented on top of the ODL Northbound interface by means of an orchestration application. The application obtains the L2/Ethernet topology to ODL via Topology REST API and, upon reception of a connectivity request between OF nodes, calculates the Shortest Path within the L2 topology. If no path is found, connectivity is requested: a LSP establishment request between the border optical nodes is sent using the PCEP speaker Northbound REST API. This is shown in Fig. 13.
Note that this scenario is quite similar to the one detailed in Section V-A, without the need for a dedicated OF agent that maps the OFP messages to a provisioning interface (based on PCEP or not). However, by using the improved topology management capabilities of ODL (for example, obtaining the detailed GMPLS topology via BGP-LS extensions [8]) more efficient network control is available, and can be directly conveyed into the PCEP PCInitiate messages, instead of a raw mapping of a flow operation. Details on the implementation and performance evaluation can be found in [19].

D. Hierarchical AS-PCE
Finally, the last considered scenario, the multiple SDN controller hierarchical model has been implemented based on the concept of hierarchical PCE (H-PCE) [21], extended with stateful and OF capabilities [22]. In this setting, the parent PCE (pPCE) orchestrates the provisioning of services with generalized identifiers and each cPCE acts as a proxy for each domain, either integrated with an OF controller, or delegating to an underlying GMPLS/PCE CP (see Fig. 14). PCEP is used for provisioning, rerouting and reporting [23].
The main procedure to set up an end-to-end connection is as follows: After an initial north bound interface request of the pPCE (1), the pPCE requests the virtual shortest path trees (VSPT) from the source and destination nodes to their corresponding border nodes (2) and uses the VPSTs and virtual links to perform domain selection (3) and, in turn, requests the segment expansion: each cPCE performs routing and spectrum assignment and returns usable path and frequency slots (4). The pPCE combines the segments into an end-to-end ERO (5) and then requests the establishment of the corresponding segments with PCinitiate messages (6). OF domains use an extended OFP to program the ROADMs and GMPLS domains delegate the establishment to the underlying CP (7). Finally, PCRpt messages communicate establishment (8) within the GMPLS domain, cPCE to pPCE and pPCE to the initiator (see Fig. 15).

VI. CONCLUSION
GMPLS deployments are integrated within centralized management systems (NMS), which can benefit from customizing the network behavior in software, becoming part of a broader SDN architecture. SDN is highlighting the need for interoperable components, open and standard interfaces and common models with the right abstractions, avoiding vendor lock-in. Emerging use cases involve heterogeneous multi-domain scenarios, requiring interworking and orchestration, abstracting and scoping such domains, e.g. behind a single entry point such an AS-PCE. Mesh, hierarchical or hybrid solutions, combining distributed and centralized elements, need to address scalability (CP dimensioning), security and efficiency. For this, the maturity, robustness and proven status of the protocols of the GMPLS/ASON architecture should not be ignored.

ACKNOWLEDGMENT
The shorter version of this paper, entitled "Interworking of GMPLS and OpenFlow domains: overarching control of flexi grid optical networks," was presented in the 40th European Conference and Exhibition on Optical Communication (ECOC 2014) as an invited paper.