Backhaul Routing and Base Station Sleep Mode Engagement in Energy Harvesting Cellular Networks

Future dense mobile networks will imply much higher costs both in access and backhaul. This paper analyzes the effect on wireless mesh backhaul routing performance when energy saving policies are present at the radio access network (RAN). We consider an heterogeneous two-tier network where small cells (SC) with energy harvesting capabilities extend the capacity of the macro base stations (MBS), and can autonomously switch on-off in order to increase the energy efficiency of the network based on a Q-learning (QL) algorithm. Instead of calculating new routes for each SCs activation pattern, we propose to agnostically adapt to the RAN traffic demands using a non-route-based backpressure routing policy for the wireless mesh backhaul to even the network resource usage amongst SCs. We used the ns-3 simulator to integrate the different mobile network segments: RAN, wireless mesh backhaul, and evolved packet core (EPC). Simulation results show an achieved reduction of the %37% of the RAN energy consumption while satisfying traffic demands with an improvement of up to a factor of 10 of delay performance in the backhaul during peak hours.


INTRODUCTION
Long Term Evolution (LTE) dense deployments formed by both Small Cells (SC) and macro base stations (MBS), known as HetNets, are expected to accommodate the rising demand of mobile data services. Being very unlikely to count with a fiber connection at each SC location, it is recognized that SCs will include wireless capabilities, allowing the deployment of wireless mesh networks between SCs for backhauling purposes.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. Consequently, future large-scale/dense mesh SC topologies will substantially increase network resource redundancy. Additionally, ultra-dense SC deployments pose some concerns in terms of energy consumption. First, not all SCs may be directly connected to the power grid. Second, even if they were, this approach would not scale. Instead of connecting these SCs to the energy grid, a solution could be to equip them with energy harvesters and storage systems to reduce their energy consumption. On this matter, photovoltaic (PV) technology represents a valuable choice due to its good efficiency, competitive cost, and availabililty. Therefore, appropriate resource management will be key for dynamic, complex, and heterogeneous future mobile networks. Their high density will imply much higher radio access network (RAN) costs, and also the need for an appropriate backhaul network. Wireless mesh networks are expected to play a key role in this respect. It is therefore fundamental to reduce RAN and backhaul resource consumption. In this paper, we evaluate the global resource consumption when combining a RAN energy management scheme and an efficient backpressure-based routing scheme in the backhaul.
State of the art routing adopted for the mobile backhaul (MPLS) uses one static single path. Such strategy can under-utilize the resources of a dense wireless mesh network. In contrast, backpressure approaches [8] offer, in theory, the possibility to exploit such resources. For instance, BP-MR [4] has shown to bring throughput and latency improvements by taking on-the-fly per-packet routing decisions. On the other hand, a promising method for energy-constrained radio resource management (RRM) is to dynamically reconfigure the RAN. This could be achieved by enabling sleep mode in the SC and offloading traffic to the macro cell based on the harvested energy and the traffic demand [10]. Indeed, a key question to answer is whether and how a routing strategy for the backhaul is adapting to the RAN reconfiguration triggered by the energy-constrained RRM.
In this paper, we consider a heterogeneous two-tier network where SCs with energy harvesting capabilities extend the capacity of the MBS, and that can distributedly and autonomously switch on-off in order to increase the network energy efficiency based on a distributed Q-learning (QL) algorithm presented in our previous work [10]. As for backhaul resource management, instead of calculating new routes for each SC activation pattern, we propose to independently adapt to the RAN traffic demands using a non-route-based backpressure routing policy for the wireless mesh backhaul. Overall, this combination provides resource management at two different timescales, first, by deciding SC hourly activation patterns and second, by dynamically adapting to backhaul traffic demand variations at packet-level timescale.
The analysis is performed using a flexible end-to-end cellular network test environment based on the popular ns-3 network simulator and an ad-hoc octave simulator to perform large scale energy characterization of the system.
The herein presented results demonstrate an efficient resource consumption in the RAN and backhaul when using a per-packet backpressure-based routing protocol in combination with the QL energy algorithm. Simulation results reveal remarkable gains in peak hours in throughput and latency (up to 10% and a factor of ten, respectively). Finally, we show that the distributed energy harvesting management can achieve high energy savings (up to 37%) with respect to when the SCs are connected to the electrical grid.
Up to our knowledge, this is the first attempt to study the reconfiguration capabilities of the wireless mesh backhaul to the self-organized dynamics enabled by energy-constrained RRM mechanisms. Usually, routing in meshed backhaul, energy harvesting, and BS sleep mode are not tackled together. In fact, there are several analysis made in the field of wireless mesh networks facing the routing with energy harvesting nodes, such as those in [2], [9] or [3]. However, they do not consider backhaul mesh networks for the mobile RAN. Other examples study backhaul-aware user association or load balancing with tree or star topology backhaul, as in [12] and [7].
The remainder of this paper is organized as follows. Section 2 provides an explanation of the node architecture together with background on energy-constrained RRM and flexible backpressure routing strategies, before conducting the aimed performance evaluation in Section 3. Section 4 concludes the paper with the main conclusions.

NODE ARCHITECTURE AND RESOURCE MANAGEMENT SCHEMES
In our vision, SCs count with a LTE radio access plus wireless backhaul capabilities to reach the EPC via a mesh topology. In addition to the standard protocol stacks, we propose to insert (i) a distributed Energy Manager module, as part of the required Operation Administration and Maintenance (OAM) activities to be conducted at the RAN level, and (ii) a backpressure-routing module taking routing decisions on a per-packet basis on IP packets. The working details of each module are covered in subsequent subsections. This decoupled, self-organized and distributed architecture allows the two modules to work independently, giving a high grade of flexibility and scalability to the system, suitable for the requirements of future 5G networks.

Energy Management
In the last years, the use of renewable energy sources (RES) in the cellular network started to generate interests in the research community [1]. The integration between renewable energies and mobile networks plays an important role in the roadmap to 5G to reduce the consumed energy used by the mobile operators. Regarding the RAN, a deployment optimization has been presented in [5], where a design for the management of k-tier HetNets powered by RES has been presented. This model optimizes the fraction of ON time of each tier. Considering on-line optimization algorithms, [13] proposes a dynamic control of the BS power consumption as a function of the energy reserve and the expected amount of renewable energy. However, the two works above do not consider the temporal variations in traffic and in harvested energy processes, as typical in a real scenario.
In order to overcome the above mentioned problems, in [10] we proposed a distributed on-line solution based on a multiagent Reinforcement Learning (RL) algorithm, known as distributed Q-learning (QL). Thanks to RL, the agents can independently learn its proper RRM policy through realtime interactions for capturing the dynamic conditions of the environment, in terms of energy inflow and traffic demand and it is able to jointly maximize the system performance in terms of throughput, drop rate and energy consumption. This paper extends this analysis interacting with a backpressure based backhaul resource management scheme.

Backpressure Routing
The origins of the backpressure concept lies on the seminal paper of Tassiulas and Ephremides [8]. The root concept consists in a centralized policy which routes traffic in a multi-hop network by minimizing the Lyapunov drift in the network, that is, minimizing the sum of the queue backlogs in the network amongst time slots.
Despite throughput optimality, this work presents several drawbacks: centralized control mode, high queuing complexity (per-flow queuing system), and poor delay performance. Recently, many proposals have been presented to alleviate the effect of these issues [8]. From this set of proposals, in this paper, we refer to BP-MR [4] because of its scalability and performance improvements in wireless mesh backhauls. Based on the Lyapunov-drift-plus penalty approach, routing decisions are taken distributively at each node on a perpacket basis combining queue backlog information (backpressure component) with geographic information.
Specifically, BP-MR performs dynamic per-packet routing decisions following a two-stage process: (i) data packet classification in a per-interface queue system according to its destination, and (ii) the use of geographic and congestion information to compute the best possible next-hop. The per-interface queue system presents lower complexity and a better delay performance than the original per-flow queueing system, contributing to the protocol scalability.

EVALUATION
This section evaluates via simulations the adaptability of the backhaul routing to the RAN reconfiguration, triggered by energy saving mechanisms. In particular, we compare BP-MR with OLSR against a distributed QL mechanism, which enables sleep mode in the SCs [10]. OLSR is a static single path routing protocol that, in absence of node mobility and failures, is equivalent to MPLS, a reference transport technology for mobile backhaul networks. Figure 1 shows the evaluated scenario consisting of a mesh deployment of non-overlapping SCs assisting a macrocell forming a plain grid of 2 × 3 elements deployed over a 1Km 2 according to use cases covered by 3GPP documents TR 36.872 and TR 36.842. Each SC counts with an LTE radio access plus multiple wireless interfaces simulating high directive µwave links. One of these SCs has a wired connection to the LTE Evolved Packet core (EPC), which connects to the Internet. The PV model considered is based on a Panasonic N235B solar module. Each SC is equipped with an array of 16 × 16 solar cells. The storage system is based on a battery of 2 kWh. The panel and battery sizes have been chosen so that the SC battery is replenished in a full winter day. The SolarStat tool [11] has been used for generating the hourly harvested energy profile. The requested traffic is generated by 120 users (UEs) uniformly placed within the coverage area of each SC. We adopt the daily traffic pattern described in [6] and updated with the requirements provided by a telco operator collaborating in the EU-H2020-SANSA project 1 . According to this model, 50% of the UEs are configured as heavy users (their data volume is 2.25GB/h), while the remaining UEs are ordinary users (0.45GB/h). Furthermore, simulations consider UDP constant bit rate (CBR) downlink (DL) traffic.

Scenario Description
Link rates in the backhaul network are dimensioned according to the mentioned traffic demands, where there is a certain percentage of active users (e.g., 16-17% during peak hour). They have been selected lower (from 12Mbps to 36Mbps) than rates offered by commercial equipment to better show the exploitation of scarce wireless resources by each of the evaluated protocols. The backhaul network is connected to the EPC through a 1Gbps wired connection.
Simulation results have been conducted with ns-3 simulator using the latest version of LENA LTE model 2 conveniently updated to include custom backhaul topologies.

RAN Reconfiguration
In this work, the energy characterization of the system has been obtained averaging 10 runs over a simulated year to evaluate the behavior of the QL energy algorithm for different months and energy harvested inflow. We used an ad-hoc octave simulator since ns-3 does not allow simulating time frames of this range in a reasonable time. Figure 2 presents the energy drained from the grid during each of the hours in a day and averaged during all days in a year. This figure compares the QL algorithm with when all the SCs are powered by the grid. QL guarantees an overall energy saving of 37% during the whole day. The consumption spike around hour 4 of the QL is due to the fact that many SCs are in sleep mode, generating a peak of energy consumption of the MBS. The QL algorithm decides a more intensive sleep mode in the RAN, thus saving the harvested energy in the storage system to use it later to satisfy traffic demands during peak hours. Figure 3 depicts the energy efficiency of the system in one day and averaged during one year. QL is able to reach up 1 http://sansa-h2020.eu/ 2 http://networks.cttc.es/mobile-networks/softwaretools/lena/ to the double of the energy efficiency during peak hours, whereas it has a lower energy efficiency when the sleep behavior is more intensive (hours 5 to 7). Figure 4 presents the daily average throughput and delay distribution variation experienced by the traffic served by the mesh network of activated SCs determined hourly by the QL algorithm. The results are obtained over 10 repetitions of a given sample day. The boxes in the statistical packet delay distribution show from the 25 th to the 75 th percentiles, and the whiskers from the 5 th to the 95 th percentiles.
BP-MR experiences significant gains both in terms of throughput and delay, especially during traffic peak hours. In particular, BP-MR increases throughput performance by up to a 10% and achieves a reduction in latency of about a factor of ten with respect to OLSR (notice the logarithmic axes on the delay graph). It distributes the traffic to follow the less congested path and is able (if needed) to deviate from the shortest path, in terms of number of hops, hence, reducing queuing delays. OLSR calculates static equivalent paths between two endpoints independently from the SC traffic requirements, hence not fully utilizing the backhaul capacity and deriving in high delays due to excessive queueing. These results indicate the compatibility of a dynamic backpressure-based protocol, such as BP-MR, with the QL energy algorithm in a mesh backhaul network providing resources redundancy.

Backhaul Reconfiguration
Durink peak hours, most of the SCs are active, hence requiring the full backhaul. However, when the requested capacity decreases, some of the backhaul links are partially used and they can be deactivated to save energy.
For this purpose, we setup a simulation where the link from the SC1 to the SC4 (see Figure 1) is deactivated due to reduced activity. Figure 5 shows a zoom of the temporal evolution of the average network throughput around t = 15s, when the link is deactivated. This event is transparent to BP-MR due to its dynamic per-packet routing approach and the enough amount of available network resources. The  behavior of OLSR is the opposite: the reconfiguration of a single wireless backhaul link yields a transitory sudden degradation in throughput performance (around 400ms) due to the recomputation of the actual end-to-end paths, being this time directly related with the size of the network.
Concluding, BP-MR presents good reconfiguration capabilities, since the route is discovered on-the-fly while packets traverse the network. This trend motivates further research on energy-constrained backpressure-based routing protocols for saving energy in backhaul networks.

CONCLUSIONS
In this paper, we evaluate the combination of a two level resource management scheme combining 1) Q-learning-based energy management in the RAN that hourly switches ON/ OFF SCs, and 2) backhaul resource management in the form of backpressure routing in the wireless mesh network that acts on a per-packet basis. Simulation results show that the QL energy algorithm reduces network energy consumption up to 37% and that the backpressure-based protocol efficiently adapts the backhaul resource consumption to the RAN reconfigurations determined by the QL energy algorithm. More specifically, the dynamic per-packet backpressure routing policy attains delay improvements by a factor of 10 during peak hours, compared to a route-based protocol such as OLSR. The analysis performed in this paper further encourages research on schemes for energy-constrained scenarios integrated within the backpressure-based protocol to dynamically (de)activate wireless backhaul links.

ACKNOWLEDGMENTS
This work was partially funded by the EC under grant agreement no 645047 (H2020 SANSA project) and by the Spanish Ministry of Economy and Competitiveness under grant TEC2014-60491-R (5GNORM).