Design and Simulation of the Multi-RAT Load-balancing Algorithms for 5G-ALLSTAR Systems

This paper introduces algorithms for the multi-RAT load balancing function to maximize QoE in terrestrial and satellite combined system developed in the 5G-ALLSTAR project. The pros and cons of the considered algorithms are described and the simulator is also described in the paper with the on-going performance evaluation processes.


I. INTRODUCTION
This paper introduces candidate algorithms for the multiconnectivity that are studied in the 5G-ALLSTAR project which stands for 5G AgiLe and fLexible integration of SaTellite And cellulaR and jointly developed by five EU companies and five Korea companies. The architecture considered in the project regarding multi-connectivity is shown in figure 1. In the figure, the two pieces of User Equipment (UEs) are able to connect to several Access Points (APs) of different Radio Access Technologies (RATs) at the same time, even for the same PDU session. The remainder of the paper consists of Section II and III which describe the proposed algorithms and the network simulator respectively, and then Section IV draws the conclusions.

II. PROPOSED ALGORITHMS AND EVALUATIONS
This section briefly introduces four of the algorithms that are being developed in the 5G-ALLSTAR project for multi connectivity and provides to the interested reader the relevant references where their complete formulation is presented.

A. Wardrop Equilibrium-based traffic steering
The first algorithm proposed in the suite developed in the 5G-ALLSTAR project is based on Wardrop's theory on adversarial routing and its related equilibria [2]. The rationale behind the algorithm is as follows: i.
Every access point (AP) is associated to a function (latency function in the Wardrop literature) of its state that measures a performance metric of interest that the controller aims at minimizing (e.g., overall resource block usage as a function of bitrate provided, power usage as a function of connections served, etc…). ii.
Every connection, which is distributed over a set of APs, migrates from every AP a portion of its traffic towards the APs whose current latency values are lower, hence improving unilaterally its routing.
Provided that the amount of traffic migrated is determined by a properly designed proportional control law, as in [3], the solution proposed can be proved to steer the network toward the Wardrop Equilibrium state, in which all the latencies of the used AP are equalized [2].

B. Network selection based on Markov games and friendor-foe RL
The second algorithm developed for the 5G-ALLSTAR project is based on Markov Games and in particular on Friend-or-Foe Reinforcement Learning [4] in order to find a solution to the network selection problem in a distributed way.
In the algorithm each AP compete with the others to try to allocate the incoming UE's request, in order to collect a positive reward depending on the type of request; different service request types are defined to model different service classes and so different Quality of Service (QoS) profiles. In case of the impossibility for the AP that wins the game to actually provide the requested resources, then the environment gives a negative reward and the request is dropped.
From [5] it is possible to show that it is possible to allocate more network resources to the more prioritized service classes at the cost of dropping more bitrate; making instead all the service classes equally weighted it is possible to obtain the same performances of centralized least loaded network selection.

C. Network selection and traffic steering based on RL
In a centralized framework, in which the AP collaborate to archive a better traffic steering, a traditional RL controller was proposed by the authors in [6] based on Q-learning and state space aggregation. In this work, different traffic profiles were considered to also include Quality of Experience maximization in the routing decisions.
In the problem formulation, the states with very close minimum capacity allocation are aggregated in order to avoid the curse of dimensionality. A state-dependent action space is then defined. In order to let the RL algorithm visit each state with the defined action space, the Q-table updating rule is modified accordingly.

D. Multi-RAT Load Balancing
For load balancing in 5G multi-RAT consisting of terrestrial cells and a satellite cell, load condition of each cell is measured in each RAT. Based on the load condition of each RAT, intra-RAT and inter-RAT offloading of the users can be performed. Due to ignorance of resource allocation differences among RATs, previous algorithms [7][8] cannot be implemented in a 5G multi-RAT network, which require common load metric to measure the load condition of each RAT. In [9] a common load metric is introduced to measure the cell load of each RAT and a load balancing algorithm a multi-RAN is proposed with an adaptive threshold to determine overloaded cells.
The proposed load balancing algorithm consists of two steps across 5G cells: intra-RAT load balancing and inter-RAT load balancing. Based on measured load of a cell, the algorithm first performs intra-RAT load balancing by offloading the appropriate edge UEs of an overloaded cell to underutilized neighboring cells. If the measured load of the cell is still higher than a predefined threshold, then inter-RAT load balancing is performed by offloading the delay-tolerant data flows of UEs to a satellite link. Offloading of the users to different RAT cells increases the number of users satisfying QoS as well as network throughput in the 5G multi-RAT network. [9] III. 5G-ALLSTAR NETWORK SIMULATOR In the context of 5G-ALLSTAR project a multi-RAT network open-source simulator [10] has been developed. This simulator handles in a simplified but effective way the resource allocation for multiple RATs like 4G LTE, 5G NR and Satellite and considers both User-Equipment (UE) mobility and Access Point (AP) mobility (i.e., Drone APs or Relays).
In this simulator the UE that makes a connection request/update has to specify just the bitrate it desires to obtain from a certain AP (belonging to a certain RAT) and the AP itself will be in charge of computing the actual physical resources to be assigned to the UE request in order to satisfy the desired bitrate.
The RAT-specific physical resources to be assigned are computed considering the pathloss between the UE and the AP and also considering inter-RAT and intra-RAT interference, as well as thermal noise.
The simulator has also the possibility to add Connection Admission Control logic both at UE-side (User-Centric, RAN-Assisted) and at RAN-side (RAN-Assisted, RAN-Controlled), in order to simulate different control approaches within the same tool.
The connection procedure is as follows: the UE measures the receiving power for each visible AP and chooses one (or more) AP(s) to connect to; this choice is made according to the Connection Admission Control algorithm implemented; at this point each of the selected APs allocates its resources based on the bitrate request made by the UE in a first-comefirst-served way (due to the scarcity of resources at the AP side, some UE connections may receive resources for a lower bitrate than the requested one). At each time step, given the mobility of the UEs and of some of the APs, the connections already established need to be updated according to the new state of the system; this process is done in a way similar to the connection procedure.

a. Receiving Power and Path Loss computing
The receiving power is computed starting from the characteristics of the AP's antennas. In details, the power received by a UE from an AP is computed as: where is the output power of the AP's antenna, is the antenna gain, are is the feeder loss and is the path loss between UE and AP . If , then the AP is considered not visible by the UE . The path loss is computed in different ways depending on the considered RAT: for satellite RATs the path loss is computed as Free Space Path Loss (FSPL) , where is the Euclidean distance between UE and AP , is the carrier frequency used for the wireless transmission and is the speed of light; for terrestrial RATs as 4G LTE and 5G NR the path loss is computed using the COST-HATA [11] model, Resources Allocation for 5G NR RATs

b. Resource Allocation for 5G NR RATs
In 5G NR the minimum allocation unit is the Resource Block (RB); an RB is composed by 12 frequency subcarriers in a single time-slot. The frequency bandwidth of the subcarriers, as well as the duration of the time-slot depends on the numerology chosen for the 5G NR RAT. The number of OFDM symbols is fixed to 12 (or 14 for normal Cycle Prefix) and the duration of a time-frame is 10ms. The number of RB depends on the available bandwidth, as defined in [12][13] [14] Once a UE requests a connection to a 5G NR AP with a certain bitrate , the SINR (Signal-over-interference-plus-noise-ratio) is computed by the AP. The noise is computed ad thermal noise � � = � � �� and the interference is computed as in [15], considering the receiving power of the neighbor APs � that utilize the same carrier frequency, their utilization ratio ���� � and the utilization ratio of the target AP ���� � ℐ ��� = ∑ � ��� ⋅ ���� � � ⋅ ���� � , where � ��� = 1, if UE � is connected to AP � and � ��� = 0 otherwise, � ��� ��) is the number of RB allocated to user � by AP � and � is the time window for the computation of the average ����.
Once computed the SINR, it is possible to estimate the bitrate that can be provided by a single RB using the Shannon Formula (� ��� = � �� log � �1 � ���� ��� )), as in [16], and so computing the number of RBs to be allocated for the connection: As said before, since the allocation is done in a first-comefirst-served way, the number of resource blocks to be allocated for an UE can be less than the requested ones if there are not enough RBs remaining at the AP side.

c. Resource Allocation for Satellite RATs
In Satellite RATs the minimum allocation unit is the Symbol since Time Division Multiple Access (TDMA) is used, according to the example showed in [17], with antennas parameters taken by [18] [19]. In a time frame of ���, a certain number of symbols � is available and at for UE connection some symbols have to be left unused as guard spaces and some headers have to be added to the actual payload. Moreover, it is not possible to allocate an arbitrary number of symbols, but the allocation can be made in multiples of 64 symbols.
Once a UE � requests a connection to a Satellite AP � with a certain bitrate � � , the SINR is computed by the AP, in the same way as in the 5G NR RAT case, considering again the interference of the neighbors Satellite APs and the thermal noise. Using the Shannon Formula, it is possible to estimate the bitrate � ��� a single symbol can provide, and so it is possible to compute the number of blocks to be used as To this number of blocks to be allocated, the guard spaces and the headers have to be added in order to compute the actual allocation for the Satellite AP.

IV. CONCLUSION
This paper provides information about the multi-RAT load balancing algorithms developed by 5G-ALLSTAR project.
The project mainly focuses on millimeter cellular and satellite combined system with originally targeted on service continuity with providing high quality QoE in a multiconnectivity framework. With these purposes, the paper shows the possible multi-connectivity algorithms, detailed performance evaluation is on-going, and the open-source network simulator of the project.