On Fairness Evaluation: LTE-U vs. LAA

In this paper, we propose a statistical framework to systematically evaluate the fairness offered by different LTE (Long Term Evaluation) technologies when they coexist with Wi-Fi in unlicensed band. In particular, we study the coexistence performance of both 3GPP LAA (Licensed Assisted Access) and LTE-U (LTE in unlicensed spectrum), as specified by LTE-U forum. We map the generally accepted 3GPP definition of fairness onto the stochastic dominance concept. We use the two-sample one-sided Kolmogorov-Smirnov test (KS-test) to test the specific hypothesis of fairness defined through throughput and latency performance, as proposed by 3GPP. We evaluate throughput and latency by means of the ns-3 simulator for LTE and Wi-Fi coexistence. According to both the simulation results and the statistical analysis, there is a need for proprietary solutions to operate on top of what has been standardized, which will able to improve the fairness in the coexistence scenario. On the other hand, through the comparative analysis of LAA and LTE-U coexistence performance, we confirm what could be expected, i.e., that LAA better performs in terms of fairness in both throughput and latency and LTE-U introduces more collisions.


INTRODUCTION
The quest for increasing the network capacity and to reduce the price per megabyte in a cost-effective way is the Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. main challenge of network operators; various solutions are being considered, from the densification of small cells to the offload of traffic in unlicensed band. We refer to "unlicensed LTE" (ULTE) as an evolution of LTE (Long Term Evaluation) technology that enables simultaneous operation, in a licensed and unlicensed band, enabled by the principle of carrier aggregation. Currently, different paradigms are under design, to allow harmonious coexistence in unlicensed spectrum. One of the principles for design are the different regional regulatory regimes, which may or may not require the access to the medium based on Listen Before Talk (LBT) procedures.
The first coexistence paradigm is specified by the LTE-U Forum [12], an industry consortium interested in introducing a proprietary solution, referred as, LTE-U, into the market for the access in unlicensed band intended for regulatory regimes without LBT requirements. Based on this paradigm, LTE duty cycles its transmissions, i.e., alternates ON and OFF periods. Proprietary solutions may adapt the duty cycle by estimating the most appropriate channel share that it should occupy, depending on the activity of the other networks in the channel. The most representative algorithm is the Qualcomm CSAT/eCSAT (Carrier-Sensing Adaptive Transmission) [13].
The second coexistence paradigm is promoted by 3GPP to meet ETSI's clear channel assessment/Listen Before Talk requirements, and it is referred to as Licensed Assisted access (LAA). This ULTE paradigm is designed and standardized to become a global solution framework allowing compliance with any regional regulatory requirement. A study item on this has been recently finalized and has produced a Technical Report TR36.889 [4]. In the rest of the paper, when we refer to ULTE we will be referring to both LAA and LTE-U.
The use of LTE in unlicensed spectrum generates multiple challenges, since LTE has been designed to work in licensed spectrum on the basis of uninterrupted and synchronous operation. Consequently, the efficient use of unlicensed bands to offload LTE traffic is of main concern for both ULTE and widely deployed Wi-Fi networks. According to 3GPP, for a fair co-existence with Wi-Fi, LAA must not harm a Wi-Fi network more than an additional Wi-Fi network on the same band. Based on this definition, in the literature different contributions try to evaluate the coexistence performance of ULTE and Wi-Fi. In [7], the impact on the fairness of two LAA-LBT based channel access schemes with the Wi-Fi network has been studied. The performance analysis is done by comparing the mean and the Cumulative Distribution Function (CDFs) of latency and throughput under different signal/energy detection thresholds by LAA-LBT procedure. In [10], the authors have proposed two channel sensing schemes for LTE in unlicensed band, and the fairness of both the schemes is established on the basis of mean user throughput of the Wi-Fi network. In [6], the performance of various co-existence methods based on LBT and Duty Cycle (DC) have been evaluated using the Monte-Carlo simulations. The fairness of these methods is evaluated on the basis of average number of collisions and average latency experienced by the coexisting Wi-Fi network. We observe that the conclusion on the fairness in these works are mainly driven by qualitative comparisons of average values and CDFs.
Differently, in this paper, we focus on statistically evaluate the fairness offered by both LTE-U and LAA paradigms to a coexisting Wi-Fi network. We perform a simulation study of the coexistence behavior by following the 3GPP methodology presented in [4], when Wi-Fi coexists with both technologies. We have extended the ns-3 simulator to support ULTE-Wi-Fi coexistence scenarios. In particular, we have designed and developed 3GPP and LTE-U Forum compliant models for both LAA and LTE-U. To the best of our knowledge this is the only open source simulator available for the evaluation of ULTE-Wi-Fi coexistence studies. The openly available code can be found here [1]. This simulator has the strong advantage of offering complete full-stack models for Wi-Fi and ULTE. We discuss in detail the behavior of the throughput and latency curves and we try to draw conclusion in terms of fairness. We observe, however, that the coexistence curves may converge in some areas and diverge in others, and it is not so intuitive to claim if ULTE is actually fair to Wi-Fi, or not. In particular, this qualitative approach leaves much space to partial interpretations. Therefore, in this initial study, our main focus is towards the use of more rigorous approach which allows us to statistically evaluate the concept of fairness and not just only the comparison of the two ULTE paradigms, which has already been discussed in many works in the literature. In this context, statistical analysis offers interesting tools and concepts which can serve the purpose. We employ these concepts to map the 3GPP fairness definition onto the concept of stochastic dominance.
To this end, the main contribution of this paper is not only the modeling and evaluation of LTE-U and LAA performance, but also a statistically evaluate the fairness offered by those technologies to Wi-Fi. Specifically, we want to prove that Wi-Fi throughput distribution curves when coexisting with ULTE, stochastically dominate those of Wi-Fi when coexisting with Wi-Fi. For latency we want exactly the contrary, i.e., we do not want that the CDF of the latency when WiFi coexists with ULTE, stochastically dominates the one, when Wi-Fi coexists with another Wi-Fi, as low latencies are desirable for a fair coexistence. We test these hypotheses through a non-parametric one-sided two sample Kolmogorov-Smirnov test (KS-test), which is a useful tool to test the first order stochastic dominance, especially with our data, since it does not require any hypothesis on the distribution of the data.
The rest of the paper is organized as follows. In section 2, we present the statistical framework that we propose to use for the fairness evaluation. In section 3, we present the models for Wi-Fi, LTE-U and LAA. Section 4 discusses the simulation results and the data analysis campaign. Finally section 5 concludes the work with final thoughts on the fairness evaluation and suggestions for future works.

STATISTICAL FRAMEWORK
The fairness as defined by 3GPP [4], is the capability of an LAA network not to impact Wi-Fi networks active on a carrier more than an additional Wi-Fi network operating on the same carrier, in terms of both throughput and latency. The same fairness concept also has been proposed to evaluate fairness for LTE-U [13]. We evaluate fairness according to the scenarios defined by 3GPP, in which there are two operators, i.e., operator A and operator B. In the first step, both operators deploy Wi-Fi, while in the second step, operator A substitutes Wi-Fi with ULTE. We claim ULTE is fair to Wi-Fi if, when switching from Wi-Fi to ULTE the performance of operator B is not negatively affected.
However, based on this qualitative definition provided by 3GPP, we choose to compare fairness by comparing the CDFs distributions of throughput and latency. If the curves overlap or the Wi-Fi performance after substitution is improved, we can safely say that ULTE is coexisting fairly with Wi-Fi. Contrarily, if these curves partially overlap or diverge in some areas (this is what we observe in our results), one cannot do much beyond explaining the reasons for the divergence and it is still debatable if the ULTE behavior should be considered as fair or not. Therefore, we believe that there is a need of a more detailed analysis of the data, which can systematically tell us whether, up to a certain tolerable extent, ULTE behavior is fair or not. Statistical data analysis offers tools to compare the statistical behavior of data, and we believe that it could add value to the evaluation of fairness in ULTE and Wi-Fi coexistence. In particular, we rely on the concept of first order stochastic dominance, which assumes that a distribution X stochastically dominates a distribution Y, if the CDF of X lies on the right side of CDF of Y. One way for 3GPP definition of fairness to be quantified is to leverage the concept of stochastic dominance. We obtain empirical CDFs (ECDFs) of the key performance parameters and we measure the extent to which one CDF dominates the other CDF. Specifically, it could be expressed as follows [14]: Where T wl (x) is the CDF of the throughput of a Wi-Fi network when it coexists with an ULTE network and Tww(x) is the CDF of the same Wi-Fi network, when it coexists with another Wi-Fi network. In practice this means that, for ULTE to be fair, the same or higher throughput must be obtained by operator B, after the substitution. So, when operator A substitutes Wi-Fi with ULTE, the throughput CDF of operator B must stochastically dominate the one, obtained before the substitution of ULTE. In terms of latency, the fairness could be achieved if the CDF of the latency distribution of a Wi-Fi network, when coexisting with ULTE network, does not stochastically dominate the baseline Wi-Fi distribution, since improved latency means smaller latency values. In other words, Where L wl (x) is the CDF of the latency in the ULTE-Wi-Fi coexistence scenario and Lww(x) is the CDF of the latency in the baseline scenario.
Statistical hypothesis testing is a procedure in which sampled data are employed to test a hypothesis about a single population or the relationship between two or more populations. This hypothesis is either a null hypothesis H0, i.e., a statement about the distribution of observations we want to test or an alternative hypothesis H1, i.e., an alternative statement in the case a null hypothesis is failed [11]. There are many statistical techniques which could be applied to test the hypothesis under study and they are mainly divided into parametric and non-parametric statistical tests. Following the method presented in [5], we use a non-parametric two sample one sided Kolmogorov-Smirnov test (KS-test) to test the first order stochastic dominance of two Wi-Fi distributions. This kind of tool is particularly useful in our problem, because it does not require any hypothesis on the underlying distribution of the data (e.g. many methods require normal distribution), and it offers no restrictions on the size of samples [9]. Following the steps of hypothesis testing, we state our H0 and H1 hypothesis for throughput and latency as follows, 1. Throughput: H0: The throughput distribution of Wi-Fi when coexisting with ULTE stochastically dominates the baseline throughput distribution.

H1:
The throughput distribution of Wi-Fi when coexisting with ULTE does not stochastically dominate the baseline throughput distribution.

Latency:
H0: The latency distribution of Wi-Fi when coexisting with ULTE does not stochastically dominate the baseline latency distribution.

H1:
The latency distribution of Wi-Fi when coexisting with ULTE stochastically dominates the baseline latency distribution.
As a result of this test, two values are obtained: • Dmax: It is the maximum measured distance between the two ECDFs.
In the literature, Dmax is mentioned as the test statistic value for the KS test. Generally, to test the hypothesis in the test, the calculated Dmax value is compared with a critical D value (Dcrit) obtained from the KS table, at a certain significance level α. The null hypothesis is rejected, if the value of Dmax is greater than Dcrit value [11]. Following a common statistical practice in the literature, we use α = 0.05. In our case, by using the Table G in [11], for m=60 and n=62, the value of Dcrit is 0.22. Where m and n are the number of IP level flows in our simulation.
• P-value: It is the probability of a null hypothesis being true (with a certain significance level α).
The interesting point about the P-value is that it quantifies the dominance of the whole distribution and not simply how close/distant they are at a certain reference point or how similar the obtained average values are. If the Pvalue obtained in the test is less than α, it indicates that we have less evidence for our null hypothesis to be true and vice versa. Specifically, if the P-value is below the significance level α we reject the null hypothesis and claim that the results obtained show unfairness to Wi-Fi. Therefore, the lower the P-value the more likely the technology evaluated is unfair to Wi-Fi. On the other hand, for values higher than the significance level α, the more likely it is that the technology evaluated is fair to Wi-Fi. To summarize, in this study we propose the following steps to systematically evaluate the fairness: 1. Build throughput and latency ECDFs of the different flows crossing the Wi-Fi network of operator B, when it coexists with Wi-Fi network of operator A, and use it as baseline.
2. Repeat the procedure in step 1, when Wi-Fi network of operator B coexists with ULTE network of operator A.
3. Apply one sided two sample KS-test.
4. Use the P-value to accept or reject the null hypothesis.

SYSTEM MODEL
We have extended the ns-3 simulator to support ULTE-Wi-Fi coexistence scenarios. To the best of our knowledge this is the only open source simulator available for the evaluation of ULTE-Wi-Fi coexistence studies.

Wi-Fi Model
Currently, the Wi-Fi module of ns-3 supports a number of IEEE standards [3]. We use 802.11n standard operating on 5 GHz band with 20 MHz of bandwidth. It uses an Enhanced Distributed Coordinated Function (EDCA) to prioritize channel access depending on the traffic access category. The MAC layer is configured to use a one-level frame aggregation method called aggregated MAC layer protocol data unit (A-MPDU), to leverage the MAC layer enhancements provided by 802.11n. The module also supports the Clear Channel Assessment (CCA) based on Energy Detection (CCA-ED) and Preamble Detection (CCA-PD), to sense the medium for the transmissions from Wi-Fi and other radio access technologies (RATs). We have used the default threshold, i.e., -62 dBm and -88 dBm for CCA-ED and CCA-PD, respectively, in both coexistence scenarios. Additionally, it offers an abstract 2x2 MIMO (Multiple Input Multiple Output) model to achieve higher physical rates offered by 802.11n. The nodes in our simulations are configured with 2x2 TX/RX antennas, supporting the rates up to Modulation and Coding Scheme (MCS) 15 with a long guard interval. The error rate model used here is based on Additive White Gaussian Noise (AWGN) channel. The Wi-Fi rate control provided by this model is an extension to the ns-3 IdealWifiManager. This rate control adaptively updates the transmission rate based on the last reported signal to noise ratio (SNR) level for a successful reception by the receiver.

LAA Model
The LBT capability of the LAA model in ns-3 is based on category 4 design [4], according to which the contention window (CW) varies based on the exponential backoff. The size of the CW changes between CWmin=15 and CWmax=63 in case of collision. The CW is updated on the basis of HARQ feedback from the UEs, if the 80% of the feedbacks are NACKs [8]. According to [4], this model supports CCA-ED functionality to provide fair coexistence to Wi-Fi. The CCA-ED threshold used in our simulations is -72dBm. We consider that both LAA nodes and Wi-Fi nodes use the same channel 36 of 20 MHz. The transmitter can occupy the channel if the medium is considered to be idle more than the initial CCA (ICCA) defer time; otherwise an extended CCA (ECCA) producer is applied [4]. The ICCA and ECCA defer period and their slot time is set to 43µsec and 9µsec, respectively.
Reservation signals are modeled to occupy the channel until the first subframe of data, to force non LAA nodes to backoff. The length of these reservation signals is uniformly distributed 0 and 1 msec. The discovery reference signals (DRS) are sent with the set periodicity of 80 msec during the discovery measurement timing configuration (DMTC) window of 6 msec between subframe 0 and subframe 5 in the unlicensed band. In absence of the data, DRS is modeled to occupy 1 msec, otherwise it is embedded with data. The model channels the Master Information Block (MIB) and System Information Block 1 (SIB1) through the primary cell. Currently, the LTE module of ns-3 supports both SISO and MIMO implementations, in this study we only use the MIMO model supporting up to MCS 28.

LTE-U Model
We have improved the ns-3 lte-wifi-coexistence module available here [1], with LTE-U functionalities [12] [13]. Similar to LAA model, here we also assume that both LTE-U nodes and Wi-Fi nodes use the same channel 36 of 20 MHz. We support a LTEU implementation based on the specifications provided by LTE-U forum in [12]. The LTE-U discovery signals (LDS) are modeled in the same way as DRS in LAA model and are transmitted with the same 80 msec periodicity. In contrast to LAA, MIB and SIB1 are sent through the secondary cell (i.e., in Unlicensed band) with the periodicity of 10 ms and 20 ms, respectively, when data is available. MIB minimum periodicity is 160 ms, meaning that if it has not been sent during 160 ms, it will be transmitted without data. We consider a symmetric scenario, where both operators deploy the same number of cells, and are injecting the same amount of traffic. The duty cycle period in our simulations is 80 msec. The maximum ON time a LTE-U node could transmit consecutively is 20 msec with puncturing of 1 msec between two consecutive 20 msec transmission blocks. We note that, in this work, we consider a fixed duty cycle of 50% to not evaluate specific proprietary solutions, but only the specifications defined by LTE-U forum.

RESULTS
In this section, after discussing the simulation scenario, we first present LAA, LTE-U and Wi-Fi performance by ana- lyzing the CDF plots of their throughput and latency, when they coexist with another Wi-Fi network in the indoor scenario as defined by 3GPP in [4]. In ns-3, we are calculating throughput and latency by using the built-in FlowMonitor tool that tracks per-flow statistics at the IP layer. We then post-process these results to obtain the CDFs. Later, on the basis of the statistical approach discussed in Section 2, we perform a statistical analysis on the throughput and latency distributions to evaluate the fairness of LAA and LTE-U networks towards Wi-Fi.

Simulation Scenario
As shown in Fig. 1, we consider the indoor simulation scenario proposed for coexistence evaluations by 3GPP [4]. In the indoor scenario, two operators deploy four small cells in a building with the dimension 120x50 meters with no walls. The four base stations for each operator are equally spaced, while the base stations from the two operators are placed with an offset on the X axis. There are 20 UEs/STAs per operator which are randomly dropped inside the building. The unlicensed 20 MHz band at 5.180 GHz is shared by both the operators. The licensed band for ULTE networks are not simulated here.
As for the traffic model, we consider the FTP 1 model proposed by 3GPP in [4]. We consider the maximum allowed λ=2.5. Moreover, for the propagation model, we use 802.11ax indoor model for all the small cells in the scenario. Simulations are run for 52 sec for all scenarios. 2. LTE-U over Wi-Fi: LTE-U network of operator A coexists with Wi-Fi network of operator B. In this scenario, operator A network is substituted by LTE-U and we present the throughput of both LTE-U and Wi-Fi networks.

LAA vs LTE-U performance analysis
3. LAA over Wi-Fi: LAA network of operator A coexists with Wi-Fi network of operator B. In contrast to previous scenario, operator A network is substituted by LAA and the throughput of both LAA network of operator A and Wi-Fi network of operator B are presented.
Let us start by analyzing the LAA coexistence behavior. When Wi-Fi coexists with LAA, we observe that in most of the cases the LAA network is coexisting fairly with the Wi-Fi network, allowing it to achieve the same throughput as it has achieved in Wi-Fi over Wi-Fi scenario. However, in some cases Wi-Fi experiences medium and low throughput flows mainly due to: 1) Transmission of reservation and control signals, i.e., periodic transmissions of discovery reference signals (DRS) which occupy one subframe (1 msec), in case they have to be transmitted without data. These control signals occupy more channel than Wi-Fi beacons and interrupt Wi-Fi flows, causing more contention, and increase in latency as shown  Fig. 3. This prevents these flows from reaching the maximum throughout.
2) Collisions due to hidden terminals. We observe that 1.27% of signals experience collisions caused by nodes below the Wi-Fi CCA-ED threshold (-62dBm) and LAA ED threshold (-72dBm). These low throughput flows would benefit from further lowering the LAA and Wi-Fi ED thresholds to -82dBm or from LAA supporting CTS2Self [3]. This would enable the Wi-Fi nodes to backoff upon detecting the CTS2Self messages, which happens at -82dBm or below. As per LAA throughput, the great majority of the flows achieve the maximum throughput, while three very low throughput flows are observed due to the hidden terminal problem, as mentioned above. In terms of latency, the same three flows suffer from high latency, as shown in Fig. 3. As discussed above for Wi-Fi, these low throughput flows can also be recovered by implementing CTS2Self functionality in LAA. When co-existing with LTE-U, Wi-Fi throughput is mainly affected by the fact that LTE-U starts transmitting without first listening to the medium, as explained in Section 1. This increases the chances to collide with on going Wi-Fi transmissions. In particular, we observe 1.27% of collisions in case of LAA and 2.83% of collisions in LTE-U, which represents a 55% increment. On the other hand, LTE-U performance as compared to LAA, is mainly degraded due to three reasons: 1) Increased number of possible collisions with the ongoing Wi-Fi transmissions due to the duty cycle transition from OFF period (T-OFF) to ON period (T-ON), as mentioned earlier.
2) The lack of LBT capability makes LTE-U nodes not backoff to each other and so they may happen to coexist with frequency reuse 1. This increases the spectral efficiency of LTE-U network at the the cost of increased inter-cell interference. As a result, we observe lower MCS values and an increased number of Hybrid-ARQ (HARQ) retransmissions.
3) High latency experienced by the flows which are interrupted by LTE-U OFF periods. In our simulation, with the maximum achievable throughput, one file transfer of 0.5 MB takes ∼30 msec to complete. Consequently, the flows which are unable to complete during one T-ON period, i.e., 40 msec in our case with 50% duty cycle, have to wait for additional T-OFF time of 40 msec to resume. Therefore, these flows experience high latencies as compared to LAA, as it is shown in Fig. 3. Contrarily, those flows which in turn are able to complete their transmission during one T-ON period are able to achieve high throughput values comparable to those offered by LAA.
From this analysis of results, we observe that Wi-Fi curves when coexisting with LAA and LTE-U are similar, but they both show some divergence. We intuitively can say that LAA seems more fair, but we are unable to state up to which extent LTE is fair with Wi-Fi. As a result, in the following subsection we proceed to statistically analyzing and comparing the throughput and latency distribution to get more insights on the fairness of the coexistence performance.

Statistical analysis of fairness
As per our discussion in Section 2, we use a single-sided two sample KS-test to systematically estimate the fairness achieved in both LAA and LTE-U scenarios. In particular, we use the Stats package of R [2], which is a widely used open source tool in statistical studies. As shown in Table 1, the obtained P-values for throughput and latency are less than α (i.e., 0.05 in our case). This indicates that at significance level of 0.05, KS-test rejects the null hypothesis, i.e., the throughput distributions obtained from Wi-Fi over LAA do not stochastically dominate the throughput distribution of Wi-Fi over Wi-Fi, and the latency distributions obtained in the same scenario do stochastically dominate the throughput distribution of Wi-Fi over Wi-Fi. Similarly, for the KS-test of Wi-Fi over LTE-U, the resulting P-values for throughput and latency are much less than the significant level of α, indicating that we have no evidence for H0 to be true. The P-values which were very low and statistically insignificant are represented by P 0.05 in Table 1. We reach to the same conclusion when analyzing the Dmax in Table 1. We observe that in all cases Dmax is higher than the Dcrit value.
On the basis of these statistical results, we can conclude that for the indoor scenario defined by 3GPP that we evalu-

CONCLUSION
In this paper, we have presented the comparative study of a coexistence performance of the ULTE (i.e., LTE-U and LAA) network, when coexisting with the Wi-Fi network. We have started from the 3GPP definition of the fairness and methodology for evaluating the fairness. Specifically, we have followed the same methodology and evaluated the coexistence performance in terms of the throughput and latency in the indoor scenario. For that we have designed and modeled both ns-3 LTE-WiFi coexistence models of LAA and LTE-U, which allows for full protocol stack and 3GPP and LTE-U forum compliant evaluations. After analyzing the results, we conclude that, it is not easy to claim in a quantitative way that the behavior of the ULTE network is actually fair/unfair towards the Wi-Fi network. Therefore, we have proposed a formal framework based on the statistical data analysis to evaluate the concept of fairness. In particular, we have mapped the 3GPP fairness concept onto the first order stochastic dominance concept. We have tested the hypothesis of such a defined fairness through the KS-test.
Based on the results, ULTE behavior when 3GPP or LTE-U forum specs are followed, needs the support of proprietary solutions on top to allow a fair coexistence with WiFi technology. As a result, further work is required to improve the management of spectrum resources and solutions strictly complying the specifications are not enough for guaranteeing fairness. On the other hand, from the analysis conducted, it emerges that LAA, which provides access to the channel in a much similar way as Wi-Fi does, appears as a preferred technology for fair coexistence, when compared to LTE-U. This result should be further confirmed after considering different traffic models, and different scenario configurations, in the context of full protocol stack evaluations, which can be enabled by the open source simulator that we have modeled.
Finally, future works in the area of fairness will deal with generalizing the proposed framework by considering also second order stochastic dominance concepts, which would allow concluding on the fairness even in cases of non-monotonic curves intersecting between each other.