S3: the Spectrum Sharing Simulator

Changes in U.S. spectrum licensing require on-going shared use of spectrum bands between commercial and government services. This requires accurate tools to estimate mutual interference between different technologies, over different spatial scales, numbers of transmitters, and using complex protocols such as LTE (Long Term Evolution) operating in multiple bands simultaneously. Current interference simulation tools are inadequate. The Spectrum Sharing Simulator Project is enhancing the open source ns-3 simulator to support much larger problems, easier parallel execution, and enhanced LTE models.


INTRODUCTION
The U.S. Federal Communications Commission has developed rules for the auction of frequency bands previously dedicated to government use. The intended use of these bands is for licensed advanced wireless services (AWS), expected to be primarily wireless broadband communications utilizing the 3GPP standards, such as LTE and New Radio (NR) 1 [1, 2]. The AWS-3 auction instituted sharing regimes with incumbent systems which cannot be relocated. To coordinate shared use, which in some cases may be permanent, U.S. Government Departments have instituted assessment mtehods to ensure interference-free operation between Federal and commercial systems as they enter the band.
Understanding in detail how LTE operation potentially leads to interfering emissions is key to these analysis processes: • Which specific LTE signals are most harmful, in aggregate?
• Can specific LTE sectors be operated to reduce emissions? • Do urban or rural settings and terrain impact emissions?
• How do high-density deployments (e.g, sporting and concert events) differ in emissions? • How should one aggregate interference from all sectors and handsets in a large metropolitan region?
Here we introduce S3: the Spectrum Sharing Simulator. The goal of the S3 Project is to enhance the open-source ns-3 simulator in order to address these questions [3]. This requires significant development of ns-3 as a parallel discrete event simulator, the addition of new methods to model shared channels in a distributed simulation, and the implementation of new LTE/NR features within ns-3 to keep pace with successive 3GPP releases. This paper is organized as follows: Section 2 provides background on the motivation for this project, and the types and scale of wireless simulation required; Section 3 discusses existing simulation capabilities and their limitations; Section 4 introduces S3: the Spectrum Sharing Simulator; Section 5 discusses enabling technologies in simulation and LTE modeling that make this project possible; and finally Section 6 concludes with a summary of the project development timeline.

BACKGROUND
The Project focus is assessing interference with systems of the US Department of Defense (DoD). The Project sponsor, the DoD Defense Spectrum Organization, is responsible for enabling early entry of AWS-3 licensees into the band and managing permanent sharing regimes for DoD systems which will continue to operate in band indefinitely. In particular, the Spectrum Sharing Test and Demonstration Program (SST&D) has two objectives: facilitate expedited and expanded early entry of licensee deployments into Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s There are two aspects to this mission. First, LTE systems may receive interference from DoD transmitters. Second LTE emissions can interfere with DoD receivers. In this case the goal is to determine how LTE aggregate emissions behave over a wide, metropolitan area and over extended periods of time, including effects of terrain and special events. Many of these conditions can only be simulated and cannot be measured which makes a dependable and validated simulation model critical.

EXISTING WIRELESS SIMULATORS
There are two primary challenges for modeling interference in large scale wireless models: first, simulating many times many base stations (eNodeB) and handsets (UE) utilizing a single shared channel, including full node mobility throughout a large metropolitan region with a million UEs; and second, accurately modeling interference from this large set of spatially and temporally distributed transmitters. Neither of these is particularly difficult if the entire model scenario can be run in a single process memory space, but this becomes intractable beyond a few dozen eNodeB and few thousand UEs. Executing such scenarios would be excruciatingly slow and require vastly more system memory than commonly available.
While there are a variety of LTE simulators, most are single process only; and the few that support distributed (multi-process) execution have not demonstrated anything near the required scale. For example, LTE-Sim is a standalone LTE simulator, but works only in a single process space [4] [5]. OMNeT++ is a general purpose discrete event simulator with a limited parallel execution capability, which has been tested to a limited scale [6]. When run sequentially OMNeT++ and ns-3 (see below) have similar execution and memory performance [7]. The INET module for OMNeT++ supports network simulation generally. This module has been enhanced with threading support and demonstrated speedups of 30% with 10 threads [8]. A different effort added distributed parallelism support to INET and achieved speedup of eight on 12 processors [9].
By contrast, ns-3 scales to 500K simulated nodes and 300 processors, at least for wired networks [3,[10][11][12][13][14]. ns-3 is a portable, research-oriented, discrete-event network simulator Development started in 2005, with 29 releases since 2008, supported by funding from the US NSF, INRIA in France, and several other public and private organizations. ns-3, and its predecessor ns-2 are the most frequently cited tools used in computer network research 2 . Consequently, the S3 Project is based on enhancing the ns-3 simulator to address the challenges described above.

S3: THE SPECTRUM SHARING SIMULATOR
The objective of the S3 Project is to develop a scalable (parallel, optimistic, dynamic load-balancing) discrete event simulator ("the Simulator") capable of analyzing mutual interference between current and near future LTE 4G/LTE-Advanced/5G-New Radio cell phone systems and DoD assets. Also as part of this project the Simulator will be used to: • Cross-validate with the UE equivalent isotropic radiated power (EIRP) cumulative distribution curves developed by the U.S. Commerce Department for urban and rural (point) scenarios (CSMAC) [15], • Conduct parameter studies around the CSMAC points.
• Extend those studies to include dynamic features of LTE, including behavior driven by eNodeB scheduler algorithms; • Analyze regional-scale scenarios; and • Analyze scenarios using LTE in DoD tactical operations.
An integral part of this work will be to validate results against real-world data acquired by SST&D. To support these analyses in real-world settings and at regional scale, the Simulator will be developed to model: real propagation effects over real terrain; specific dynamic features of current or near-future LTE/NR standards; and LTE/NR interference and mitigating control features between modeled LTE/NR cells in large scale scenarios. Each major modeling feature will be validated against SST&D data. In order to provide the best execution performance, the Simulator will be a Parallel Discrete Event Simulator (PDES) using "optimistic" synchronization and supporting dynamic load balancing across the computing cluster.
The ns-3 LTE module, commonly referred to as LENA, combines a simulated channel and physical layer model with an implementation of the LTE and EPC protocol stack that closely follows 3GPP specifications. Most of the current version of LENA was developed between 2011 and 2013 as part of an industrial project funded by Ubiquisys Ltd. (now part of Cisco), and carried out by Centre Tecnològic de Telecomunicacions de Catalunya (CTTC), with the aim of developing an open-source productoriented LTE/EPC Network Simulator, allowing LTE small/macro cell vendors to design and test Self Organized Network (SON) algorithms and solutions.
In the current version ns-3 models LTE Release 13 at the resource block level (0.5 ms and 180 kHz, typically), with nearly arbitrary time and frequency resolution for asynchronous interference/noise sources. New Release 15 NR features are also available in a separate NR module recently released, including support for NR frame structure, flexible numerologies, variable TTIs, beamforming, bandwidth part managements, etc. To support regional-scale scenarios the Simulator will be capable of representing This scale will also require new configuration tools to manage such large numbers of eNodeBs and UEs. These goals represent one to three orders of magnitude improvement in current LTE simulation capabilities, compared to any currently available simulation platform, including the current release of ns-3.

ENABLING TECHNOLOGIES
The scope and scale required to address this problem space is daunting, however there have been a number of developments in parallel discrete event simulation and LTE models which make this scope feasible.

Parallel Discrete Event Simulation
One of the key issues in PDES is synchronization of the multiple processes so that the parallel execution produces exactly the same results as would have been obtained in a sequential execution. The main difficulty is managing execution of events in the local queue with respect to event messages being sent by other processes. Historically, there have been two approaches to this problem: conservative execution, in which the simulation processes coordinate so that each process can determine which events in its queue are safe to execute; and optimistic execution, in which each process speculatively executes events, and reverses those executions ("rolls back") if it turns out that an event sent by a remote process should have been executed first [16].
Generally optimistic simulation executes faster because of the reduced synchronization overhead, despite the need for additional code and storage to support event roll back, which has been a major impediment. (On the other hand, conservative execution requires additional information from the model ("lookahead"), which is often difficult to extract.) We have developed a compilerassisted approach, dubbed Backstroke, which automatically instruments sequential model code to support roll back [17,18]. This tool supports all C and C++ constructs, including the C++ Standard Template Library. Backstroke implements a source code to source code transformation which adds instrumentation to assignment statements in model code and provides a library to record those assignments and reverse them when necessary. With Backstroke one writes just the sequential model code yet obtains the ability to execute it optimistically.
Recently we have generalized the conservative/optimistic divide into a Universal Virtual Time (UVT) approach, where each simulation process determines dynamically whether to execute the next event conservatively or optimistically [19]. This has the potential to be faster than either alone. With UVT and Backstroke a model developer can focus on writing correct sequential model code, and the simulator will enable optimistic execution. As one gains experience with the model, one can add the lookahead information and enable conservative execution, which will now act as an accelerator.
The second major issue in large scale PDES simulation is dynamic load balance. Even with a very well-behaved Gaussian distributed work load, across 1M objects about 1000 will be 3sigma behind. Charm++ offers an alternative messaging infrastructure (compare to the MPI support in ns-3 now), which supports dynamic object migration for load balancing [20]. This has been shown to be an excellent platform for PDES [21].
For S3 we plan to introduce the concept of a logical process to the ns-3 core simulator, then release the optimistic simulator, including Backstroke and UVT, running over Charm++, as a separate model. Prior work has established that only a limited number of features in ns-3 will need to be refactored [22]. Further enhancements to underlying ns-3 infrastructure will include enhanced propagation and clutter models, and a new approach to modeling distributed interference based on the fast multipole method. While the focus of the Project is on LTE interference, the resulting methods will be applicable to any ns-3 model using the spectrum-aware channel model.

LTE Model Development
We are extending ns-3 to model additional and proposed LTE/NR standards, such as: For example, these technologies anticipate refining the LTE scheduling unit, from the current 0.5 ms resource block down as small as the symbol level, or 62 µs., representing an increase in complexity of one to two orders of magnitude.
Recently, multiple lines of development of LTE models for ns-3 have emerged. A significant enhancement was announced late last year which adds support for extended RRC functionalities, compared to those previously supported in ns-3 which primarily addressed the RRC CONNECTED state. Examples of thefeatures are support for radio link failure, handover failure, cell reselection, and long-awaited support for IDLE mode, paging, etc. These features are being merged into the release version of ns-3 [24].
In parallel, two groups have been developing models for LTE and NR features. The NYU and University of Padova mmWave module is focused on physical layer modeling in the mmWave bands, and leverages ns-3 LENA for the higher protocol layers [25]. CTTC has developed a new 5G protocol model for ns-3 forked from the mmWave work. In this project, the roadmap is defined to leverage the mmWave module, along with the RRC work mentioned above, develop and release a comprehensive new model of LTE and its evolution, and New Radio, within ns-3, with the exciting computational improvements that we have discussed in previous sections [26][27][28].