Colorado WAM separations standards targets of opportunity and flight test analysis

The Federal Aviation Administration (FAA) Surveillance and Broadcast Services (SBS) Program Office and the Colorado Department of Transportation are implementing Wide Area Multilateration (WAM) in Non-Radar Airspace (NRA) to improve air traffic services at and around routes and fixes that support operations at Hayden/Yampa Valley (HDN), Craig-Moffat (CAG), Steamboat Springs/Bob Adams Field (SBS), and Garfield County Regional - Rifle (RIL) airports. In particular, the lack of comprehensive radar surveillance at and below 10,000 ft. in these regions requires controllers to use procedural separation standards for the Instrument Flight Rules (IFR) arriving/departing aircraft. While this is a safe means of providing service, it is inefficient for current traffic and especially for expected demand growth. Wide Area Multilateration is a distributed surveillance technology that utilizes a constellation of ground stations to provide surveillance coverage within a defined region. This technology makes use of signals transmitted from Air Traffic Control Radio Beacon System (ATCRBS) (Modes A and C) and Mode S transponders, in response to interrogations. The Mode S transponders also provide the squitter message once per second. The implementation, certification, and commissioning of WAM would enable air traffic controllers to apply more efficient separation standards for aircraft operating in the affected airspaces. For this operational environment, with WAM as the surveillance technology, the FAA seeks to achieve 5 NMI lateral/longitudinal separations. This paper describes the technical results from the data modeling, controlled flight test, and targets of opportunity analysis for the WAM sensor constellations near HDN and RIL to support separation standards within the Host Computer System (HCS) automation platform environment at Denver Center (ZDV). Comparative analysis was conducted between WAM and Secondary. U.S. Government work not protected by U.S. Copyright. Surveillance Radar (SSR) to evaluate and validate WAM performance to support separation services.

Wide Area Multilateration is a distributed surveillance technology that utilizes a constellation of ground stations to provide surveillance coverage within a defined region. This technology makes use of signals transmitted from Air Traffic Control Radio Beacon System (ATCRBS) (Modes A and C) and Mode S transponders, in response to interrogations. The Mode S transponders also provide the squitter message once per second.
The implementation, certification, and commissioning of WAM would enable air traffic controllers to apply more efficient separation standards for aircraft operating in the affected airspaces. For this operational environment, with WAM as the surveillance technology, the FAA seeks to achieve 5 NMI lateral/longitudinal separations. This paper describes the technical results from the data modeling, controlled flight test, and targets of opportunity analysis for the WAM sensor constellations near HDN and RIL to support separation standards within the Host Computer System (HCS) automation platform environment at Denver Center (ZDV). Comparative analysis was conducted between WAM and Secondary Surveillance Radar (SSR) to evaluate and validate WAM performance to support separation services.

Background
The FAA's SBS program formed the Separation Standards Work Group (SSWG) whose charter is to evaluate and assess new surveillance technologies to support existing FAA separation standards, as defined in FAA Order 7110.65. The SSWG was initially created to conduct technical activities necessary to evaluate and assess Automatic Dependent Surveillance -Broadcast (ADS-B) as a surveillance sensor to support separation services at the various key sites identified within the ADS-B program. The SSWG was tasked, in addition to assessing ADS-B as a surveillance sensor, to evaluate and assess WAM in an enroute environment interfacing with the HCS.
The SSWG organized and conducted controlled flight testing in Colorado during February 2009 to collect position data in airspaces where both WAM and secondary surveillance coverage existed, and GPS-based "Truth" data could be simultaneously recorded on board the test aircraft. Aircraft leased from Ohio University and the FAA Aviation Systems Standards (AVN) organization conducted flight profiles in both the HDN and RIL areas. Flights were only operated in airspace where WAM coverage was determined to meet its performance requirements.
Data was collected from both the MSSR and WAM prior to input into the automation system as well as the System Analysis Recording (SAR) data from the automation system. The radar data was collected by the FAA's William J. Hughes Technical Center, Atlantic City, NJ (the Tech Center), and processed into individual files for analysis. The HCS data obtained through Denver Center was replayed at the Tech Center, thereby generating SAR data that was further processed and subsequently used in this analysis.

Coverage Regions
Figure 1 depicts the MSSR and WAM coverage regions for the HDN and RIL surveillance areas (as limited by terrain). These are the areas where both TOO data and flight test data were collected.

Figure 1. HDN/RIL Coverage Regions
The green areas show where only WAM coverage is available, whereas the blue areas show were only MSSR data is available, and the red areas show where both MSSR and WAM coverage are available.
The lower set of navigation aides refers to the RIL virtual radar area, and the upper set refers to the HDN radar area.

Targets of Opportunity
A TOO pair is two aircraft tracks for each surveillance system that have been found to occur substantially during the same time period, and have the following analytical characteristics: • Either aircraft may be tracking in any direction relative to each other. • Both aircraft are at any altitude relative to each other.
• Both aircraft must be within GJT and RIL coverage.
• When pairing the two aircraft positions at different times to determine separation distances, the number of "close enough matching times" must be more than four (4).
• The time differences between position reports used to calculate separation distances are no more than 7.5 seconds apart. The primary analytical measure in the TOO analysis is the difference in separation distances between the two aircraft in a TOO pair. This difference is calculated between what the GJT radar indicates versus what the RIL virtual radar indicates. This is commonly called "the difference of separation distances," or simply the "separation differences." One potentially important source of bias in the data is the time difference between updates. Not every aircraft has a reported position by each of the sensors at the same absolute time. This analysis allows for up to 7.5 seconds of difference between consecutive radar and virtual radar reports. Since the two radars, GJT and RIL, update at different rates, 9 seconds versus 12 seconds, respectively, a filtering calculation must be made to normalize one system's position report with the other system's position report. The approach taken in this analysis is to utilize the analytical features of Bézier curves to interpolate the GJT position information relative to the RIL time updates. This mitigation of bias is especially important when one of the aircraft in a TOO pair is turning relative to the track of the other aircraft, especially in the presence of missing data.

TOO Data
The radar primarily used in both the RIL and HDN areas is the Grand Junction (GJT) Air Route Surveillance Radar (ARSR). This is the primary radar in all sort boxes in which data was collected. The Tech Center records both sets of radar data from the En-Route Communications Gateway (ECG). See

Figure 2. TOO Data Collection
The RIL virtual radar data is supplied from 20 Remote Units (RU) in the RIL and HDN regions. The WAM system gathers its data (updated approximately once per second) and "down samples" to provide data every 12 seconds (and reports this data in the same format as would a non-virtual radar). The GJT radar provides updates every 9 seconds.

TOO Data Processing
TOO data processing consists of four steps (see

Track Comparisons
The two sets of data, RIL and GJT, were extracted and put into a common data format and stored into separate comma-delimited ASCII files for the convenience of further processing as text as well as data. The beacon codes were sorted and a beacon code list was produced for each file. An indexing matrix was also produced that identified which data item belongs to which beacon code.
The two beacon code lists were compared for commonality and only those which were common to both RIL and GJT data sets were accepted for further processing. The matched beacon codes and their indexing matrix are then used to find common time elements. The data points in the RIL track are then compared to the data elements in the GJT track to find common points that are within +-7.5 seconds from each other. See Figure 3. If a point cannot be found in the GJT data that is within the +-7.5 seconds of a RIL data point, then that data point is removed from the RIL data set.
If after completing this time comparison the track is reduced below 4 accepted data points, the whole track is removed from the track list. At the end of this process a list of usable tracks has been identified which consists of a GJT and RIL data set of same length and common in time (to within +-7.5 seconds).

Interpolation
The track pair list, consisting of filtered GJT and RIL data, was then interpolated so that the GJT position data is shifted in time to match the RIL time. The interpolation process uses Bézier curve fitting calculations that utilize only four points.
During the interpolation process, if a point were unable to be modified and shifted in time to match the RIL time, the point was removed from the track pair. Additionally, if after the interpolation process the track pair count became less than 4 points, the track was removed from the list. See Figure 4.

Figure 4. TOO Data Track Comparison
The position data in the interpolated track list was then converted from range and azimuth to decimal WGS-84 latitude and longitude. The range and azimuth was first converted into Cartesian coordinates, then the senor origin was used to map into decimal latitude and longitude values. Once the position information was converted, the time and altitude information can be retained in its original format and passed with the converted data.
There are several important analysis benefits from these interpolation methods.
• Radar raw data timestamps and Virtual Radar raw data timestamps rarely agree, and are most commonly more than a few seconds apart for the same aircraft.
• Using the raw data to calculate separation differences without accounting for the time differences introduces additional error in the summary statistics by using incompatible Radar position reports with Virtual Radar position reports.
• Bézier curve interpolation addresses the time difference problem for all types of tracks, e.g., straight line, gradual turn, sharp curve, etc.
The methods used to implement interpolation have been fine-tuned to mitigate the variability in the data.
• The interpolation reduces the additional error in the summary statistics due to time differences by syncing the position reports for Radar and Virtual Radar.
• The interpolation adjusts for the random error found in all raw data position reports.
• The separation differences using the interpolated Radar track has a significantly smaller average error due to timestamp differences than when using the raw data. The analytical benefits to the analysis from these interpolation methods accomplish several important goals.
• Reduction in average additional error in summary statistics. • Minor increase in variability of results which does not affect conclusions. • Elimination of "geometry" as a source of additional error.

Finding TOO Pairs
The interpolated track pair list was now parsed and every track was compared iteratively with all other tracks using the same Track Comparison process as before. Every point in Track 1 was compared to every point in Track 1+n where n is the length of the track list. If the time comparison produces a data set with less than 4 points, that particular TOO pair was not considered further. See Figure 6.
The output from this step was a data set with 4 tracks, each with the same number of points. The TOO pair data set consists of: •

Statistical Calculations
Once the TOO pairs had been extracted and were of equal lengths, the separation differences were calculated. See Figure 7. The following formulas were used:

Figure 7. TOO Pair Statistical Calculations
The statistical values that were calculated from the separation differences were:

Flight Tests
Six types of flight test profiles were developed to position two aircraft 5 NMI apart in trailing and 4.D.3-6 parallel configurations. All six profiles place the two aircraft over both the Rifle (RIL) and Haden (HDN) coverage areas. The flight tests required that both WAM and Radar have surveillance coverage over the test area at 15,000 ft MSL. GPS "Truth" data were collected from both aircraft to compare to the data collected off the RIL Virtual Radar and GJT radar.
The primary analytical measure in the flight test analysis is the difference in "true" separations, that is, the difference between the GJT radar position and the "true" position, versus the difference between the RIL virtual radar position and the "true" position. It is this "difference of differences" that measures the extent to which MSSR and WAM report an aircraft's position differently.

The Flight Profiles
The six flight profiles were developed to space two aircraft 5 NMI apart in multiple types of trailing and parallel configurations. This simulates the relative movement of aircraft that TOO pairs would also exhibit. The six profile types are: • In Trail Radial Approach procedure information and existing navigational aides, e.g., VOR, aided in the scenario development, and are the primary reference points for each scenario.

In Trail Radial
This scenario features two aircraft flying on a radial extending out (constant azimuth) from the GJT Radar in an in-trail formation, separated by ~5 NMI. See Figure 8.

Parallel Radial
This scenario features two aircraft flying tangential to the GJT radar in an in-trail formation, separated by ~5 NMI. See Figure 9.

Parallel Arc
This scenario features two aircraft flying tangentially to a VOR within coverage of both surveillance GJT and RIL sensors separated by ~5 NMI in co-centric orbits 10 NMI and 15 NMI, for both the HDN and RIL regions. See Figure 11.

Parallel Tangential
This scenario features two aircraft flying in parallel tracks separated by ~5 NMI on radials towards/from a VOR within the coverage of both surveillance sensors for both the HDN and RIL regions. See Figure 12.

Data Processing Elements
The data processing of flight test data consists of three elements (see Figure 14): • Common Formatting The three sets of data, "Truth," RIL, and GJT, were processed into a common data format based on desired beacon codes. The RIL and GJT data were extracted into separate comma-delimited ASCII files for the convenience of further processing as text as well as data. The range and azimuth form of radar data was transformed into decimal latitude and longitude by first converting into Cartesian coordinates then using the sensor origin to map into WGS-84 latitude and longitude values. Once the position information was converted, the time and altitude information was retained in its original format and passed with the converted data.

Scenario Extraction
Once the data had been extracted and converted it had to be filtered so that the retained data only had elements that were common between all three sets ("Truth," RIL, and GJT). This means for every point in the RIL data a match close enough in time for the GJT and "Truth" data must be found. If a match close in time cannot be found then that RIL data point was dropped. A data point is considered close in time if it is within 7.5 seconds, either earlier or later. Figure 13 depicts a flight test where the data points for all three data sets are within 1 second of each other, except for several data points that were removed due to the 7.5 second limit. Since some parts of the scenarios fall outside the WAM coverage area the limiting sensor was RIL (the virtual radar data).
Once the totality of the data had been filtered to only include common points, the data was then broken into the six scenarios based on the recorded times during the flight test monitoring.
Each extracted scenario data set resulted in six data sets (tracks) all of equal lengths: • Radar Aircraft 1 • Radar Aircraft 2 • Virtual Radar Aircraft 1 • Virtual Radar Aircraft 2 • GPS Truth Aircraft 1 • GPS Truth Aircraft 2

Statistical Calculations
Once the scenarios had been extracted and were of equal lengths, the separation error calculations were computed. The following formulas were used. The following statistics were also calculated to measure the extent and direction of the separation differences: •

TOO Pairs
Since there is no "Truth" data for TOO analysis purposes, two calculation issues must be considered in interpreting any results: (a) Error attributable to the 7.5 second allowance made in calculating separation differences between TOO pairs, and (b) the use, or non-use, of analytical interpolation techniques to account for missing data during significant time periods.
To demonstrate the effect of the 7.5 second allowance, consider the histogram and cumulative distribution graph of the separation differences (as measured in nautical miles [nm or nmi]) as shown in Figure 16 with summary statistics in Table 1.  Now compare the results shown in Figure 17 with summary statistics in Table 2, where the same data was used as in Figure 16 and Table 1, but this time only a 2 second allowance is allowed. Notice that the number of matching points decreases significantly when a maximum of 2 seconds is allowed between position reports for the two TOO pair aircraft (as compared to the 7.5 second allowance). Notice also that the measures of centrality and extent of variability have also significantly decreased (moved towards zero).  These graphics demonstrate how the allowance affects the summary statistics, with the general pattern that the measures of centrality and extent of variation are generally reduced towards zero as the calculation allowance for track matching is reduced.
In fact, further reducing the allowance for separation difference calculations to 1 second results in a further reduction towards zero of the summary statistics while retaining the same shape and position for the histogram and cumulative graphic, and while retaining a large enough critical value for meaningful inference. However, the amount of data available for analysis using 1 second is so small as to compromise the confidence with which the results may be stated.
Furthermore, the use of Bézier curve interpolation to "fill in" the gaps of missing data during significant time periods was found to have a minimal effect on the summary statistics, and did not materially affect the inference of the calculation results. In effect, when both TOO pair tracks had missing data at the same time, the interpolation methods used in the analysis produced position 4.D.3-11 estimates whose differences were consistent with those found in the observed data. When only one of the TOO pair track had missing data, the interpolation methods tended to increase the separation differences, especially when the track was a straight line. Even with these inflated separation differences due to interpolation, the consistently low chi-squared statistics (compared to their critical values) demonstrated that the interpolation method did not materially affect the conclusions of the analysis. Figure 18 depicts the histograms and cumulative distribution graphs of the separation differences observed from the 191 usable points in the flight scenarios flown near the RIL region. The two graphics on the left refer to the separation differences between position reports from the GJT radar and the "Truth" source, while the two graphics on the right refer to the separation difference between position reports from the RIL virtual radar and the "Truth" source. The values shown in Table 3 are the calculated statistics for these graphics.  Note that the scale of the histogram on the right is significantly smaller than for the histogram on the left. This makes the distribution on the right appear to be more spread out relative to the distribution on the left, even though the standard deviations in Table 3 prove otherwise. Note also that the mean and median are essentially zero, and the chi-square value is significantly smaller than the critical value. Only 10% of the data is more than one-tenth of a nautical mile from zero, and these results apply equally to Radar/Truth data and Virtual Radar/Truth data.

4.D.3-12
In particular, for the parallel arc flight test profile, Figure 19 depicts the point-by-point separation differences for a short period of the test, and The summary statistics for this flight test profile indicate the separation errors were consistent with the summary statistics for the entire RIL region, even though the measures of centrality were slightly higher here. The time delta between tracks discrepancies (lower right graph in Figure 17) were more than 1 second on one occasion, when a "Truth" report was missed (note that the top vertical value on this scale has an exponent of 10 to the -11). This missing datum does not appear to account for any particular large separation errors. However, the size of the critical value indicates any inference we make from these statistics may be made with a high level of certainty.
In general, the graphics and summary statistics for the HDN region look similar to those shown for the RIL region, except for the observation periods (when the flight tests were actually performed).

Conclusions
Based on the totality of TOO data analysis, and taking into account the sensitivities of the interpolation methods and time limits for calculating separation differences, the following conclusions may be stated at a high level of certainty.
• Day after day, with very little variation, the cumulative graph curve consistently crosses 0 at the same cumulative percentage (80%). This indicates the negative bias found in the data is due to calculation "features," rather than a systematic difference in how the 4.D.3-13 virtual radar system reports separations compared to the radar system. • The average calculated separation differences between radar and virtual Radar is consistently at -0.25 nautical mile (with 7.5 second allowance) to -0.08 nautical mile (with 2 second allowance), with little variation (with fixed allowance). This is the negative bias.
• The chi-square statistic strongly indicates that the negative biases are due to calculation limitations in the analysis, rather than a systematic difference in how the two systems report separations.
• With hundreds of TOO pair calculations, the chi-square statistic has a lot of "power" (the ability to detect significant differences even when those differences are small relative to the magnitude of the numbers in the data).
• Since the chi square statistic is always less than the corresponding 95% critical value, by as much as a factor in the thousands in some cases, the bias may be considered insignificant, since it may be eliminated by more stringent separation calculations.
• Therefore, regardless of distance from origin, relative movement speed and direction, type of aircraft, or any other observed characteristic, the virtual radar system reports the separation of two aircraft with essentially the same relative accuracy as does the radar system. • These results strongly indicate that the virtual radar system performs with indistinguishable relative accuracy compared to the radar system. • The amount of data available, even during short time periods, makes these conclusions highly reliable, in the sense that the probability that some unobserved phenomena explain the data in a more precise manner is very, very small.
With respect to the flight test data, the following conclusions follow from the analysis.
• Virtual radar reported separation was highly accurate for both RIL and HDN areas.
• The mean and standard deviation of separation errors for the virtual radar were smaller and more accurate than the separation errors measured for the GJT, a Mode S radar. • Individual scenarios confirm that the larger sets of statistics, for the entire RIL and HDN regions, properly characterize the data; the individual scenarios were self-similar to the larger distributions.
• Sources of anomalies observed were primarily due to timing issues (matching data between flight tracks and with "Truth" data). No significant errors were observed during the flight test.
• In particular, each of the summary statistics tables confirms that the radar and the virtual radar data may be used interchangeably for separation calculations, as both systems report separations with essentially the same measure of centrality (mean and median) and extent of variability (standard deviation and chi-square). Statistical analysis methods confirm these results with a high level of certainty.
• In addition, missing and clearly erroneous data, and communication drop-out periods, account for all outlier results and anomalous situations found in flight test scenario data.

Summary
The Separation Standards team conducted separate comparative assessments for WAM against MSSR, which consisted of controlled Flight Testing and Targets of Opportunity.
Targets of Opportunity pairs were selected within a volume of airspace with both WAM and MSSR coverage in Colorado. Analysts filtered the observed data to find comparable pairs of aircraft whose separations could be assessed in a statistically significant manner. The assumption of acceptable asynchrony in sensor updates used as a data filter property (default of ±7.5 seconds) is the dominant factor in the constant bias, and not a systematic error issue between radar and virtual radar systems. Statistical analysis methods confirm these results with a very high level of certainty.

4.D.3-14
Specific flight profiles were flown by Ohio University and the FAA AVN organization in airspace with overlapping coverage by WAM and MSSR. The results showed regardless of the relative position and movement of the targets, separation error with virtual radar data was smaller in standard deviation and 5 th /95 th percentile values than the GJT. Statistical analysis methods confirm these results with a high level of certainty.
The results for the analytical activities shows WAM performance related to separation services is the equivalent of secondary surveillance radar. Missing and clearly erroneous data from both WAM and radar, and communication drop-out periods, account for all outlier results and anomalous situations found in both TOO and flight test data. More stringent criteria for accepting data would only strengthen the certainty of these conclusions.