Issues and research needs concerning the use of head-up displays in air transports

Head-up Displays (HUDs) are being installed in air transports in order to allow manual approaches, landings, and takeoffs in poor visibility. HUDs make it possible to overlay and augment the real-world image with conformal symbols such as a flight path marker (FPM) which indicates the direction the aircraft is heading within the out-the-window (OTW) view. While HUDs have been operational on civil transports for over 10 years, design issues remain to be resolved with further research. Relatively extensive research has highlighted HUDs' special sensitivity to clutter, but further research is needed to give designers the tools needed to compare HUD designs in a specific flight deck application. Research on unusual attitude recognition and recovery with HUDs has yielded solutions that when combined are only minimally adequate. Limitations in HUD coding abilities call out for a standard for indicating alerts, while limitations in HUD optics require more knowledge on pilot head activity.


Introduction
Background in air transports in order to allow manual approaches, landings, and takeoffs in poor visibility. HUDs make it possible to overlay and augment the real-world image with conformal symbols such as a flight path marker (FPM) which indicates the direction the aircraft is heading within the out-the-window (OTW) view. HUD symbology is also collimated, focusing well in front of the aircraft, which allows the pilot to switch rapidly between monitoring the OTW view and instruments without appreciable re-focusing or moving of the head [1] [2].
to maneuver toward the runway while primary flight indications remain in view. To land an aircraft, a pilot can simply maneuver the aircraft into a heading and attitude such that the FPM lies on the end of the runway. In poor visibility, a flight path angle appropriate for an approach can be established by bringing the FPM to the proper position below the conformal artificial horizon also provided in the HUD [3]. The pilot can then rapidly make the transition to visual when the runway comes into view. This makes it possible for pilots to manually fly approaches and landings with remarkable precision [2][4] [5]. Pilot performance while using a HUD in this manner was sufficiently accurate and reliable that by 1989, HUD-equipped airliners were certified for landing in Category IIIA conditions [1] [4].

Head-up Displays (HUDs) are being installed
With these capabilities, a HUD allows a pilot

Problem Statement
However, as with any new technology added to the flight deck, one must be careful that it does not introduce substantial new risks. Being the display for the pilot, one must in particular carefully evaluate the human factors issues associated with HUDs to ensure safety concerns do not outweigh benefits. HUDs of one form or another have been in operational aircraft since the 1950s [ 1] [3], so it would seem unlikely that there is any fundamental problem with the technology. However, concerns have been raised about specific design implementations [3] [6]. This paper reviews the research on four selected issues of concern, summarizing the conclusions and suggesting places requiring further research.
Specifically, this paper addresses the following issue areas related to HUD design:   [29]. Symbology for unusual attitude recognition and recovery has received special attention, probably in reaction to an unusual number of cases of pilot disorientation in extreme attitudes for HUD-equipped warplanes of the US military forces. These are relevant for providing design guidance for maximizing taskdisplay compatibility.

Review of Issues
There is also substantial research on Display In formation Accessibility balance must be sought between providing the useful information for the pilot but not overwhelming the presentation so that critical information is difficult to find. This is a particular concern for HUDs because HUDs are designed to display symbology that overlays other useful information that arrives OTW. Thus, HUD symbology always has the potential to occlude or mask important information. The need to minimize occlusion also requires designers to minimize the use of frames or rules to promote visual separation and organization of symbology, removing one the means of mitigating clutter.
Clutter is also a particular concern because HUDs generally have monochrome symbology (typically green). Thus, color cannot be used to perceptually separate symbology within the HUD. Color HUDs may soon be economically practical for certain high-end applications, but monochrome HUDs can be expected to be designed for the foreseeable future if for no other application than for low-cost general aviation applications.
HUD is redundant with the HDD. Therefore whatever is not on the HUD is still available to the pilot flying, although at the cost of looking head down. Thus, the fact that a piece of information is necessary is not sufficient to allow its inclusion on the HUD. It must be sufficiently important and Excess clutter is a concern on any display. A With some exceptions, most symbology on the used with sufficient frequency that looking for it head down is not acceptable.
The issue of clutter is thus more complicated with HUDs. For each indicator added to the HUD, the designer must determine both the potential for the indicator to obscure the OTW view and the potential for the indicator to interfere with other HUD symbology. These must then be weighed against the cost of locating the indicator head down.
There is little research on the effects of excluding significant indicators from the HUD in order to minimize clutter. For example, there is no research on the potential that a pilot may fail to perceive an HDD event such as a warning or caution when using a HUD. There is, however, a fair body of research on the detection of OTW events while a HUD is used. In general, the detection of OTW events is as fast or faster for pilots using HUDs than for pilots using HDDs [2][18] [21]. For HUDs with stroke-drawn indicators, effects of masking the OTW image are offset by the reduction of the need to look head down while monitoring the OTW view. This may not apply, however, to HUDs that display a raster image, such as those used with enhanced vision systems.
There is also research that indicates that events in an indicator are generally detected faster when displayed head up than head down [18] [22]. This is true especially in visual meteorological conditions (VMC), when presumably the pilot is more likely to be scanning OTW [22]. Tracking when using alphanumeric indicators is also improved when they are displayed on the HUD rather than on a separate HDD [ 181. It would appear that a collimated indicator proximal to the OTW view can be scanned more often or accurately.
However, clutter is still a concern. The display of information that is not required for the current task on a HUD has been shown to be disruptive, slowing a pilot's reaction to important OTW events such as a traffic conflict [3] [22] and also events within the HUD itself [22]. A generally accepted rule of thumb is to include only indicators that are necessary for the current task, where "necessary" means that the pilot must refer to the indicator or symbol repeatedly throughout the normal conduct of the task. For approach and landing, this includes the Basic T and vertical speed. If available and applicable, the HUD should also show lateral and vertical path deviation, the flight director, and an FPM [3]. Additional indicators may also be justified, but designers must resist pressure from customers to increase the clutter of their HUDs with indicators that can adequately be monitored head down [4]. Designers may also need to consider the likely impact of having two pilots on the flight deck. For example, an important indicator for the task may exist only head down if it is sufficient that it be monitored by a non-HUD-using pilot alone.
To mitigate clutter effects of a given set of indicators, several techniques are available. The designer can maximize the use of conformal symbology as there is some reason to believe that conformal symbology interferes less with detecting OTW events than non-conformal symbology [22].
Nonconformal symbology can be placed in the upper periphery of the display, away from the region in the OTW that contains the most critical information for the pilot, namely, directly ahead, and, for approach and landing, the lower portion of the HUD [3]. This is especially true if key information found in these indicators can be deduced from other smaller centrally located indicators. For example, if a speed error ''worm" and acceleration cue is displayed on the FPM, then the airspeed indicator may be more peripheral.
in the upper and outside portions of the display may minimize any attention trapping effects of HUD symbology due to differences in relative motion between the symbology and the OTW view [11] [17]. There is some evidence that eye motion to periphery of a HUD may discourage attention trapping [20]. This motion can be induced by placing important indicators that are part of a trained pilot's normal scan out on the periphery of the HUD. Operational experience suggests that placing such symbology between 10 and 25 degrees above or to the side of the center keeps the center adequately clear while still allowing fast access to the information in the symbology [30]. One HUD design has gone so far as to place its non-conformal heading indicator at the top of the display [30], in direct violation of the standard position below the attitude indicator [3 1] [32].
lighting or dimming of the less essential HUD Also, such placing nonconformal symbology There are some weak indications that low-

2.A.4-3
indicators can reduce their tendency to mask OTW events [21] [22]. However, it does not appear to completely eliminate the effects of clutter on detecting the OTW view and has not been shown to improve the detection of events in non-low-lighted indicators in the HUD. In general, there is apparently little one can do to mitigate the effects of clutter on the HUD itself. Increasing the brightness of the HUD overall may improve the detection of HUD events, but at the cost of speedy detection of OTW events [21]. Thus, an optimal level of brightness needs to be found. sufficient guidance to designers. There is an absence of metrics needed to evaluate designs. Clutter could be measured by calculating the simple physical masking of the OTW view, calculating the angular area covered with symbology, perhaps weighting each indicator by distance from the center, and perhaps also taking into account the symbology's brightness. This angular area measure must take into account the masking potential of the symbology, not just the actual area covered by a stroke [33].
However, measuring raw clutter alone is not sufficient to make design decisions. For example, if the measure implies that the clutter of a particular HUD is "too high," a designer may be forced to move a relatively critical indicator to the HDD. This may actually increase the probability of a pilot missing important information, either on the indicator because it is head down or OTW because the pilot is forced to look head down for the indicator. It appears the former case has not received serious study. Once a HUD is installed with some important symbology, the pilot's normal scan may be disrupted reducing likelihood of noticing HDD events. No one has studied the possibility that HUD use may inhibit perception of HDD events.
What is needed is a measure that indicates the effects of locating an indicator head up versus head down. A designer cannot design the HUD in isolation, but must include the context of the HDD in the allocation of indicators to the HUD. The measure must evaluate the entire cockpit for the effectiveness of the location of all the information, OTW, HUD, and HDD. This evaluation procedure will determine if the pilot has a sufficiently high probability of quickly receiving information The research to date provides some but not whatever its location. It must take into account not only the clutter of a display but the characteristics of each indicator such as size and contrast. For example, larger indicators with high contrast. exhibiting large motions can be located less centrally as events are easier to detect in them even parafoveally. Finally, it must be applicable to nontraditional indicators such as an acceleration cue or speed error indicator that provide alternate forms of information found in other indicators. The measure. can do this by assessing the effectiveness of the information provided, not of the indicators displayed.

Unusual Attitude Recognition and Recovery
While traditional HDD attitude indicators use relatively large areas of color and shade to distinguish sky from ground, HUDs are hampered in this by a lack of color and a need to minimize clutter [2]. Attitude indication is thus limited to the pitch ladder including a representation of the horizon line. While conventional HDD attitude indicators typically display on the order of 40 degrees of pitch range at any moment, HUDs display half of that or less owing to their limited size and use of conformal pitch lines. The net effect is that while HUDs very effectively represent pitch at normal attitudes (partly due to their conformity), unusual attitudes can be both difficult to detect and difficult to recover from when the pilot refers to the HUD alone. Traditional pitch bars look the same above and below the horizon, so, without the shaded and colored background, being pitched sharply down appears the same as a being pitched sharply up. For similar reasons being inverted appears much like being right side up. As a result, pilots are slower to detect an unusual attitude [2] [3].
Once an usual attitude is recognized, the appropriate maneuver for recovery can be difficult to determine with a HUD, delaying a time-critical response. With its limited display range of pitch and lack of background to provide an integrated image, the scene can become a confusing shower of pitch bars sweeping across the display too fast to read the value labels.
In an effort to improve the HUDs capacity to aid the pilot in unusual attitude recognition and recovery, some efforts have focused on modifying the traditional pitch bars so as to: Distinguish above the horizon from below. Graphically indicate the extremity of pitch. Indicate the direction towards the horizon or zenith. Above versus below the horizon pitch bars can differ in length, gap width, weight, color (when it becomes available), use of dashed lines, position of vertical rung marks, position of value label, and means to indicate extremity [3][24] [34]. It appears necessary to use combinations of these to achieve adequate differentiation [29][30].
Compression of the pitch scale (in the range of 1.5: 1 to 2: 1 compared to conformal for a HUD that occupies 20 degrees of the field of view) at extreme attitudes has been shown to improve unusual attitude recovery[ 151, while still retaining the benefits of conformal pitch bars at normal attitudes. Two other methods for indicating extremity of pitch have been tested: tapering (varying the length of the pitch bars) and articulation (varying the orientation of the pitch bars). Both appear to work well, although articulation should only be used with a centered FPM during unusual attitudes, either occurring automatically or through a separate l'caged" FPM symbology [3] [27]. Tapering and articulation may be combined in a single display to provide a means to distinguish pitch above versus below the horizon [27] [29][30].
The direction towards zenith or the horizon can be indicated with vertical rung marks at the ends of the pitch bars that "point" in the appropriate direction [3][23] [35]. If articulation is used to indicate extremity, this can also be used to indicate direction. However, use of the pitch bar shape alone is not sufficient because fast moving pitch bars will be difficult to read [23].
modifying the pitch bars is to add symbology dynamically during unusual attitudes to aid recognition and recovery of unusual attitudes. To control the level of clutter, symbology unnecessary for unusual attitude recovery, such as the heading indicator, can be dynamically removed.
Another approach that may be combined with

2.A.4-4
Such direction indicating symbology includes arrows or chevrons on the FPM or pitch ladder respectively. Another alternative is a "ghost" horizon, being attenuated horizon symbology at the "pegged" position. Non-conformal attitude indicators that fit inside the display have also been researched, including electronic variations of the traditional mechanical "eight ball" attitude indicator, and novel indicators such as the "orange peel" and "grapefruit." Properly implemented with multiple cues for attitude, a pilot can perform unusual attitude recognition and recovery equally well with a HUD as with a standby head-down attitude indicator [29][36]. Specifically, pilots using the military standard HUD recognized and responded to an unusual attitude within 1.5 seconds and their initial inputs were correct over 90% of the time [ 3 I]. This may be considered minimally adequate performance. In a sense, this is disappointing considering the battery of research-based solutions applied to the problem. One may prefer that performance would be equal to that achieved with a primary attitude indicator, but further research is needed to reach this goal. One potential approach is to investigate command rather than status displays to cue the pilot on the correct action for recovery [23]. This may be especially important for HUDs intended for transport pilots for whom extreme attitudes are both serious and rare, but can be recovered from using standard techniques [37].

Display-Display Coding Consistency
To minimize pilot confusion when transferring between HUD and HDD, symbology within the two displays use consistent codes for the information they present. Because HUDs are often retrofits to the flight deck, the onus of achieving consistency typically falls on the HUD rather than the HDD designer. Coding of individual parameters on the flight deck uses the following: Shape (e.g., font, pointer appearance).

Position.
Of these, color, shade, weight, and proximal symbols all represent issues for HUDs. Color is not available on current HUDs and, even if it were available, the apparent color on a HUD is also affected by colors of the OTW view in the background, making color an unreliable coding method for HUDs. While a clear distinction can be made with electronic HDDs between shade coding (e.g., gray versus black lettering) and weight coding (e.g., normal vs. bold lettering), the two coding methods may be more easily confused with one another in a HUD owing to blooming effects associating with bright symbology drawn on the cathode ray tubes used for HUDs. Also, even with automatic brightness adjustment, relative brightness and contrast of HUD symbology varies with the brightness of the OTW background, unlike HDD, where a fixed foreground-background brightness can be assured. Thus, a pilot cannot be expected to judge the meaning of a shade or weight coded HUD symbol in isolation. The problem with proximal symbols is that they add clutter, potentially occluding the OTW view. Some recommend that removal of the indicator symbology indicate a fault in the indicator in order to minimize clutter [3]. However, removal of symbology was associated with significantly slower reactions to failures than proximal symbology such as overlaying X's on the indicator [26], thus implying a tradeoff that is difficult to weigh.
Because of these limitations of available means for coding information on the HUD, designers frequently must necessarily deviate from HDD conventions that rely on color, shade, weight, and, occasionally, proximal symbols. Fortunately, shade and weight are rarely prominent codes on the flight deck for indicators that are likely to be represented on the HUD. In some cases the HUD can rely on redundant codes used with the HDD. For example, some HDDs distinguished armed and engaged autopilot modes are by position as well as color, and potentially position coding alone would be sufficient for the HUD. Performance limits are typically indicated with color (red) on indicators, but often this is redundant with frequency coding (the "barber pole"), and some HUDs have adopted the use of frequency coding by itself [35], although this is not universal [3][30]. below the horizon in the HDD attitude indicator, and achieving consistency on this in the HUD is daunting. Coding for sky and ground is treated elsewhere in this paper (see Unusual Attitude Recognition and Recovery, Page 4).
code for alerts (warnings and cautions, respectively). Inherent limitations precluded extending this to HUDs. Green can appear amber when projected against a reddish OTW background as when flying into a setting sun; red may not be adequately visible at all. The need to have clear and consistent coding for alerts warrants the development of a monochrome standard to be used by all such displays. A likely candidate would be to draw from performance limits conventions and use frequency coding (e.g., diagonal striped box around warnings). The effects of additional clutter introduced by such an approach are minor since alerts typically appear briefly and are pilot cancelable. However, comprehensive research is necessary before such a standard should be declared.
Shade and color are used to code for above and Color (red and amber) is also used on HDDs to

Head Motion
One consequence of the collimating optics used in HUDs is that the displayed symbology is only visible when the pilot's eye is within a specific volume of space called the cockpit head motion box (CHMB) [3 13. In operational HUDs, this volume is on the order of a few inches in each dimension, opening the possibility that involuntary head motion, such as that induced by vigorous control inputs or turbulence may displace the pilot's eyes outside the CHMB. Obviously, the larger the CHMB the better, but the size of the CHMB is dictated by the optical design of the HUD, with larger CHMB being technically difficult and expensive to manufacture. As HUDs are developed for the lower-end general aviation market, one can anticipate smaller CHMBs in an effort to contain costs.
While recommendations exist for minimal CHMB dimensions [3][3 11, there is little hard data on relevant pilot head activity. Simply put, how much does a pilot's head move? A field study is needed to address this, measuring the typical motion of various pilot's heads for various conditions, especially turbulence. Such a measure would need to exclude periods when the pilot is clearly not attending to the primary flight reference.
Such a descriptive study would set the preferred lower bounds for the CHMB. It is possible, however, that the CHMB can be made smaller assuming that the pilot can compensate by deliberately minimizing head motion. Follow-up experiments could manipulate the allowed head motion in order to assess the effects on workload, fatigue, and strain.

Sum ma ry
transports for over 10 years, design issues remain to be resolved with further research. Relatively extensive research has highlighted HUDs' special sensitivity to clutter, but further research is needed to give designers the tools needed to compare HUD designs in a specific flight deck application. Research on unusual attitude recognition and recovery with HUDs has yielded solutions that when combined are only minimally adequate. Limitations in HUD coding abilities call out for a standard for indicating alerts, while limitations in HUD optics require more knowledge on pilot head activity.
While HUDs have been operational on civil