Document
Audio-VisuM Situational Awareness forGeneral AviationPilots
Suresh K. Lodha Lilly Spirkovska NASA Ames Research Center University of California, Santa Cruz lodha@cse.ucsc.edu spirkov@email.arc.nasa.gov Abstract Weather is one of the major causes of general aviation accidents. Researchers axe addressing this problem from various perspectives including improving meteorological forecasting techniques, collecting additional weather data automatically via on-board sensors and "flight" modems, and improving weather data dissemination and presentation. We approach the problem from the improved presentation perspec- tive and propose weather visualization and interaction methods tailored for genera/aviation pilots. Our system, Aviation Weather Data Visualization Environment (AWE), utilizes information visualization techniques, a direct manipulation graphical interface, and a speech-based interface to improve a pilot's situational awareness of relevant weather data. The system design is based on a user study and feedback from pilots.
1 Introduction General aviation (GA) flight safety rests on four pillars of situational awareness: position, terrain, traffic, and weather. Loss of weather situational awareness is a main cause of GA accidents and has been attributed as a factor in over 30% of accidents and over 15% of fatal accidents [2]. One possible cause is that the pilot may not absorb and retain all the weather information he/she is required to review prior to flight. Because more data is provided than is applicable to a given flight and the data is presented poorly, it requires much time and cognitive effort to develop a "big picture" view of the weather, especially in an unfamiliar area. In this work, we focus on providing weather situational awareness to general aviation pilots.
There are several components of weather that a pilot needs to be aware of in order to maintain weather- related situational awareness. These components include surface winds and winds aloft, cloud conditions, and visibility conditions. Important classifications of weather related data and information on weather trends to alert the pilot for impending danger or assist him in path planning or replanning is important.
Communication of all this information effectively so that the pilot can assimilate and act on the information is of the utmost importance.
A number of researchers have addressed the poor presentation problem by developing graphical displays of aviation weather for use prior to flight. However, most of these systems are tailored for use by commercial airline pilots [7]. The concerns of commercial airline pilots are different and have been discussed by us in our earlier work [11]. Our system AWE (Aviation Weather Data Visualization Environment) focuses on providing weather data tailored to the needs of general aviation pilots using visual displays [11]. In particular, AWE focuses on graphical displays of three weather elements, namely, meteorological observations (METARs), terminal area forecasts (TAFs), and winds aloft forecasts and maps them onto a cartographic grid specific to the pilot's area of interest [11].
AWE was evaluated by pilots at NASA Ames Research Center, California, and found to be useful [11].
However, the pilots also gave a number of suggestions for improving the system. In this work, we describe a few important improvements over our previous work. First, we describe the enhancements in information displays that provide much improved situational awareness. Second, we present a completely new speech processing functionality that will reduce head-down time for the pilots significantly.
Section 2 presents a brief background followed by our previous work on AWE in Section 3. Section 4 describes the new information displays that incorporate pilots' feedback on including weather trend infor- mation, and display of multiple forecast elements or multiple winds aloft. Section 5 presents speech-based interactions within AWE. We conclude with our future plans.
r 2 Background Significant advances have been made in the last decade in both weather forecasting and weather visual ization for a variety of audiences including scientists, forecasters and the general public [4]. Professional TV production systems such as the TriVis system [9], interactive 3D weather visualization systems such as VISUAL operating in the German Meteorological Office (DWD) in collaboration with Fraunhofer IGD, and personalized Weather-on-Demand products through the internet are available. Augmented reality weather visualization systems are also being developed. Numerous aviation weather visualization efforts are under way including those at the Federal Aviation Administration (FAA), the National Oceanic and Atmospheric Administration (NOAA), the Naval Research Lab, Rockwell Science Center, Honeywell International, WSI Corporation, BFGoodrich, MIT Lincoln Lab, Echo Flight, and Jeppesen Inc.
In spite of all this progress, however, like other audiences, pilots are requesting tailored visualization tools for their specific needs. In particular, existing and developing weather forecasting and visualization technology needs to be harnessed appropriately for the benefit of pilots. The goal, as outlined by Stough, the manager of NASA's aviation weather information system (AWIN) project, is to provide "weather information relative to the pilot's flight path, present it to the pilot in the cockpit in an easy-to-interpret graphical format, and give him decision-making aids to help him use that information ..." [5].
The Aviation Digital Data Service (ADDS) system [10], a joint effort of NOAA, the National Center for Atmospheric Research (NCAR), and the Aviation Weather Center, perhaps comes closest to fulfilling the AWIN goal. Initially released in 1997, it provides unofficial graphical displays of a variety of aviation weather briefing report elements and allows pilots to zoom in to get more specific information about an area of interest. Although it presents a great variety of weather graphics, so-called terminal area forecasts (to be described later in Section 3) are displayed only textually without any filtering. It also does not relate the data to the pilot's planned path or schedule.
The most important official source of aviation weather reports for general aviation pilots in the United States is Direct User Access Terminals (DUATs) [3]. However, most of these briefings are textual or verbal.
In particular, DUATs does not provide visualization of three of the most important elements of a weather briefing: airport-specific current weather observations (meteorological observations, or METARs), terminal area forecasts (TAFs), and winds aloft forecasts. A more thorough description of related systems including details on the DUATs system and the data it provides can be found in our previous work [11].
3 AWE: Aviation Weather Data Visualization Environment We now present an overview of our previous work on AWE (Aviation Weather Data Visualization Environ- merit) [11]. We also provide details on some specific displays in order to explain several improvements we have incorporated in AWE to increase the situational awareness of pilots.
In AWE, we have designed and tested three visual displays for communicating current and forecast conditions at airports including clouds, visibility, and surface winds, and one display for communicating winds aloft forecasts. A pilot can use a graphical user interface to interact with the weather data and plan his flights quickly and easily.
VFR charts as Display Background: The weather data is overlayed on a VFR (Visual Flight Rules) sectional aeronautical chart, as shown in Figure 1. The chart shows the location of airports (magenta or blue circles or short lines that mimic the runway layout), airways ("highways" in the sky shown as light blue straight lines), navigation aids (mostly depicted by a compass rose), controlled and special use airspace, obstructions, natural terrain features (such as water and hills, depicted using color coded altitudes), demographic features (such as cities, depicted in yellow), and maximum elevation in each area (depicted with numbers with superscripts). Although the background may look cluttered and complex, the chart background texture gives pilots a familiar environment with which to interact and provides them with additional information for making their "go/no-go" decision. For example, winds aloft speeds of 40 knots are much more problematic if you will be flying 2000 feet above mountains than if you'll be flying 2000 feet over fiat terrain. Overlaying the weather on the chart consolidates weather and terrain situational awareness, allowing the pilot to make a decision by looking only at one source. Displaying weather information next to the airport it applies to improves a pilot's ability to perceive the big picture, especially in an unfamiliar area where he otherwise needs to look at a chart to determine where each airport is located and how it relates to his flightpath.Displaying surface windinformation near the airport also improves the pilot's ability to visually determine a landing runway and determine the amount of crosswind to expect. It provides information, while still away from the airport, that is commensurate with the information he receives from the airport wind sock. We decided on using a VFR Sectional chart as the background after considering several alternatives as described in our previous work [11]. The use of VFR aeronautical chart drew unanimous enthusiastic response from the pilots.
Figure 1: Current cloud, visibility and winds conditions shown using the symbolic and textual formats. The background image is an aeronautical chart for the San Francisco Bay Area.
Input: AWE begins with an official briefing downloaded from DUATs and extracts current airport condition reports, airport forecast reports, and winds aloft forecast reports. A DUATs current airport conditions report, shown in Figure 2, provides data about surface wind velocity, cloud altitude and coverage amount, visibility and obstructions to visibility, temperature, dew point, and barometric pressure setting.
A DUATs airport forecast report (terminal area forecast), shown in Figure 3, provides predictions about surface wind velocity, cloud altitude and coverage amount, and visibility and obstructions to visibility.
Visual Displays: We now describe the three visualization formats used in AWE.All three displays use the common fields listed in the previous paragraph; the textual format adds a few additional, unique fields.
For ease of interpretation, we chose to use similar display methods for both current and forecast conditions.
Symbolic Visual Display: The symbolic format is shown in Figure 1. The top rectangle of the symbol depicts wind velocity. Wind direction is shown by the direction of the arrow, while wind speed is shown by the thickness of the arrow. This method allows a pilot to quickly glance at the display and visually detect strong winds. The border of the rectangle is also color-coded (red for very strong winds, yellow for somewhat strong winds, and black for reasonable winds) to further enhance the pilot's ability to quickly determine wind KSQI_ 101646Z 18014KT 8SM NO FE_012 5CT050 BKND88 11108 A3022 KSFO 18T656Z 24000k'T 105M MO FE_012 BKN060 11/08 A3022 KPAO 101646Z ooo05Kr !05M NO FE%_915 SL'T050 11/'08 #,3021 KS.TC101853Z aoOOOKr 9SM Mo FE_BO O /OZ A3021 KFIHV 1916532 0100OKT IOSM MO FE_i015 S£T100 10107 A3022 KHND 191654;' 04OOISK'r IOSM NO SCT050 OVClO0 11/07 A3021 KOAK 181653Z 07007RT 1QSM 110 FEW020 5CT040 OVC070 11109 h3023 KLVK 181653Z 90000KT _5M HZ ELR 09/06 A3021 KECR I81653Z 05004KT IOSM NO FEW035 0V£060 1"1108 A3023 KAI=C181653Z 03007h'T 2OS_ NO FEW030 = 5C/:050 15/'08 A3023 KSCK 18t656Z 27006KT tSM NO OV_O01 07/07 A3021 KS_ 181058Z 33004KT 0.25Si9 FG 0V_001' 67107 A3022 KMO0 181650Z 08003KT 5SM BR FEW015 SCTO50 04/C4 A3021 KMRY 101854Z 05016KT 3SM RA BK_O20 OVCO41D 16[04 A301_ K5N5 I81853Z ttO22KT 8SM SH SCTO20._BKN035t4/07 A3016 KWVI 1610532 11012KT 9SM BR FEWt)3(_ BKN060 16/07 A3016 KVEB 181653Z 35007KT IOSM NO CLI_ t5/07 83019 " 181653Z 00000KT 4SM t42. SCT012 GKN025 14/96 A3019 WE 1813s3z 0000OKT ?S_, HZ CL,. I_J08 A3013 Figure 2: Sample current airport conditions file. Each line represents a report for a specific airport, specified by the airport identifier (first element). It also specifies the time of observation, wind direction and wind speed, visibility, visibility restrictions, cloud layers (with coverage amount and altitude), temperature, dew point, and barometric pressure.
KSFO 181730Z 181818 05007KT PGSM FE'd010 5CT050; 8ECMC 2223 28010KT" PBSM SCTIO0 SCT200; FM0300 30005KT P6SM SCT080 8KNIO0 8KN200; BECMG 0708 07005KT PBSM 5CT200; ks_c 18173oz 18i818 oooo3_ P_SM SC_0o; 8ECMG 2223 32017Kr PGSM FE_050 SCT200; FM0300 32008K1" PGSM SCT050; FMO600 O00O3KT P65M 5CT040 8KN080; FM0900 18020KT 3SM BR BKN020 OVC080.: FM1600 OOOl)3KT PBSM SCT200; ....
KOAK % 8173OZ 1818_ 8 070OTKT PBSM SCT200; BECMG 2223 28007KT PGSM BKNIO00YC200; FMO300 31005K'I" P{_SM 5CTI00 OVC200 ; 8ECMG 0708 O8005KT PGSM S£T200; kscK _oi73sz _8i810 3000SKT P_SM: SKC: BECMG 0102 O00O3KT P6SM SKQ FM0800 0000OKT 3SM:BR SKC ; FMIO00 O00OOKT 15M OR SKC ; TEMPO 1015 0.255M FG W002; FM1800 00003Kl" 3SM BR SKC ; Figure 3: Sample airport forecast file. A forecast for an airport extends from the airport identifier to the dot ("."). Each forecast provides the time the forecast was created followed by the effective times of each forecast element. Each element specifies the wind direction and speed, visibility and restrictions to visibility, and cloud layers (with coverage amount and altitude).
conditions.
The middle rectangle depicts cloud information. The rectangle represents altitudes from 0 feet (at the bottom) to 12,000 feet (at the top, below the wind rectangle). This rectangle is then filled _ith sub- rectangles to represent cloud altitude and coverage amount. The FAA has defined five ranges for cloud coverage: the sky can be reported to be clear, may have .few clouds (FEW = less than 1/8 of the sky is covered), scattered clouds (SCT = between 1/8 and 3/8 of the sky has clouds), broken clouds (BKN = 4/8 to 7/8 of the sky), or be completely overcast (OVC = 8/8 of the sky). The FAA also provides a list of standard contractions for visibility obscurations. Our examples are not for comprehensive coverage, but rather for general understanding, so not all options are described. Using information visualization methods promoted by Tufte [12, 13] and Bertin Ill, we map these five ranges to five gray levels: white for clear skies; progressively darker gray for few, scattered, and broken clouds; and black for an overcast sky. Cloud coverage and altitude are measured by ground-based instruments located at the airport and can be reported as layers.
That is, cloud information may be reported as few @ 1200 if, sct @ 3000 if, ovc @ 12,000 ft. Hence, cloud coverage is depicted by layering appropriately colored sub-rectangles at a vertical position determined by a linear mapping from 0 to 12,000 feet. Similar to the wind rectangle, the cloud rectangle is color-coded to ease interpretation. The color-coding takes the cloud layers into consideration as well as the information KSFO 11 11 0 10 10 2 £ 8 -4 7 5 '10 27:7 -23 25 1:9 -34 24 3:6 4924 325525:24 58 KSAC 11 12 O 20 20 1 18 i9 -'5 15 29-:I:0=3l 35:_24:16 29 -35 24 33 4924:31 55:26 20 58 Figure 4: Sample winds aloft file. The airport identifier is followed by groups of three elements: wind direction, wind speed, and temperature. These groups of three are available for prespecified altitudes from 3000 feet to 39000 at various increments.
displayed as text in the bottom rectangle: visibility distance and any obstructions to visibility, such as fog, haze, or smoke. The border is colored black if the clouds and visibility allow a pilot to fly under visual rules (lowest broken or overcast cloud layer, or ceiling, at an altitude greater than 3000 ft and visibility better than 5 miles), yellow if the ceiling or visibility is marginal (ceiling greater than 1000 ft but less than 3000 ft; and visibility between 3 and 5 miles), or red if the ceiling or visibility is low enough to require flight using instrument flight rules (ceiling less than 1000 ft; visibility less than 3 miles).
Both current and forecast airport conditions can be displayed using these symbols. They provide most data needed to plan flights. Other data available in current condition reports is not usually necessary but is sometimes useful.
Textual Display with Color-Coded Borders: If a pilot wants to view the above mentioned data textually or if he wants to see precise numbers for the other elements, he can ask for a textual presentation, as shown in Figure 1. This format shows all the elements provided. The text rectangle is color-coded using the same coding we used for the middle rectangle of the symbolic display. That is, it represents the visibility and cloud coverage combination and can be red, yellow, or black.
Triangular Iconic Display: The last format a pilot can request is the iconic format which provides an overview of wind, visibility, cloud altitude, and cloud coverage data. In the current conditions icon, the temperature - dew point spread (that is, temperature minus dew point; useful for detecting possible fog formation) is also shown. It displays these elements using four sub-triangles arranged into a single triangle, as shown in Figure 5. Each sub-triangle is color-coded using the colors white for good conditions, yellow for marginal conditions, and red for adverse conditions, as defined for the symbolic format above. Because the airport forecast reports do not provide temperature or dew point information, the center sub-triangle is colored gray when displaying forecast data. The iconic display is useful in providing a very quick overview of conditions. A pilot can quickly get the big picture of the weather situation. For example, if the bottom section of the triangles is red in a wide area, the pilot can quickly understand that there is widespread low fog. On the other hand, if a geographic area shows all white triangles, he can easily see that at least the elements shown graphically are conducive to flying.
Winds Aloft Display: Finally because they are complementary to current and forecast airport condition reports, we also provide winds aloft forecasts graphically. The winds aloft forecasts report, shown in Figure 4, provides data about the wind velocity and temperature at predefined altitudes, generally speaking ranging from 3000 ft to 39,000 ft in increments of 3000 ft. We found that pilots preferred a very simple representation - one that does not require much interpretation. Thus, we display the winds aloft for a selected altitude as a directional arrow, as shown in Figure 5. The direction of the arrow specifies the direction the wind is coming from. The wind speed is shown textually on the arrow. The pilot selects an altitude using a slider provided on the graphical user interface. As the pilot drags the slider, the wind arrows change to reflect winds at the selected altitude using interpolation to compute winds at altitudes not given in the winds aloft forecast report. This helps the pilot choose an appropriate cruising altitude and appropriate flight path without having to perform any computation.
All of the above formats can be displayed for an entire area or just along a pilot-selected route of flight.
For winds aloft displayed along a route, we use distance-based interpolation of the two closest wind reporting sites to compute winds at selected locations. Similarly, if current conditions or forecasts are requested for airports that do not provide reports, the nearest available reports are displayed. This provides the pilot an estimate of what the weather may be like at his selected airports.
We conducted a user study to test the usability of the above formats. Through interviews and a question- naire, we determined that pilots preferred our graphical formats over seven formats from four competitive Figure5: Current weather conditions shown using theiconic format.Windsat a pilot-selected altitude shown witha cyan arrow.
systems. Thestudy also revealed areas forfurther improvement. In thenexttwosections, wediscuss some ofthemostimportant feedback fromtheuserstudy.
4 Information Displays The focus of this work is to provide additional information displays to genera] aviation pilots to better main- tain weather situational awareness. The functionality provided by these additional displays were suggested by pilots, as mentioned before, in a user study. They requested three additional functions: display of trend information for a selected airport, display of multiple forecast elements for a selected airport, and display of multiple altitude winds aloft forecasts. The following subsections describe these new displays.
4.1 Trend Information Display In our original implementation of AWE, we provided a symbolic representation of METAR/TAF display as shown in Figure 1 and discussed earlier in Section 3. In the original version, AWE displays only the most recent report of winds, clouds, and visibility conditions. The user study revealed that the pilots wanted the ability to look back at previous reports as well. An important feedback from the pilots was to provide a display of weather trends based on predictions. An important related question is how do we determine whether these predictions are accurate or not.
In order to address this feedback, we present two layers of weather trend display. On the top layer, we display the forecast applicable at time t - 2, t - 1, t, and t + 1 using the symbolic representation as shown in Figure 6. On the bottom layer, the actual conditions at time t - 2, t - 1, and t are shown symbolically and aligned with the same time stamps of the top layer as shown in Figure 6. The bottom layer of information associated with a given airport provides a pilot with information on how the weather actually changed.
When compared with the forecast for that time period on the top layer, the pilot can decide whether the current conditions in wind, cloud, or visibility displayed in the bottom layer support or refute predictions shown in the top layer. Although statistical and artificial intelligence based approaches can be used to derive statistical measures of uncertainty of predictions, most pilots are more comfortable and trusting of raw data rather than interpretations of the raw data by algorithms they do not understand. It is also very easy for a pilot to perform a visual pattern comparison. However, if algorithms are designed to derive uncertainties that gain the trust of the pilots, such measures can easily be displayed within AWE either augmenting or replacing the suggested displays.
By studying Figure 6, a pilot can see the discrepancies between the actual readings and the predictions.
Observe that the surface winds are predicted to be calm consistently while the actual readings consistently have a wind in the easterly direction with stable speed. For visibility conditions, the predictions are that the fog (and associated low visibility) will dissipate after 14:00 and the visibility will improve to 1 mile (from 0.25 mile). Observe that in actuality, the visibility is improving more quickly than forecast and is already at 3 miles. The clouds are also lifting so the forecast for ceilings above 12,000 ft (shown by the white rectangle) is plausible.
Although here we have discussed the trend information to be displayed using symbolic representation only, we have implemented the weather trend display using other visual methods such as the triangular iconic displays discussed earlier in Section 3.
There are certainly other display possibilities for weather trend information such as the direct graphical displays of the three elements using both actual and predicted conditions. While such a display may be more usefirl for longer time periods because they may occupy less screen area, we preferred to build on the original displays designed in AWE because of two reasons. First, the trend display is an extension of the static one time display and therefore, does not cause any additional cognitive workload. Second, continuous graph-based displays seem to be less suitable for picking out exactly where in time the pilot is in comparison to the discrete displays that we have designed. Certainly, we need to conduct another user study to determine the efficacy of the new information displays presented in this work.
Figure 6: Display of trend information for an airport. The top layer displays the winds, cloud, and visibility as predicted; the bottom layer displays the winds, cloud, and visibility as measured. Correlation or discrepancy between the two layers allows pilots to determine how reliable predictions are likely to be.
4.2 Multiple Forecast Element Display In the original implementation, AWE automatically selects the appropriate forecast for the destination airport based on the pilot's estimated time of arrival, ETA. However, the user study revealed that pilots also wanted the ability to see the forecast for nearby times to better determine how certain the forecast is for their ETA. That is, if the weather is predicted to change 30 minutes after their ETA, it may be prudent for the pilot to have a backup plan in case the change occurs earlier. Similarly if they are planning to arrive 20 minutes after the fog is forecast to clear; they may want to plan for an alternate destination if the fog does not clear on schedule.
We have incorporated this suggestion so that AWE now has the capability to display the predicted weather for all time periods provided by the DUATs report for pilot-selected airports. Rather than having AWE select the appropriate forecast for the estimated arrival time, AWE can now display the entire series of forecast elements. We refer to this display as multiple forecast element display. AWE can use any one of the three visualization methods - textual display with color-coded borders, overview triangular visual displays, or symbolic visual displays - to visualize multiple forecasts at several airports simultaneously. Figure 7 shows an example of a triangular overview display for three airports simultaneously. The fourth airport shown with a single triangle is one the pilot has not requested additional information about. Hence, only the element applicable to the AWE-computed arrival time is shown.
Fromthis display, thepilot canseethat,although therearefourelements given in theTAF for one of his selected airports (Oakland International, left side of the image, shown with 4 triangles), it will have acceptable surface winds, visibility, and cloud conditions during all periods. In contrast, he can very quickly see that for San Jose airport (shown on the bottom of the image, with 6 elements) the winds will be moderate during one forecast duration, improve for a time, and then deteriorate along with poor visibility and lowered cloud ceiling. If the single element display for his ETA showed only a white triangle, he may not be able to develop a realistic big picture of the expected conditions nearby his ETA. Note that all elements for the 24 hour forecast period provided by DUATs are shown. The pilot is unable to determine when a change occurs.
To get such details, he needs to reference the symbolic or textual formats. Using the overview display, he can only determine if a change is forecast.
Figure 7: Overview of weather forecasts for all periods provided by DUATs at three airports using the triangular display. Rather than showing just the forecast element applicable to the AWE-computed estimated time of arrival (as shown for the left-most airport), it shows all the elements associated with a particular airport, as provided by DUATs. This kind of display is likely to be useful to the general aviation pilot to get a quick overview of forecast conditions.
If a change is forecast, the pilot can ask for additional details to determine how it may affect his flight.
He can mix-and-match different kinds of displays to extract just the information he is interested in. For example, Figure 8 shows a single forecast element selected by AWE as applicable at the flight's ETA, similar to the display of Figure 1, together with all weather predictions for 3 airports simultaneously, one using the textual format to get all the details, and the other two using the symbolic visual representation. The pilot can now see that Oakland airport has four elements (the white triangles shown in Figure 7) because the wind conditions are forecast to vary significantly. He can also see that the decreased visibilities at the upper right airport (SCK, Stockton) are due to mist ('BR") and fog ("FG") and will be a factor from 8:00 Zulu to 17:00 Zulu as fog and low clouds develop and then dissipate from California's Central Valley. He can also see that the conditions at San Jose airport (bottom of the image) are caused by a front moving through, with associated decreasing ceilings.
4.3 Multiple Altitude Winds Aloft Display AWE allows the pilot to evaluate potential cruising altitudes interactively by displaying winds aloft velocities only one at a time as he manipulates the altitude selector slider. The user study revealed that pilots wanted to see forecasts for all winds aloft altitudes simultaneously to formulate a better big picture. To formulate the same picture without this functionality, the pilot needs to watch multiple arrows as he moves the slider and remember winds for previous altitudes. With simultaneous display, all the information would be available concurrently to allow for a quick visual evaluation.
AWE now provides multiple altitude winds aloft display. DUATs winds aloft reports provide forecasts for wind velocities at 3000, 6000, 9000, 12,000, 18,000, 24,000, 30,000, 34,000, and 39,000 feet. Aircraft flown by the target audience for AWE rarely fly above 12,000 ft. The 18,000 ft forecast provides additional information about the potential for thunderstorm activity. Thus, we limited the multiple altitude winds aloft display to 18,000 ft. An example for the San Francisco airport is shown in Figure 9. Winds are displayed fl'om 3000 ft at the bottom to 18,000 ft at the top. The wind direction is displayed using a directional arrow and the wind speed is displayed as text. The rectangle for each altitude is color-coded for easier detection of strong winds. We chose a modified stop-light color coding scheme with green for calm winds (< 10 kts), yellow (< Figure 8: Overview ofweather forecasts attwoairports using thesymbolic format andoneusing thetextual format. Thedisplay forSFO airport shows onlytheAWEselected element applicable atthepilot'sETA.The twoairports displaying symbolic trendinformation show fog forecast to develop (upper right airport) and a front moving through (lower center airport) with associated decreasing visibilities and ceilings. The textual trend information shows that although conditions at OAK will be good throughout the forecast period, the surface winds will be changing significantly. This kind of display is likely to be useful to the general aviation pilot in path re-planning to decide whether to Iand at an alternative airport or expedite or delay the flight arrival time.
20 kts) for moderate winds, orange (< 30 kts) for stronger winds, and red (> 30 kts) for very strong winds.
The altitude associated with each rectangle is not shown in order to decrease clutter. It's always the same altitudes and the altitudes follow standard DUATs reports; hence, they are easily internalizable by pilots.
Figure 9: Winds aloft forecast shown for multiple altitudes simultaneously. The individual altitude rectangles are color-coded using a modified stop-light scheme to quickly inform the pilot of wind conditions.
5 Speech-based User Interface The original implementation of AWE has only visual elements. One of the most important feedbacks from the pilots was to incorporate a speech-based user interface. It is hoped that with this interface, the pilot would not have to dedicate much eye time and concentration on interpreting the textual data. In an eyes- busy, hands-busy environment such as flying an aircraft, a speech-based user interface will allow the use of an additional modality to interact with the system distributing the cognitive overload between different senses. Experimental evidence indicates that if a user is performing multiple tasks, performance improves if those tasks can be managed over independent input/output channels, or modalities [81. Moreover, it is expected that a speech-based interface would allow easier access without the disadvantages associated with WIMP (windows, icons, menus, pointing device) interfaces in mobile applications.
To make AWE easier to use in flight, we have now added a speech-based user interface. Speech has a number of advantages. Using speech, the pilot can extract information from AWE without having to devote much visual attention to the task. Speech has low panel real-estate needs. Small GA cockpits do not allow for much other input-output paraphernalia. Speech as an input modality is quicker than either typing or writing. A person can speak several hundred words per minute, an excellent typist can produce 8(I-100 wpm, andone canhand writeatless thanhalfthetypingrate.Speech also hasdisadvantages. Listening isslower thanreading. It'salso temporal, serial, "bulky", andmay notbeprivate enough insome situations. Also, the technology isstill immature. It works bestwitheasily distinguishable vocabularies andlimitedgrammars.
Last, the output tends to sound unnatural and can be misunderstood until you become accustomed to its accent.
The advantages outweigh the disadvantages in the GA domain for a number of reasons. First, the official language for aviation is English, even in many non-English speaking countries. However, non-English speakers (pilots, air traffic controllers, and flight service specialists) must learn enough English to transfer necessary information. Second, pilots are accustomed to specialized vocabularies and grammars. Because of congested radio communications, conversations are often terse and follow a standard phraseology. Pilots are also accustomed to communication using a headset or handheld microphone. Finally, pilots are accustomed to the speech synthesizer accent because some weather data is already synthesized.
Given the advantages and limitations of speech, we have implemented a SUI (Speech-based User Inter- face) for AWE. We did not implement a natural language understanding system. Rather, we focused on a specialized vocabulary and grammar that general aviation pilots often use. We used speech as an input and output modality. The objective is to provide the pilot with a mechanism to express needs and obtain services from AWE. We also chose not to implement a dialog tree [6]. They are useful in situations where the user will not get any pre-use training, such as telephone bank tellers. Pilots using AWE will get initial training.
Also, dialog trees do not mimic any communications typically found in GA and can be frustrating to use.
Command and control interfaces better mimic other GA communications and pilots should find them more pleasant to use.
AWE was implemented in C++, uses OpenGL and GLUT for the GUI, and runs under LintLx. For the SUI, we chose IBM's ViaVoice for Linux products for speech recognition and speech synthesis. To improve speech recognition accuracy, we designed a grammar that defined the directives that can be issued to the recognition engine. The challenge here is to design the grammar so the pilot has sufficient expressive power to ask for what he needs, but the grammar remains limited enough to get good recognition accuracy. The grammar is written in the Speech Recognition Control Language (SRCL), which is basically a BNF (Backus- Naur form) grammar adapted to speech recognition. Like a BNF grammar, it has a set of production rules with non-terminals, terminals, and some predefined terminals that allow the developer to specify optional words or repeated words. The AWE grammar is constructed to constrain the pilot to request current airport conditions, airport forecasts, and winds aloft forecasts, and to set program parameters aurally. Some example sentences the grammar accepts include • show area current weather as symbols • show route forecast as icons at 15 30 zulu • hide route winds aloft • say winds at Palo Alto • say forecast visibility for Monterey at 18 hundred • set display type to icon • show trend information for Stockton as icon AWE can respond to the pilot's directives by either graphical feedback (for show, hide, or set commands) or aural feedback (for say commands). We chose an implicit confirmation strategy for aural feedback. AWE does not explicitly ask the pilot "Did you say ...?" and then wait for a yes or no before answering. Rather it provides a succinct answer to a query but also provides enough context so the pilot knows the request was understood correctly. If the pilot detects that an incorrect question is being answered, he would reissue the query, just as he would if he were speaking to a Flight Service Station specialist who misunderstands a query issued over the aircraft radio. We feel this is justified because AWE provides answers to queries. Therefore, a recognition error results in a delay in the pilot getting the information, but is otherwise harmless.
]0 6 Summary and Future Work AW'E (Aviation Weather Data Visualization Environment) has been tailor designed to suit the needs of general aviation pilots. It focused on the gaps found in other aviation weather visualization systems by providing displays related to METARs, TAFs, and winds aloft display. Our previous user study with pilots resulted in useful feedback about the usability of AWE.
In this work, we have provided additional functionality based on the pilots' feedback. In particular, we have made several enhancements to weather information displays and added a speech-based user interface.
Our next step is to conduct another user study to determine whether our implementation of these additional features satisfies pilots' needs.
We then plan to further enhance AWE to incorporate more situational awareness. In its current imple- mentation, AWE relies on the pilot to determine which area of the chart to look at. It would be usefffl to automatically obtain an aircraft's current position and use that to provide only relevant data to the pilot.
We will accompIish this by interfacing to a GPS unit.
Finally, to further decrease a pilot's workload, we will enhance AWE to take the initiative in providing information it determines may be of use to the pilot. For instance, when new weather updates are received, AWE can automatically look for inconsistencies between the forecasts and the evolving conditions. If it detects any unexpected conditions, it can spontaneously warn the pilot. Of course, all the data will also be available to the pilot for his own exploration.
AWE already provides useful functionality to help a pilot maintain an awareness of the weather situation.
With planned future functionality, he will be able to maintain weather situational awareness with further reduced workload.
References [1] Jacques Bert/n. Semiology of Graphics: Diagrams, Networks, Maps. University of Wisconsin Press, Madison, WI, 1983.
[2] AOPA 1998 Aviation Fact Card. 1998.
[3] DynCorp DUATS. World Wide Web. wwwl.duats.com.
[4] H. Haase, M. Bock, E. Hergenrother, C. Knopfle, H.-J. Koppert, F. Schroder, A. Trembilski, and J. Wei- denhausen. Meteorology meets computer graphics - a look at a wide range of weather visualizations for diverse audiences. Computers and Graphics, 24:391-397, 2000.
[5] T. S. Perry. Tracking weather's flight path. fEEE Spectrum, 37(9):38-45, September 2000.
[6] R. Rosenfeld, D. Olsen, and A. Rudnicky. Universal speech interfaces. Interactions, 8(6):34-44, Novem- ber/December 2001.
[7] Charles H. Scanlon. Cockpit graphical weather information shown to enhance efficiency, safety, and situation] awareness. In Proceedings of the 391h Annual Corporate Aviation Safety Seminar (CASS), Flight Safety Foundation, pages 83-94, April 1994.
[8] C. Schmandt. Voice Communication with Computers. Van Nostrand Reinhold, New York, NY, 1994.
[9] F. Schroder and M. Lux. Trivis: Professional television weather presentation. World Wide Web, October 1997. http://www.igd.fhg.de/www/igd-a4/projects/docs/trivis/trivis_e.html.
[101 NOAA National W'eather Service. World Wide Web.
www.awc-kc.noaa.gov/awc/aviation_weather_center.html.
[11] L. Spirkovska and S.K. Lodha. Awe: Aviation weather data visualization environment. Computers and Graphics, 26(1), January 2002. To appear.
[12] Edward R. Tufte. The Visual Display of Quantitative Information. Graphics Press, Cheshire, CT, 1983.
[13] Edward R. Tufte. Envisioning Information. Graphics Press, Cheshire, CT, 1990.