Single-Pilot Workload Management in Entry-Level Jets

extended by Dr. Durand Begault in conducting the voice analyses for this study. Finally, we extend our sincere appreciation to the Cessna Mustang pilots who completed questionnaires and who served as participants in this study. Their desire to further the cause of aviation safety, their willingness to participate, and the insights they so generously shared, were essential to the success of the study. We are very grateful. v CONTENTS Introduction -----------------------------------------------------------------------------------------------------------------------------------1 Jet Single-Pilot Workload ----------------------------------------------------------------------------------------------------------------1 Approaches to Measuring Workload ---------------------------------------------------------------------------------------------------1 Automation Use ---------------------------------------------------------------------------------------------------------------------------2 The Current Report -----------------------------------------------------------------------------------------------------------------------3 Methods -----------------------------------------------------------------------------------------------------------------------------------------3 Participants---------------------------------------------------------------------------------------------------------------------------------3 Materials------------------------------------------------------------------------------------------------------------------------------------3 Design ------------------------------------------------------------------------------------------------------------------------------------ 12 Procedure --------------------------------------------------------------------------------------------------------------------------------- 12 Data Management and Preparation----------------------------------------------------------------------------------------------------- 13 Simulator Flight Performance Data and Data Extraction ------------------------------------------------------------------------- 13 Graphs ------------------------------------------------------------------------------------------------------------------------------------ 16 Google Earth Plots ---------------------------------------------------------------------------------------------------------------------- 17 Flight Communication Transcription ------------------------------------------------------------------------------------------------ 18 Voice Analysis---------------------------------------------------------------------------------------------------------------------------- 18 Video Data ------------------------------------------------------------------------------------------------------------------------------- 19 Results------------------------------------------------------------------------------------------------------------------------------------------ 20 Participant Demographics ------------------------------------------------------------------------------------------------------------- 20 Autopilot Use During the Experimental Flight ------------------------------------------------------------------------------------- 21 Analysis of Workload and Task Management of Four En Route Events -------------------------------------------------------- 22 Event 1: Interception of the Broadway (BWZ) Radial ---------------------------------------------------------------------------- 22 Event 2: Reroute and Descent to Meet a Crossing Restriction at a Waypoint ------------------------------------------------- 30 Event 3: Expedited Descent ---------------------------------------------------------------------------------------------------------- 37 Event 4: Communication Assistance for a Lost Pilot ------------------------------------------------------------------------------ 41 Voice Analyses Across the Four High Workload Events --------------------------------------------------------------------------- 51 Overall Performance across the Four High Workload Events--------------------------------------------------------------------- 54 Discussion ------------------------------------------------------------------------------------------------------------------------------------- 55 Participants------------------------------------------------------------------------------------------------------------------------------- 56 Workload Management ---------------------------------------------------------------------------------------------------------------- 56 Automation Use ------------------------------------------------------------------------------------------------------------------------- 57 Successful Task Completion and Errors---------------------------------------------------------------------------------------------- 58 Study Limitations and Recommended Future Studies----------------------------------------------------------------------------- 60 Recommendations for Workload Management and Automation Use----------------------------------------------------------- 60 Conclusion ------------------------------------------------------------------------------------------------------------------------------------ 61 References------------------------------------------------------------------------------------------------------------------------------------- 61 Appendix A. Demographic Data Questionnaire ------------------------------------------------------------------------------------------A1 Appendix B. Advanced Avionics and Automation Questionnaire ----------------------------------------------------------------------B1 Appendix C. Citation Mustang and G1000 Cockpit Set-Up Preference Questionnaire ------------------------------------------- C1 Appendix D. Pilot Briefing Package -------------------------------------------------------------------------------------------------------- D1 Appendix E. Post-Study Interview Questions --------------------------------------------------------------------------------------------- E1 Appendix F. Observed “Best Practices” and Other Things to Consider Best Practices ----------------------------------------------- F1 vii LIST OF FIGURES Figure 1. Familiarization Scenario Route of Flight----------------------------------------------------------------------------------------4 Figure 2. Experimental Leg 1 Scenario Route of Flight ----------------------------------------------------------------------------------5 Figure 3. Experimental Leg 2 Scenario Route of Flight ----------------------------------------------------------------------------------6 Figure 4. Excerpt of the Familiarization Scenario Script ---------------------------------------------------------------------------------7 Figure 5. Cessna Citation Mustang Flight Simulator and Projection System ---------------------------------------------------------8 Figure 6. Simulator G1000 Avionics Suite and Out-The-Window View --------------------------------------------------------------8 Figure 7. Experimenter’s Station. The Simulator and Visual System can be Seen in the Background --------------------------- 10 Figure 8. Researchers and ATC at the Experimenter’s Station ------------------------------------------------------------------------ 11 Figure 9. Sample Portion of a Concurrent Task Timeline ----------------------------------------------------------------------------- 12 Figure 10. Excel Spreadsheet Produced for 11 Specific Flight Parameters ------------------------------------------------------------ 15 Figure 11. By AGL Stacked Graphs of Simulator Data ---------------------------------------------------------------------------------- 16 Figure 12. Example flight trajectory plotted in Google Earth -------------------------------------------------------------------------- 17 Figure 13. Flight Path Trajectory With Additional Aircraft Data Selected ------------------------------------------------------------ 17 Figure 14. Sample Flight Communication Transcription ------------------------------------------------------------------------------- 18 Figure 15. Four Camera Views of Cessna Mustang Simulator Cockpit -------------------------------------------------------------- 19 Figure 16. ATC Dynamic Navigation Map and Flight Parameter Display ----------------------------------------------------------- 19 Figure 17. Time Required to Complete Expedited Descent as a Function of AP Use ---------------------------------------------- 40 Figure 18. ISA Workload Rating During the Expedited Descent as a Function of AP Use ---------------------------------------- 40 Figure 19. ILS or LOC RWY 25 Approach to KHSP (AOPA, 2012) ---------------------------------------------------------------- 44 viii LIST OF TABLES Table 1. Experimental Flight ISA Rating Prompts ---------------------------------------------------------------------------------------9 Table 2. Experimental Flight NASA TLX Task Rating Events -------------------------------------------------------------------------9 Table 3. Sample Flight Simulator Variables --------------------------------------------------------------------------------------------- 14 Table 4. Participant Flying History------------------------------------------------------------------------------------------------------- 20 Table 5. Personal Experience With Advanced Avionics and Automation ----------------------------------------------------------- 21 Table 6. Expected Strategies for Programming the BWZ Radial Intercept --------------------------------------------------------- 23 Table 7. Errors Committed During High Workload Event 1 ------------------------------------------------------------------------ 25 Table 8. Sequences of Lateral AP Modes Used by Participants During Event ----------------------------------------------------- 26 Table 9. Sequences of Vertical AP Modes Used by Participants During Event 1 Climb ------------------------------------------ 26 Table 10. Strategies for Setting up the BWZ 208o Radial Intercept ------------------------------------------------------------------- 27 Table 11. Pilot Flying History in Hours by Major Task, Success Status, and Experience of FD Failure ------------------------- 29 Table 12. ISA and NASA RTLX1 Ratings by Major Task Success Status and Experience of FD Failure ------------------------ 30 Table 13. Correctly Copying Reroute Clearance----------------------------------------------------------------------------------------- 32 Table 14. Errors Committed During Event 2 -------------------------------------------------------------------------------------------- 33 Table 15. Descent Performance in Event 2 ----------------------------------------------------------------------------------------------- 34 Table 16. Time Required for Programming Reroute and Descent to Meet the DQO Crossing Restriction -------------------- 35 Table 17. Pilot Demographics by Problems Encountered With the Reroute or Meeting the Crossing Restriction at DQO ---------------------------------------------------------------------------------------------------------------------------- 36 Table 18. RTLX Workload Ratings for Event 2 ------------------------------------------------------------------------------------------ 37 Table 19. Autopilot Modes Used During Expedited Descent -------------------------------------------------------------------------- 38 Table 20. Timing of Autopilot Re-Engagement------------------------------------------------------------------------------------------ 38 Table 21. Time Lapsed for Participant Response and Altitude Gained Prior Aircraft Descent ----------------------------------- 39 Table 22. Pilot Participant TLX-Ratings of the Expedited Descent Procedure ------------------------------------------------------ 41 Table 23. Errors Committed During High Workload Event 4 ------------------------------------------------------------------------ 43 Table 24. Relationship of Timing of Approach Briefing and Programming to Encountering Difficulties in Programming or Conducting the ILS Runway 25 Approach Into KHSP ----------------------------------------------------------------- 45 Table 25. Lower-Left PFD Inset Map Configurations ---------------------------------------------------------------------------------- 46 Table 26. Pilot Demographics by Problems Encountered With the Crossing Restriction and Instrument Approach --------- 48 Table 27. NASA RTLX Ratings by Problems Encountered With the Crossing Restriction 15 nm Before MOL -------------- 49 Table 28. ISA and NTLX Ratings for the Lost Pilot Scenario by Problems Encountered With the Crossing Restriction and Instrument Approach ------------------------------------------------------------------------------------------------------- 50 Table 29. ISA and NASA RTLX Ratings by Problems Encountered With the ILS or LOC RWY 25 Instrument Approach Into KHSP ------------------------------------------------------------------------------------------------------------------------ 51 Table 30. Fundamental Frequency Descriptive Statistics (Hz) ------------------------------------------------------------------------- 52 Table 31. Articulation Rate Descriptive Statistics ---------------------------------------------------------------------------------------- 53 Table 32. Success in Accomplishing Major Tasks in the Four High Workload Events --------------------------------------------- 54 ix ACRONYMS AND ABBREVIATIONS AC .............Advisory Circular AC_ID .......Aircraft Identifiers ADDS ........Aviation Digital Data Service AFM ..........Aircraft Flight Manual AGL ...........Above Ground Level AIM ...........Aeronautical Information Manual AIRMETS..Airmen’s Meteorological Information ALAR .........Approach and Landing Accident Reduction AOM .........Aircraft Operating Manual AOPA ........Aircraft Owners and Pilots Association ARTCC......Air Route Traffic Control Centers ASOS .........Automated Surface Observation System ATC ...........Air Traffic Control ATIS ..........Automated Terminal Information Service ATP ...........Airline Transport Pilot AWOS........Automated Weather Observing System b/t ..............Between CAMI ........Civil Aerospace Medical Institute CFIT..........Controlled Flight Into Terrain CFR ...........Code of Federal Regulations CRM..........Crew Resource Management CSV ...........Comma-Separated Value DH ............Decision Height DIS ............Distance DME .........Distance Measuring Equipment ELJs ...........Entry Level Jets EMRRS .....Enhanced Mission Record and Review System ETA ...........Estimated Time of Arrival FAA............Federal Aviation Administration FAF ............Final Approach Fix FARs ..........Federal Aviation Regulations FL ..............Flight Level FMS ...........Flight Management System FPL ............Flight Plan FPM...........Feet Per Minute FSF ............Flight Safety Foundation FSI .............Flight Safety International GA .............General Aviation G/A ............Go Around GPS ...........Global Positioning System IAF.............Initial Approach Fix ICAO .........International Civil Aviation Organization IF ...............Intermediate Fix ILS .............Instrument Landing System IFR ............Instrument Flight Rules IMC ...........Instrument Meteorological Conditions ISA .............Instantaneous Self-Assessment KML ..........Keyhole Markup Language kts ..............Knots LAN...........Local Area Network LINE..........Line Number Variable LOSA .........Line Operations Safety Audit MDA .........Minimum Descent Altitude METARs ....Meteorological Aerodrome Report MFD..........Multi-functional Display MOE .........Margin Of Error MSL ...........Mean Sea-Level MTS ..........Movable Type Script NextGen ....Next Generation Air Transportation System NMT .........Not More Than NOAA .......National Oceanic and Atmospheric Administration NOTAMs...Notices to Airmen PFD ...........Primary Flight Display PIREPs .......Pilot Reports POM..........Pilots Operating Manual PTS ............Practical Test Standards QRH..........Quick Reference Handbook RNAV ........Area Navigation RTLX .........Raw NASA Task Load Index Scores SDHC .......Secure Digital High-Capacity SIGMETS ..Significant Meteorological Information SRM ..........Single Pilot Resource Management TAFs ..........Terminal Area Forecasts TAS ............True Airspeed TDZ ..........Touchdown Zone TDZE ........Touchdown Zone Elevation TERPS .......Terminal Instrument Approach Procedures TLX ...........Task Load Index UNK ..........Unknown VBA ...........Visual Basic for Applications VFR ...........Visual Flight Rules VLJs ...........Very Light Jets VMC .........Visual Meteorological Conditions VOR ..........Very High Frequency Omni Directional Radio Range Vref .............Landing Reference Speed WAV ..........Waveform Audio File Format WPT ..........Waypoint WQXGA ....Wide Quad Extended Graphics Array xi EXECUTIVE SUMMARY Researchers from the NASA Ames Flight Cognition Lab and the FAA’s Flight Deck Human Factors Research Laboratory at the Civil Aerospace Medical Institute (CAMI) examined task and workload management by single pilots in Very Light Jets (VLJs), also called Entry-Level Jets (ELJs). Fourteen certificated Cessna Citation Mustang (C510-S) pilots flew an experimental flight with two legs involving high workload management under Instrument Flight Rules (IFR) in a Cessna Citation Mustang ELJ level 5 flight training device1 at CAMI. Eight of the pilots were Mustang owner-operators, and the other six flew the Ci- tation Mustang as part of their jobs as professional pilots. In addition to the Cessna Citation Mustang simulator, data col- lection included the use of a non-invasive eye tracker (mounted to the glare shield), instantaneous self-assessment of perceived workload, NASA Task Load Index (TLX) workload measures, researcher observations, final debriefing interviews, and three questionnaires: Cockpit Set-up Preferences, Demographics, and Automation Experiences and Perceptions. This exploratory study of VLJ/ELJ single-pilot workload management and automation use was conducted to answer the following questions: • How do single pilots in small jets manage their workload? • Where do they have problems managing their workload and what might be some reasons why? • Are there any workload management approaches that might be characterized as “best practices” and why? • How do automation and advanced technologies help or hinder single jet pilots in their workload management and what might be some reasons why? This study was also intended to generate baseline data to be used relative to future NextGen-oriented studies. Because of the complex nature of the study and the sub- stantial amount of data analysis required, overall analysis of the data was separated into phases. The analyses described in this report pertain to the management of workload, completion of tasks, and automation use by single pilots flying ELJs during four scripted high workload events occurring during climb out and the en route phase of flight. 1Although technically a flight training device, for simplification it will be referred to as a “simulator” in this report. The four high workload events analyzed were: 1. setting up the automation to intercept the 208o Broadway (BWZ) radial following the completion of the departure procedure out of Teterboro, NJ (KTEB) in leg one, 2. programming a reroute while at cruise and meeting a waypoint crossing restriction on the initial descent from cruise in leg one, 3. the completion of an expedited descent to accommodate another aircraft with an emergency in leg two, and 4. descent to meet a crossing restriction prior to a waypoint and preparation for the approach into Hot Springs, VA (KHSP) while facilitating communication from a lost pilot who was flying too low for air traffic controllers to hear. Approximately two-thirds of the major tasks in the four events were accomplished by the participants with no difficulties. Participants who were successful or encountered no problems in accomplishing a task tended to rate their performance much higher than those who were unsuccessful or did have problems, often by a substantial margin. We found no differences in performance due to pilot age or pilot type (owner-operator or professional pilot). Furthermore, we found a significant effect on task performance success related to hours of experience only for the first event. Some type of error using the G1000 avionics was at the root of the problem for most participants who had difficulty accomplishing one or more of the tasks. All participants committed a variety of errors during all four high workload events (e.g., readback error, airspeed violation), but most were not directly related to overall task success. Implications of the findings are discussed, and techniques demonstrated by our participants that we have characterized as “best practices” have been identified. Recommended strategies for automation use and countermeasures to task overload and workload breakdowns have also been provided. 1 Single-Pilot Workload ManageMent in entry-level JetS INTRODUCTION The development and production of personal jets such as entry level jets (ELJs) and very light jets (VLJs) have made a wider range of operations and missions available to private and professional pilots alike. Private, corporate, and charter pilots can now fly higher and faster than ever before. These jets, as with some of their slightly larger brethren, are typically certified for single-pilot operations as well as for operation by crews of two pilots. The automation and advanced technology aboard these aircraft are essential features that make flight by single pilots possible. However, automation and advanced technology are not a panacea. The design of glass cockpit systems currently used in these aircraft places a heavy cognitive load on the pilot in terms of long-term, working, and prospective memory; workload and concurrent task management; and developing correct mental models as to their functioning (Burian & Dismukes, 2007, 2009). These cognitive demands have been found to have a direct relationship to pilot errors committed during flight (Dismukes, Berman, & Loukopoulos, 2007). Burian (2007) found a signifi- cant correlation between poor workload and time management (i.e., poor crew and single-pilot resource management, which are abbreviated CRM and SRM, respectively) and problems using advanced avionics. Additionally, almost two-thirds of the accident reports she analyzed involved at least one of six differ- ent cognitive performance problems (e.g., distraction, memory problems, risk perception). She found that these problems were experienced at similar rates by pilots flying professionally and those flying for personal reasons. Thus, workload management is a crucial aspect of SRM. Best practices for single-pilot flight task and workload manage- ment must be better understood within the current operating environment and beyond, as we move to an era of optimizing the National Airspace System as outlined in NextGen concepts (FAA, 2012). The accessibility of these ELJs to owner-operators, who may fly less frequently than professional pilots, compels an examination of their proficiency in task and workload manage- ment, in addition to that demonstrated by professional pilots who fly these jets more regularly (National Business Aviation Association, 2005). Jet Single-Pilot Workload An individual has to dedicate finite cognitive and physical resources towards performing any given task. Some of these re- sources include visual and auditory attention, working memory, and vast stores of declarative and procedural knowledge stored in long-term memory (Anderson, 2000). Higher order cogni- tive processes such as decision-making and reasoning will be required for determining strategies to properly prioritize and perform tasks. Energy is also required to perform tasks, both mental and physical. Cognitive resources have been conceptual- ized in various ways, including as a singular shared resource or as multiple resources dedicated to specific modalities, such as vision or hearing (Wickens, 2008). Workload can also be associated with interrupting discrete tasks that take resources away from ongoing tasks. Within aviation, there are a number of discrete tasks that can interrupt the ongoing tasks associated with the aviate-navigate-communicate (ANC) task prioritization scheme. When individuals perform a visually-intensive interrupting task, such as searching their surroundings for obstacles or inbound traffic, they have fewer cognitive resources to attend to ongoing tasks such as navigating along a predetermined flight path. When ATC contacts an aircraft and provides a reroute instruction, that interruption requires that pilots devote auditory resources as they listen and reduces available visual resources as they write down the new clearance. When programming the new route, pilots’ visual resources are narrowly allocated toward the multifunction display (MFD), and memory resources are taxed as they recall the procedure for inputting new waypoints. The constant stream of interrupting and ongoing tasks requires that pilots shift attention among them in an intricate dance commonly referred to as multitasking or concurrent task management (Chou, Madhaven, & Funk, 1996; Hoover & Russ-Eft, 2005; Loukopoulos, Dismukes, & Barshi, 2003). However, when performing multiple tasks there is a decrement in performance caused by the time required to switch between tasks (Gopher, Armony, & Greenshpan, 2000). Pilots must recall what other tasks are waiting to be performed or where they left off when returning to an interrupted task. Thus, research has found a tendency to delay switching tasks because of the chal- lenges involved (Wickens & Hollands, 2000). In modern crewed operations, two pilots divide the work- load between them. One pilot may be managing the entry of waypoint information, while another is communicating with ATC. The result is that fewer cognitive resources are drawn from any single crew member. In single-pilot operations, however, all of the workload must be managed alone. Part of the workload management task for the single pilot is to determine how to best use outside resources, such as cockpit automation, to help complete flight tasks (Burian & Dismukes, 2007, 2009). As described below, cockpit automation is a boon to the single pilot in accomplishing many flight tasks but one that comes with a cost. Pilots must first tell the automation what to do, through programming, and then carefully monitor it to make sure it does what the pilot intended (Roscoe, 1992). At first, it might seem reasonable to conclude that the ad- dition of advanced technology liberates the pilot by taking over the role of a second pilot. However, the automation that is currently available is unable to completely fulfill that role. Automation generally cannot recognize when an error has been made, respond to ATC instructions, reset the altimeter, and it cannot recognize when the pilot needs assistance. Single-pilot operations, therefore, introduce a single point of failure in an aircraft (Deutsch & Pew, 2005; Schutte et al., 2007). Approaches to Measuring Workload The study of workload has resulted in the development of several instruments and measures. Often these instruments measure one’s perception of how difficult a particular task is to perform. The information gained can be used with other, less subjective, data to improve training, procedures, or device interfaces to reduce workload. 2 One of the most well-known instruments is the NASA Task Load Index (Hart & Staveland), more commonly known as the NASA-TLX or simply, TLX. The TLX is an instrument that originally had two main steps. The first assesses the perceived difficulty of a task along six subscales: mental demand, physical demand, temporal demand, performance, effort, and frustration level. The second component weights the importance of each subscale to account for individual differences to compute a final TLX score (Hart & Staveland, 1988; Hart, 2006). Over the years, the TLX has been implemented in a variety of ways. One of the variations has included using the unweighted scores for each of the subscales, thereby eliminating the need to complete a secondary rating scale. The result simplifies the analysis procedure for the researcher and makes the scale easier to complete for the respondent without sacrificing measurement sensitivity. This approach is referred to as Raw TLX, or simply, RTLX (Byers, Bittner, & Hill, 1989; Hart, 2006; Miller, 2001). Another subjective measure of workload is the Instantaneous Self-Assessment (ISA) technique (Castle & Legget, 2002). The ISA, unlike the TLX, is a unidimensional measure of work- load. ISA measures consist of a rating on a scale of one (low) to five (high) of the perceived level of workload, as well as the respondents’ reaction time to provide the rating. The ISA has the advantage of being quick to administer and is minimally intrusive, unlike the TLX (Castle & Legget, 2002; Farmer & Brownson. 2003; Miller, 2001). Cognitive Task Analysis (CTA) is an amalgam of techniques to assess performance for a task or set of tasks. CTA commonly uses direct observation of behaviors of interest, as well as interviews to glean information about the behaviors or thought processes of individuals while they attempt to perform a task (Clark, Feldon, vanMerrienboer, Yates, & Early, 2008). Automation Use The amount and sophistication of technology in aircraft have increased dramatically over the past few decades, and it is impor- tant to understand the varying roles that advanced automation, in particular, can play. First, it can act as a substitute, replacing a function the human operator would normally perform. Such is the case when an autopilot controls pitch and roll and flies a holding pattern, and when automation calculates descent points, rates, and speeds, assists with fuel management, and performs wind corrections (Casner, 2003; Hinton & Shaugnessy, 1984). Second, it can play the role of an augmenter by providing active assistance to the pilot’s actions in the form of envelope protec- tion. Third, automation can aid pilots by collecting, integrating, and presenting information about aircraft systems, airspace, traffic, and weather. For a successful flight, pilots must be able to delegate tasks to automation to reduce their own workload so that they may free up time and cognitive resources to focus on tasks that require higher-level thinking and decision making (Palmer, Rogers, Press, Latorella & Abbot, 1994). Although there are many benefits to introducing advanced automation into general aviation cockpits, it is not without drawbacks (Aircraft Owners and Pilots Association, 2007). The automation will only do what it is programmed to do, includ- ing fly the aircraft into the ground. There are many cases of this in general and commercial aviation. For example, a Beechcraft A-36 Bonanza crashed outside of Chapel Hill, N.C., after the pilot was unable to turn the autopilot off and subsequently impacted terrain while trying to perform an emergency landing with full nose-down elevator trim (NTSB, 1992). The investiga- tion revealed that the pilot would have been required to apply 45 lbs. of aft stick force, necessitating the use of both hands, to counteract the nose-down trim forces of the autopilot and maintain level flight. It is also crucial that pilots constantly monitor the automation to ensure it is doing what is intended. In addition, pilots need to know what to do if the system is not performing as desired. Sometimes the pilot makes a programming error and the cor- rective action involves entering in the proper programming (i.e., re-programming). In other situations, abnormal or emergency procedures exist that the pilot must remember and/or access. In the Chapel Hill accident, a procedure to counteract a run- away trim/autopilot malfunction existed and could have likely prevented the fatal accident. In this circumstance, however, the pilot may not have had time or been able to physically access the procedure while struggling with an autoflight system that would not disconnect. Stress may also have impaired his ability to recall that the procedure was even available. Modern glass cockpits in general aviation aircraft are able to present more information in the same amount of space than traditional round dial gauges. They also integrate information related to aircraft control, communication, and navigation (Air Safety Institute, 2012; NTSB, 2010), as well as allowing easier monitoring of systems, more efficient flying, and improved situation awareness (Billings, 1997; Zitt, 2006). Although glass cockpits and automated systems are able to provide large amounts of information and assist in flying the aircraft, many suggest that pilot workload has not decreased; it has simply changed in nature (Hoh, Bergeron, & Hinton, 1983; Howell & Cooke, 1989; Wiener, 1988). For example, the pilot’s task has shifted from total active controller of the aircraft to supervisory controller over the automated systems, which requires that the pilot know how the automated system operates in order to be able to understand, predict, and manipulate its behavior (Sarter, Woods, & Billings, 1997). If the automated systems suggest a potentially dangerous action, it is important that pilots are able to recognize and disregard the suggested ac- tion. Layton, Smith, and McCoy (1994) found that computer generation of a suggestion or recommendation significantly impacted the operator’s decision even if, unbeknownst to the operator, the recommendation was poor and had potentially harmful consequences. Increased cognitive workload with higher levels of automa- tion may be a function of an increasing memory burden, with pilots having to remember how and what the machine was pro- grammed to do and what it is supposed to be doing over long periods of time. Increasing memory burden requires pilots to use prospective memory, in which they must remember to remember when to perform a task whose execution must be delayed. In the meantime, unrelated tasks are performed, which increases the possibility that pilots will forget to complete the delayed task when it is time to do so (Dismukes, 2010). Furthermore, although automated systems are able to perform procedural and predictable tasks, it is the human operator who is ultimately responsible for tasks requiring inference, judgment, and decision making. When pilots get overloaded with information, their situation awareness, judgment, and decision making become impaired (Burian & Dismukes, 2007). 3 Mode awareness is the ability of an operator to track and an- ticipate the behavior of an automated system (Sarter & Woods, 1992). A moded system is one that produces different behaviors depending on which mode is currently in use (Casner, 2003). A major factor in the safe use of automation lies with the opera- tor knowing what is happening and why. Pilots must be able to evaluate the automation’s intentions through its actions and performance. Mode errors typically occur because the automa- tion interface fails to provide the user with salient indications of its status and behavior (Sarter & Woods, 1995). It is important for manufacturers of airplanes with glass cockpits to ensure that pilots are provided the necessary cues to understand what mode is in use and how to address issues pertaining to possible mode confusion (GAMA, 2005). The design of modern glass cockpits must take into account how many buttons are feasibly able to be placed on the glass panel and how many different layers of menus within those buttons can be used until the pilot becomes confused (GAMA, 2000, 2005). With glass cockpits having layered menus and softkeys that do different things depending on previous button presses, there is a greatly increased demand on memory and attention (Burian & Dismukes, 2007). An NTSB (2010) report on the introduction of glass avionics found that complex integration of data and confusion caused by multiple display modes are some of the leading causes of glass panel accidents. With increased levels of automation, it is vital that pilots avoid becoming complacent in the cockpit and are constantly ensuring that the system is providing the desired action. Wiener (1981) defines complacency as a psychological state characterized by a low index of suspicion that results from working in highly reliable automated environments. It has been established that automation use can lead to complacency in monitoring and a decrease in mode awareness (Parasuraman, Molloy & Singh, 1993; Sarter & Woods, 1995). There is also evidence for the role of personality in automation use as well. In a study conducted by Prinzel (2002), it was demonstrated that self-efficacy (i.e., the belief in oneself as competent and capable) is a moderating variable when identifying pilots who are likely to succumb to automation-induced complacency. Those with low self-efficacy were more likely to suffer from complacency-induced errors. The Current Report This report focuses on ELJ single-pilot workload strategies and performance during four high workload events that oc- curred during the climb out and en route portions of flight. Performance was evaluated against airline transport pilot and instrument rating practical test standard criteria (FAA, 2008a, 2010), as well as the successful completion of the scripted tasks. Because this was an exploratory study, instead of developing a number of detailed hypotheses to test, we designed situations that we believed would increase workload and embedded them in experimental scenarios for our study participants to fly. These scenarios involved flight in the relatively demanding operational environment of the U.S. east coast corridor from the New York City area through and to the southwest of Washington, DC. We were interested in learning about how single-pilots flying an ELJ manage their workload and use automation in such an environment. We were interested in examining problems they encountered, determining possible reasons why, and identifying strategies for task management and automation use that worked out particularly well (i.e., “best practices”). We also wished to gather baseline information on single-pilot operational behavior for reference in future studies. The data from the current study provided an opportunity to begin constructing a model of nor- mative behavior and workload management strategies involved in single-pilot jet operations. METHODS Participants The FAA Airmen Certification Branch provided the names of all pilots who possessed a C510-S type rating at the time of our request. From that list, 321 pilots were identified as living in the contiguous 48 United States of America. These pilots were mailed recruitment letters briefly describing the study and invited them to contact the NASA Ames Human Systems Integration Division Testing and Participant Recruitment Office if they were interested in participating. One hundred one pilots responded and were sent, via email, a copy of the NASA Informed Consent form and three questionnaires: Demographics, Advanced Avionics and Automation, and Schedule Availability. Forty-six pilots (3 females and 43 males) returned the completed questionnaires, and 14 male pilots were selected for participation in the simulation portion of the study. Participation in the study was voluntary, and pilots were allowed to terminate their participation in the study at any time, though none chose to do so. They were paid a rate of $50.00 per hour of participation and were reimbursed for all travel costs and provided a per diem for the cost of meals. Materials Demographic questionnaire. Background information was solicited from potential participants to screen for pertinent flight certification and history that was essential for the study. A portion of this information was used to identify potential participants representing the population of interest (Mustang owner-operators), as well as others (i.e., professional pilots) who flew the experimental scenarios in the simulator. In addition to the type of flying performed and hours of experience, par- ticipants were asked to rate their experience and perceived skill levels regarding the use of various avionics packages and cockpit technologies such as the Garmin G1000™ and autoflight systems. As indicated earlier, 46 participants completed the demographics questionnaire, which can be referenced in Appendix A. Advanced avionics and automation questionnaire. An ad- vanced avionics and automation questionnaire was also completed by 46 participants. This questionnaire was designed to gather information with regard to participant attitudes toward advanced technologies such as glass cockpits/primary flight displays and multifunction displays. The participants were polled on which features they preferred most and least, as well as on issues related to advanced avionics and automation design, functionality, use, training, and maintaining proficiency, among other things. The questionnaire can be found in Appendix B. Citation Mustang and G1000 Cockpit Set-up Preferences questionnaire. The 14 pilots who participated in the simulator portion of the study completed a questionnaire to indicate their preferred Garmin G1000 default settings. This information was then used to set up the G1000 in the study simulator prior to their session to match those settings in the actual aircraft that they flew. For example, temperatures on the G1000 displays can 4 be expressed in degrees Celsius or degrees Fahrenheit. Similarly, pilots can choose among 12 different variables, such as distance (DIS), estimated time of arrival (ETA), and true airspeed (TAS), for display in four fields at the top of the G1000 Multifunction Display (MFD). The Citation Mustang and G1000 Cockpit Set-up Preferences questionnaire can be seen in Appendix C. Flight bag materials. A flight bag was provided for pilots to use during their flights in the simulator. Items in the flight bag included a knee-board with paper; pencils and pens; three dif- ferent types of flashlights; colored sticky tabs; a stopwatch/timer; a baseball cap; current Visual Flight Rules (VFR) sectional and terminal charts; current paper Jeppesen high and low altitude Instrument Flight Rules (IFR) en route navigation charts; com- plete Jeppesen Airway Manuals with current paper departure, arrival, and approach plates; and current Airport and Facilities Directories. Pilots were allowed to take as much or as little of the flight bag materials with them into the simulator as desired. However, once the scenario began, pilots were not allowed to leave the simulator to retrieve flight bag materials they had left behind in the pre-flight briefing room. Flight briefing materials. Prior to each scenario, pilots were provided with a binder of briefing materials (see Appendix D). Each binder included: • The purpose of the flight, airports of departure and destina- tion, the current date, proposed time of departure, aircraft location on the field at the departure airport, and planned aircraft parking at the destination airport • A departure airport diagram (downloaded from the Web) with the aircraft’s location indicated • A completed flight plan on FAA Form 7233-1 • A navigation log • Completed weight and balance information, including a weight and balance diagram • A complete weather briefing package including an area forecast and synopsis, current satellite conditions, sig- nificant meteorological advisories (SIGMETs) and airmen weather advisories (AIRMETs), weather and sky conditions, pilot reports (PIREPs), meteorological aerodrome reports (METARs), and terminal area forecasts (TAFs) and radar returns for departure and destination airports, winds aloft forecast for the route of flight, en route METARs and ter- minal area TAFs, and a complete set of notices to airmen (NOTAMs). Some of this material was downloaded (and modified as necessary) from the National Oceanic and Atmospheric Administration (NOAA) Aviation Weather Center Aviation Digital Data Service (ADDS) on a day with similar conditions as that in the scenarios (see http:// www.aviationweather.gov/adds/ ). Familiarization and experimental flight scenarios. With the help of a Cessna Citation Mustang and other jet pilot subject matter experts (SMEs) and in consultation with ATC SMEs, two flight scenarios were designed for use in this study. The first flight was developed so that participants could become familiar and comfortable with the research environment, including the simulator, the panel mounted eye-tracker, and the ISA measure (described below). The familiarization flight was an IFR flight lasting approxi- mately 30 minutes from Clinton-Sherman Airport (KCSM) in Oklahoma, to Will Rogers World Airport in Oklahoma City, Oklahoma (KOKC). Pilots performed the same flight tasks that they would complete for the experimental flight, including re- viewing the pre-flight briefing packet materials, pre-flight cockpit preparation, conducting a takeoff and an instrument departure, instrument en route navigation, communicating with ATC, and completing an instrument approach and landing. Although pilots were completing an IFR flight, the weather for the familiariza- tion flight was visual meteoro- logical conditions (VMC). The scenario was designed to produce relatively low workload, although on two occasions the pilots were informed of traffic crossing their route of flight that was not a conflict (i.e., “not a factor”) by ATC. Following the familiariza- tion flight, the participants were asked if they had any questions and if they understood how to use the ISA device. No data from the familiarization flights were analyzed. Figure 1 illustrates the route of flight for the familiariza- tion scenario. The experimental flight con- sisted of two legs, each approxi- mately one hour in length. Each leg was designed to include a number of high workload tasks that would be typical of the type experienced by pilots flying along the scripted routes. In the first leg, pilots departed from Teterboro Figure 1. Familiarization Scenario Route of Flight. Not to be used for Navigation Figure 1. Familiarization Scenario Route of Flight. 5 Airport in New Jersey (KTEB) and landed at Martin State Airport (KMTN) just outside of Baltimore, Maryland. In addition to normal piloting tasks such as reviewing briefing materials and conducting en route navigation, the participants were confronted with the following high workload tasks and conditions: • TEB6 Departure off runway 24, KTEB • Intercept the Broadway (BWZ) 208o radial • In-flight reroute • Meet a crossing restriction at a waypoint • Hold at a waypoint • RNAV (GPS) Rwy 33, circle to land Rwy 15 at KMTN • IMC conditions throughout, although not down to mini- mums and no convective weather • Traffic, although none was intended to be a factor for the participant pilots After a break for lunch, lasting 30-60 minutes, pilots then completed the second leg of the experimental flight in which they departed from Martin State Airport (KMTN) for a destination of Hot Springs/Ingalls airport (KHSP) in Virginia. The high workload tasks and conditions of this leg included: • Radar vector departure from KMTN • Expedited descent to accommodate another aircraft with an emergency • The pop of the anti-skid circuit breaker approximately half-way through the scenario • Meet a crossing restriction 15nm prior to a waypoint • Asked to assist in relaying communication to a Washing- ton Center controller from a lost pilot at the same time as meeting the crossing restriction and preparing for the approach and landing • Perform the ILS or LOC Rwy 25 approach at KHSP • Deal with a temporarily disabled aircraft on the runway at KHSP (typically by going around or performing the missed approach procedure) • IMC conditions throughout, although not down to mini- mums and no convective weather • Traffic, although none was intended to be a factor for the participant pilots with the exception of the disabled aircraft at KHSP Figures 2 and 3 illustrate the route of flight and major work- load tasks for the experimental flight Legs 1 and 2, respectively. Background chatter. An essential part of pilot workload in busy airspace is attending to background chatter on the radio, in part to monitor for a call from ATC but also to be alert to surrounding aircraft activity in case there might be some effect upon one’s own flight. An elaborate script of background chatter involving over 100 other aircraft was developed and recorded for use in this study (Burian, Pruchnicki, & Fry, 2013). Un- fortunately, unanticipated problems were experienced with the simulator audio system and we were unable to use it. We did, however, have a few occasions where “other pilots,” such as the “lost pilot” during the second leg of the experimental flight, interacted with ATC and with the study pilots over the radio during the three scenarios. Figure 2. Experimental Leg 1 Scenario Route of Flight. Not to be used for Navigation Figure 2. Experimental Leg 1 Scenario Route of Flight. 6 All “other pilot” communications were scripted and per- formed by members of the research team in real time (i.e., not pre-recorded) as the scenarios unfolded. Study scripts. Detailed scripts were developed for all three study scenarios and were used to guide all communications from ATC and other pilots as well as the triggering of all events, such as the circuit breaker pop during the second leg of the experimental flight. The scripts included the following: aircraft location, active radio frequency, triggers for all ATC calls to the participant pilots (such as the aircraft’s location), notes and alternate actions that may be necessary, a description of pilot tasks (to facilitate situation awareness among the ATC and researchers), and all exact communications from ATC and other (non-participant) pilots. Figure 3. Experimental Leg 2 Scenario Route of Flight. Figure 3. Experimental Leg 2 Scenario Route of Flight. 7 An excerpt of the familiarization scenario script can be seen in Figure 4. All of the scripts developed for this study are included in their entirety in Burian, Pruchnicki, and Fry (2013). Cessna Citation 510 Mustang flight simulator. The flight simulator used in this study was a Frasca level 5 flight training device that features a realistic Mustang flight deck with a G1000 avionics suite, digital control loaders, and a high-fidelity digital surround sound system that accurately replicates flight, engines, system, and environmental sounds. The out-the-window (OTW) display system included a 3D Perception 225 degree (lateral angle) spherical projection screen that gave the pilot a realistic field-of-view. Figure 4. Excerpt of the Familiarization Scenario Script. Figure 4. Excerpt of the Familiarization Scenario Script. 8 Six wide-quad-extended-graphics-array (WQXGA) (1920x1200) projectors were driven from six high-end Intel server class computers at 60 Hz. The projection screen used embedded sensors to detect the alignment, brightness, and edge blending quality of the projected images. The projection system was used to display high-fidelity MetaVR™ terrain imagery and 3D computer models of the airports that the pilots would encounter during the study. Pictures of the simulation environment can be seen in Figures 5 and 6. Eye tracker. Eye movements of participants were tracked using a FaceLab™ v5 system consisting of non-invasive cameras, IR emitters, and software from Seeing Machines, Inc. Camera set-up and calibration procedures were followed, as described in the FaceLab user manual, except where modified for use in the simulator cockpit. Specifically, the dual eye tracking cameras were mounted on the left-seat cockpit dash, above the level of the control yoke column without blocking the view of either the outside or the cockpit instruments. In addition, during calibration procedures, the pilot (rather than the experimenter) held the calibration target up to the camera while seated in the cockpit to ensure that the distance to the cameras were consis- tent and tailored for each pilot’s height and seating position. Image quality, camera focusing, and calibration were confirmed by the experimenter on a computer laptop located just outside and below the left cockpit window and initially required 10-15 minutes. Recalibration of the eye-tracker took only a minute or less and was performed every time the participant re-entered the simulator cockpit following a break. Due to calibration errors, events in the simulated flight could not be related to tracked eye movements in a manner required for monitoring time-dependent cognitive workload; therefore, analysis of the eye tracking data was not possible. It is recom- mended that a system of video and audio time-event markers, called “time hacks,” be included in future eye tracking/flight simulator studies. Figure 5. Cessna Citation Mustang flight simulator and projection system. Figure 5. Cessna Citation Mustang Flight Simulator and Projection System. Figure 6. Simulator G1000 avionics suite and out-the-window view. Figure 6. Simulator G1000 Avionics Suite and Out-The-Window View. 9 Instantaneous self-assessment (ISA). The ISA device consisted of a small rectangular box with a red light at the top and five numbered buttons arranged vertically below it. Pilots were prompted to perform an instantaneous self-assessment of workload by pressing one of the five numbered buttons (with 5 being associated with “very high” workload and 1 meaning “very low” workload) when the red light was illuminated. Research- ers controlled when the light would illuminate remotely from the experimenter’s station. Once illuminated, the light would stay on for up to 60 s or until the participant pressed one of the numbered buttons. Prior to the familiarization flight, pilots were briefed on the use of the ISA rating system and were provided a printed card, retained for their reference during flight, which reiterated how the ISA was to be used and described the mean- ing for each ISA rating. Pilots were also informed verbally and in writing that making an ISA rating when prompted was secondary to any other task. They were instructed to only make the rating when they were able and to not make a rating at all if there was no break in their primary task during the 60 s that the ISA light was illuminated. Table 1 depicts checkpoints where participants were prompted to make an ISA workload rating during the two legs of the experimental flight. NASA Task Load Index. Paper and pencil versions of the NASA TLX were administered immediately after Leg 1 and again after Leg 2. Pilots were asked to give ratings on each of the subscales for the flight overall, as well as for specific high workload tasks or phases of flight. The events for which participants completed a TLX for both legs of the experimental flight are shown in Table 2. Table 1. Experimental flight ISA rating prompts. Leg 1 Leg 2 2000 foot level-off plus 60 s Aircraft reaching 2000 feet plus 60 s Heading change for BIGGY waypoint plus 60 s Aircraft reaching 6000 feet after expedited descent Reaching COPES waypoint Aircraft reaching FL200 plus 60 s Initiation of descent from FL200 Aircraft turning over CSN VOR plus 60 s Aircraft descending through 12,000 feet Aircraft reaching MOL VOR Aircraft turns outbound after crossing JUGMO waypoint in the hold Aircraft turning inbound over AHLER waypoint on the approach plus 15 s Table 1. Experimental Flight ISA Rating Prompts. Table 2. Experimental flight NASA TLX task rating events. Leg 1 Leg 2 Leg 1 Flight Overall Leg 2 Flight Overall KTEB 6 Departure KMTN Departure Build Course to Intercept Broadway (BWZ) 208 Radial Immediate Descent for Emergency Aircraft VNAV Path to Descent Circuit Breaker Pop Event Hold at JUGMO Waypoint Assist Lost Pilot RNAV (GPS) Rwy 33 Approach and Circle to Land Runway 15 KMTN Meet Crossing Restriction Before MOL VOR ILS Approach to KHSP Deal With Disabled Aircraft and Complete Landing at KHSP Table 2. Experimental Flight NASA TLX Task Rating Events. 10 Data acquisition and storage. The Cessna VLJ Mustang simulator lab used three systems to digitally record and store audio, video, and simulator data streams. Each stream was re- corded and analyzed independently. All data recording systems were managed and controlled at the operator station. Audio recordings safe. A Zoom H4n Handy Recorder™ was used to record and store high-fidelity audio recordings of cockpit, ATC, and experimenter communications as well as post flight interviews that were conducted with each participant. The Zoom H4n Handy Recorder stores audio information in 96Khz, 24- bit, MP3 digital audio files onto standard Secure Digital High Capacity (SDHC) memory cards. Additional audio recordings of pilot and ATC communications were achieved through a high-fidelity digital recording system which employed several devices that were networked together. These audio recordings were integrated into the video recordings, discussed below. Video recordings. Four Arecont Vision IR™ video cameras were specifically selected for their high resolution color image streams. Two of the Arecont cameras were mounted on tripods placed on each side of the simulator cockpit. The camera on the pilot side recorded the pilot’s primary flight display (PFD). The camera on the co-pilot side recorded the pilot so participant well-being could be monitored as required by FAA and NASA Institutional Review Board protocol. A third camera was mounted at the aft of the simulator cab to record the MFD. The fourth camera was mounted inside the cockpit on the co-pilot’s win- dow pillar, and it recorded the co-pilot’s PFD. All four cameras operated at 60hz NTSC signal and were infrared (IR) sensitive. A Plexsys™ data recording system called Enhanced Mission Record and Review System (EMRRS™) was used in the VLJ simulator lab to record, process, and store high-quality digital video streams. EMRRS was used to combine multiple audio, video, and data streams and store them on a Plexsys media storage server. The Arecont Cameras and sound mixer were connected to the Plexsys recording system through a network hub. EMRRS synchronized all the recorded streams for accurate time-stamped playback and real-time analysis. Additionally, it provided real- time observation of pilot activity during the recording, including pausing, rewinding, and replay of the media without disturbing the recording. Simulator data stream. The Frasca simulator features a data storage capability including 5159 variables. The variables are a recording of the state of the aircraft and the immediate simulated environmental conditions. The data are stored in a Frasca proprietary file format that is exported to standard, comma delimited, or comma separated value (CSV) text files, which can be opened in a variety of spreadsheet programs. Experimenter’s station. Researchers and air traffic controllers sat at the experimenter’s station (see Figures 7 and 8) situated approximately 20 feet behind the simulator. Several monitors at the station allowed the researchers and ATC to monitor the progress of the flight and the feed from the video recorders in the cockpit. Researchers playing the role of “other pilots” and ATC wore headsets at the station and spoke on the radios by press- ing a push-to-talk switch on the headset or audio system panel. Pilot headsets. Pilots were invited to bring and use their own headsets but none did. The simulator came with a set of lower-quality foam headphones that are not noise-cancelling. They were used by one participant and resulted in some dif- ficulty in hearing ATC communications. All the remaining participants used a Bose A20 noise-cancelling headset that we provided. Figure 7. Experimenter’s station. The simulator and visual system can be seen in the background. Figure 7. Experimenter’s Station. The Simulator and Visual System can be Seen in the Background. 11 Debriefing interview. After a short break following the sec- ond leg of the experimental flight, a semi-structured debriefing interview of participants was conducted. We asked pilots about their overall impression of their experience for the day and if there were any tasks performed during the flights that increased their workload. In addition, we asked how they felt they managed their workload during the flights. For a complete description of the specific questions that were asked during the semi-structured interviews, see Appendix E. These interviews were recorded as WAV files on a digital audio recorder and were transcribed for later analysis. Task analyses. During the study design phase of this re- search, high level outlines of the two experimental flights were constructed (Burian, Christopher, Fry, Pruchnicki, & Silverman, 2013). These outlines included all the major tasks to be com- pleted by the participants during those flights. Detailed tasks analyses were then conducted with the assistance of a SME who is knowledgeable about the G1000 and serves as an instructor and mentor pilot in the Cessna Citation Mustang. In these task analyses, the major tasks were broken down into subtasks, sub-sub-tasks, and so on until each step for the completion of a task was identified down to the level of pressing a button or turning a knob. To the extent possible, cognitive tasks associ- ated with some of these physical tasks (e.g., “recall that ATC gave direction to report when reaching assigned altitude”) were also included. These task analyses were developed to classify the correct way in which each task must be completed or—when multiple ways of accomplishing a task exist—classifying one way of accomplishing the scripted tasks that represents the correct action and a superior approach to workload management and task completion, as determined by our SME. The task analyses were used during data analysis when reviewing approaches to task completion and workload management employed by the study participants. The task analyses for the two experimental flights can be seen in their entirety in Burian et al. (2013). Figure 8. Researchers and ATC at the experimenter’s station. Figure 8. Researchers and ATC at the Experimenter’s Station. 12 Concurrent task timelines. Following the completion of the task analyses for the two experimental flights, we developed Concurrent Task Timelines (CTTs) in which bars (or lines) representing the first three levels of tasks and sub-tasks included in the analyses were drawn relative to each other (the horizontal axis on the page indicates time; see Figure 9). The purpose of these timelines was to depict concurrent tasks in a format that indicted their expected length relative to each other. Again, our Cessna Citation Mustang SME assisted in the development of these timelines, which were used by researchers during the data analysis phase of the study for identifying and evaluating participant performance and workload management strategies. The complete CTTs for both experimental flights can be seen in Burian et al. (2013). Design This exploratory study of jet single-pilot workload manage- ment was observational in nature. As described earlier, detailed scripted flight scenarios which included a variety of typical but high workload tasks were developed, and pilots representing the population of interest agreed to fly the scenarios. Recently retired air traffic controllers who had experience directing traffic in the US northeast corridor (where the experimental flights took place) were hired to play all the roles of ATC in the scenarios (e.g., ground controller, local controller, departure, center, etc.). Procedure The evening before each of the participants was scheduled to complete the study, they met with one of the researchers to review the study procedures and purpose. Participants were given an opportunity to ask any questions, and they signed the FAA Informed Consent Form. They were given the flight briefing materials and associated charts and maps for the familiarization flight. Participants conducted whatever pre-flight planning they felt necessary for the familiarization flight that evening in their hotel rooms. Participants were told that they should both prepare for and fly the scenarios in the same ways as they normally did when flying in the real-world. The following morning, participants were picked up from their hotel rooms and driven to the simulator facility at CAMI. The pilots first completed a flight around the pattern at KOKC to begin getting familiar with the simulator environment. During this circuit (on downwind), pilots were cued to read a series of words printed on a card. This provided baseline audio data for use in later analyses of pilot voice communications and workload during the experimental flight. All pilot communications in the simulator (once their headset was on) were captured in WAV files. Tail numbers of the participant’s own Mustang aircraft were used during all ATC radio communications throughout familiarization and experimental flights to further a sense of familiarity for the pilots in the simulation environment. Following the completion of the circuit at KOKC, pilots were given an opportunity to review the briefing materials for the familiarization flight from KCSM to KOKC and were provided the flight bag materials. Pilots then re-entered the simulator cockpit, were briefed on the use of the ISA, participated in the initial calibration of the eye-tracker, and flew the familiarization flight, which lasted approximately 30 minutes. Pilots were then provided a brief break, typically around 10 minutes, and were offered a choice of beverages and snacks. They were given the briefing materials for the experimental flights and were told that they could review the materials for both Figure 9. Sample portion of a concurrent task timeline. Figure 9. Sample Portion of a Concurrent Task Timeline. 13 legs or only the first, whichever they preferred. The amount of time taken by participants to complete this pre-flight briefing varied according to whether both legs or only the first leg was briefed and ranged from 12 to 90 minutes. Those participants who only briefed the first leg took approximately 30 minutes to review the materials. When pilots expressed that they were ready, they flew the first leg of the experimental flight, which lasted approximately 60 minutes. In this flight, they departed from Teterboro, New Jersey (KTEB) with a destination of Martin State Airport (KMTN), near Baltimore, MD, during daylight hours in September on a moderate IMC day. The aircraft was fully fueled. Following cockpit setup and G1000 initialization, the flight was cleared to Martin State Airport via the Teterboro Six Departure. An IFR flight plan was filed and the departure weather con- sisted of rain and a slight crosswind at KTEB. Due to proximity to New York City, the departure procedure was complex. IMC was encountered during the initial climb. Once established en route with New York Center, radar vectors and route modifica- tions were assigned. Altitude restrictions were applied as well to avoid simulated traffic conflicts in busy airspace. The flight evolved normally and was representative of a typical flight in the USA Northeastern Corridor. After handoff to Washington Center, and following a brief hold, the single pilot completed the RNAV (GPS) RWY33 non-precision approach in marginal VFR conditions and circled to land on runway 15. After land- ing, the participant shut down the aircraft. At the completion of the flight, participants left the simula- tor, completed the NASA TLX measures for the first leg, and were then provided lunch. Following the lunch break, pilots were given an opportunity to review the briefing materials (or conduct a pre-flight briefing if not done earlier) for the second leg of the experimental flight. Participants’ review of the second leg briefing material ranged from 4 minutes to 45 minutes and varied according to whether the second leg had been briefed earlier as part of the Leg 1 review. When pilots indicated they were ready, they flew Leg 2 of the experimental flight. In Leg 2, the participants departed from Martin State airport (KMTN) with a destination of Ingalls Field at Hot Springs, Virginia (KHSP). This flight took place during daylight hours in September on a moderate IMC day. Following the cockpit setup and G1000 initialization, the aircraft was cleared to Ingalls Field via the radar vectors to PALEO, the Nottingham (OTT) VOR and then as filed. Runway 15 was in use for departure with an initial altitude assigned of 2000’ MSL. An IFR flight plan was filed for the KMTN departure and a slight crosswind existed. The departure procedure was straight out and simple, but the airspace in the D.C. Metroplex is complex. IMC was encountered during the initial climb, and altitude restrictions were applied to avoid traffic conflicts. During the climb to cruise altitude, the aircraft was instructed by ATC to descend immediately to accommodate another aircraft with an emergency. Once established en route with Washington Center, the flight evolved normally and was representative of a typical flight in the USA Northeastern Corridor. However, a relatively minor non-normal event occurred (the popping of a circuit breaker) which required reference to a non-normal procedure in the aircraft Quick Reference Handbook (QRH). In the final third of the flight, the pilot was asked to assist with communi- cation between Washington Center ATC and a pilot who was lost and flying too low to be heard by ATC. Upon receipt of the Automated Weather Observation System (AWOS) for KHSP, the pilot was instructed to prepare for a precision ILS approach with an expected break-out from the overcast at 600 ft above decision height (DH). As part of the experimental design, an aircraft landing prior to the participant’s aircraft was temporarily disabled on the runway, forcing the participant to go around or complete a missed approach procedure. Following the second landing attempt, the pilot secured and shut down the aircraft. NASA TLX measures for the second leg were then completed, and the participant was provided a short break before participating in the debriefing interview. At the completion of the debriefing interview, participants were thanked for their participation and provided a certificate and CAMI promotional pen as thank you gifts. Participants were reminded of reimbursement procedures for their travel expenses and were driven back to their hotels. DATA MANAGEMENT AND PREPARATION This report focuses on single-pilot workload management and performance during four high workload events that oc- curred during the en route phase of flight from the completion of the departure procedure/ initial climb to the initiation of an instrument approach procedure. We spent several months downloading and organizing data from the simulator itself, the audio and video recordings, the ISA data, and the eye tracker data. CAMI personnel placed these data on external hard drives, some of which were shipped to NASA collaborators. We also transcribed the recorded debriefing interviews conducted with participants and recorded Mustang SME comments made while reviewing the recordings of the experimental flight. We also developed and populated four databases with information from three questionnaires and NASA TLX workload measures. NASA personnel shared updated documents outlining data to be analyzed, research questions to be answered, and hypotheses to be evaluated. Biweekly, weekly, and sometimes daily teleconferences were held among NASA and CAMI research team members to discuss data management and preparation, data analysis, findings, writing assignments (which were distributed among the team), and to edit this report. Because of the qualitative nature of much of the data and the large and distributed nature of the research teams, much more coordination and communication regarding the approach to data analysis was needed than is typically the case. Simulator Flight Performance Data and Data Extraction The Frasca simulator included the capability of recording real-time flight data. The data stream contained 5,159 separate simulation variables sampled and recorded at a rate of 5Hz. Each sample constitutes a sequentially numbered “frame” in the data stream. These data included latitude, longitude, and altitude in- formation, the status of cockpit controls and displays, simulated weather settings, aircraft attitude and airspeed, and the activation and values of specific G1000 settings (e.g., barometric pressure). Following the completion of each scenario, the simulation data stream recording was stored in a proprietary data format on the local simulator drive. 14 Table 3 shows an example of some of the flight parameters, units of measure, and variable names within the Frasca software package that were used for analysis. Table 3. Sample Flight Simulator Variables. Parameter Units Variable Name Altitude (MSL) Feet AltitudeMSLExpression_Ft Indicated Airspeed Knots IndicatedAirspeedExpression_Kts Heading (Magnetic) Degrees MagneticHeadingExpression_Deg Vertical Speed Feet per Minute VertSpeed_Fpm Bank Angle Degrees BankExpression_Deg Pitch Angle Degrees PitchExpression_Deg Landing Gear Position True or false MISCOUTPUTS:NOSELDGGEARDOWNANN Flap Selection Degrees GIA1_GEA1:DOIOP_C_EAU_FLAPS_POSITION.POSITION_DEG Autopilot Engage- ment On or Off AUTOPILOT1:DOIOP_C_AFCS_1_ANNUNC.AP_ENGAGESTATE Latitude Radians LatitudeExpression_Rad Longitude Radians LongitudeExpression_Rad Autopilot Vertical Mode Ordinal AutoPilot1:doIOP_C_AFCS_1_ANNUNC.PitchCoupledMode Autopilot Horizontal Mode Ordinal AutoPilot1:doIOP_C_AFCS_1_ANNUNC.RollCoupledMode Table 3. Sample Flight Simulator Variables. 15 Figure 10 shows a spreadsheet of some of the downloaded data for several flight parameters recorded by the simulator. More information about the extraction and transformation of the simulator data in preparation for analysis can be found in Williams et al. (2013). Figure 10. Excel spreadsheet produced for 11 specific flight parameters. Figure 10. Excel Spreadsheet Produced for 11 Specific Flight Parameters. 16 Graphs Using extracted simulator data, graphs of several continu- ous variables were created in Microsoft Excel. The following variables were graphed to assist us in our analyses: altitude, airspeed, vertical speed, engine power (N1), magnetic heading, autopilot use (on/off), and autoflight modes used. The x-axis of all graphs was expressed as time in minutes, and the y-axis was indicated by a scale appropriate to each variable. To compare the multiple variables simultaneously, graphs were stacked on top of each other, aligning time markers along the x-axis. Figure 11 illustrates these graphs for one of the high workload events analyzed for this report. More information about the construction and content of the graphs can be found in Williams et al. (2013). Figure 11. Stacked graphs of simulator data. Figure 11. By AGL Stacked Graphs of Simulator Data. 17 Google Earth Plots To assist in the analysis of the data, the flight path trajecto- ries were plotted in Google Earth™. Figure 12 shows a sample flight trajectory for a circling approach and landing on runway 15 at KMTN, with a 1.3 nautical mile radius circle around the runway threshold as an obstacle clearance safe area, plotted in Google Earth. The identification of specific events during the flight such as the use of the autopilot was indicated by uniquely formatted place marks so they could be easily distinguished within the flight path trajectory. Figure 13 shows a Google Earth plot of a flight trajectory with one of the place marks selected, showing the additional information available. Developing Google Earth plots require the creation of stan- dardized OpenGIS® KML files. Details of the KML file standard are available from the maintainers of the specification. Open Geospatial Consortium, Inc. at http://www.opengeospatial.org/ standards/kml/. The procedure for creating KML files and place marks is described in detail in Williams et al. (2013). Figure 12. Example flight trajectory plotted in Google Earth. Figure 12. Example flight trajectory plotted in Google Earth. Figure 13. Flight path trajectory with additional aircraft data selected. Figure 13. Flight Path Trajectory With Additional Aircraft Data Selected. 18 Flight Communication Transcription The audio files of the flight communications were transcribed into Excel files with the use of Start Stop Universal™ software. This enabled the extraction of start and stop times for each transmission, including communications between ATC and the participant or other aircraft pilots included in the scenario. Since the participant’s cockpit headset included a “hot” mic (on and recording continuously), the transcripts also included when a participant was recorded thinking aloud, and the simulator voice aural alerts heard in the cockpit. Each transcribed file started at zero hours, minutes, and seconds (00:00:00). Figure 14 illustrates what a transcription might look like; more information about the transcription process can be found in Williams et al. (2013). Voice Analysis Previous research has found a relationship between different vocal qualities and stress or workload. For example, it has been found that speech fundamental frequency (pitch) and vocal intensity (loudness) increase significantly as workload increases and tasks become more complex (Brenner, Doherty, & Shipp, 1994; Griffin & Williams, 1987). Speech or articulation rate has also been shown to increase when the speaker is under stress associated with high workload (Ruiz, Legros, & Guell, 1990). Therefore, we decided to conduct various voice analyses as pos- sible objective indicators of participant workload in this study. To prepare each participant’s audio files for the fundamen- tal frequency (FO) and articulation rate analyses, audio files containing the flight communications of each participant were exported into Sound Forge Audio Studio (Version 10). Sections of communication for each participant to be used in the analyses, described later, were identified and labeled; all audio of the ATC, experimenter, other pilots, and simulator noises were deleted from the file. Each identified section of communication was then cut and pasted into a single WAV file so that each participant had one audio file containing all audio sections to be analyzed. For the FO analyses, these same audio sections were exported into WaveSurfer™ (Version 1.8.8p4), and FO was calculated at a rate of .01 s. The average articulation rate per section was then calculated using Praat™ software (Version 5.3.22; Boersma & Weenink, 2012; de Jong & Wempe, 2009). Articulation rates were then calculated by dividing the number of syllables by the total speaking time. Figure 14. Sample flight communication transcription. Figure 14. Sample Flight Communication Transcription. 19 Video Data Two types of video data were collected, cockpit camera footage and a recording of a navigational map combined with limited flight parameter data. During data analysis, the recorded cockpit video feed from the four cameras could be viewed on one screen, as shown in Figure 15, or video from just one of the cameras could be selected to make it easier to see what was recorded. Although post-collection examination of the cockpit video data revealed a lower video quality than expected, they still served as valuable sources to confirm simulator flight parameter data by helping to place other data in context. During data collection, video of a dynamic display of a navigation map, including a depiction of the participants’ air- craft position, was used as a radar screen for ATC and was only available at the experimenter’s station. A limited set of 40 flight parameters was displayed on the right hand side of the screen (see Figure 16), which allowed researchers and ATC to moni- tor participant performance in real-time. Video recordings of the navigation maps with the flight parameters were also used during data analysis. Figure 15. Four camera views of Cessna Mustang simulator cockpit. Starting from top left rotating clockwise – MFD, view of pilot, pilot’s PFD, co-pilot’s PFD. Figure 15. Four Camera Views of Cessna Mustang Simulator Cockpit. Starting From Top Left Rotating Clockwise – MFD, View Of Pilot, Pilot’s PFD, Co-Pilot’s PFD. Figure 16. ATC dynamic navigation map and flight parameter display. Figure 16. ATC Dynamic Navigation Map and Flight Parameter Display. 20 RESULTS Participant Demographics Fourteen male pilots, type-rated to fly the Mustang as a single pilot, participated in the simulator portion of this study. Dur- ing data collection, we discovered that one of the participants had no prior experience flying as a single pilot, so data from his flights were not included in any of the analyses reported below. In addition to a C510-S type rating, participants were either owner-operators of a Cessna Citation Mustang (n=7) or flew the Mustang as part of their jobs as corporate or contract pilots (n=6). Their ages ranged from 29 to 61 years, with a mean age of 48.9 years. In the year prior to the study, our participants reported flying the Cessna Mustang a mean of 153.7 hours (range: 68-350 hours) and flying the Mustang as a single pilot for a mean of 138.5 hours (range: 15-350 hours). General flying and Citation Mustang-specific flying history can be seen in Table 4. No significant differences in flight hours were found between study owner-operators and professional pilots. Table 4. Participant Flying History. Mean Median Range SD General Flying Total number of flight hours Owner-operators Professional pilots 3998.92 3507.85 4571.83 3950.00 2500.00 4425.00 1000 - 8130 1000 - 8130 2900 - 6381 2087.84 2590.75 1294.58 Flight hours in the past year Owner-operators Professional pilots 230.53 201.14 264.83 170.00 170.00 247.00 90 - 528 90 - 528 100 - 515 152.57 151.09 160.79 Total number of jet hours as a single pilot Owner-operators Professional pilots 331.61 287.86 382.66 210.00 230.00 168.00 100 - 1345 100 - 475 100 - 1345 329.36 136.98 481.69 Flight hours in the past 3 months Owner-operators Professional pilots 52.61 51.43 54.00 40.00 40.00 45.00 20 - 121 25 - 100 20 - 121 30.26 28.09 35.31 Citation Mustang Specific Flight hours with a mentor pilot Owner-operators Professional pilots 11.16 13.57 8.36 5.00 15.00 0.00 0 - 35.00 0 - 35.00 0 - 25.20 12.62 12.82 12.96 Flight hours in the past year Owner-operators Professional pilots 153.69 161.43 144.66 125.00 125.00 119.00 68 - 350 75 - 350 68 - 325 89.61 93.75 92.45 Flight hours in the past year as a single pilot Owner-operators Professional pilots 138.46 160.71 112.50 100.00 120.00 83.50 15 - 350 75 - 350 15 - 325 99.78 94.09 108.49 Table 4. Participant Flying History. 21 Pilots were asked to fill out a questionnaire assessing their experience with advanced avionics and automation. Analysis revealed that the pilots were fairly experienced in using the G1000, as well as other types of advanced avionics (e.g., Avi- dyne, Chelton). Some of the questions asked, along with rating means, standard deviations and ranges are presented in Table 5 (see Appendix B for the complete questionnaire). Ratings were given from 1 to 5, with a rating of 1 referring to having little experience and 5 being very experienced. No significant differ- ences in self-reported experience or skill with advanced avionics and automation were found between the owner-operators and the professional pilots. Autopilot Use During the Experimental Flight Participants turned on the simulator’s autopilot an average of 1 minute after take-off at a mean altitude of 901 ft MSL during leg 1 of the experimental flight (SD = 547 ft MSL). However, the participants fell within two distinct groups with regard to when they engaged the autopilot relative to their altitude on climb out. Nine of them turned it on at or below 854 ft MSL (M = 572 ft MSL, SD = 191 ft MSL, range: 305 to 854 ft MSL), and the other four engaged it at or above 1,408 ft MSL (M = 1,642 ft MSL, SD = 206 ft MSL, range: 1,408 to 1,886 ft MSL). The seven owner-operators engaged the autopilot at a mean altitude of 648 ft MSL (SD = 373 ft MSL), and the six professional pilots engaged it at a mean altitude of 1,195 ft MSL (SD = 599 ft MSL). Thus, in leg 1 most of the owner-operators were in the group of participants who initially engaged the au- topilot earlier (at lower altitudes), and most of the professional pilots were among the group who initially engaged the autopilot later at higher altitudes. Additionally, the altitudes at which the owner-operators engaged the autopilot were more similar (i.e., smaller range of altitudes) than those altitudes at which profes- sional pilots engaged the autopilot. Table 5. Personal Experience with Advanced Avionics and Automation. Questions Assessed Mean Median SD Overall Experience using different types of advanced avionics/ glass cockpits Owner-operators Professional pilots 3.07 3.28 2.83 3.00 4.00 2.50 1.55 1.38 1.83 Experience using the G1000 in the Citation Mustang or any other aircraft Owner-operators Professional pilots 4.00 4.14 3.83 4.00 4.00 3.50 1.08 .90 1.32 Skill level using the G1000 in the Citation Mustang or any other aircraft Owner-operators Professional pilots 4.08 4.28 3.83 4.00 4.00 4.00 0.95 1.17 .75 Experience using the G430/G50 or other similar Garmin IFR avionics systems Owner-operators Professional pilots 3.77 3.29 4.33 4.00 3.00 5.00 1.42 1.50 1.21 Experience using the other types of advanced avionics (e.g. Avidyne, Chelton, etc.) Owner-operators Professional pilots 2.83 2.57 3.20 3.00 3.00 4.00 1.69 1.51 2.04 Experience with using the FMS Owner-operators Professional pilots 2.69 2.71 2.67 2.00 3.00 2.00 1.60 1.79 1.50 Experience using stand-alone autopilot/auto flight systems Owner-operators Professional pilots 3.85 3.57 4.16 4.00 4.00 4.50 1.40 1.72 .98 1 = little experience/skill, 5 = very experienced/skilled Table 5. Personal Experience With Advanced Avionics and Automation. 22 In leg 2 of the experimental flight, participants again turned on the simulator’s autopilot an average of 1 minute after take- off but at a mean altitude of 1,148 ft MSL (SD = 584 ft MSL, range 335 – 2,014 ft MSL). Unlike the first leg, the altitudes chosen for engaging the autopilot were fairly evenly distributed throughout the range and owner-operators and professional pilots were, likewise, fairly evenly represented at all altitude levels in the range (low, medium, high) with regard to when the autopilot was engaged. During leg 1, the participants had the autopilot engaged for an average of 94.4% of the time during their flights (SD = 2.7%) from take-off to landing, with the owner-operators using the autopilot slightly more (M = 95.4%, SD = 2.1%) than the professional pilots (M = 93.3%, SD = 3.0%). The flights, from take-off to landing, lasted an average of 50.2 minutes (SD = 3.81 minutes) with the average length of the flights flown by the owner-operators and the professional pilots being almost exactly the same. During leg 2, the participants had the autopilot engaged for an average of 94.9% of the time during their flights (SD = 1.6%) from take-off to landing, again with the owner-operators using the autopilot slightly more (M = 95.7%, SD = 0.5%) than the professional pilots (M = 93.7%, SD = 1.9%). From take-off to landing, the leg 2 flights lasted an average of 57.11 minutes (SD = 4.93 minutes). When the two pilots who did not complete a missed approach procedure at KHSP are removed, the average length of the leg 2 flights rises to 58.92 minutes (SD = 2.82 min- utes), with the professional pilots (M = 57.58 minutes, SD = 1.43 minutes) generally completing the leg only slightly faster than the owner-operators (M = 59.81 minutes, SD = 3.28 minutes). Analysis of Workload and Task Management of Four En Route Events Due to time and resource limitations, we focused our analyses on four events in the two experimental flights that were specifi- cally scripted to involve high pilot workload. In the first leg from KTEB to KMTN, the two events subjected to detailed scrutiny were 1) the instruction from ATC to intercept the 208o Broad- way (BWZ) radial following the completion of the departure procedure out of KTEB and 2) programming a reroute while at cruise and meeting a waypoint crossing restriction on the initial descent from cruise. In the second leg from KMTN to KHSP, we focused our analyses on 3) the completion of an expedited descent to accommodate another aircraft with an emergency, and 4) task completion and preparation for the approach into KHSP while facilitating communication from a lost pilot who was flying too low for ATC to hear. Below are the findings of the analyses associated with these four events, individually, as well as a review of some overall findings across the two experimental legs. Due to the very small number of participants in our study, we were unable to generate sufficient statistical power. Therefore, our analyses were susceptible to type II errors, which are defined as accepting the null hypothesis when it is in fact false—meaning that significant differences between groups may not have been detected. Additionally, due to the small number of participants, the statistically significant differ- ences found among our participants, reported below, illustrate true differences among the study participants (i.e., our sample). However, caution should be exercised when generalizing our findings to other pilots who were not participants in this study (i.e., the population of single pilots flying VLJs/ELJs as a whole). In describing our findings, for the most part, we only report differences observed in the performance of owner-operators and professional pilots if the differences were statistically significant or, in the case of frequency data, appeared to be relatively large. Event 1: Interception of the Broadway (BWZ) Radial Upon completing the TEB6 departure off runway 24 at KTEB, the aircraft should have been on a heading of 280o and level at 2000 ft MSL. In our scenario, the participants were then told to continue to fly at 2000 ft MSL to accommodate crossing traffic descending into LaGuardia International Airport. At 15 nm DME from TEB, ATC told them to “fly heading 270o to intercept the Broadway, Bravo, Whiskey, Zulu, 208o radial to BIGGY, then as filed.” After reading the clearance back correctly, the participants were also given the instructions to “Climb and maintain 6000, contact New York Departure on 132.80.” Thus, in addition to looking for the crossing traffic headed to LaGuardia, there were four main tasks that had to be ac- complished: a heading change, intercepting a radial off a VOR, a climb to a new altitude, and a change in radio frequency and requirement to check in with a new controller. The participants had to remember each of these tasks with their associated num- bers (heading, radial, altitude, frequency) and consider how to accomplish them and in what order. Three of the tasks (change in heading, altitude, and frequency) are commonly performed during IFR flight, and each can be accomplished fairly quickly by proficient pilots. Therefore, we thought it likely that the subtasks required for each would be completed in their entirety before moving on to those associated with a new task, rather than interleaving them across the three tasks. For example, we expected that a pilot would verify the radio in use and then switch to another task, such as dialing in a new heading, before going back to the original task and dialing in the new radio frequency. However, one exception to our expectation that these three tasks would be performed sequentially, rather than interleaved, was that we thought some pilots, after having changed to the new radio frequency, might choose to complete other tasks, such as dialing in the new altitude and initiating the climb to 6000 ft, prior to checking in with the new departure controller. The fourth task in this clearance, intercepting the BWZ radial, is quite different from the other three tasks with regard to its cognitive and temporal demands. There are a number of ways to accomplish a radial intercept using the G1000, although none of them is as simple as pressing a button or two or locating the option in a dropdown menu. As a consequence, the participants had to consider how to use the automation, if at all, to complete an unexpected task, which is relatively uncommon. 23 The three most likely strategies pilots were expected to employ to accomplish this task using the G1000 are presented in Table 6. In the first strategy, OBS function in the G1000 is used in conjunction with the selection of the BIGGY waypoint and the desired arrival course. In the second strategy, the pilot alters the flight plan by entering the BWZ VOR prior to BIGGY, thereby creating a flight plan leg to