Simulator Evaluation of Runway Incursion Prevention Technology for General Aviation Operations

Appendix A: Rare Event Test Matrix

Evaluation Pilot EPn (n=1 to 16) RIPS Scenario Scenario 1 – arrival / takeoff hold (see Figure 8) Meteorological VMC1 = 3 nm, day, 1000 ft ceiling VMC2 = 3 nm, day, 2000 ft ceiling IMC1 = 1 nm, day, 1000 ft ceiling IMC2 = 1 nm, day, 400 ft ceiling IMC3 = 2 nm, day, 1000 ft ceiling Altitude (ft) 4412 4420 5420 7100+ (0 AGL) (8 AGL) (1008 AGL) (2688 AGL) Wind Speed (kt) 10 10 20 30 Wind Direction (deg) 260 260 260 360 Note: gradual shift of wind speed and direction Display Concept BMO = Baseline + surface map and ownship BMOT = Baseline + surface map and ownship and traffic BAMO = BMO + audible incursion alerts BAMOT = BMOT + incursion alerts (audible & visual) Note: IC = Initial Condition Approach S34RL = Straight-in approach to 34R and land. IC is 3nm out, 1010 ft AGL, 90 kt S34RW = Straight-in approach to 34R then waveoff at 200 ft AGL. IC is 3 nm out, 1010 ft AGL, 90 kt S34LSS= Straight-in approach, lined up with 34L, sidestep to 34R and land. IC is 3nm out, 1010 ft AGL, 90 kt S25L = Straight-in approach to 25 and land. IC is 3 nm out, 1800 ft AGL C-long = Non-precision approach – initial approach to 34R from base leg ending in Circling approach to 25 and land. IC is 9.56 nm out from 34R on left base (85 degree heading & 1.6 nm left offset), 2737.5 ft AGL S-long = ILS approach – approach to 34R from base leg, follow approach path and go around at 200 ft. IC is 9.56 nm from 34R on left base (85 degree heading & 1.6 nm left offset), 2737.5 ft AGL Rare Event Test Matrix EP Case No. RIPS Meteoro- Alert Display Approach Scenario logical Source Concept 1,5,9,13 1 None VMC1 N/A BMO S34RL 2 Scenario 1 VMC1 N/A BMO S34RL 3 None IMC2 N/A BMO S34RL 4 None IMC2 N/A BMO S34RL 5 None VMC1 N/A BMO S34RW 6 None VMC1 N/A BMO S34RW 7 None IMC2 N/A BMO S34RW 8 None IMC2 N/A BMO S34RW 9 None VMC1 N/A BMO S34LSS 10 None VMC1 N/A BMO S34LSS 11 None VMC1 N/A BMO S34LSS 12 None VMC1 N/A BMO S34LSS 13 None VMC2 N/A BMO S25L 14 None VMC2 N/A BMO S25L 15 None VMC2 N/A BMO S25L 16 None VMC2 N/A BMO S25L 17 None IMC3 N/A BMO C-long 18 None IMC3 N/A BMO C-long 19 None IMC2 N/A BMO S-long 2,6,10,14 20 None VMC1 N/A BMOT S34RL 21 Scenario 1 VMC1 N/A BMOT S34RL 22 None IMC2 N/A BMOT S34RL 23 None IMC2 N/A BMOT S34RL 24 None VMC1 N/A BMOT S34RW 25 None VMC1 N/A BMOT S34RW 26 None IMC2 N/A BMOT S34RW 27 None IMC2 N/A BMOT S34RW 28 None VMC1 N/A BMOT S34LSS 29 None VMC1 N/A BMOT S34LSS 30 None VMC1 N/A BMOT S34LSS 31 None VMC1 N/A BMOT S34LSS 32 None VMC2 N/A BMOT S25L 33 None VMC2 N/A BMOT S25L 34 None VMC2 N/A BMOT S25L 35 None VMC2 N/A BMOT S25L 36 None IMC3 N/A BMOT C-long 37 None IMC3 N/A BMOT C-long 38 None IMC2 N/A BMOT S-long 3,7,11,15 39 None VMC1 N/A BAMO S34RL 40 Scenario 1 VMC1 RSM BAMO S34RL 41 None IMC2 N/A BAMO S34RL 42 None IMC2 N/A BAMO S34RL 43 None VMC1 N/A BAMO S34RW 44 None VMC1 N/A BAMO S34RW 45 None IMC2 N/A BAMO S34RW 46 None IMC2 N/A BAMO S34RW 47 None VMC1 N/A BAMO S34LSS 48 None VMC1 N/A BAMO S34LSS 49 None VMC1 N/A BAMO S34LSS 50 None VMC1 N/A BAMO S34LSS 51 None VMC2 N/A BAMO S25L 52 None VMC2 N/A BAMO S25L 53 None VMC2 N/A BAMO S25L 54 None VMC2 N/A BAMO S25L 55 None IMC3 N/A BAMO C-long 56 None IMC3 N/A BAMO C-long 57 None IMC2 N/A BAMO S-long 4,8,12,16 58 None VMC1 N/A BAMOT S34RL 59 Scenario 1 VMC1 RSM BAMOT S34RL 60 None IMC2 N/A BAMOT S34RL 61 None IMC2 N/A BAMOT S34RL 62 None VMC1 N/A BAMOT S34RW 63 None VMC1 N/A BAMOT S34RW 64 None IMC2 N/A BAMOT S34RW 65 None IMC2 N/A BAMOT S34RW 66 None VMC1 N/A BAMOT S34LSS 67 None VMC1 N/A BAMOT S34LSS 68 None VMC1 N/A BAMOT S34LSS 69 None VMC1 N/A BAMOT S34LSS 70 None VMC2 N/A BAMOT S25L 71 None VMC2 N/A BAMOT S25L 72 None VMC2 N/A BAMOT S25L 73 None VMC2 N/A BAMOT S25L 74 None IMC3 N/A BAMOT C-long 75 None IMC3 N/A BAMOT C-long 76 None IMC2 N/A BAMOT S-long

Appendix B: Usability Study Test Matrix

RIPS Scenario 1 = Arrival / takeoff hold 2 = Departure / intersection departure 3 = Arrival / departure 4 = Departure / departure 5 = Taxi crossing / departure Display Concept B = Baseline round dials BMO = B + surface map and ownship BMOT = B + surface map and ownship and traffic BA = B + audible incursion alerts BAMO = BMO + audible incursion alerts BAMOT = BMOT + incursion alerts (audible & visual) BRIPS = B + perspective surface map All runs conducted under VMC1 (3nm, day, 1000 ft ceiling, no wind) Table B.1: Display Concept Evaluation for Runway Incursion Prevention Case No. RIPS Alert Display Scenario Source Concept 77 1 N/A B 78 1 N/A BMO 79 1 N/A BMOT 80 1 RSM BA 81 1 RSM BAMO 82 1 RSM BAMOT 83 1 RSM BRIPS 84 2 N/A B 85 2 N/A BMO 86 2 N/A BMOT 87 2 RSM BA 88 2 RSM BAMO 89 2 RSM BAMOT 90 2 RSM BRIPS 91 3 N/A B 92 3 N/A BMO 93 3 N/A BMOT 94 3 RSM BA 95 3 RSM BAMO 96 3 RSM BAMOT 97 3 RSM BRIPS 98 4 N/A B 99 4 N/A BMO 100 4 N/A BMOT 101 4 RSM BA 102 4 RSM BAMO 103 4 RSM BAMOT 104 4 RSM BRIPS 105 5 N/A B 106 5 N/A BMO 107 5 N/A BMOT 108 5 RSM BA 109 5 RSM BAMO 110 5 RSM BAMOT 111 5 RSM BRIPS Table B.2: RSM and PathProx™ Incursion Detection Algorithm Evaluation Display Case No. RIPS Alert Source Concept Scenario 148 1 RSM BRIPS 150 1 PathProx™ BRIPS 152 2 RSM BRIPS 154 2 PathProx™ BRIPS 156 3 RSM BRIPS 158 3 PathProx™ BRIPS 160 4 RSM BRIPS 162 4 PathProx™ BRIPS 164 5 RSM BRIPS 166 5 PathProx™ BRIPS

Appendix C: Rare Event Case Lists

Tables C.1 and C.2 lists the experimental case order for each evaluation pilot for the rare event portion of the testing. Refer to Appendix A for a detailed description of each case.

Table C.1. Rare Event Case Order for Evaluation Pilots 1 through 8 EP1 EP2 EP3 EP4 EP5 EP6 EP7 EP8 Run 1 17 36 55 74 17 36 55 74 Run 2 16 33 49 65 15 31 49 60 Run 3 10 25 47 62 11 24 51 64 Run 4 6 30 48 61 7 33 44 68 Run 5 11 23 54 73 4 28 48 71 Run 6 13 35 46 71 14 23 41 62 Run 7 7 34 43 64 13 32 50 66 Run 8 8 27 45 69 1 22 39 58 Run 9 12 26 42 67 10 35 47 61 Run 10 15 29 52 60 5 25 45 70 Run 11 19 38 57 76 19 38 57 76 Run 12 3 31 44 66 16 26 46 73 Run 13 1 22 51 70 12 20 54 65 Run 14 9 24 53 72 6 29 43 72 Run 15 14 28 39 68 8 34 52 69 Run 16 4 20 41 58 3 27 42 63 Run 17 5 32 50 63 9 30 53 67 Run 18 18 37 56 75 18 37 56 75 Run 19 2 21 40 59 2 21 40 59 Table C.2. Rare Event Case List for Evaluation Pilots 9 through 16 EP9 EP10 EP11 EP12 EP13 EP14 EP15 EP16 Run 1 17 36 55 74 17 36 55 74 Run 2 3 28 44 68 12 34 54 67 Run 3 11 32 53 71 8 27 47 60 Run 4 1 35 39 69 3 23 41 63 Run 5 13 23 42 72 1 25 46 64 Run 6 16 31 47 58 14 33 52 70 Run 7 8 33 49 62 5 32 43 68 Run 8 12 30 50 63 4 22 42 69 Run 9 14 29 51 65 6 35 39 61 Run 10 9 34 48 73 13 29 50 65 Run 11 19 38 41 76 19 38 57 76 Run 12 6 20 57 66 15 31 49 58 Run 13 10 26 54 61 7 28 44 71 Run 14 15 24 43 60 9 24 51 73 Run 15 5 27 45 64 16 20 45 66 Run 16 7 22 52 67 10 26 48 62 Run 17 4 25 46 70 11 30 53 72 Run 18 18 37 56 75 18 37 56 75 Run 19 2 21 40 59 2 21 40 59

Appendix D: Usability Study Case Lists

Tables D.1 through D.4 lists the experimental case order for each evaluation pilot for the usability study portion of the testing. Refer to Appendix B for a detailed description of each case. In Tables D1 and D2, the cases shown with ‘skip’ were not conducted.

Part 1: Display Concept Evaluation Table D.1. Case List for Display Concept Evaluation for Pilots 1 through 8 EP1 EP2 EP3 EP4 EP5 EP6 EP7 EP8 77 77 84 84 91 91 98 98 78 83 90 85 92 97 104 99 79 82 89 86 93 96 103 100 80 81 88 87 94 95 102 101 81 80 87 88 95 94 101 102 82 79 86 89 96 93 100 103 83 78 85 90 97 92 99 105 84 84 91 91 98 98 105 105 90 85-skip 92-skip 97 104 99-skip 106-skip 111 89 86 93 96 103 100 107 110 88 87 94 95-skip 102 101 108 109-skip 87 88-skip 95-skip 94 101 102-skip 109-skip 108 86 89 96 93 100 103 110 107 85 90 97 92-skip 99-skip 104 111 106-skip 91 91 98 98 105 105 77 77 92 97 104 99-skip 106-skip 111 83 78-skip 93 96 103 100 107 110 82 79 94 95-skip 102-skip 101 108 109-skip 81-skip 80 95 94 101 102-skip 109-skip 108 80 81-skip 96 93 100 103 110 107 79 82 97 92-skip 99-skip 104 111 106-skip 78-skip 83 98 98 105 105 77 77 84 84 104 99-skip 106-skip 111 83 78-skip 85-skip 90 103 100 107 110 82 79 86 89 102 101 108 109-skip 81-skip 80 87 88-skip 101 102-skip 109-skip 108 80 81-skip 88-skip 87 100 103 110 107 79 82 89 86 99 104 111 106-skip 78-skip 83 90 85-skip 105 105 77 77 84 84 91 91 106 111 83 78-skip 85-skip 90 97 92-skip 107 110 82 79 86 89 96 93 108 109-skip 81-skip 80 87 88-skip 95-skip 94 109 108 80 81-skip 88-skip 87 94 95-skip 110 107 79 82 89 86 93 96 111 106-skip 78-skip 83 90 85-skip 92-skip 97 Table D.2. Case List for Display Concept Evaluation for Pilots 9 through 16 EP9 EP10 EP11 EP12 EP13 EP14 EP15 EP16 105 105 77 77 84 84 91 91 106 111 78 83 85 90 92 97 107 110 79 82 86 89 93 96 108 109 80 81 87 88 94 95 109 108 81 80 88 87 95 94 110 107 82 79 89 86 96 93 111 106 83 78 90 85 97 92 77 77 91 91 98 98 105 105 83 78-skip 97 92-skip 104 99-skip 111 106-skip 82 79 96 93 103 100 110 107 81 80 95-skip 94 102-skip 101 109-skip 108 80 81-skip 94 95-skip 101 102-skip 108 109-skip 79 82 93 96 100 103 107 110 78-skip 83 92-skip 97 99-skip 104 106-skip 111 84 84 105 105 77 77 84 84 85-skip 90 106-skip 111 78-skip 83 85-skip 90 86 89 107 110 79 82 86 89 87 88-skip 108 109-skip 80 81-skip 87 88-skip 88-skip 87 109-skip 108 81-skip 80 88-skip 87 89 86 110 107 82 79 89 86 90 85-skip 111 106-skip 83 78-skip 90 85-skip 91 91 84 84 91 91 98 98 97 92-skip 90 85-skip 97 92-skip 104 99-skip 96 93 89 86 96 93 103 100 95-skip 94 88-skip 87 95-skip 94 102-skip 101 94 95-skip 87 88-skip 94 95-skip 101 102-skip 93 96 86 89 93 96 100 103 92-skip 97 85-skip 90 92-skip 97 99-skip 104 99 98 98 98 105 105 77 77 99-skip 104 99-skip 104 106-skip 111 78-skip 83 100 103 100 103 107 110 79 82 101 102-skip 101 102-skip 108 109-skip 80 81-skip 102-skip 101 102-skip 101 109-skip 108 81-skip 80 103 100 103 100 110 107 82 79 104 99-skip 104 99-skip 111 106-skip 83 78-skip

Part 2: Incursion Detection Algorithm Evaluation Table D.3. Case List for Incursion Detection Algorithm Evaluation for Pilots 1 through 8 EP1 EP2 EP3 EP4 EP5 EP6 EP7 EP8 148 150 148 150 152 154 152 154 150 148 150 148 154 152 154 152 154 152 154 152 162 160 162 160 152 154 152 154 160 162 160 162 156 158 156 158 148 150 148 150 158 156 158 156 150 148 150 148 162 160 162 160 158 156 158 156 160 162 160 162 156 158 156 158 164 166 164 166 164 166 164 166 166 164 166 164 166 164 166 164 Table D.4. Case List for Incursion Detection Algorithm Evaluation for Pilots 9 through 16 EP9 EP10 EP11 EP12 EP13 EP14 EP15 EP16 164 166 164 166 148 150 148 150 166 164 166 164 150 148 150 148 162 160 162 160 158 156 158 156 160 162 160 162 156 158 156 158 156 158 156 158 164 166 164 166 158 156 158 156 166 164 166 164 154 152 154 152 154 152 154 152 152 154 152 154 152 154 152 154 148 150 148 150 160 162 160 162 150 148 150 148 162 160 162 160

Appendix E: Post Run Questionnaires

The evaluation pilot completed a questionnaire (see below) at the end of each evaluation run. Mental Demand, Physical Demand, Temporal Demand, Performance (Pilot), Effort, and Frustration were all Task Load Index (TLX) measures that evaluated workload. Demand on and Supply of Attentional Resources and Understanding of Situation values were combined to derive the Situation Awareness Rating Technique (SART) rating. Level of Terrain Awareness and Stress were two other independent measures that were collected.

MENTAL DEMAND DEMAND ON ATTENTIONAL RESOURCES Low High Low High PHYSICAL DEMAND SUPPLY OF ATTENTIONAL RESOURCES Low High Low High TEMPORAL DEMAND UNDERSTANDING OF THE SITUATION Low High Low High PERFORMANCE LEVEL OF TERRAIN AWARENESS Good Poor Low High EFFORT STRESS Low High Low High FRUSTRATION Low High Workload: Situational Awareness: TLX - Mental Demand SART - Demand on Attentional Resources Physical Demand Supply of Attentional Resources Temporal Demand Understanding of the Situation Performance Effort Frustration Run Questionnaire Key: Title Descriptions MENTAL DEMAND How much mental and perceptual activity was required (e.g., thinking, deciding, calculating, remembering, looking, searching, etc.)? Was the task easy or demanding, simple or complex, exacting or forgiving?

PHYSICAL DEMAND How much physical activity was required (e.g., pushing, pulling, turning, controlling, activating, etc.)? Was the task easy or demanding, slow or brisk, slack or strenuous, restful or laborious?

TEMPORAL DEMAND How much time pressure did you feel due to the rate or pace at which the tasks or task elements occurred? Was the pace slow and leisurely or rapid and frantic?

PERFORMANCE How successful do you think you were in accomplishing the goals of the task set by the experimenter (or yourself)? How satisfied were you with your performance in accomplishing these goals?

EFFORT How hard did you have to work (mentally and physically) to accomplish your level of performance?

FRUSTRATION LEVEL How insecure, discouraged, irritated, stressed and annoyed versus secure, gratified, content, relaxed and complacent did you feel during the task?

Please rate your overall impression of the scenario in terms of how much attention and effort was required to perform the scenario successfully. Things to consider are the degree of instability, complexity, and variability that you perceived while flying the scenario.

DEMAND ON ATTENTIONAL RESOURCES Low High Please rate your overall impression of the scenario in terms of the amount of “spare” attention to give to other tasks. Was 100% of your attention directed towards successfully completing the scenario? Or, could you have completed other sub-tasks while flying the scenario? Things to consider include your level of arousal, level of concentration, and if you attention was divided across many sub-tasks.

SUPPLY OF ATTENTIONAL RESOURCES Low High Please rate your overall understanding of what was happening with the aircraft. Mark on the line below the degree to which you felt confident that you were aware of the elements in your environment. Things to consider include the level of information quantity and quality as well as familiarity that you felt you had with what was taking place during the scenario.

UNDERSTANDING OF THE SITUATION Low High Please rate you overall understanding of the terrain environment you were operating within. Things to consider for this response are how comfortable were you with your terrain awareness.

LEVEL OF TERRAIN AWARNESS Low High Please rate your overall level of stress that you experienced while completing the experimental test run.

STRESS 1. Please rate your level of traffic awareness experienced with the display concept during the approach.

1 = Low; 10 = High 2. Please rate your level of awareness of where you were (ownship position awareness) using the display concept during the approach. 1 = Low; 10 = High 3. Please rate your ability to stay on path (flight path awareness) using the display concept during the approach. 1 = Low; 10 = High

Appendix F. Rare Event Run Questionnaire Results

The following questionnaire was administered to the EP at the conclusion of the rare event runway incursion run. Four different display conditions were used for the rare event testing as follows: EP # Display Condition 1, 5, 9, 13 BMO 2, 6, 10, 14 BMOT 3, 7, 11, 15 BAMO 4, 8, 12, 16 BAMOT Only half of the EPs received incursion alerts for the rare event test run.

1. If applicable (BAMO and BAMOT only), please rate the timeliness of the alert (did the alert happen in a timely manner) in terms of providing enough time to take evasive action.

1 = Poor; 10 = Excellent EP BAMO EP BAMOT 3 8 4 7 7 5 8 8 11 9 12 10 15 9 16 9 Mean 7.75 Mean 8.5 Std. Dev. 1.89 Std. Dev. 1.29 2. Upon receiving an incursion alert, was your immediate reaction to : Confirm the hazard; Take evasive action; Other: Explain Reason EP # Confirm Hazard Take Evasive Action Other Reason 1 N/A 2 N/A 3 Would have queried ATC with warning  4  Already confirmed traffic prior to alert 5 N/A 6 N/A 7 Need radio response. Initiate each, can   continue if valid or realign if not.

 9 N/A 10 N/A 11  12 Unfamiliar with warning  13 N/A 14 N/A 15 Saw first – alert confirmed visual   Total 5 4 3. Please indicate where you first perceived the location of the incursion aircraft.

Window; Surface Map (if applicable); Audible Alert (if applicable) If applicable, upon seeing the incursion aircraft, where did you then focus your attention?

Window; Surface Map (if applicable); Audible Alert (if applicable) If applicable, please estimate how useful each of the following were in helping you to detect and avoid the runway incursion situation (total 100%) Window; Surface Map (if applicable); Audible Alert (if applicable) First Perceived Location Focus Attention Detection Usefulness (%) EP # Window Map Audible Window Map Audible Window Map Audible 1 N/A N/A N/A 100 0 N/A   2 N/A N/A 100 0 N/A   3 N/A 10 0 90   4 40 60 0   5 N/A N/A N/A 100 0 N/A   6 N/A N/A 80 20 N/A   7 N/A 95 0 5   8 50 50 0   9 N/A N/A N/A 100 0 N/A   10 - - N/A - - N/A - - N/A 11 N/A 100 0 0   12 0 40 60   13 N/A N/A N/A 100 0 N/A   14 N/A - - N/A - - N/A  15 N/A 80 0 20   16 33 33 34   Total 12 1 2 10 3 1 70.6 14.5 26.1 4. If applicable, how did you use the surface map to detect and locate the incursion aircraft?

EP 1 – N/A EP 2 – N/A EP 3 – N/A EP 4 – Saw traffic location on the map and then confirmed the traffic by looking out the window. Looked at map afterwards for additional confirmation.

EP 5 – N/A EP 6 – N/A EP 7 – N/A EP 8 – I was looking at the display to confirm my visual out-the-window look when it lit up red.

EP 9 – N/A EP 10 – N/A EP 11 – N/A EP 12 – Upon audible, I confirmed on map then visually.

EP 13 – None EP 14 – No response EP 15 – Out of field of view.

EP 16 – Confirm visual.

If applicable, please indicate whether the surface map alone was sufficient to help you detect and locate the intrusion aircraft? Yes; No; Likelihood of detection (%) EP # Yes No Likelihood of Detection (%)  2 0  3 N/A 4 75  5 N/A 6 N/A 7 N/A 8 100      13 N/A    Total 1 10 What enhancements could be made to the surface map to raise the likelihood?

EP 2 – More attention getting symbol, location of display is too far from site picture EP 4 – Highlight traffic before alert generated. A longer warning on map with audible warning meaning imminent danger.

EP 6 – Location EP 7 – Traffic of hazard to this aircraft, should have bright/flash indication of hazard with warning EP 11 – Adding traffic EP 12 – Positioning EP 14 – Placement EP 15 – Place within field of view.

EP 16 – Too cluttered.

5. If applicable, please indicate whether the audible alert alone was sufficient to help you detect and locate the intrusion aircraft? Yes; No; Likelihood (%) EP # Yes No Likelihood of Detection (%) 1 N/A 2 N/A 3 90  4 No answer 5 N/A 6 N/A 7 25   9 N/A 10 N/A 11 100   13 N/A 14 N/A   Total 5 2 What enhancements could be made to the audible alert (e.g. 3-D Audio) to raise the likelihood?

EP 3 – Position EP 7 – Need clear beep/squawk to alert about the alert. Plus voice distinct from ATC.

EP 11 – Audio is OK EP 12 – None EP 15 – Tone EP 16 – Female with British accent.

6. Please indicate your rating of how much aid the display concept provided in detection of the incursion aircraft. 1 = None; 10 = Exceptional EP BMO EP BMOT EP BAMO EP BAMOT 1 1 2 1 3 8 4 8 5 1 6 4 7 2 8 10 9 1 10 1 11 8 12 9 13 1 14 2 15 7 16 8 Mean 1 Mean 2 Mean 6.25 Mean 8.75 Std. Dev. 0 Std. Dev. 1.41 Std. Dev. 2.87 Std. Dev. 0.96 Collective EP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 1 8 8 1 4 2 10 1 1 8 9 1 2 7 8 Mean 4.5 Std. Dev. 3.58 7. Please indicate the likelihood that the display concept would prevent similar runway incursions in the real world (based on your experiences). 1 = Low; 10 = High EP BMO EP BMOT EP BAMO EP BAMOT 1 1 2 10 3 9 4 9 5 1 6 9 7 4 8 10 9 5 10 8 11 8 12 10 13 1 14 5 15 10 16 10 Mean 2 Mean 8 Mean 7.75 Mean 9.75 Std. Dev. 2 Std. Dev. 2.16 Std. Dev. 2.63 Std. Dev. 0.5 Collective EP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 10 9 9 1 9 4 10 5 8 8 10 1 5 10 10 Mean 6.88 Std. Dev. 3.5 8. Please rate your level of perceived safety you believe you would experience if you had this system onboard your aircraft during a similar runway incursion event. 1 = Low; 10 = High EP BMO EP BMOT EP BAMO EP BAMOT 1 1 2 10 3 7 4 8 5 1 6 9 7 7 8 10 9 5 10 7 11 6 12 10 13 5 14 5 15 10 16 9 Mean 3 Mean 7.75 Mean 7.5 Mean 9.25 Std. Dev. 2.31 Std. Dev. 2.22 Std. Dev. 1.73 Std. Dev. 0.96 Collective EP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 10 7 8 1 9 7 10 5 7 6 10 5 5 10 9 Mean 6.88 Std. Dev. 2.94 9. Please rate (if applicable) the effectiveness of the display concept compared to what is currently available onboard Part 23 aircraft for prevention of runway incursions. 1 = None; 10 = Exceptional EP BMO EP BMOT EP BAMO EP BAMOT 1 1 2 10 3 9 4 10 5 1 6 10 7 4 8 10 9 1 10 7 11 8 12 10 13 5 14 5 15 8 16 9 Mean 2 Mean 8 Mean 7.25 Mean 9.75 Std. Dev. 2 Std. Dev. 2.5 Std. Dev. 2.22 Std. Dev. 0.5 Collective EP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 10 9 10 1 10 4 10 1 7 8 10 5 5 8 9 Mean 6.75 Std. Dev. 3.45 Please indicate the reason for your rating.

EP 1 – The display concept did not show aircraft on it.

EP 2 – No airport display / map in current aircraft.

EP 3 – One more piece of information.

EP 4 – No current indications of traffic available. This display concept gives exceptional traffic information.

EP 5 – System is not indicating traffic.

EP 6 – Like having co-pilot and second pair of eyes.

EP 7 – Definite potential for improvement.

EP 10 – Enhances traffic awareness if used the technology & knew how to use it & part of scan EP 11 – Less subjective type of information EP 12 – Only visual on Part 23 in low visibility conditions. Audible and surface map make a big difference.

EP 13 – No display of traffic.

EP 15 – Focus attention where many tasks may be involved EP 16 – Gives more information than just visual – gets pilot’s attention.

10. Please provide any other comments or suggestions for improvement of the efficacy of the display concept for prevention of runway incursions.

EP 1 – Needs to show aircraft on it.

EP 2 – Ability to use software to make numbers, dials, letters, symbols bigger/smaller, brighter/dimmer.

EP 6 – Automatic range scaling.

EP 8 – Because of current vs prior experience.

EP 10 – On instrument approaches would rely more on ATC.

Appendix G: Usability Study Display Evaluation Questionnaire Results

The EPs evaluated the seven display conditions after each test scenario. The acronyms for the display conditions are defined as follows: B Baseline BMO Baseline with Surface Map and Ownship BMOT Baseline with Surface Map and Ownship and Traffic BA Baseline with Audible Incursion Alerts BAMO Baseline with Surface Map and Ownship and Audible Incursion Alerts BAMOT Baseline with Surface Map and Ownship, Traffic, and Incursion Alerts BRIPS Baseline with Perspective Surface Map 1. Please rate the likelihood that the display configuration would prevent the runway incursion scenario evaluated. 1 = low; 10 = high Scenario 1: arrival / takeoff hold EP # B BMO BMOT BA BAMO BAMOT BRIPS 1 6 6 8 7 7 10 9 2 1 1 5 5 5 8 8 3 1 1 4 6 6 7 8 4 3 3 4 7 7 10 10 5 1 1 8 10 10 10 10 6 0 0 3 8 8 10 8 7 1 4 5 4 5 8 9 8 1 1 4 10 10 10 10 9 1 1 6 7 7 9 9 10 1 1 5 7 7 8 8 11 1 1 5 8 8 10 10 12 1 1 5 8 8 10 10 13 1 2 5 4 6 8 9 14 1 1 2 7 7 9 8 15 1 1 3 7 7 8 9 16 1 1 3 7 7 9 9 Mean 1.38 1.63 4.69 7 7.19 9 9 Std. Dev. 1.36 1.5 1.66 1.71 1.42 1.03 0.82 Scenario 2: departure / intersection departure EP # B BMO BMOT BA BAMO BAMOT BRIPS 1 6 6 8 7 7 10 10 2 1 2 6 6 6 8 9 3 1 1 3 6 6 8 8 4 2 2 7 7 7 10 10 5 1 1 3 9 9 9 10 6 0 0 3 8 8 10 8 7 1 3 4 4 5 9 8 8 1 1 4 10 10 10 10 9 1 3 6 7 8 9 9 10 1 1 5 6 6 9 9 11 1 1 6 8 8 10 10 12 1 1 5 10 10 10 10 13 1 2 5 4 7 9 8 14 1 2 2 7 7 9 8 15 1 1 2 7 7 7 8 16 1 2 3 7 7 9 9 Mean 1.31 1.81 4.5 7.06 7.38 9.13 9 Std. Dev. 1.3 1.38 1.79 1.73 1.41 0.89 0.89 Scenario 3: arrival / departure EP # B BMO BMOT BA BAMO BAMOT BRIPS 1 4 4 5 6 6 10 9 2 1 2 5 7 8 9 10 3 1 1 4 7 7 8 8 4 2 2 4 7 7 10 10 5 1 1 3 8 9 10 10 6 1 3 6 8 8 10 10 7 1 3 5 2 5 8 9 8 1 1 1 10 10 10 10 9 1 1 4 8 8 8 8 10 1 1 5 7 7 8 8 11 1 1 5 8 8 10 10 12 1 1 4 8 8 10 10 13 1 1 5 5 5 8 8 14 1 1 1 2 2 3 3 15 1 1 2 8 8 9 10 16 1 1 3 7 8 9 9 Mean 1.25 1.56 3.88 6.75 7.13 8.75 8.88 Std. Dev. 0.77 0.96 1.5 2.14 1.89 1.77 1.78 Scenario 4: departure / departure EP # B BMO BMOT BA BAMO BAMOT BRIPS 1 4 4 6 6 6 10 10 2 1 2 6 8 8 9 10 3 1 1 4 7 7 8 8 4 1 2 5 6 6 9 9 5 0 1 3 10 10 10 10 6 0 2 6 8 8 10 8 7 1 4 5 5 6 9 9 8 1 1 3 10 10 10 10 9 1 1 4 8 8 8 8 10 1 1 6 7 8 8 8 11 1 1 7 8 8 10 10 12 1 1 4 10 10 10 10 13 1 3 5 5 7 9 8 14 1 1 3 7 7 9 8 15 1 1 3 7 7 8 8 16 1 1 3 7 7 9 9 Mean 1.06 1.69 4.56 7.44 7.69 9.13 8.94 Std. Dev. 0.85 1.08 1.36 1.59 1.35 0.81 0.93 Scenario 5: taxi crossing / departure EP # B BMO BMOT BA BAMO BAMOT BRIPS 1 4 4 6 7 7 10 10 2 1 3 6 7 8 9 10 3 1 1 4 6 7 7 8 4 2 2 7 5 5 9 9 5 1 1 8 10 10 10 10 6 0 3 6 8 8 10 8 7 1 2 6 4 5 9 8 8 1 1 10 10 10 10 10 9 1 1 6 7 7 9 8 10 1 1 5 7 7 10 10 11 1 1 8 8 8 10 10 12 1 1 6 8 8 10 10 13 1 3 5 6 7 9 8 14 1 1 5 7 7 9 8 15 1 1 2 8 8 9 9 16 1 2 2 5 7 9 9 Mean 1.19 1.75 5.75 7.06 7.44 9.31 9.06 Std. Dev. 0.83 1.0 2.05 1.65 1.36 0.79 0.93 2. Please rate the efficacy of the display concept for runway incursion prevention with and without alerting for the following display concepts. 1 = low; 10 = high Scenario 1 – arrival / takeoff hold EP # B + no alert B + audible alert BMO + no alert BMO + audible alert BMO + visual alert BMO + audible & visual alert BMOT + no alert BMOT + audible alert BMOT + visual alert BMOT + audible & visual alert BRIPS + no alert BRIPS + audible alert BRIPS + visual alert BRIPS + audible & visual alert 1 5 5 5 6 6 6 6 8 9 10 7 7 8 9 2 1 6 1 5 4 6 5 6 5 8 5 5 5 8 3 1 5 1 6 4 4 4 7 6 7 4 8 7 8 4 3 7 3 7 4 8 8 9 8 10 8 9 8 10 5 1 10 1 10 8 10 8 10 8 10 8 10 8 10 6 0 8 0 8 3 8 3 8 8 10 3 8 8 8 7 1 4 4 5 5 7 5 7 6 8 6 8 7 9 8 1 10 1 10 4 10 4 10 4 10 4 10 4 10 9 1 7 1 7 6 7 6 9 6 9 6 9 7 9 10 1 7 1 7 6 7 5 7 6 8 5 7 6 8 11 1 8 1 8 6 8 5 8 6 10 5 8 6 10 12 1 8 1 8 8 10 5 10 8 10 5 10 8 10 13 1 4 2 6 3 7 5 7 6 8 5 6 6 9 14 1 7 1 7 1 7 2 7 2 9 2 7 2 8 15 1 7 1 7 4 7 3 8 6 8 4 9 7 9 16 1 7 1 7 5 7 3 9 7 9 3 9 7 9 Mean 1.31 6.94 1.56 7.13 4.81 7.44 4.81 8.13 6.31 9.0 5.0 8.13 6.5 9.0 Std. 1.14 1.73 1.31 1.45 1.83 1.59 1.68 1.26 1.74 1.03 1.71 1.45 1.67 0.82 Dev.

Scenario 2 – departure / intersection departure EP # B + no alert B + audible alert BMO + no alert BMO + audible alert BMO + visual alert BMO + audible & visual alert BMOT + no alert BMOT + audible alert BMOT + visual alert BMOT + audible & visual alert BRIPS + no alert BRIPS + audible alert BRIPS + visual alert BRIPS + audible & visual alert 1 6 7 6 7 7 7 8 9 9 10 8 9 9 10 2 1 6 2 6 5 5 6 6 5 8 6 8 7 9 3 1 6 1 6 4 7 3 6 4 8 4 6 4 8 4 2 7 2 7 4 8 7 9 8 10 7 9 8 10 5 1 10 1 10 8 10 7 10 8 10 8 10 9 10 6 0 8 0 8 3 8 3 8 6 10 3 8 6 8 7 1 4 3 5 4 6 4 8 7 9 5 7 6 8 8 1 10 1 10 1 10 4 10 4 10 4 10 4 10 9 1 7 3 8 4 8 6 8 6 9 5 9 6 9 10 1 6 1 6 5 6 5 6 5 9 5 6 5 9 11 1 8 1 8 7 9 6 10 9 10 7 10 9 10 12 1 10 1 10 8 10 1 10 8 10 1 10 8 10 13 1 4 2 7 5 7 5 7 7 9 4 7 7 8 14 1 7 2 7 2 7 2 7 2 9 2 7 2 8 15 1 7 1 7 2 6 2 7 3 7 3 7 3 8 16 1 7 2 7 5 8 3 9 7 9 3 9 7 9 Mean 1.31 7.13 1.81 7.44 4.63 7.63 4.5 8.13 6.13 9.19 4.69 8.25 6.25 9.0 Std. 1.3 1.82 1.38 1.5 2.09 1.54 2.07 1.5 2.13 0.91 2.09 1.44 2.18 0.89 Dev.

Scenario 3 – arrival / departure EP # B + no alert B + audible alert BMO + no alert BMO + audible alert BMO + visual alert BMO + audible & visual alert BMOT + no alert BMOT + audible alert BMOT + visual alert BMOT + audible & visual alert BRIPS + no alert BRIPS + audible alert BRIPS + visual alert BRIPS + audible & visual alert 1 5 6 5 6 6 6 8 9 9 10 8 9 9 10 2 1 7 2 8 7 8 5 8 7 9 6 7 7 10 3 1 7 1 7 4 7 4 7 6 8 3 7 4 8 4 2 7 2 7 3 8 4 10 4 10 4 10 4 10 5 1 8 1 9 2 9 3 10 2 10 3 10 3 10 6 1 8 3 8 6 10 6 8 6 10 6 8 6 10 7 1 2 3 5 4 6 5 6 7 8 5 8 7 9 8 1 10 1 10 1 10 1 10 1 10 1 10 1 10 9 1 8 1 8 4 8 4 8 4 8 4 8 4 8 10 1 7 1 7 6 8 5 7 7 8 5 7 7 8 11 1 8 1 8 6 9 5 9 7 10 6 9 7 10 12 1 8 1 8 4 10 4 8 8 10 8 10 8 10 13 1 5 1 5 4 7 5 7 6 8 5 7 7 8 14 1 2 1 2 2 2 1 3 2 3 1 3 2 3 15 1 8 1 8 3 8 2 9 4 9 2 9 4 10 16 1 7 1 8 4 9 3 7 4 9 3 7 4 9 Mean 1.31 6.75 1.63 7.13 4.13 7.81 4.06 7.88 5.25 8.75 4.38 8.06 5.25 8.94 Std. 1.01 2.14 1.15 1.89 1.71 2.01 1.81 1.78 2.32 1.77 2.16 1.81 2.29 1.81 Dev.

Scenario 4 – departure / departure EP # B + no alert B + audible alert BMO + no alert BMO + audible alert BMO + visual alert BMO + audible & visual alert BMOT + no alert BMOT + audible alert BMOT + visual alert BMOT + audible & visual alert BRIPS + no alert BRIPS + audible alert BRIPS + visual alert BRIPS + audible & visual alert 1 5 6 5 6 6 6 8 9 9 10 8 9 9 10 2 1 8 2 8 7 8 6 8 8 9 7 8 7 10 3 1 7 1 7 6 8 4 7 6 8 4 8 5 8 4 1 6 2 6 5 7 5 7 5 9 5 7 5 9 5 1 10 1 10 3 10 3 10 3 10 3 10 3 10 6 0 8 2 8 2 8 6 8 6 10 2 8 6 10 7 1 5 4 6 5 7 5 7 6 9 5 7 6 9 8 1 10 1 10 5 10 3 10 5 10 2 10 2 10 9 1 8 1 8 4 8 4 8 4 8 4 8 4 8 10 1 7 1 8 6 8 6 8 6 8 6 8 6 8 11 1 8 1 8 8 10 7 10 10 10 7 10 10 10 12 1 10 1 10 4 10 4 10 4 10 4 10 4 10 13 1 5 3 7 5 7 5 7 6 9 3 6 5 8 14 1 7 1 7 2 7 3 7 3 9 2 7 2 8 15 1 7 1 7 3 7 3 7 4 8 3 7 4 8 16 1 7 1 7 5 7 3 7 5 9 5 9 7 9 Mean 1.19 7.44 1.75 7.69 4.75 8 4.69 8.13 5.63 9.13 4.38 8.25 5.31 9.06 Std. 1.05 1.59 1.24 1.35 1.69 1.32 1.58 1.26 2.00 0.81 1.89 1.29 2.24 0.93 Dev.

Scenario 5 – taxi crossing / departure EP # B + no alert B + audible alert BMO + no alert BMO + audible alert BMO + visual alert BMO + audible & visual alert BMOT + no alert BMOT + audible alert BMOT + visual alert BMOT + audible & visual alert BRIPS + no alert BRIPS + audible alert BRIPS + visual alert BRIPS + audible & visual alert 1 4 6 4 6 5 6 6 8 7 10 6 8 7 10 2 1 7 3 8 7 8 6 8 7 9 6 8 8 10 3 1 6 1 5 4 7 4 7 6 7 5 7 6 8 4 2 5 2 5 5 6 7 9 7 9 6 9 6 9 5 1 10 1 10 9 10 8 10 9 10 8 10 9 10 6 0 8 3 8 6 8 6 8 6 10 8 8 6 8 7 1 4 2 5 4 6 6 8 7 9 5 7 6 8 8 1 10 1 10 5 10 10 10 10 10 10 10 10 10 9 1 7 1 7 5 7 6 9 6 9 6 8 6 8 10 1 7 1 7 4 8 5 7 6 10 5 7 6 10 11 1 8 1 8 9 9 8 10 9 10 8 10 9 10 12 1 8 1 8 6 8 6 10 9 10 6 10 9 10 13 1 6 3 7 6 8 5 8 8 9 6 7 7 8 14 1 7 1 7 2 7 5 8 5 9 5 8 4 8 15 1 8 1 8 7 8 2 8 7 9 2 8 7 9 16 1 5 2 7 5 9 2 9 5 9 9 9 9 9 Mean 1.19 7.00 1.75 7.25 5.56 7.81 5.75 8.56 7.13 9.31 6.31 8.38 7.19 9.06 Std. 0.83 1.67 1.00 1.53 1.82 1.28 2.05 1.03 1.5 0.79 1.92 1.15 1.64 0.93 Dev.

3. Please rate surface traffic awareness for the following display concepts. 1 = low; 10 = high Scenario 1 – arrival / takeoff hold EP # B BMO BMOT BRIPS 1 4 4 8 10 2 1 2 8 8 3 1 1 6 7 4 1 1 10 10 5 1 1 10 10 6 0 0 8 6 7 1 5 8 9 8 1 1 10 10 9 1 1 6 6 10 1 1 9 9 11 1 1 8 8 12 1 1 8 8 13 1 1 6 5 14 1 1 9 8 15 1 1 3 4 16 1 2 7 7 Mean 1.13 1.5 7.75 7.81 Std. Dev 0.81 1.26 1.84 1.87 Scenario 2 – departure / intersection departure EP # B BMO BMOT BRIPS 1 4 4 9 10 2 1 2 8 9 3 1 1 6 6 4 1 1 9 9 5 1 1 8 10 6 0 0 8 6 7 1 5 9 8 8 1 1 10 10 9 1 3 6 8 10 1 1 9 9 11 1 1 8 8 12 1 1 6 6 13 1 3 7 6 14 1 1 6 5 15 1 1 4 5 16 1 2 5 5 Mean 1.13 1.75 7.38 7.5 Std. Dev. 0.81 1.34 1.71 1.90 Scenario 3 – arrival / departure EP # B BMO BMOT BRIPS 1 4 4 10 10 2 1 2 9 10 3 1 1 6 6 4 1 1 8 8 5 0 0 8 8 6 0 3 8 6 7 1 6 8 9 8 10 10 10 10 9 1 1 4 4 10 1 1 9 9 11 1 1 8 8 12 1 1 5 6 13 1 1 5 5 14 1 1 3 3 15 1 1 4 4 16 1 1 5 5 Mean 1.63 2.19 6.88 6.94 Std. Dev. 2.39 2.56 2.28 2.35 Scenario 4 – departure / departure EP # B BMO BMOT BRIPS 1 5 5 9 10 2 1 2 8 10 3 1 1 6 6 4 1 1 7 7 5 1 1 3 3 6 0 0 6 6 7 1 2 9 9 8 1 1 10 10 9 1 3 4 4 10 1 1 9 9 11 1 1 10 10 12 1 1 4 4 13 1 3 8 7 14 1 1 3 2 15 1 1 3 4 16 1 1 7 7 Mean 1.19 1.56 6.63 6.75 Std. Dev. 1.05 1.21 2.55 2.74 Scenario 5 – taxi crossing / departure EP # B BMO BMOT BRIPS 1 5 5 10 10 2 1 3 9 10 3 1 1 6 7 4 1 1 8 7 5 1 1 10 10 6 0 0 10 8 7 1 4 9 8 8 1 1 10 10 9 1 1 8 8 10 1 1 9 9 11 1 1 8 8 12 1 1 6 6 13 1 3 6 5 14 1 1 8 7 15 1 1 2 2 16 1 1 9 9 Mean 1.19 1.63 8 7.75 Std. Dev. 1.0 5 1.36 2.13 2.14

Appendix H: Usability Study Detection Algorithm Evaluation Questionnaire

Results

The EPs evaluated the two incursion detection algorithms after each test scenario. The scenarios are defined as follows: Scenario 1 arrival / takeoff hold Scenario 2 departure / intersection departure Scenario 3 arrival / departure Scenario 4 departure / departure Scenario 5 taxi crossing / departure 1. Please rate the effectiveness of the alerting for preventing runway incursions for each display concept (based on the runway incursion scenario experienced). 1 = low; 10 = high Note: All evaluations were made using the BRIPS display concept (Baseline with perspective map).

Scenario 1 Scenario 2 Scenario 3 Scenario 4 Scenario 5 EP # Single Two Single Two Single Two Single Two Single Two Stage Stage Stage Stage Stage Stage Stage Stage Stage Stage 1 10 10 10 9 10 6 10 7 10 5 2 9 10 10 10 10 10 10 10 9 10 3 7 8 7 7 7 8 7 8 7 4 8 9 9 9 7 7 8 7 7 4 5 10 10 10 10 3 10 10 10 10 3 6 7 10 10 10 10 10 10 9 10 9 7 8 9 9 9 8 9 9 8 9 8 8 10 6 10 8 10 10 10 10 10 10 9 8 9 8 8 8 8 8 8 8 10 8 9 8 8 8 8 8 8 8 9 11 9 10 10 10 10 10 10 10 10 10 12 9 10 10 9 9 10 10 10 10 13 8 9 9 8 9 9 9 9 8 14 8 9 8 8 8 8 8 7 8 8 15 10 9 10 10 9 10 4 10 8 16 9 9 9 9 9 9 9 9 9 Mean 8.63 9.13 9.19 8.88 8.44 8.8 9.19 8.31 9.06 7.88 Std. Dev. 1.02 1.02 0.98 0.96 1.79 1.48 0.91 1.66 1.00 2.16 Comments: EP1 Scenario 2 two stage – alerted later EP4 Scenario 1 two stage – more time, chance to evaluate EP7 Scenario 1 two stage faster Scenario 3 based on how landed, thought two stage alerted sooner Scenario 4 single stage faster Scenario 5 single stage faster EP9 Scenario 5 caution didn’t buy anything EP13 Scenario 5 caution not beneficial – too close to warning EP14 Scenario 5 single stage – did not perceive difference in alerting two stage – not useful as implemented EP15 Scenario 5 two stage – no need for two alerts in this case 2. In general, which type of alerting is better for single pilot general aviation operations?

Single stage alerting or two stage alerting For the choice selected, how much more effective is that alerting type for runway incursion prevention?

1 = low; 10 = high Please provide a reason for your answer.

EP # Single Effectiveness Two Effectiveness Reason Stage Stage 1 1 It gave more time to react with single stage in sim. I’d rather have two stage alerting with the alerting time of single stage.

2 1 2 Both are excellent, two stage provides a higher comfort level.

3 1 4 It is better to have the caution which seems to give you more time to evaluate the situation and react accordingly.

4 1 4 1 6 Two stage for approaches only. Single stage alerting allowed more time for response during ground ops. The two stage, however, allows the pilot to know when to start looking on approach for other aircraft when concentrating on other tasks.

5 1 3 Seems redundant but a second warning may help if the first is not heard.

6 1 8 With one you first react. With two your mind makes you analyze what is happening.

7 1 8 Two stage seems to provide earlier alerting of hazards, enabling pilot attention and decision making – proactive responses. Like selective on stage in certain cases (takeoff …) 8 1 8 (no comment) 9 1 2 Earlier notification 10 1 1 The two stage gave me a chance to look at the display prior to the warning. Better SA.

11 1 3 Single is very good, but two stage offers a little more cushion.

12 1 3 Caution can be confusing 13 1 4 More time to prepare before acting 14 1 3 Early alert for more reaction time 15 1 9 Late alert for two stage 16 1 4 Couldn’t tell a big difference Total 6 6.4 11 3.64 3. With respect to latency, were the incursion alerts provided in a timely manner allowing sufficient time to react to the potential conflict?

All 16 EPs answered YES Did either single stage or two stage alerting provide more time for response to the RI?

Please estimate how much more time the alert provided.

If not, when should the alert have occurred?

EP # Single Two Stage Seconds Comment Stage 1 2  2 5 to 10  3 3 to 4 Single stage for taxi and departure, two stage for approach   4 5 to 10 Two stage taxi (scenario 5) not enough time. Two stage more time  on approach.

5 Same If any thing, alert should have occurred when the other plane begins takeoff roll. Two stage for scenario 3 & 5 was late.

6 1 to 1.5 Except for departure, alerts should be sooner.

 7 1 to 4 Potentially a lifetime.

 8 Equal 9 2 Most times single stage is all that is needed. Two stage beneficial  on approach only.

10 Neither 11 2 to 3  12 1  13 3  14 2 Single stage for takeoff, two stage for landing   15 3  16 10 to 15 Two stage on approach  Total 8 7

Appendix I: Final Semi-Structured Interview Questionnaire Results

A post-test questionnaire was administered to each EP. The acronyms for the display conditions are defined as follows: B Baseline BMO Baseline with Surface Map and Ownship BMOT Baseline with Surface Map and Ownship and Traffic BA Baseline with Audible Incursion Alerts BAMO Baseline with Audible Incursion Alerts, Surface Map and Ownship BAMOT Baseline with Incursion Alerts, Surface Map and Ownship, Traffic BRIPS Baseline with Perspective Surface Map I. RIPS General Safety 1. In general, please rate the level of safety felt during runway incursion incidents using the following equipage. 1 = Not Safe; 10 = Completely Safe EP # B BMO BMOT BA BAMO BAMOT BRIPS 1 4 4 5 6 6 10 10 2 5 6 7 8 9 10 10 3 1 2 6 6 6 7 8 4 2 2 5 5 5 10 10 5 1 1 7 8 8 9 10 6 0 0 3 8 8 10 8 7 1 5 6 6 7 9 8 8 5 5 8 10 10 10 10 9 1 3 4 7 7 9 9 10 6 6 9 10 10 10 10 11 1 1 8 8 8 10 10 12 1 1 5 10 10 10 10 13 1 3 4 5 6 9 8 14 2 3 3 5 5 9 8 15 1 1 3 7 7 8 9 16 1 1 3 6 7 9 9 Mean 2.06 2.75 5.38 7.19 7.44 9.31 9.19 Std. Dev. 1.84 1.95 2.00 1.76 1.67 0.87 0.9.1 II. RIPS General Advantages and Disadvantages 2a. What do you consider the best feature of the RIPS displays/technology you evaluated today?

Can the feature be improved? If so, how?

EP # Best Feature Improvement 1 Liked perspective map better; however, the traffic Showed movement of other aircraft on airport icons were harder to see 2 Provides enhanced situational awareness, ownship location, airport environment & traffic position & direction moving Capability to vary font size 3 Audible alert "Warning, traffic 34R" not descriptive enough 4 Airport layout system for increased situational awareness Text clutter adjustments (ID labels) 5 Audio alerts Little sooner to match the graphic takeoff roll 6 Top down view, audible & traffic N# no help 7 Dynamic and real-time situational awareness and hazard warning (no comment) 8 N/A N/A 9 Audible alert (no comment) 10 Traffic Warning late sometimes / latency 11 Traffic + audible altering (no comment) 12 Audible (no comment) 13 Combination visual & audible alerts Alert a little sooner 14 Taxi clearance (no comment) 15 Show ownship with larger symbol, use larger Large red warning symbol with audio alert traffic symbols at threshold 16 Give warning of incursions when traffic was difficult to see (no comment) 2b. What do you consider to be the worst feature of the RIPS displays/technology you evaluated today?

Can the feature be improved? If so, how?

EP # Worst Feature Improvement 1 Text warning didn't get my attention as well as Maybe use a short flashing box that highlights the audible and text warning 2 Location of display in cockpit - too much eye movement for cross check Ability to declutter ID tags readily 3 Top down map looked busy Less clutter 4 (no comment) (no comment) 5 Did not like to watch the rotation of the top down view Not really - leave it in as a choice 6 Perspective view (no comment) 7 No "worst" feature Minor tune ups 8 N/A N/A 9 None (no comment) 10 (no comment) (no comment) 11 Location (no comment) 12 Caution chevron (yellow) hard to see; placement of display Flashing chevron - reposition panel 13 Position of screen Move it 14 During warning, change display to show only Clutter, color of symbology conflict, traffic, speed, distance, time to conflict 15 With 2 stage alert - make warning earlier - Small symbol for ownship & traffic & delayed increase size of symbols of a/c moving on surface warning on 2 stage alert (traffic and ownship) 16 Display a little cluttered in top down display (no comment) 2c. What are the advantages and disadvantages of the RIPS displays/technology you evaluated compared to the baseline condition (baseline round dials only)?

EP # Advantages Disadvantages 1 Pin points your location in relation to the other a/c on airport and warnings (audible and text) No disadvantage 2 Situation awareness Possible sunlight washout of screen 3 Better situational awareness Training, more to look at 4 Traffic advisories allow for a much higher sense The chance a pilot may focus on guidance line of awareness during approach 5 Pilots can be too dependent on the technology and The pilot can see traffic and know their direction less on looking out the window. What if some and location even in poor visibilities and plane is not compatible with the technology and is distance locations obscured by airport facilities not displayed?

6 Situational awareness greatly enhanced none 7 Great improvement in SA and for intuitive and information Essentially zero 8 N/A N/A 9 Audible + perspective + graphic alert None 10 Would take training to ensure attention distribution optimally and how to interpret the Situation awareness warnings 11 More aware of traffic and surrounding Left placement of screen 12 Audible alerts + SA Audible caution was confusing 13 Gives information you need - my position relative to traffic & airport features (no comment) 14 (no comment) (no comment) 15 Audio alert + red (large) warning lettering & symbols on display None 16 Gave excellent traffic warning especially when audio alert used None III. RIPS Surface Ops Evaluation 3. Please indicate your level of agreement with each of the following statements for all display concepts. 1 = Low; 10 = High 3a. Where am I? The display concept provides sufficient awareness of my ownship position with respect to runways, taxiways, and stationary objects.

If unequal rating, please indicate reason.

If not “completely agree” for Surface Map conditions, what can be done to improve the concept to increase awareness?

EP # B BMO BMOT BA BAMO BAMOT BRIPS Reasons Improvement 1 2 10 10 2 10 10 10 Baseline doesn't show my orientation on airport (no comment) 2 1 10 10 1 10 10 10 (no comment) (no comment) 3 1 6 6 1 6 6 7 (no comment) (no comment) 4 2 8 8 2 8 8 8 No vertical (no comment) indications 5 1 10 10 1 10 10 10 (no comment) (no comment) 6 1 8 8 1 8 10 8 Easier to interpret top down (no comment) 7 1 10 10 2 10 10 9 Perspective slightly poor due to relative visibility of certain areas Same comment 8 6 8 8 6 8 10 10 (no comment) (no comment) 9 1 9 9 1 9 9 9 Audible provides no info for my position awareness Nothing 10 4 4 7 6 8 10 10 Airport environment + traffic + alerting N/A 11 1 8 8 2 8 8 10 Perspective view (no comment) 12 1 10 10 1 10 10 10 Surface map provides all the information you need (no comment) 13 1 8 9 1 9 9 8 Prefer top down view on airport surface; traffic alert without map position not Move display; as helpful adjust scale 14 1 8 8 1 8 8 8 (no comment) (no comment) 15 1 10 10 2 10 10 10 (no comment) (no comment) 16 1 9 9 1 9 9 9 Draw picture (no comment) of clearance Mean 1.63 8.5 8.75 1.94 8.81 9.19 9.125 Std. 1.41 1.67 1.24 1.65 1.17 1.17 1.02 Dev.

3b. Where am I relative to other moving objects? The display concept provides sufficient awareness of my ownship position with respect to moving traffic, such as vehicles and other aircraft.

If unequal rating, please indicate reason.

If not “completely agree” for surface map with traffic conditions, what can be done to improve the concept to increase awareness?

EP # B BMO BMOT BA BAMO BAMOT BRIPS Reasons Improvement 1 1 1 9 1 1 9 10 No traffic info Perspective without map view gave a (baseline rounds broader view of dials only) airport traffic 2 1 1 8 5 5 10 10 Audible/ visual alerting enhances SA to (no comment) moving traffic 3 2 2 6 2 2 6 7 (no comment) (no comment) 4 1 1 8 3 3 9 9 Alert brings any incursions to the users notice (no comment) 5 1 1 10 2 2 10 10 Audible alerting provides info for one a/c not all (no comment) 6 0 0 8 6 6 10 8 Like top down (no comment) 7 1 2 8 2 2 10 9 Need to see traffic + self on map (no comment) 8 4 4 8 6 6 10 10 (no comment) (no comment) 9 1 1 6 1 1 8 8 Msg & audible alerts adds value on other traffic position and need to respond (no comment) 10 4 4 7 6 6 10 10 Elements are shown on map + relevant features, identifiable landmarks, etc. (no comment) 11 5 5 8 7 7 10 10 (no comment) (no comment) 12 1 1 10 5 5 10 10 (no comment) (no comment) 13 1 3 8 6 6 9 8 Position of display; change Prefer top down scale 14 1 1 8 5 5 9 9 (no comment) (no comment) 15 1 1 10 1 1 10 10 The 1's don't show traffic (no comment) 16 1 1 9 7 9 9 9 If unit could show a potential conflict in a (no comment) different color Mean 1.63 1.81 8.19 4.06 4.25 9.31 9.19 Std. 1.41 1.42 1.22 2.26 2.54 1.08 0.98 Dev.

IV. RIPS Display and Alerts 4. Please indicate the effectiveness of the perspective map display compared to the top-down views for prevention of runway incursion. 1 = Low; 10 = High Reason for rating.

EP # Rating Reason 1 9 The perspective gave a broader view of airport traffic 2 2 Preference 3 4 Easier to pick out ownship in less time 4 0 No difference 5 5 Top down showed more, perspective was easier to look at 6 2 Fast orientation to top down 7 8 Focus on approach environment (for approaches only, perspective more effective) 8 5 (no comment) 9 1 Same information and equally as readable 10 0 Same effectiveness; preference is for 2D top down 11 1 (no comment) 12 3 Top down was easier to interpret, perspective items get lost 13 3 Top down gives broader view to sides & rear 14 1 (no comment) 15 0 No difference - both maps displayed the alert warnings satisfactorily 16 2 Map less cluttered Mean 2.88 Std. Dev. 2.73 5. When traffic was presented on surface map, was incurison traffic easily discernable?

Yes or No All 16 EPs responded Yes.

If yes, please indicate how discernable. 1 = Low, 10 = High EP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 10 10 7.5 10 9 8 10 5 9 9 6 8 7.5 5 10 10 Mean 8.38 Std. Dev. 1.77 6. Is traffic presentation necessary on surface map to prevent runway incursions?

Yes or No EPs 8 and 15 responded No. The other 14 EPs responded Yes.

If so, please indicate how much additional situation awareness is provided with the addition of traffic presentation on the surface map.

Comparing surface map with ownship TO surface map with ownship and traffic, situation awareness enhancement is: 1 = Low; 10 = High EP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 10 9 8 7 10 8 9 4 10 8 8 10 10 8 9 9 Mean 8.56 Std. Dev. 1.56 7. Is graphical presentation of alerts necessary on surface map to prevent runway incursions?

Yes or No If so, please indicate how much additional situation awareness is provided with the addition of graphical alerting on the surface map. 1 = Low; 10 = High EP # Yes No Rating Comment 1 10  2 Graphic not needed if audible alert available  3 3  4 But is helpful  5 9  6 8  7 9  8 3   10 8  11 3  12 4  13 8  14 4    Total 10 6 6.27 Mean 2.83 Std. Dev.

8. If a runway incursion situation occurs, which of the following is most likely to bring the event to your attention? If multiple, please indicate the rank order.

OTW = out-the-window Map = surface map Audible = audible alerting Note: For B and BMO, OTW is the only method of acquiring incursion traffic.

map map map EP # BA / OTW OTW OTW OTW BMO / BMO / audible audible audible audible BRIPS / BRIPS / BRIPS / BMOT / BMOT / BAMOT/ BAMOT/ BAMOT/ BA / OTW 1 2 1 1 2 1 2 2 3 1 2 3 1 2 1 2 2 1 2 1 2 1 3 2 1 3 3 2 1 2 1 2 1 3 1 2 3 1 2 4 2 1 1 2 1 2 2 3 1 2 3 1 5 2 1 1 2 1 2 2 3 1 2 3 1 6 1 2 2 1 2 1 2 1 3 2 1 3 7 2 1 2 1 2 1 3 1 2 3 1 2 8 1 2 1 2 1 2 1 2 3 1 2 3 9 2 1 2 1 2 1 3 1 2 3 1 2 10 1 2 2 1 2 1 3 1 2 3 1 2 11 1 2 1 2 1 2 3 2 1 3 2 1 12 1 2 2 1 2 1 2 1 3 2 1 3 13 2 1 2 1 2 1 3 1 2 3 1 2 14 1 2 1 2 1 2 1 2 3 1 2 3 15 1 2 2 1 2 1 3 1 2 3 1 2 16 2 1 2 1 2 1 3 1 2 3 1 2 Mean 1.5 1.5 1.63 1.38 1.63 1.38 2.38 1.56 2.06 2.38 1.56 2.06 Std. 0.52 0.52 0.5 0.5 0.5 0.5 0.72 0.81 0.77 0.72 0.81 0.77 Dev.

9. Upon receiving an incursion alert, was your immediate reaction to: Confirm the hazard, Take evasive action, or Other Please give a reason for your answer.

EP # Confirm Evasive Action Other Reason 1 I saw traffic first and the audible confirmed traffic incursion  and I took action.

2 Limited time to react  3 (no comment)  4 Evaluate what the evasive action should be  5  The audio alerts were given after I saw the incursion 6 Training that warning requires action  7 Initiate immediately and confirm, then decide   8 (no comment)  9  Need to act quickly 10 Wasn’t in an emergency situation so was safe to get to safe  altitude 11 Best course of action  12 Trusted the alert  13 I’m a skeptic  14 (no comment)  15 At slow speeds best to abort  16 May miss an approach or create a dangerous situation if  hazard is not confirmed first Total 4 13 0 What do you feel is the appropriate procedure on receipt of a RI alert for the following?

Landing, Takeoff, and Taxi EP # Landing Takeoff Taxi 1 Go around Abort Stop 2 Go around Abort Stop 3 Go around Abort Stop 4 Go around Abort Stop 5 Go around Abort Stop 6 Go around Abort Stop 7 Go around Abort Stop 8 Go around Abort Stop 9 Go around Abort Stop 10 Go around Abort Stop 11 Go around Abort Stop 12 Go around Abort Stop 13 Go around Abort Stop 14 Go around Contention Stop 15 Go around Abort Stop 16 Go around Abort Stop 10. Which display combination would provide a minimal configuration to be effective for runway incursion prevention.

EP # Audible Alert Surface Map with Surface Map with Baseline Round Dials Ownship only Ownship & Traffic                  Total 10 0 6 1 For an optimal configuration?

EP # Audible Alert Surface Map with Surface Map with Baseline Round Dials Ownship only Ownship & Traffic                                 Total 16 0 16 0 11. Please rate the effectiveness of the terms used for the incursion alerts. 1 = Low; 10 = High EP # “Warning, “Caution, “ Warning, “Caution, traffic “Warning, “Caution, traffic on traffic on traffic departing departing traffic traffic RWY” RWY” RWY” RWY” approaching approaching RWY” RWY” 1 10 10 10 10 10 10 2 10 10 10 10 10 10 3 5 5 8 8 8 8 4 10 10 10 10 10 10 5 9 9 9 9 N/A N/A 6 10 7 10 7 10 7 7 9 9 9 9 9 9 8 10 10 10 10 10 10 9 9 9 9 9 9 9 10 10 10 10 10 10 10 11 10 10 10 10 10 10 12 10 5 10 5 10 5 13 9 9 9 9 9 9 14 8 8 8 8 8 8 15 8 4 4 8 3 3 16 10 10 10 10 10 10 Mean 9.25 8.44 9.13 8.88 9.07 8.53 Std. Dev. 1.29 2.06 1.54 1.41 1.83 2.10 If not “completely effective”, would another term be more effective or descriptive?

How could the alert term be improved (i.e.wording)?

The following EPs responded to this question: EP 3 More specific about location of incursion traffic EP 12 Delete cautions EP 13 Repeat warning – may not be understood first time EP 15 Caution traffic crossing left, to right, right to left 12a. For single pilot operations, was the two stage alerting (“Caution” & “Warning”) more effective in preventing runway incidents than single stage alerting (“Warning” only)? Yes or No 12b. Did the two stage alerting (“Caution” & “Warning”) enhance your situation awareness (SA) more than the single stage alerting (“Warning” only)? Yes or No 12c. If yes, please indicate how much more the two stage alert enhanced your SA compared to the one stage alert. 1 = Minimal; 10 = Substantially What was the reason for your rating?

What are the benefits, if any, of two stage alerting?

12a 12b 12c EP # Yes No Yes No Reason Benefits 1 Gives the pilot time to react   to threat of incursion 2 4 Increased comfort level with   having more perceived time for decision making 3 4 Felt like I had more time Reaction time, scene   evaluation 4 4.5 Allowed more time for response Same comment   and reaction when giving a caution and warning 5 Only needed if the first alert   is not heard 6 Better that none, single more   effective 7 8 Draws attention to hazard earlier Same comment   & provides assessment + decision time advantage   9 2 Very slight improvement in Personal option to act sooner   timing of information 10 4 Gave opportunity to look at More tim to analyze   surface map to confirm situation 11 4.5 Provided more response time Same Comment   12 Caution was confusing N/A   13 1.5 Not a big difference More time to confirm /   consider 14 7 Single for takeoff; two stage for Early warning   landing 15 None     Total 6 10 9 7 4.39 Mean 2.07 Std. Dev.

13. For each display concept, indicate which type of alerting provides the greatest amount of runway incursion prevention awareness and the amount of situation awareness enhancement (based on the choice you selected) for the following displays: 1 = Minimal; 10 = Substantially S = Single stage alert T = Two stage alert SA = Situation awareness enhancement EP # BA + S SA BA + T SA BAMO + S SA BAMO + T SA BAMOT + S SA BAMOT + T SA BRIPS + S SA BRIPS + T SA 1 1 8 1 8 1 10 1 10 2 1 6 1 7 1 9 1 9 3 1 5 1 5 1 7 1 8 4 1 5 1 5 1 9 1 9 5 1 5 1 5 1 9 1 10 6 1 8 1 8 1 10 1 8 7 1 8 1 8 1 8 1 8 8 1 4 1 4 1 4 1 4 9 1 2 1 2 1 3 1 3 10 1 3 1 3 1 3 1 3 11 1 4 1 4 1 2 1 2 12 1 10 1 10 1 10 1 10 13 1 2 1 2 1 2 1 2 14 1 2 1 2 1 2 1 2 15 1 7 1 8 1 9 1 10 16 1 2 1 2 1 2 1 2 Total 6 10 6 10 7 9 7 9 Mean 6.5 4.2 6.7 4.3 7.7 5.0 7.6 5.2 Std. 2.95 1.99 3.01 2.11 3.30 3.16 3.23 3.42 Dev.

14. Would you like to be given escape guidance on the maneuver to take to avoid the conflict in addition to the runway incursion alert?

On Final On Takeoff When What type of advisory / guidance would you like to see Approach Roll Taxiing during landing, during takeoff, or during taxi across a Across a runway?

Runway?

EP # Yes No Yes No Yes No 1    Safest maneuver to use to avoid collision 2    Missed approach instructions 3    Simple escape guidance (e.g. Climb right, descend left) 4    Guidance by heading to go behind the conflicting aircraft 5    Landing: direction to go around; takeoff: “abort takeoff”; taxi: “stop taxi” 6    Go around or missed approach, climb & steering commands 7    Whatever appears to provide relatively maximal probability of incident avoidance, resolution of imminent hazards, and aircraft/crew survival 8    If advising, simple command: “hold short” 9    Landing: course & climb decision; takeoff: direction change if needed; taxi: just stop 10    Explicit aural annunciation and display 11    Stop!

12    13    If at controlled airport, issue vector, clearance etc.; if at uncontrolled field, short statement of bese evasive action 14    15    16    Direction – stop – turn – heading? Altitude?

Total 12 4 8 8 9 7 15. Please provide any additional comments that will help us in our evaluation and development of the synthetic vision system concepts.

The following EPs responded to this question: EP 1 Increase volume on the aircraft noise in cockpit.

EP 4 Velocity vector display on map EP 9 Value of surface map may have increased if in a different location, closer to primary displays in multifunction display area EP 13 Why not put traffic lights at hold lines EP 15 Location of display needs to be within pilots heads-up field of view; land-and-hold-short planning for all cross-runway landings

Appendix J: Paired Comparisons Questionnaire

Overview: This questionnaire is designed to allow statistical analysis of your subjective assessment of situation awareness and runway incursion prevention for each of the following display configurations you evaluated today. Please look at the pictures of the display concepts when making your comparisons.

Paired Comparison Rating Instructions: Each paired comparison will be listed on the left side of the questionnaire. You will be asked to make separate comparisons based on what you experienced during the flying tasks in the simulator.

Situation Awareness: If situation awareness is not equal, indicate the magnitude of the difference by marking the appropriate box on the scale to the right of the comparison. The following definition of situation awareness should be used for reference: The pilot’s awareness and understanding of all factors that will contribute to the safe flying of their aircraft under normal and non-normal conditions.

Runway Incursion Prevention Awareness: If runway incursion awareness is not equal, indicate the magnitude of the difference by marking the appropriate box on the scale to the right of the comparison.

The following definition should be used for reference: The pilot’s awareness and understanding of the dynamic environment and degree to which he or she is aware of, and can successfully avoid, potential incursions with other aircraft or vehicles in the airport environment.

The following example shows how to make the comparisons. Do not take an excessive amount of time on each comparison; your first impression is usually best. However, please feel free to correct any comparisons. Also, the data will be checked for consistency; if the results are inconsistent, you may be asked to clarify your responses.

Situation Awareness Comparisons If not equal, how much more or how much less?

Barely Substantially Display Concept ‘X’ Provides (_  _ more)(__ equal)(__ less) SA than  Display Concept ‘Y’

Situation Awareness Comparisons If not equal, how much more or how much less?

Barely Substantially Baseline round dials only (B) Provides (__ more)(__ equal)(__ less) SA than B + surface map with ownship only (BMO) Baseline round dials only (B) Provides (__ more)(__ equal)(__ less) SA than B + surface map with ownship + traffic (BMOT) Baseline round dials only (B) Provides (__ more)(__ equal)(__ less) SA than B + audible alerting only (BA) Baseline round dials only (B) Provides (__ more)(__ equal)(__ less) SA than B + audible alerting + map with ownship only (BAMO) Baseline round dials only (B) Provides (__ more)(__ equal)(__ less) SA than B + audible & graphical alerting + map with ownship + traffic (BAMOT) Baseline round dials only (B) Provides (__ more)(__ equal)(__ less) SA than B + audible & graphical alerting + perspective map (BRIPS) B + surface map with ownship only (BMO) Provides (__ more)(__ equal)(__ less) SA than B + surface map with ownship + traffic (BMOT) B + surface map with ownship only (BMO) Provides (__ more)(__ equal)(__ less) SA than B + audible alerting only (BA) B + surface map with ownship only (BMO) Provides (__ more)(__ equal)(__ less) SA than B + audible alerting + map with ownship only (BAMO) B + surface map with ownship only (BMO) Provides (__ more)(__ equal)(__ less) SA than B + audible & graphical alerting + map with ownship + traffic (BAMOT) B + surface map with ownship only (BMO) Provides (__ more)(__ equal)(__ less) SA than B + audible & graphical alerting +perspective map (BRIPS)

Situation Awareness Comparisons If not equal, how much more or how much less?

Barely Substantially B + surface map with ownship + traffic (BMOT) Provides (__ more)(__ equal)(__ less) SA than B + audible alerting only (BA) B + surface map with ownship + traffic (BMOT) Provides (__ more)(__ equal)(__ less) SA than B + audible alerting + map with ownship only (BAMO) B + surface map with ownship + traffic (BMOT) Provides (__ more)(__ equal)(__ less) SA than B + audible & graphical alerting + map with ownship + traffic (BAMOT) B + surface map with ownship + traffic (BMOT) Provides (__ more)(__ equal)(__ less) SA than B + audible & graphical alerting + perspective map (BRIPS) B + audible alerting only (BA) Provides (__ more)(__ equal)(__ less) SA than B + audible alerting + map with ownship only (BAMO) B + audible alerting only (BA) Provides (__ more)(__ equal)(__ less) SA than B + audible & graphical alerting + map with ownship + traffic (BAMOT) B + audible alerting only (BA) Provides (__ more)(__ equal)(__ less) SA than B + audible & graphical alerting +perspective map (BRIPS) B + audible alerting + map with ownship only (BAMO) Provides (__ more)(__ equal)(__ less) SA than B + audible & graphical alerting + map with ownship + traffic (BAMOT) B + audible alerting + map with ownship only (BAMO) Provides (__ more)(__ equal)(__ less) SA than B + audible & graphical alerting +perspective map (BRIPS) B + audible alerting + map with ownship + traffic (BAMOT) Provides (__ more)(__ equal)(__ less) SA than B + audible & graphical alerting +perspective map (BRIPS)

Runway Incursion Prevention If not equal, how much more or how much less?

Barely Substantially Baseline round dials only (B) Provides (__ more)(__ equal)(__ less) potential for RI prevention than B + surface map + ownship only (BMO) Baseline round dials only (B) Provides (__ more)(__ equal)(__ less) potential for RI prevention than B + surface map + ownship + traffic (BMOT) Baseline round dials only (B) Provides (__ more)(__ equal)(__ less) potential for RI prevention than B + audible alerting only (BA) Baseline round dials only (B) Provides (__ more)(__ equal)(__ less) potential for RI prevention than B + audible alerting + map + ownship (BAMO) Baseline round dials only (B) Provides (__ more)(__ equal)(__ less) potential for RI prevention than B + audible & graphical alerting + map + ownship + traffic (BAMOT) Baseline round dials only (B) Provides (__ more)(__ equal)(__ less) potential for RI prevention than B + audible & graphical alerting + perspective map (BRIPS) B + surface map + ownship only (BMO) Provides (__ more)(__ equal)(__ less) potential for RI prevention than B + surface map + ownship + traffic (BMOT) B + surface map + ownship only (BMO) Provides (__ more)(__ equal)(__ less) potential for RI prevention than B + audible alerting only (BA) B + surface map + ownship only (BMO) Provides (__ more)(__ equal)(__ less) potential for RI prevention than B + audible alerting + map + ownship (BAMO) B + surface map + ownship only (BMO) Provides (__ more)(__ equal)(__ less) potential for RI prevention than B + audible & graphical alerting + map + ownship + traffic (BAMOT) B + surface map with ownship only (BMO) Provides (__ more)(__ equal)(__ less) potential for RI prevention than B + audible & graphical alerting +perspective map (BRIPS)

Runway Incursion Prevention If not equal, how much more or how much less?

Barely Substantially B + surface map + ownship + traffic (BMOT) Provides (__ more)(__ equal)(__ less) potential for RI prevention than B + audible alerting only (BA) B + surface map + ownship + traffic (BMOT) Provides (__ more)(__ equal)(__ less) potential for RI prevention than B + audible alerting + map + ownship (BAMO) B + surface map + ownship + traffic (BMOT) Provides (__ more)(__ equal)(__ less) potential for RI prevention than B + audible & graphical alerting + map with ownship + traffic (BAMOT) B + surface map + ownship + traffic (BMOT) Provides (__ more)(__ equal)(__ less) potential for RI prevention than B + audible & graphical alerting + perspective map (BRIPS) B + audible alerting only (BA) Provides (__ more)(__ equal)(__ less) potential for RI prevention than B + audible alerting + map + ownship (BAMO) B + audible alerting only (BA) Provides (__ more)(__ equal)(__ less) potential for RI prevention than B + audible & graphical alerting + map + ownship + traffic (BAMOT) B + audible alerting only (BA) Provides (__ more)(__ equal)(__ less) potential for RI prevention than B + audible & graphical alerting +perspective map (BRIPS) B + audible alerting + map + ownship (BAMO) Provides (__ more)(__ equal)(__ less) potential for RI prevention than B + audible & graphical alerting + map + ownship + traffic (BAMOT) B + audible alerting + map + ownship only (BAMO) Provides (__ more)(__ equal)(__ less) potential for RI prevention than B + audible & graphical alerting +perspective map (BRIPS) B + audible alerting + map + ownship + traffic (BAMOT) Provides (__ more)(__ equal)(__ less) potential for RI prevention than B + audible & graphical alerting +perspective map (BRIPS) Form Approved REPORT DOCUMENTATION PAGE OMB No. 0704-0188 The public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number.

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.

1. REPORT DATE (DD-MM-YYYY) 2. REPORT TYPE 3. DATES COVERED (From - To) 01 - 2011 01- Technical Publication 4. TITLE AND SUBTITLE 5a. CONTRACT NUMBER Simulator Evaluation of Runway Incursion Prevention Technology for General Aviation Operations 5b. GRANT NUMBER 5c. PROGRAM ELEMENT NUMBER 6. AUTHOR(S) 5d. PROJECT NUMBER Jones, Denise R.; Prinzel, Lawrence J., III 5e. TASK NUMBER 5f. WORK UNIT NUMBER 609866.02.07.07.02 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) NASA Langley Research Center Hampton, VA 23681-2199 L-19917 9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR'S ACRONYM(S) National Aeronautics and Space Administration NASA Washington, DC 20546-0001 11. SPONSOR/MONITOR'S REPORT NUMBER(S) NASA/TP-2011-217045 12. DISTRIBUTION/AVAILABILITY STATEMENT Unclassified - Unlimited Subject Category 03 Availability: NASA CASI (443) 757-5802 13. SUPPLEMENTARY NOTES 14. ABSTRACT A Runway Incursion Prevention System (RIPS) has been designed under previous research to enhance airport surface operations situation awareness and provide cockpit alerts of potential runway conflict, during transport aircraft category operations, in order to prevent runway incidents while also improving operations capability. This study investigated an adaptation of RIPS for low-end general aviation operations using a fixed-based simulator at the National Aeronautics and Space Administration (NASA) Langley Research Center (LaRC). The purpose of the study was to evaluate modified RIPS aircraft-based incursion detection algorithms and associated alerting and airport surface display concepts for low-end general aviation operations. This paper gives an overview of the system, simulation study, and test results.

15. SUBJECT TERMS Runway safety; Runway incursions; Incursion detection algorithms; General aviation 19a. NAME OF RESPONSIBLE PERSON 18. NUMBER 17. LIMITATION OF 16. SECURITY CLASSIFICATION OF: OF ABSTRACT STI Help Desk (email: help@sti.nasa.gov) a. REPORT c. THIS PAGE b. ABSTRACT PAGES 19b. TELEPHONE NUMBER (Include area code) 86 (443) 757-5802 U U U UU Standard Form 298 (Rev. 8-98) Prescribed by ANSI Std. Z39.18