A46: Validation of Visual Operations Standards for Small Uncrewed Aircraft Systems – Final Report

2 RESEARCH QUESTIONS ..................................................................................................... 2 3 RESEARCH TASKS AND FINDINGS ................................................................................. 3 3.1 Task 1 – Literature Review and Market Analysis ............................................................ 3 3.1.1 Task 1-1: Literature Review – Conclusions.............................................................. 3 3.2 Task 2 – Updated Research Task Plan ............................................................................. 7 3.3 Task 3 – Initial Test and Analysis .................................................................................... 7 3.3.1 Task 3-1: Develop an Experimental Plan ................................................................. 7 3.3.2 Task 3-2: Initial Flight Test Plan ............................................................................ 10 3.3.3 Task 3-3: Draft Data Analysis Plan ........................................................................ 13 3.3.4 Task 3-4: Test and Data Analysis Plan Review ...................................................... 15 3.3.5 Task 3-5: VO Training PowerPoints....................................................................... 15 3.3.6 Task 3-6: Completed Initial Flight Test and Data Collection ................................. 18 3.3.7 Analysis of Aircraft Projected Area for NMSU and KSU Aircraft ........................ 37 3.4 Task 4 – Flight Testing and Data Analysis .................................................................... 43 3.4.1 KSU Aircraft ........................................................................................................... 43 3.4.2 KSU Participant Breakdown ................................................................................... 45 3.4.3 Location of VO Stations and UAS Reference Path ................................................ 47 3.4.4 KSU Flight Test Results ......................................................................................... 47 3.5 Task 5 – Lessons Learned Document ............................................................................ 74 4 CONCLUSIONS ................................................................................................................... 74 5 REFERENCES ...................................................................................................................... 77 v TABLE OF FIGURES Figure 1. Wide View of the NMSU Testing Area. ....................................................................... 12 Figure 2. Bird-eye view of the operational location for the NMSU A46 event............................ 13 Figure 3. NMSU Visual Observer Training in the Field. ............................................................. 19 Figure 4. Visual Observer East Location. ..................................................................................... 20 Figure 5. Visual Observer South Location. .................................................................................. 21 Figure 6. Visual Observer North Location. .................................................................................. 21 Figure 7. Visual Observer Location Set-Up and Reference Notes. .............................................. 22 Figure 8. View From VO North Location. The east location is to the left of the image, and the Central and South Locations are visible to the right. .................................................................... 22 Figure 9. sUAS Pilot and VO During the Testing and Data Collection. ...................................... 23 Figure 10. View to the Northeast. ................................................................................................. 23 Figure 11. View to the Northwest. ................................................................................................ 23 Figure 12. VO locations are marked with yellow pins (all are near the center crossing of the dirt runway). ........................................................................................................................................ 24 Figure 13. Full Flight Path for September 21, 2022. .................................................................... 25 Figure 14. Flight Path Crossing for September 21, 2022 ............................................................. 25 Figure 15. Close-up view of the flight path crossings on September 21, 2022. ........................... 26 Figure 16. Full Flight Path for September 22, 2022. .................................................................... 26 Figure 17. Flight Path Crossings on September 22, 2022............................................................. 27 Figure 18. Close-up View of the Flight Path Crossings on September 22, 2022. ........................ 27 Figure 19. Full Flight Path for September 23, 2022. .................................................................... 28 Figure 20. Flight Path Crossings on September 23, 2022............................................................. 28 Figure 21. Close-up View of the Flight Path Crossings on September 23, 2022. ........................ 29 Figure 22. Example of the Random Intruder Approaches on Day 1 of NMSU testing. ............... 34 Figure 23. Potential Alternative Intruder Flight Paths. ................................................................. 35 Figure 24. Percentile Distribution for Intruder Detection Distance Calculated for the NMSU Flight Test Encounters. ............................................................................................................................ 36 Figure 25. Positions of the Intruder Aircraft at Detection on a Map for the NMSU Flight Test Encounters..................................................................................................................................... 37 Figure 26. NMSU's CTLS Light Sport Aircraft. .......................................................................... 39 Figure 27. CTLS Shown from Different Angles. ......................................................................... 39 Figure 28. Side View of CTLS Used for Projected Area Assessment.......................................... 40 Figure 29. Grid Overlay of Side View of the CTLS. .................................................................... 40 Figure 30. CTLS Side View Grid Overlay with Estimated Area. Each Block is 2.7 inches square or 7.34in2 . There are 1,126 blocks. ............................................................................................... 41 Figure 31. KSU Cessna 172 Skyhawk. ......................................................................................... 44 Figure 32. KSU Cirrus SR20. ....................................................................................................... 44 Figure 33. KSU Great Shark 330. ................................................................................................. 45 Figure 34. Location of Operating Stations and UA Reference Flight Path for the KSU Flight Test Encounters..................................................................................................................................... 47 Figure 35. Percentile Distribution for Intruder Detection Distance Calculated for the KSU Flight Test Encounters. ............................................................................................................................ 50 vi Figure 36. Positions of the Intruder Aircraft at Detection on a Map for the KSU Flight Test Encounters..................................................................................................................................... 51 Figure 37. Scatter Plot for Intruder Aircraft Detection Distance vs. Light Intensity Levels at Detection for the KSU Flight Test Encounters. ............................................................................ 52 Figure 38. Scatter Plot for Intruder Detection Distance vs. Light Intensity at Detection for the KSU Flight Test Encounters. ................................................................................................................. 53 Figure 39. Scatter Plot for Intruder Detection Distance vs. Light Intensity at Detection for the KSU Flight Test Encounters. ................................................................................................................. 53 Figure 40. Images from GoPRO camera mounted on the VOs during the KSU flight test encounters (a) one of the runs from Day 2 tests with overcast conditions, and (b) one of the runs from Day 6 tests with bright sunny conditions................................................................................................. 54 Figure 41. Scatter plot for Intruder Detection Distance vs. Noise Level at Detection (Adjusted for outliers) for the KSU Flight Test Encounters. .............................................................................. 55 Figure 42. Percentile Distribution of Intruder Aircraft Distance as a Function of the VO Aviation Experience Categories for the KSU Flight Test Encounters. ....................................................... 56 Figure 43. Percentile Distribution of Intruder Detection Distance as a Function of the VO Visual Acuity Categories for the KSU Flight Test Encounters................................................................ 57 Figure 44. Scatter Plot for Intruder Detection Distance vs. Intruder Speed at Detection for the KSU Flight Test Encounters. ................................................................................................................. 58 Figure 45. Example KSU Encounter Flight Paths with Different Intruder Flight Path Angles (P1 VO Station, Day 1 - 11/01/22), Run #4. ....................................................................................... 60 Figure 46. Example KSU Encounter Flight Paths with Different Intruder Flight Path Angles (PI VO Station, Day 1 - 11/01/22), Run #2. ....................................................................................... 60 Figure 47. Example KSU Encounter Flight Paths with Different Intruder Flight Path Angles (P1 VO Station, Day 1 - 11/01/22), Run #6. ....................................................................................... 61 Figure 48. Example KSU Encounter Flight Paths with Different Intruder Flight Path Angles (P1 VO Station, Day 1, 11/01/22), Run #15........................................................................................ 61 Figure 49. Example NMSU Encounter Flight Paths with Different Intruder Flight Path Angles (East VO Station, Day 1 - 09/21/22), Run #1. .............................................................................. 62 Figure 50. Example NMSU Encounter Flight Paths with Different Intruder Flight Path Angles (East VO Station, Day 1 - 09/21/22). Run #3. .............................................................................. 62 Figure 51. Scatter Plot for Intruder Detection Distance vs. Projected Visual Area for the KSU and NMSU Flight Test Encounters...................................................................................................... 63 Figure 52. Percentile Distribution for Intruder Detection Distance Calculated for the CTLS Aircraft and Cessna 172 head-on Detection Runs. ..................................................................................... 64 Figure 53. Percentile Distribution for Intruder Visual Angle at Detection Calculated for the KSU and NMSU Flight Test Encounters. .............................................................................................. 65 Figure 54. Breakdown of Avoidance Maneuvers Suggested by the VOs for the KSU Flight Test Encounters..................................................................................................................................... 66 Figure 55. Breakdown of avoidance maneuvers suggested by the VOs for the KSU flight test encounters separately for each intruder aircraft (a) Cessna 172 and (b) SR20. ............................ 67 Figure 56. Example of the Relationship between Detection Distance and Position of the Intruder and UA in EVLOS Operations: (a) Intruder far away from the UA, and (b) Intruder near the UA. ....................................................................................................................................................... 68 vii Figure 57. Scatter Plot for Response Times to Initiate the Avoidance Maneuver vs. Distance Between the Intruder and the UA at Detection for the KSU Flight Test Encounters ................... 69 Figure 58. Example Encounter Flight Paths for the UA and Intruder (P1 VO Station, Run #4, Day 1-11/01/22). ................................................................................................................................... 70 Figure 59. Percentile distribution for change in CPA (slant & horizontal distance) calculated for the KSU flight test encounters. ..................................................................................................... 71 Figure 60. Percentile Distribution for Change in CPA (vertical distance) calculated for the KSU Flight Test Encounters. ................................................................................................................. 72 viii TABLE OF TABLES Table 1. Flight Design CTLS. ....................................................................................................... 11 Table 2. NMSU Subject Data ....................................................................................................... 30 Table 3. CLTS Run Time for September 21, 2022 ....................................................................... 33 Table 4. Descriptive Statistics for Intruder Detection Distance Calculated for the NMSU Flight Test Encounters. ............................................................................................................................ 36 Table 5. Descriptive Statistics for Intruder Speed at Detection for the NMSU Flight Test Encounters..................................................................................................................................... 37 Table 6. Estimated Projected Area for the CTLS, Cessna 172 Skyhawk, and Cirrus Design SR20 Aircraft. ......................................................................................................................................... 42 Table 7. KSU Participants' Snellen Test Results, Aviation Experience, and Fatigue Levels ....... 45 Table 8. Dependent and Independent Variables ........................................................................... 48 Table 9. Descriptive Statistics for Intruder Detection Distance Calculated for the KSU Flight Test Encounters..................................................................................................................................... 49 Table 10. Wind and Gust Speeds, Direction for the Eight KSU Flight Test Days. ...................... 54 Table 11. Descriptive Statistics for Intruder Speed at Detection for the KSU Flight Test Encounters..................................................................................................................................... 58 Table 12. Projected Visual Area (Front and Side) for the KSU and NMSU aircraft. .................. 59 Table 13. Descriptive Statistics for Response Times to Initiate the Avoidance Maneuver. ......... 69 Table 14. Average Reaction Time for Collision Avoidance Procedure Tasks ............................. 70 Table 15. Well Clear and NMAC Violations for Mitigated and Unmitigated encounters in the KSU flight tests ...................................................................................................................................... 73 ix TABLE OF ACRONYMS AARC Applied Aviation Research Center ADS-B Automatic Dependent Surveillance-Broadcast AGL Above Ground Level ANOVA Analysis of Variance ASSURE Alliance for System Safety of UAS Through Research Excellence BVLOS Beyond Visual Line of Sight COA Certificate of Authorization CPA Closest Point of Approach DAA Detect and Avoid EP External Pilot EVLOS Extended Visual Line of Sight FAA Federal Aviation Administration GPS Global Positioning System HMD Horizontal Miss Distance JER Jornada Environmental Range KSU Kansas State University MSU Mississippi State University NM Nautical Miles NMAC Near Mid Air Collision NMSU New Mexico State University NTSB National Transportation Safety Board RFP Request for Proposal RP Remote Pilot RPIC Remote Pilot in Command RTP Research Task Plan SDT Signal Detection Theory sUA small Uncrewed Aircraft sUAS small Uncrewed Aircraft System TCAS Traffic Collision Avoidance System UA Uncrewed Aircraft UAS Uncrewed Aircraft System VAD Visual Acquisition Distance VLOS Visual Line of Sight VMD Vertical Miss Distance VO Visual Observer WSU Wichita State University 1 EXECUTIVE SUMMARY The Alliance for System Safety of UAS through Research Excellence (ASSURE) A46 research team investigated Visual Observer (VO) effectiveness in maintaining safe separation from crewed aircraft during a set of uncrewed aircraft (sUA) flight operation trials. This research investigated two aspects of VO performance: (1) visual detection of a crewed aircraft and (2) the avoidance of a crewed aircraft. As a component of this research, the team (1) assessed the current state of the industry and identified key elements of previous VO experimental designs, (2) developed a flight test plan based on these findings, and (3) conducted flight test experiments that enabled the team to generate data on VO performance in terms of effectiveness in an Extended Visual Line of Sight (EVLOS) operational testing paradigm. A thorough literature review demonstrated the variability in the current state of VO/Remote Pilot (RP) “see and avoid” research. Researchers have studied the principle of see and avoid, which relies on the visual detection capabilities of crewed aircraft pilots to avoid a potential collision with other traffic, for several decades. While most of the observations noted on the topic of the see and avoid principle are not directly related to VO performance, a few provided useful insight into visual detection and scanning strategies that could prove beneficial in VO tasks. The literature highlighted the importance of including team-based skills associated with situational awareness, problem-solving/decision-making, and communication in future VO training and training standards. As an experimental construct, the research team leveraged a flight test campaign to (1) assess the accuracy of detection of a crewed aircraft by a VO, (2) explore the capacity for VOs to use visual references for avoidance, and (3) explore the capacity for the VO and RP to give way to crewed aircraft. This research used a mixed methods design utilizing a triangulated approach to capture both quantitative and qualitative data concurrently during the data collection phase of this study. These flight tests identified three common themes: (1) VOs with no prior aviation experience perform less effective than VOs with previous aviation experience, and (2) ambient conditions (e.g. contrast), ambient light, aircraft speed, and aircraft configuration (e.g., Cessna 172 versus Cirrus SR20) may have an impact on VO performance. Previous research ascertained VO ineffectiveness in estimating collision potential between an intruder crewed aircraft and the UA when the UA is within EVLOS. This research suggested that participants were 89.5% effective in identifying an intruder aircraft within 1 mile, with a significant degradation in performance effectiveness to 22.6% at 2 miles, and 3.3% at 3 miles. These findings ascertain that VOs may not be the most reliable as an effective source for collision avoidance. In fact, VOs did not perceive that an avoidance maneuver was necessary in 22 out of 54 (40.7%) encounters highlighted with Near Mid-Air Collision (NMAC) violations in the data set. Further research is deemed necessary to better understand the extent to which these independent variables impact VO/RP performance. 2 1 INTRODUCTION & BACKGROUND The Alliance for System Safety of UAS through Research Excellence (ASSURE) A46 research team investigated the effectiveness of Visual Observer (VO) performance for Extended Visual Line of Sight (EVLOS) operations. More specifically, the A46 research team explored methods to quantify VO performance, identify potential human visual detection limitations, uncrewed aircraft (UA) avoidance decision-making limitations, and inform safety training for Visual Line of Sight (VLOS) and EVLOS operations. The research tasks relevant to this report were as follows: • Task 1 – Literature Review • Task 2 – Updated Research Task Plan • Task 3 – Initial Test and Analysis • Task 4 – Flight Testing and Data Analysis • Task 5 – Lessons Learned • Task 6 – Final Report As a whole, this approach enabled the research team to (1) identify the current state of research related to visual observer performance in terms of effectiveness, (2) develop initial flight test plans and analysis techniques to gather relevant VO performance data using both live and simulated aircraft in a real-world testing environment, (3) conduct revised flight testing based on lessons learned from initial flight test campaigns, and (4) generate a lessons learned document based on the findings from the flight tests conducted as a component of this work. This approach provided a means to address research questions regarding VO performance, how different physiological aspects may impact VO performance, and how training may impact VO performance. Overall, these tasks provided a means to gather data to depict the effectiveness of VOs at identifying crewed aircraft in the airspace and their accuracy towards maneuver suggestions based on specific flight test regimes. 2 RESEARCH QUESTIONS The following questions guided this research methodology. The research questions provided an idea of the scope and scale of the project. Additionally, these questions informed the structure of research tasks and addressed the key requirements for this work. Research Questions • How well are VOs/RPs able to recognize potential collision hazards (other aircraft, terrain, obstacles, etc.) and avoid them through visual means during the day, during civil twilight, and at night? • What are the primary variables, variable relationships, and failure modes that are important in regard to VO/RP functions to see and avoid collision hazards? • What information on optical illusions and decision-making guidance to avoid collision hazards should VO/RP training standards include? • What are recommended safe operational ranges from a small uncrewed aircraft to a visual observer that are acceptable for EVLOS operations? What safety justifications can be used to support these ranges? What additional training is needed for EVLOS operations? 3 Research tasks described in the following sections addressed the research questions listed above. As previously mentioned, this research emphasized VO effectiveness, focusing on the human ability to detect and avoid intruder aircraft during multiple trials of an Extend Visual Line of Sight (EVLOS) flight operation. 3 RESEARCH TASKS AND FINDINGS The research team performed the following tasks for the A46 research effort. These tasks addressed research questions while generating deliverables that guided the research team. What follows is a description of each task, an overview of the findings, and a description of the conclusions and their relation to research goals. The research team omitted tasks that did not generate deliverables from this report. This report satisfies the requirements for A46 Task 6 – Final Report. 3.1 Task 1 – Literature Review and Market Analysis Task 1 consisted of a literature review. This literature review was essential for providing the research team with a background on the industry's current state regarding Uncrewed Aircraft Systems (UAS) VO visual acquisition and avoidance of potential collision hazards. Additionally, Task 1 provided an opportunity for the research team to identify critical gaps associated with the effectiveness of VOs in UAS operations. The following sections outline the approach, methodology, and findings from the literature review. This task informed future tasks while bounding the project's scope. 3.1.1 Task 1-1: Literature Review – Conclusions The team conducted a thorough literature review that explored multiple facets of VO/RP performance as stated in the research questions, visual acquisition, and avoidance of potential collision hazards, including avoidance of other aircraft, terrain, and obstacles. The primary purpose was to provide the FAA with a clear understanding of the following: 1. The current state of research on VO/RP visual acquisition and avoidance of other aircraft, terrain, and obstacles, 2. EVLOS operations, and 3. Operations in VLOS in which specific scenarios challenge visual conspicuity. The following sub-sections offer vital insights and key takeaways from the A46 literature review. These sub-sections outline conclusions from the literature that framed the scope for future tasking and identified areas where more work may help quantify VO/RP performance. Appendix A of this report provides the complete literature review associated with Task 1 of this research. 3.1.1.1 Human Factors Related to VO/RP Performance Human Visual System Limits The human visual system is limited by the following factors: blind spot, acuity threshold, accommodation of the eye, empty field myopia, and focal traps (FAA, 2016a). The human visual system, specifically visual scanning, can be affected by attention and response to traffic movement, refocusing eyes with and without switching views, eye movement, threat spotting, retinal 4 eccentricity, contrast threshold, small visual angle, visual obstructions, and visual search requirements. Detecting and recognizing an intruder aircraft, assessing its collision potential, making an avoidance decision, and initiating and completing an avoidance maneuver takes a pilot a minimum of 12.5 seconds, according to the FAA Advisory Circular 90-48D. Significant caveats must be placed on this metric when considering human performance (FAA, 2016b). The reference Advisory Circular does not describe any encounter geometry used to generate the timing of this performance metric. The Advisory Circular provides a general idea of reaction time, which may not be typical or conservative. The Advisory Circular breaks down the elements into segments, with five seconds attributed to the pilot becoming aware of the collision geometry and four seconds for deciding an evasive response. This lends itself to the Advisory Circular's comment that the concept of “see and avoid” alone is insufficient for collision avoidance. One must consider these important caveats when using the 12.5-second reaction time as a human observer performance baseline for Detect and Avoid (DAA) compliance. Human Visual Performance Models Woo et al. (2020) developed a new mathematical model based on rendered images of sUA. The model from Woo et al. (2020) indicated that sUA is typically not visible to crewed aircraft pilots with enough time to avoid a collision. Woo et al. (2020) noted that at the minimum distance that allows pilots enough time to react and avoid a collision, many sUA had rendered image sizes were near or below one arc-minute. The study established this as the lower limit for human visual acuity. Therefore, according to the model shown by Woo et al. (2020), the see and avoid principle is unreliable for sUA detection. According to this model, the two most important factors that affect the sUA detection are the size of the sUA (positively correlated with detection) and the speed of the crewed aircraft (negatively correlated). Similarly, Wallace et al. (2019) noted that the visibility of the UA dropped to fewer than ten arc-minutes when operated at over 400 feet in altitude. They also stated that apart from the largest sUA, most sUA were difficult to see for operations that exceeded 4,000 ft. sUA Visual Detection by VO/RPs Based on this literature review, VOs are poor at estimating distance and altitude correctly (Crognale, 2009) and are 2.5 times more likely to overestimate distances rather than underestimate distances (Vance et al., 2017). Detection rates and detection distances vary significantly across the studies evaluated as a component of this literature review. Typically, detection distances are higher for larger sUA due to a larger visible cross-section. However, size alone cannot predict the visibility of sUA. Color and background contrast also have a significant impact on the visibility of the vehicle. Key factors that could hinder visual detection include the sUA size, sun position, and any visual obstructions encountered by VOs. In general, the visibility of an aircraft (crewed and uncrewed) during daytime is determined by the aircraft’s physical size and contrast against the sky and clouds. In contrast, lighting systems establish visibility during night operations. Experience of VOs, or lack thereof, and corrected vision (20/20) may have a negligible impact on a VO's ability to detect sUA visually. 5 The Ability of VO/RPs to Avoid a Potential Collision According to the literature review, VOs are poor at estimating the collision likelihood of UA with surrounding traffic (Crognale, 2009). When evaluating the ability of VOs to assess the potential for a collision between a Boeing-Insitu Scan Eagle and an intruding aircraft, Crognale (2009) found that VOs cannot evaluate the likelihood of a collision unless they can see the crewed intruder aircraft and the UA at the same time. Crognale (2009) also found that without audible signals, Traffic Collision Avoidance System (TCAS), or radio announcements, visual detection by VOs is unlikely to contribute to collision avoidance in a significant way. VOs may significantly contribute to collision avoidance by utilizing trajectory estimation strategies, like tracking an object projected on a flat plane rather than maintaining a linear optical trajectory. Vance et al. (2017) found that based on the error intercept time with respect to the average intercept time, there is a significant risk of RPs not having enough time to avoid a collision. RPs could take longer than the 12.5-second estimate to follow the required procedures for collision avoidance. Last, Vance et al. (2017) also found that VOs perceive a worse collision potential than reality and overestimate the closure rate rather than underestimate it. 3.1.1.2 Factors Related to Aircraft Visual Conspicuity Lighting Systems Lighting systems make the aircraft more visible to other aircraft and to ground traffic. The increased visibility reduces the chances of collision. The two external lighting systems currently used on crewed aircraft are position and anticollision lights. Dolgov (2016) found that factors such as background illumination, presence of other lights in the background, low level of contrast between the UA equipped with lights and the background, and time required for the human eye to adapt to the dark adversely affect the ability of a VO/RP to detect anticollision lights. Dolgov (2016) also noted that lighting systems on aircraft (crewed and uncrewed) offer a contrast against overcast or dark skies, enabling VOs to track a sUA more efficiently during dusk and night conditions. Paint Schemes Hobbs (1991) suggests that a paint scheme that maximizes the contrast of the aircraft color with its background is more useful in increasing the visibility of the aircraft. However, the contrast of the aircraft against the background also depends on the background luminance. A light-colored aircraft is less visible against a light background on a dark day. However, increasing the background luminance will reduce the contrast of the aircraft with the background. The reduction, in contrast, will reduce the visibility of the aircraft. Considering these factors, Nelson et al. (2020) stated that paint on the top surface of the UA rather than the bottom surface provides a better contrast against the ground. Nelson et al. (2020) also found that fluorescent paint is more effective than no fluorescent paint in increasing the conspicuity of an aircraft and that adding reflective strips on larger UA may be beneficial in increasing UA detectability. 6 3.1.1.3 The Current VO/RP Training Paradigm There are no standardized training requirements for VO; however, many universities and institutions have their own training guidelines. While the number of categories covered and the depth of training by subject did vary, the Test Sites and university materials reviewed had central core topics such as airspace knowledge, Certificate of Authorization (COA) requirements, waivers, FAA requirements, and communication procedures. Many training programs reviewed by the research team detail topics not specific to VO tasks, such as site-specific information, including state and local regulations, wildlife interactions, and weather safety. • The top-level “Training Topics” included the following: o COA Requirements and Waivers o Federal Aviation Requirements (General Knowledge) o Federal Aviation Requirements (VO Specific) o Airspace Knowledge o Part 107 - Operating Limits o Part 101 – Moored Balloons, Kites, Amateur Rockets, Uncrewed Free Balloons, and Certain Model Aircraft. o Team Composition and Reporting o Responsibilities for Primary (Inside) Observer o Responsibilities for Secondary (Outside) Observer o Responsibilities for RPIC o VO Placement o Communications o Situational Awareness o UAS Observer Issues o Spatial Disorientation o Techniques o Emergency Procedures o Practical training/application – demonstrated knowledge and field demonstrations for training. o Site-specific knowledge and Safety Training o Other Implementing a demonstration of knowledge or practical skills, such as a successful test operation where the VO demonstrates their ability to ensure the separation of the UA from other aircraft, would be beneficial for determining whether a trainee can perform VO tasks successfully. 3.1.1.4 The Role of VO/RP in Testing of DAA Researchers investigated the VO and RP’s roles in testing DAA systems. These two roles are commonly part of all operations. The desire was to assess if there were any differences or changes 7 in their roles during DAA system testing. DAA system testing involves planned encounters with an intruder aircraft, which is more complex than just flight operations that look for and assess random air traffic that may or may not impact the UA flight. Planned encounters require another level of planning and safety to maintain safe separations and in-flight situational awareness. DAA testing involves planned encounters with piloted aircraft. During the mission-specific briefs, there is a discussion about the specific flight profiles. RPs and VOs are fully aware of the encounter set and the safe separation criteria designed into the test plan, individual test cards, and predetermined vertical and lateral offsets for safe separation to aid the VOs in their assigned duties. Mission planning highlights the communication and roles of the RP and VO as they are integral to safe operation. However, the roles of VO/RP are formally undefined beyond what is best practice for all flight testing. 3.2 Task 2 – Updated Research Task Plan The Research Task Plan (RTP) was a living document throughout this project. The RTP received updates as needed in coordination with the sponsor. The RTP informed modifications to the methodology as researchers developed and exercised initial test plans. The sponsor received the final draft of the RTP on February 1, 2023, capturing the methods used for the flight testing within associated tasks/subtasks. Appendix B of this report contains the RTP. 3.3 Task 3 – Initial Test and Analysis Task 3 consisted of six sub-tasks. The six sub-tasks included (1) the development of an experimental plan, (2) the development of a draft flight test plan, (3) the development of a draft data analysis plan, (4) flight test and data analysis peer review coordination meetings, (5) VO training development, and (6) an updated flight test plan. The findings from the subsequent literature review in Task 1 guided the initial flight test planning. 3.3.1 Task 3-1: Develop an Experimental Plan With clear definitions, an evaluation plan, identified variables, and data collection and processing procedures, the research team developed an experimental methodology to inform the initial flight testing and data analysis as a component of sub-tasks 3-3 and 3-5. The following sections summarize the key elements associated with the experimental plan. Below are a series of questions developed by the performing team to guide the flight test plan methodology and flight test campaign established from four of the research questions outlined in this report and captured in the RFP. Appendix C provides the finalized Experiment Plan. 1. What is the visual detection time for a ground-based visual observer? 2. At what point does the visual observer (a) see the intruder aircraft, (b) alert the RPIC, and (c) suggest an avoidance maneuver? 3. How many suggested avoidance maneuvers were accepted or rejected by the RPIC? 4. How long does it take for the RPIC to decide regarding the suggested avoidance maneuver? 5. How long does it take to complete the avoidance maneuver? 6. How does distance/angular distance and/or the associated portion of the field of view between the VO and the intruder aircraft affect the probability of detection? 7. Do ambient light levels affect detection performance? 8 • What may improve VO performance? This will be answered anecdotally based on input from subject matter experts. • What was the closest point of approach (CPA) for each trial (see Section 3.7 for the Data Analysis)? 3.3.1.1 Methodology and Experiment Design The research team utilized a sUAS flight test campaign in Kansas to collect data associated with this experimental design. This experimental design emphasized the human factors complexities of VO tasking during a sUAS EVLOS operation. Flight tests conducted afforded the research team the ability to measure specific dependent variables related to VO and RPIC performance, such as: 1. The time it takes for a participant to detect an intruder aircraft visually. 2. The distance at which a participant visually detects an intruder aircraft. 3. The time for the participant to decide whether an avoidance maneuver is required to ensure safe separation between aircraft. 4. The time it takes for the RPIC to initiate the maneuver. 5. The time it takes to complete the maneuver. The experimental design offered the benefit of directly addressing the research questions by: 1. Enabling the research team to evaluate the ability of VOs and RPs to detect conventional aircraft accurately. 2. Exploring the capacity for VOs and RPIC to use visual references for avoidance. 3. Identifying challenges associated with VO/RPIC communications. 4. Exploring the capacity of the RPIC to give way to the conventional aircraft. The experimental design allowed the research team to collect general information that may have impacted VO detection performance, such as ambient noise, light levels, and individual physiological differences related to visual acuity, color deficiency, and hearing capabilities. This information served as a baseline for considering future training criteria for VOs and RPICs. NMSU conducted a series of preliminary test runs of the experiment design in New Mexico before the final data collection flights in Kansas. This initial testing assessed personnel layout, data collection methods, flight path geometries, data gathering approaches, and other testing elements to ensure successful testing with participants in Kansas. The initial tests validated the test design and maximized data collection potential before conducting flight tests in Kansas. Additionally, lessons learned from the New Mexico flights were applied to the data collection flight test methods. A mixed methods design utilizing a triangulated approach was used for this research to capture both quantitative and qualitative data. Qualitative and Quantitative data were equally weighted during the data collection phase of this study. The primary advantage of implementing a mixed methods experimental design was to counterbalance the weaknesses presented by each method. For instance, this method leveraged the strengths of the qualitative data collection methods (e.g., data about the context) to offset the weakness of the quantitative data collection methods (e.g., ecological validity). Likewise, the strengths of the quantitative data collection methods (e.g., generalizability) offset the weaknesses associated with the qualitative data collection methods 9 (e.g., context dependence; Mills & Gay, 2012). As such, the research team validated this design as well suited to investigate the parameters associated with VO effectiveness. The rationale for choosing a mixed methods design stemmed from the notion that quantitative data gathered via the Cessna-172 and SR20 flight logs, ground control station (GCS), and in-situ environmental observations could be collected concurrently with the qualitative data from the VO participants. While the quantitative and qualitative data were gathered concurrently, the analyses were performed separately; after both analyses were complete, the team compared results to draw overall conclusions. 3.3.1.2 Validity and Reliability Maximizing internal validity and reliability was a crucial element during the experimental design phase of this research. Mills and Gay (2012) described internal validity as the degree to which observed differences in the dependent variable result solely from manipulating the independent variables and not from any uncontrolled extraneous variables. Of foremost importance was to collect data in a manner that was credible and reliable to minimize any threat to the internal validity of this experiment (Mills & Gay, 2012). Thus, instituting standardized procedures, establishing a fixed location, and ensuring consistent data collection methods enhanced the internal validity of this methodology. This experimental design used a double-blind procedure to enhance internal validity. The VO participants, RPIC staff, and researchers did not know the chosen flight path for each experimental run. Only the C-172 pilot and an "Air Boss" on the ground were aware of the selected flight paths. Double-blind procedures helped to minimize the possible effects of participant and research bias (Frey, 2018). In turn, enhancing the study's internal validity also enhanced the external validity of the results, allowing the results to serve as a baseline representation of the types of data required when evaluating VO and RPIC performance. Reliability refers to the degree to which a test (or qualitative research data) consistently measures whatever it measures and includes both the instruments and tests and the techniques being used to collect data (Mills & Gay, 2012). To enhance the reliability of this study, experienced RPICs with FAA Remote Pilot Certificates and previous flight experience were utilized to evaluate VO participant performance. The construction, planning, and testing of all instruments were also documented and described, and the researcher's relationship with the groups and setting was detailed and captured. 3.3.1.3 Limitations and Assumptions The research team considered seven limitations and seven assumptions when developing the methodology for the flight test campaign. These limitations and assumptions resulted from (1) individual human performance limitations, (2) a limited participant pool, and (3) challenges associated with generalizing results across a wide variety of unique or novel sUAS use cases. Limitation 1: Human performance could vary significantly between individuals/participants. Limitation 2: Since participants had little to no VO experience, conclusions about VO performance may not translate to VOs with more experience. 10 Limitation 3: Kansas State University Salina has a smaller campus population, limiting the number of participants. Limitation 4: The brief time available for this task was limited to the ability to recruit and schedule participants. Limitation 5: Flight tests simulated a fixed-wing hybrid VTOL sUA; thus, the time it took for this system to initiate and complete an avoidance maneuver varied compared to other sUA platforms. Therefore, the results may not translate to operations using other sUA platforms. Limitation 6: BVLOS flight operations used for this experiment were limited to daytime operations under Visual Meteorological conditions (VMC). The results from this study may not apply to operations conducted under different conditions, such as nighttime operations. Limitation 7: The data collected for the time it took a participant to see an intruder aircraft may be skewed because as the variability in time increases across participants, the data will become positively skewed. The following assumptions drove the experimental design and flight test campaign. Assumption 1: Experienced VOs would perform differently. What constituted an experienced VO was undefined by previous literature. For the sake of this research, a VO was grouped into the experienced category if they had an FAA Remote Pilot Certification and/or any Private, Commercial or Airline Transport certification. If a VO had no prior aviation experience they were categorized as having no experience. Assumption 2: Trained and experienced RPICs would be better at confirming if the suggested VO avoidance maneuver was the best option. Assumption 3: The VO training provided to the participants was an adequate amount of information to help them understand the roles and responsibilities of a VO during a sUAS operation. Assumption 4: Participants would follow the processes given during the VO training, including using correct callouts and the specified scanning techniques. Assumption 5: The participants would not look to their fellow participants for clues about where the intruder aircraft is located. Assumption 6: Participants would follow instructions and not yell their callouts and/or not use hand gestures to identify where the intruder aircraft is in the airspace. Assumption 7: The RPICs and the research administrators would not scan the airspace and give inadvertent clues to the participants as to where the intruder aircraft is located. 3.3.2 Task 3-2: Initial Flight Test Plan The New Mexico State University Unmanned Aircraft Systems Test Site (NMSU UASTS) developed and documented an initial flight test plan before initial flight testing. This document was titled "A46 Visual Observer Assessment Test Plan" and dated May 9, 2022. The research team previously provided a copy of this initial flight test plan to the FAA, attached in Appendix D. Items discussed in the Initial flight test plan and the general methodology are shown below: 11 1. Introduction 1.1 Project Overview 1.2 Scope of Testing 2. Test Architecture 2.1 Delayed Assets 3. Aircraft and Support Equipment 4. Flight Locations 5. Encounter Geometrics 6. Success Criteria 7. Participants and Roles 8. Schedule 8.1 General Flight Day 9. Data Management Plan 9.1Metadata Spreadsheet 9.2 Test Card Data 9.3 Flight Data 9.4 VO Data and Other Test Data 10. Flight Day Communications Plan A few items in this initial test plan were worth highlighting to provide basis, background, and clarity. This includes information on the intruder aircraft, test location, and various encounter geometries. The NMSU UASTS utilized a Flight Design CTLS, owned and operated by NMSU, as the intruder aircraft. Table 1 provides the CLTS aircraft information card. Table 1. Flight Design CTLS. The Flight Design CTLS is a two-seater Light Sport Aircraft designed around the FAA LSA regulation. It is all composite construction and uses a Rotax 90HP engine. The CTLS served as the intruder aircraft during the flights at NMSU. Wingspan 28ft 8 in Cruise Speed 80-90 knots Maximum Takeoff Weight 1,600 pounds UAS Operator NMSU Fuel Capacity 34 gallons Crewed aircraft took off from Las Cruces International Airport (KLRU). The test area was ~18 NM Northeast of Las Cruces International Airport (Figure 1). Flight maneuvers took place away from the airport to minimize the impact on general aviation operations. The area is sparsely populated, and all maneuvering of the crewed aircraft was performed at or above 500 ft Above Ground Level (AGL). The simulated sUA in this testing maneuvered at or below 400 ft AGL. To 12 maintain safety, the crewed aircraft involved in the operation had ADS-B in/out installed so that all pilots had high situational awareness. The VO test subjects were located at the Jonada Range. Figure 1. Wide View of the NMSU Testing Area. 13 Figure 2. Bird-eye view of the operational location for the NMSU A46 event. The encounter geometries provided are referred to as "wagon wheel" crossings. The encounters were laid out around the compass rose centered where the VOs were situated (Figure 2). Encounters started at a distance beyond the visual line of sight of the VOs. Nominally, for these tests, the encounters started at 5 miles. The intruder came in from one of the depicted starting points. The sUA flew various patterns within a .25 nautical mile (NM) radius of the bullseye. The intruder then exited this location by aligning along the defined exit vector. The exit direction was aligned to minimize the flight time outside the 5-mile diameter ring. Each subsequent intruder crossing was along a different random "spoke of the wheel." Dedicated visual observers were posted on the ground and in the crewed aircraft to maintain safety. For more details on each encounter, please see the associated test cards. Appendix E presented the finalized test cards prepared for the New Mexico flights. As noted above, the initial flight test plan was developed for the first round of testing conducted in New Mexico. KSU used the lessons learned from this testing when developing the flight test plan for Task 4. 3.3.3 Task 3-3: Draft Data Analysis Plan This section summarizes key elements of the data analysis plan, which is a part of the experiment plan provided in Appendix C. The research team identified the main dependent and independent variables for the initial flight testing. The primary dependent variable was the intruder detection distance. The finalized variables are listed and discussed in Section 3.4.5.1 of this report. The proposed list of data collected in the experiment is consistent with the previous work done at New Mexico State University’s UAS test site (Dolgov, 2016 & Dolgov et al., 2012). The data 14 collection initially proposed in the data analysis plan was slightly altered before the start of flight testing. The finalized data collected before and during flight tests is listed below: • Test Site Information o Test Area Dimensions, Dates and Times of Experiments, Latitude and Longitude coordinates of the test area o Test Site Weather (Temperature, Dew Point, Barometric Pressure, Wind Speed and Direction, Rainfall Rate) o Test Site Ambient Light Level o Test Site Ambient Noise Level • VO Information o Background o Aviation Experience o Visual Acuity and Hearing Levels o Fatigue Levels • Audio and Video Recordings o VO and Remote Pilot in Command (RPIC) Communications o Video recording of the Ground Control Station (GCS) areas o Photographs (GCS and test area, test setup, crewed aircraft) • VO-Specific Responses o Acknowledgment of visual acquisition of intruder aircraft o The suggestion of a maneuver to RPIC if a potential collision is identified. • Crewed and Uncrewed Aircraft Information and Global Positioning System (GPS) data: o Latitude, Longitude, Altitude, Speed, Heading at a minimum frequency of 1 Hz. Wichita State University developed scripts and algorithms to process the raw data, compute the quantitative parameters of interest, and perform graphical and statistical analysis on the quantitative parameters. The steps involved in the processing and analysis of the data are listed below: • Obtain the intruder aircraft GPS and flight data from the Garmin-1000, ForeFlight, or GPS Puck logs. • Obtain the sUA GPS and flight data from the Mission Planner software logs. • Identify the start and end times of useful “runs” based on a 4.5-mile distance between the intruder aircraft and VO locations. • Compute the horizontal and vertical distances between Intruder aircraft and UA, Intruder aircraft and VO locations, sUA and VO locations (Distances are based on geodesic arc lengths between points defined on the WGS84 ellipsoid). • Compute the unmitigated flight path for the sUA. • Compute the unmitigated and mitigated CPA and corresponding horizontal and vertical miss distances: o CPA occurs at the smallest slant distance between the intruder aircraft and sUA for a given run. o Slant distance is the Euclidean distance between the intruder aircraft and sUA. 15 • Determine Well Clear (2000 ft x 250 ft) and near mid-air collision (NMAC) (500 ft x 100 ft) violations for both mitigated and unmitigated flight test encounters. • Determine the intruder's distance, altitude, speed, and heading from the GPS data during VO detection. • Determine the ambient light and noise levels during VO detection. • Determine the response times to initiate and complete an avoidance maneuver when performed. • Compute the intruder's visual angle and angular visual area at the time of VO detection. • Compute descriptive statistics and percentile distribution curves for the relevant dependent variables. • Compute statistical relationships and correlations between the relevant dependent and independent variables using common methods like Analysis of Variance (ANOVA), Regression model fit, and Spearman’s correlation test. The ANOVA analysis used in this study was the one-way method. A one-way ANOVA analysis determines whether different groups of a single independent variable affect a dependent variable differently. The ANOVA analysis is useful for both numeric and categorical independent variables. The Spearman correlation test is a bivariate analysis that measures the strength and direction of a linear relationship between two variables. Positive values of the correlation coefficient close to 1 indicate a strong positive relationship, and negative values relative to -1 indicate a strong negative relationship. The Spearman correlation test was preferred in this study over the Pearson correlation test (one of the most widely used correlation statistics) since the Pearson test assumes both variables to be normally distributed. The Spearman correlation is a non-parametric statistic with no requirement of normality. The data processing and statistical analysis methodologies described in this section are consistent with several of the VO performance-related experiments mentioned in the literature review (Dolgov, 2016; K. W. Li et al., 2019a., K. W. Li et al., 2019b, Li et al., 2020; Vance et al., 2017; Woo et al., 2020). 3.3.4 Task 3-4: Test and Data Analysis Plan Review Task 3-4 consisted of the data analysis plan discussed in Section 3.3.4. The detailed Data Analysis Plan can be found in Appendix C. 3.3.5 Task 3-5: VO Training PowerPoints Task 3-5 involved creating VO Training PowerPoint slides intended to train the participants on their role of the VO during the flight testing. The team created two training versions while building the requisite PowerPoint presentations; the first focuses on training the VO participants on the topics most applicable to the flight testing in a condensed 30-minute training. The second version is an extended version of the training provided to the FAA with a PowerPoint that encompassed the topics needed to train a VO for safe EVLOS operation. Appendix F provides the PowerPoint slides for the condensed version, and Appendix G provides the extended version. The performing team used four different training courses to identify the most pertinent topics needed for this training: (1) New Mexico State University's FAA UAS Flight Test Site Training, 16 (2) the Alaska Center for Unmanned Aircraft Systems Integration Training, (3) Kansas State University Salina's Applied Aviation Research Center Training, and (4) the Public Safety Unmanned Response Team Training. This section offers an overview of the topics included in the training PowerPoints. • Federal Aviation Requirements (General Knowledge) o FAR § 107.3 – Definitions o FAR § 107.31 – Visual Line of Sight Aircraft Operation o FAR § 107.37 – Operation Near Aircraft; Right of Way Rules o FAR § 107.39 – Operation Over Human Beings o FAR § 107.51 – UAS Operating Limits o FAR § 107.17 – Medical Conditions o FAR § 107.23 – Hazardous Operations o FAR § 107.27 – Alcohol and Drugs o Federal Aviation requirements (VO Specific) • Airspace Knowledge o FAR § 107.33 – Visual Observer o FAR § 107.33a – Effective Communication o FAR § 107.33b – See the aircraft throughout the flight and accurately determine UAS altitude and direction. o FAR § 107.33c – Coordination • UAS Part107 Operating Limitations o Operating Requirements • Team Composition and Reporting o Definition of mission support teams' roles and responsibilities  Remote Pilot in Command (RPIC)  Flight Team (VO, Team Leader, Air Boss) o Defined reporting structure o Responsibilities of RPIC  Clearly define the roles and responsibilities of the entire support team • Responsibilities for Primary Observer o Deployed at launch/landing site o UAS Tracking o Late-game collision avoidance o External pilot assistance o Interface between VO and other personnel o Interference with non-participants • VO Placement o Geography • Communications o Hand-held radios o Call signs o Observer to pilot  Aircraft tracking information  Maneuver recommendations 17 o Pilot to Observer  Heads up for inbound traffic  No factor aircraft calls  Communication Procedures o Phraseology  Object name "Intruder aircraft."  Object heading "Cardinal direction."  Object heading relative to the UA (toward, away, parallel, etc.)  Object altitude relative to the UA (high, co-altitude, low altitude)  Maneuver (maintain, climb, descent, turn)  Example callouts: • "Intruder aircraft, headed south, toward UA, high, maintain." • "Intruder aircraft, headed southeast, toward UA, co-altitude, recommended descent." • "Intruder aircraft, headed southeast, toward UA, low altitude, recommend climb." o Communication standards (phonetic alphabet, figures/numbers, altitudes and flight levels, direction, speed, time) o Emergency terminology • Situational Awareness o Know your directions. • UAS Observer Issues o Size and orientation of the UA o Paint schemes and lights o Engine noise (or lack of) o Environmental and terrain effects  Sun, clouds, haze, dust  Mountains in the background o Accurate altitude and distance estimates for non-participating aircraft • Spatial Disorientation o Visual Illusions o Autokinesis o Flicker Vertigo o False Perceptions o False Horizons o Lost Horizons o Black Hole Syndrome/ Black Hole Approach • Techniques o Scanning Technique 1  10-degree sectors through the area of responsibility  Horizon to operating altitude. o Engine noise may be the first indication. o Compass Use  N, S, E, W, not "left" and "right." 18  Give bearing from our location. • Emergency Procedures o Follow "Air Boss's" Instructions 3.3.6 Task 3-6: Completed Initial Flight Test and Data Collection Task 3-6 consisted of the final round of initial flight testing. The conclusion of initial flight testing and data collection informed the methodology for subsequent flight test activities and was derived from lessons learned in Task 2. NMSU conducted two rounds of flight testing as outlined above. The sections below discuss the initial flight testing completed by New Mexico State University and the subsequent flight testing by Kansas State University. 3.3.6.1 NMSU Flight Testing An initial round of flight tests took place at the Jornada Experimental Range (JER) near Las Cruces, NM, from September 21 to 23, 2022. This first round of testing ran through the planning, training, and execution of the designed test protocols. Eight VO test subjects were part of the testing over three days. The objectives were to gather data and validate test protocols. The three sections below cover the flight's set-up and execution, collected data, and the lessons learned. 3.3.6.2 NMSU Flights and Data Collection Tasks 3-4 of the proposed work required NMSU to complete initial flight testing and data collection. All tests had a similar plan, timing, and set-up procedure. Four static locations were set up. The team established a central command and control location where the test subjects were given their VO training, shown in Figure 3. All test subject locations were approximately 200 feet from this central location. The other three static locations were located in the East, North, and South of the central location, as shown in Figures 4, 5, and 6. Figure 7 presents images of the VO location setup. Figures 8, 9, 10, and 11 present broader views of the testing area. One can see clearly that the test area is flat and offers few obstructions, maximizing visibility. The images below were from Wednesday, September 21, 2022. 19 Figure 3. NMSU Visual Observer Training in the Field. 20 Figure 4. Visual Observer East Location. 21 Figure 5. Visual Observer South Location. Figure 6. Visual Observer North Location. 22 Figure 7. Visual Observer Location Set-Up and Reference Notes. Figure 8. View From VO North Location. The east location is to the left of the image, and the Central and South Locations are visible to the right. 23 Figure 9. sUAS Pilot and VO During the Testing and Data Collection. Figure 10. View to the Northeast. Figure 11. View to the Northwest. 24 The intruder for these tests was a CTLS Light Sport aircraft. The NMSU team employed four systems for collecting the intruder aircraft's GPS coordinates to ensure accuracy. These systems consisted of two self-contained GPS Pucks, GPS data from a device onboard the CTLS running the ForeFlight application, and a Garmin GPS recording device. Figure 12 below shows the approximate location of the VO's during the testing. All VOs were approximately 200-230 feet away from each other. The approximate GPS positions of the VO test subjects were as follows: North VO: 32.596904° -106.740488° East VO: 32.596735° -106.739860° South VO: 32.596287° -106.740274° Figure 12. VO locations are marked with yellow pins (all are near the center crossing of the dirt runway). Flight plots for each day are shown below in Figures 13 through Figure 21. There were three flight images for each day of flight testing. The first image in each set shows the entire flight of the intruder with the aircraft taking off from the Las Cruces International Airport. Figure 14 presents an overview of the crossing patterns, and Figure 15 depicts a close-up view of the crossing area to show the dispersion of the flights and just how close the aircraft approached the center of the target area. The red line to the right in the images represents the boundary for the White Sands Missile Range restricted airspace where the intruder aircraft could not fly. That is why all the approaches turned before reaching this limit. 25 Figure 13. Full Flight Path for September 21, 2022. Figure 14. Flight Path Crossing for September 21, 2022 26 Figure 15. Close-up view of the flight path crossings on September 21, 2022. Figure 16. Full Flight Path for September 22, 2022. 27 Figure 17. Flight Path Crossings on September 22, 2022. Figure 18. Close-up View of the Flight Path Crossings on September 22, 2022. 28 Figure 19. Full Flight Path for September 23, 2022. Figure 20. Flight Path Crossings on September 23, 2022. 29 Figure 21. Close-up View of the Flight Path Crossings on September 23, 2022. Over the three days of testing, there were a total of 143 runs, which were broken down as follows: • 9/21: 3 subjects, 18 runs each = 54 potential data points • 9/22: 2 subjects, 19 runs each = 38 potential data points • 9/23: 3 subjects, 17 runs each = 51 potential data points The initial runs were set with a start point of 5 miles out, but it was clear that the aircraft could not be seen at this distance. The starting point was moved to 4 miles out. 3.3.6.3 NMSU VO Subject Data Collection A total of eight ground-based VOs served as participants in the initial flight testing at NMSU. Each operating station included an RPIC, a researcher, and a single ground-based VO. Table 2 shows the participants’ IDs, visual acuity, aviation experience, and fatigue levels. 30 Table 2. NMSU Subject Data Participant ID Snellen Test (Best Eye) Aviation Experience Fatigue Level D1P1 20/20 None Well-rested D1P2 20/20 None Well-rested D1P3 20/30 None Well-rested D2P1 20/20 None Well-rested D2P2 20/25 None Neutral D3P1 20/30 None Neutral D3P2 20/20 None Well-rested D3P3 20/20 None Well-rested 3.3.6.4 NMSU Flight and Testing Lessons Learned The NMSU team consolidated their initial notes and lessons learned from this testing event and provided them to the KSU team before subsequent flight testing occurred. The notes provided below are unfiltered items collected during post-testing. The entire test team provided inputs to the list below. Items listed run the gamut of recommendations, ranging from specific actions to general impressions and feelings arising from testing. Researchers did not attempt to resolve conflicting comments, thoughts, or impressions. The feedback and notes sometimes represented points that contrasted with the overall testing plan. The listing below represents the raw feedback from the NMSU team. Some items are germane to the New Mexico testing, and some are to the overall effort. All the following points were reviewed and discussed before the round of testing at KSU. If necessary, KSU revised the test planning and execution. The items below are in no specific order and are grouped by topic area. • Logistics o Participants met the research team at a local business and followed the research team to the testing site, guaranteeing no participants got lost. o All participants and research team members were encouraged to use the facilities before entering the field, as no restroom facilities were on site. o The drive from the local business to the testing site was approximately 30 minutes. • Setup o Generators used in the field should be placed far enough away from the group/lecture/presentation area to not interfere with the talks/discussions. o Take photos of all areas for the test setup for reference - staged with team personnel and not with participants. 31 o A dedicated weather station on site for the sound and light meters is essential to ensure a sterile data collection environment. o Copies of the desired callouts were taped to each table by the VOs – Note: several subjects asked for a compass to help with cardinal directions. Thus, a picture of a compass was taped to the VO tables for the KSU flight testing. o Flight simulations were set for a takeoff point at each VO location. o In the future, and in the interest of the flight crew’s safety, do not set up a monitoring station in the middle of the east-west runway. If the intruder aircraft had a problem at the low altitudes they were flying, that would be the emergency runway. The aircraft may not always be within gliding distance of JER, but they were for a good part of the time. • VO Briefing/Training o The VO training was difficult to conduct outside because of noise and glare on the computer screen, making the PowerPoint hard to see. Based on this observation, KSU presented the VO training PowerPoint inside their sUAS command trailer. o The initial VO brief took ~30 minutes. o Specific notes from the briefer who did all the VO training:  An individual with no experience will struggle with the PowerPoint presentation. It lacked materials for them to be successful.  Provide an array of scanning options for detecting aircraft. The slides suggest there are more options, so they should be provided.  Provide deviations in communication from VO to RPIC. There is no singular way to articulate an intruder aircraft.  Include an order of precedence if an intruder aircraft goes unnoticed in a rapid communications process. "Drop altitude NOW, Intruder aircraft." When the aircraft is safe, then relay additional information.  When the VO has no experience, specific steps in communication should be listed as optional. The main information that needs to be divulged is stating intruder, relative location, heading towards/away/no factor, and altitude.  When the VO has no experience, how can you expect them to recommend a maneuver accurately? The suggested maneuver is a dangerous metric to measure by. It is simulated, but putting this idea into a beginner's mind could form a hazardous habit.  When the VO has no experience, describing the altitude of an aircraft has no real meaning. They do not know what 500ft above ground looks like. o Provide a large sample of photos, video, and audio that characterizes an intruder aircraft.  A low-flying aircraft flying directly towards the VO. What does the profile of the plane look like? What does an aircraft sound like at 500 ft? What does a plane look like at 500ft, 1000ft, 1500ft, etc.? o Provide more practice slides or quizzes with actual videos of intruder aircraft. This can be a video of the open sky spanning several minutes to show what "normal" looks like while training. 32 o Keep other support personnel and radio traffic away from the briefing area and impose a buffer zone. Based on these observations by NMSU, KSU adjusted the training slides to include more practice slides for VOs. The research team member presenting the slides also quizzed each participant multiple times on these slides to gauge whether they had a grasp of the information presented. The VO trainer also utilized a local cell tower and power lines to demonstrate different heights. The local power lines are approximately 16 feet tall, whereas the cell tower is approximately 120 feet tall. While this was not the most pertinent comparison, it allowed the participants to attain a real-world view of what 120 feet looked like so they could grasp what the sUA operational height of 400 feet looked like. • Communications o Utilizing two sets of radios allows for (1) communication linking all ground personnel and (2) communication linking the test director to the intruder aircraft. The VO teams could not hear the communication between the test director and the intruder aircraft. o For future testing, a better practice is not to have individuals carrying both sets of radios to ensure that aircraft radio calls do not get out to the VOs and ground team (and vice versa). We had no issues but realized splitting the two radio roles was best. o Each VO test location needs a unique identifier – NMSU used North, South, and East. • Flight Test Execution o Test operational planning requires the test team to go through an entire practice session to ensure they know how to start the tests, record data, end the test, store test materials, etc. Practicing these procedures multiple times before the first test day ensured streamlined data collection. o The test cards had to be modified in real time due to the shorter test subject time. All flights were at 120 meters (~394 feet) AGL. o Timestamps were called out on the radio at 15-minute intervals and were captured on the three computers recording data in attempts to ease post-processing. o Aircraft range after crossing:  The aircraft were sent to 5 nm out for the first two crossings.  Team leads assessed the loss of aircraft visibility at ~3.5 to 4 miles out.  After the first two runs, the distance was changed to 4 miles out. The shorter distances decreased the cycle time between runs.  Post-testing, the VOs said they could not see the aircraft after each run except in one run by one person at a specific orientation. It appears that for our testing with this specific aircraft, 4 miles out was sufficient for a reset for the subsequent runs under these conditions.  See Table 3; the average time for the 4 miles out runs was ~6 minutes between passes. o During participant debriefing, participants voiced that they knew the sUA was a simulation and that it was easy to recognize the pattern as the aircraft was flying 33 at a constant altitude. Future testing would benefit from deviating the altitude during testing. o Deviate the flight paths to have offset flybys. An example of alternating the wagon wheels' center points is shown in Figure 23. o Once the participants realized the aircraft would fly only to this fixed point at a certain altitude, it created a high error margin. o Conceptually, this does not simulate normalcy. The crewed aircraft incursions typically do not occur back-to-back. The VO is on edge, specifically watching for a specific intruder aircraft, which does not reflect typical operations. o For example, runs on Wednesday, September 21, 2022, cross-over point to the center of the test area – aircraft directly overhead, approximate times between crossings. This "cycle time" meant approximately ten runs per hour. Table 3. CLTS Run Time for September 21, 2022 Run Time of Day Distance Time Between Runs Run 1 2:22 PM Run 2 2:32 PM 5 miles out 0:10 minutes Run 3 2:37 PM 4 miles out 0:05 minutes Run 4 2:42 PM 4 miles out 0:05 minutes Run 5 2:47 PM 4 miles out 0:05 minutes Run 6 2:54 PM 4 miles out 0:07 minutes Run 7 2:59 PM 4 miles out 0:05 minutes Run 8 3:01 PM 4 miles out 0:02 minutes Run 9 3:13 PM 4 miles out 0:12 minutes Run 10 3:19 PM 4 miles out 0:06 minutes Run 11 3:24 PM 4 miles out 0:05 minutes Run 12 3:30 PM 4 miles out 0:06 minutes Run 13 3:36 PM 4 miles out 0:06 minutes Run 14 3:44 PM 4 miles out 0:08 minutes Run 15 3:51 PM 4 miles out 0:07 minutes Run 16 3:57 PM 4 miles out 0:06 minutes Run 17 4:03 PM 4 miles out 0:06 minutes Run 18 4:08 PM 4 miles out 0:05 minutes Average Run Time 0:06 minutes A rough sketch of the approach orientations from the 18 runs completed on September 21 is detailed below in Figure 22. Figure 22 shows how random the approaches were; subsequent days were equally random. 34 Figure 22. Example of the Random Intruder Approaches on Day 1 of NMSU testing. • Post Testing o Closedown procedures for post-flight are essential to ensuring all recording devices are stopped and all data is captured. o A crewed aircraft is about 75 dB(A) when flying overhead. o Lay the lux meter and sound level meter on a flat surface away from other structures and noise sources. Use the Slow, Low Range, dB(A) settings on the sound level meter. o Put a blank for the time on the post-run RPIC questionnaires. This will help to identify associations between responses and given runs. o Work with the RPICs to standardize interactions with the subjects and their responses on the post-run questionnaires as much as possible. o Consider completing the VO training immediately before subjects complete the flight tests. They will forget some of the details otherwise. o Pin a mic to the subject to ensure you get a clear audio recording of their responses. Participants will not always be standing near the RPIC's laptop when making responses, so recording audio with the laptop's mic is not always ideal. • Perceptual Testing o The hearing screening is initiated at 30 dB, with octave frequencies from 250 to 8000 Hz. If the subject has trouble with an ear/frequency combination at 30, boost in 5 dB steps until they can hear it, then record the stimulus level. Start with the right ear, then do the left. Subjects are seated in a sound-attenuated booth with the door open. o A quiet, indoor facility is necessary for hearing and vision testing. At NMSU, the ambient noise in the sound-attenuated booth with an open door is 19 dB(A). o The Snellen chart was used, with subjects located 20 feet away. o The Ishihara 38-plate booklet was utilized to determine color deficiency. Subjects went through the booklet and read out the numbers they saw on the plates. • Post-test, additional inputs from participants were collected. o VOs asked when to reset as the intruder aircraft turned near the test area. N 1 5 3 2 4 N 10 7 8 9 N 16 14 15 6 11 12 18 17 13 35 o VOs stated they could not predict the incoming direction. Restricted airspace did help them narrow this down. o Subjects said they always heard the aircraft before they saw it, but they only responded when they saw the aircraft. o Many participants were confused about the altitude part of the response. Participants said the aircraft's altitude was low relative to where airplanes usually are, so they always responded low. o Subjects said that based on the VO Training, they knew the UA should never be above the crewed aircraft; thus, the command prompt "climb" did not apply to the operation. o The tear-down for flight testing took 30 minutes. Figure 23. Potential Alternative Intruder Flight Paths. 3.3.6.5 NMSU Flight Test Results This section provides results for the intruder detection distance, intruder speed at detection, and intruder position at detection for the initial flight tests. Table 4 provides the descriptive statistics for intruder detection distance calculated for the NMSU flight test encounters with the CTLS intruder aircraft. The detection distances were computed for all 143 runs. The VOs detected the CTLS intruder aircraft at an average distance of 1.14 miles. This is an improvement over the average visual detection distance of 0.79 miles obtained by Dolgov (2016). In his study, Dolgov (2016) obtained visual acquisition distances of the CTLS aircraft in daytime conditions with 3 participants acting as VOs for 67 trials. 36 Table 4. Descriptive Statistics for Intruder Detection Distance Calculated for the NMSU Flight Test Encounters. Intruder Detection Distance (miles) Intruder Aircraft: CTLS, Sample Size = 143 runs Min. Max. Mean Median Std. Deviation 0.05 3.49 1.14 1.06 0.50 Figure 24 shows the percentile distribution for intruder detection distance calculated for the NMSU encounters with the CTLS intruder aircraft. The VOs detected the intruder aircraft at a distance of at least 1 mile in 57.1% of the runs, at least 2 miles in 5.6% of the runs, and at least 3 miles in 0.9% of the runs. Figure 24. Percentile Distribution for Intruder Detection Distance Calculated for the NMSU Flight Test Encounters. Table 5 provides the descriptive statistics for intruder speed at detection for the CTLS intruder aircraft used in the NMSU flight tests. The intruder aircraft speed was only available for 127 out of 143 runs in the flight data logs. The average speed at detection for the CTLS intruder aircraft was 95.8 kts. 37 Table 5. Descriptive Statistics for Intruder Speed at Detection for the NMSU Flight Test Encounters. Intruder Speed at Detection [kts] Intruder Aircraft: CTLS, Sample Size =127 runs Min. Max. Mean Median Std. Deviation 68.2 113.6 95.8 97.7 9.2 Figure 25 shows the positions of the intruder aircraft at detection, VO stations, and the mission flight path for the NMSU flight test encounters. The center of the wagon wheel for the intruder aircraft tracks was directly above the VO locations. Figure 25 shows that most intruder detections were evenly distributed in all directions except in the east direction. Figure 25. Positions of the Intruder Aircraft at Detection on a Map for the NMSU Flight Test Encounters. 3.3.7 Analysis of Aircraft Projected Area for NMSU and KSU Aircraft Researchers used three different intruder aircraft between two rounds of VO testing. These were a CTLS Light Sport, a Cessna 172 Skyhawk, and a Cirrus Design SR20. These aircraft are distinctly different in size and shape, resulting in perceptible differences by observers. Since the size of the aircraft differed from every viewing angle, researchers attempted to provide general approximations of the shape, profile, and projected area an observer may see. Approximations accounted for the projected area when viewing the aircraft from the front, top, and side. These approximations are not intended to be exact. Still, they provided a relative value to assess how much the aircraft may fill the observers’ field of view at different ranges (i.e., subtended arc within the field of view). 38 The approach used to estimate the projected areas is one that the research team has used in the past for estimating balloon flight payload impact areas and safety analysis. There are several methods to do this, and three different methods have been used for previous analyses. The three different approaches are "Blocks, Bubbles, and Bulk" techniques, all requiring a reference measurement. • Blocks – estimation by grid overlay (what was used here). • Bubbles – tracing the entire area in a computer-aided design program and looking at the resulting area of the generated shape. There are challenges with this approach based on the image quality and optics used to create the images (perspective and depth). • Bulk – also called the "paper doll" method, where one prints two sheets of paper with the image. After carefully cutting out the image on one sheet, one weighs the entire sheet and the "paper doll cut out." This can be converted into an area based on the mass ratio. The block grid estimation is appropriate for this analysis. Previous efforts have shown that results using all three approaches are comparable. The block area approach allows one to go back easily and further estimate component areas (e.g., wing area from a top-down view or fuselage area only from the front if you eliminate the wing area). One must count the blocks for the desired area towards a mathematical conclusion. A simple description of the approach used was to (1) find a graphic of aircraft with measurements; (2) place on a grid overlay of known dimensions of aircraft for all three orientations (front, side, and top); (3) identify which blocks were at least 50% overlay of the aircraft; (4) count the number of blocks; and (5) calculate. The assessments of the projected areas of NMSU's CTLS from the side view are shown below in Figures 26 to 30. The projected areas for the CTLS from the side and top view and the C-172 and SR20 from the top, side, and front angle can be found in Appendix H. 39 Figure 26. NMSU's CTLS Light Sport Aircraft. Figure 27. CTLS Shown from Different Angles. http://flightdesignusa.com/wp -content/gallery/specifications/ctls -lsa-inch -2.jpg NMSU’s CTLS Light Sport Aircraft http://flightdesignusa.com/wp -content/gallery/specifications/ctls -lsa-inch -2.jpg d area of installed tem not accounted n calculations 40 Figure 28. Side View of CTLS Used for Projected Area Assessment. Figure 29. Grid Overlay of Side View of the CTLS. 41 Figure 30. CTLS Side View Grid Overlay with Estimated Area. Each Block is 2.7 inches square or 7.34in2. There are 1,126 blocks. The estimated projected areas in both inches squared and feet squared were made using the estimated block sizes for each view and counting the total number of blocks. Further estimates were made for the wings, body, tail, and landing gear for the front profile. This was done because the wings are long, thin, and often not as easy to see when viewing the aircraft head-on. The aircraft's body section presents the most straightforward element to see visually. Table 6 summarizes the estimated projected areas for all three aircraft. 42 Table 6. Estimated Projected Area for the CTLS, Cessna 172 Skyhawk, and Cirrus Design SR20 Aircraft. These estimated projected areas serve only as general references. Observers seldom see an aircraft precisely from these perspectives. The actual view from the observer will be a blend of these perspectives. The size difference between the three aircraft is also noteworthy. The CTLS is smaller than the Cessna 172 Skyhawk and the Cirrus SR20. The Cessna 172 Skyhawk and the Cirrus SR20 are similar in size. A comparison of the aircraft sizes based on projected areas in Table 5 considers the three perspectives as follows: Aircraft View Section Block side dimension (in) Area of each Block (in^2) Number of Blocks Area (in^2) Area (ft^2) Side N/A 2.71 7.34 1126 8,269.46 57.43 Front All 2.91 8.47 762 6,452.69 44.81 Wings 305 2,582.77 17.94 Body 327 2,769.07 19.23 Tail 44 372.60 2.59 Landing Gear 86 728.26 5.06 Top 5.64 31.81 684 21,757.77 151.10 Side 3.684 13.57 1157 15,702.64 109.05 Front All 3.646 13.29 652 8,667.24 60.19 Wings and struts 318 4,227.27 29.36 Body 214 2,844.77 19.76 Tail 59 784.31 5.45 Landing Gear 61 810.89 5.63 Top 6.261 39.20 984 38,572.92 267.87 Side 5.47 29.92 486 14,541.56 100.98 Front All 5.41 29.27 284 8,312.14 57.72 Wings and struts 129 3,775.58 26.22 Body 102 2,985.35 20.73 Tail 25 731.70 5.08 Landing Gear 28 819.51 5.69 Top 5.6 31.36 1150 36,064.00 250.44 Cirrus Design SR20 Front 5.41 29.27 CTLS 172S 2.91 8.47 3.646 13.29 Front Front 43 • The side view projected area of the Cessna 172S is ~190% greater than the CTLS. • The front view projected area of the Cessna 172S is ~134% greater than the CTLS. • The top view projected area of the Cessna 172S is ~177% greater than the CTLS. • The side view projected area of the Cirrus SR20 is ~176% greater than the CTLS. • The front view projected area of the Cirrus SR20 is ~128% greater than the CTLS. • The top view projected area of the Cirrus SR20 is ~166% greater than the CTLS. • The side view projected area of the Cessna 172S is ~108% greater than the Cirrus SR20. • The front view projected area of the Cessna 172S is ~104% greater than the Cirrus SR20. • The top view projected area of the Cessna 172S is ~107% greater than the Cirrus SR20. The size difference may correlate to the resulting detection distances and may be used to extrapolate the detection distance of aircraft smaller or larger than these three. 3.4 Task 4 – Flight Testing and Data Analysis Following the development and initial testing of the experimental plan within Task 3, the research team carried out the designated experiments in Task 4. At the conclusion of this task, the research team consolidated data from the final flight testing at KSU to capture the results. This section offers an overview of the data and results. A database with the relevant parameters for all valid flight test encounters was generated and is provided in Appendix I. 3.4.1 KSU Aircraft KSU flight tests used two aircraft types for the intruder aircraft: a Cessna 172 Skyhawk and a Cirrus SR20. Using two distinct aircraft platforms allows for comparisons of VO performance as a function of aircraft size and speed, as the Cessna 172 and the SR20 have different cruise speeds, wingspans, heights, and lengths, as broken down below. The simulated sUA was a Great Shark 330 simulated using Mission Planner Software; participants reviewed the specifications of this aircraft during the VO Training. 3.4.1.1 Cessna 172 Skyhawk Figure 31 depicts the Cessna 172 Skyhawks in the KSU fleet. The Cessna 172 has a wingspan of 36 feet, a length of 27.17 feet, and a height of 8.92 feet. The cruise speed was 90 knots, and the altitude for this flight testing was 500 feet AGL. 44 Figure 31. KSU Cessna 172 Skyhawk. 3.4.1.2 Cirrus SR20 Figure 32 displays one of the Cirrus SR20s in the KSU fleet. The Cirrus SR20 has a wingspan of 38.3 feet, a length of 26 feet, and a height of 8.9 feet. The cruise speed is 110 knots, and the altitude for this flight testing was 500 feet AGL. Figure 32. KSU Cirrus SR20. 3.4.1.3 Great Shark 330 Figure 33 displays the Great Shark 330 with a wingspan of 11 feet, a cruise altitude of 400 feet AGL, and a cruise speed of 45 knots. 45 Figure 33. KSU Great Shark 330. 3.4.2 KSU Participant Breakdown A total of 19 ground-based VOs served as participants in this study. Each participant was assigned an ID based on the test day and their operating station. Each operating station included an RPIC, a researcher, and a single ground-based VO. A minimum of two operating stations were active on any given test day. Three operating stations were active on three out of the eight test days. The participants' assigned ID, test day and date, Snellen test results (visual acuity), aviation experience, fatigue levels, and hearing test results are reported in Table 7. The participants on the first four days detected the Cessna 172 aircraft, and the participants on the remaining four days detected the SR20 aircraft. Table 7. KSU Participants' Snellen Test Results, Aviation Experience, and Fatigue Levels ID Test Day and Date Snellen Test Results (R and L) Aviation Experience Fatigue Level Hearing Test Results (Best Ear) D1P1 Day 1 (11/01/22) 20/15 20/15 None Not reported All Frequencies at 30dB D1P2 Day 1 (11/01/22) 20/20 20/20 Remote Pilot Well rested All Frequencies at 35dB D2P1 Day 2 (11/10/22) 20/20 20/25 Remote and Private Pilot Neutral All Frequencies at 30dB D2P2 Day 2 (11/10/22) 20/15 20/15 Remote Pilot Well rested All Frequencies at 30dB D2P3 Day 2 (11/10/22) 20/13 20/13 Remote and Private Pilot Extremely well rested All Frequencies at 30dB D3P1 Day 3 (03/28/23) 20/15 20/15 Remote and Private Pilot Well rested All Frequencies at 30dB 46 D3P2 Day 3 (03/28/23) 20/13 20/13 Remote Pilot Well rested All Frequencies at 30dB except 8kHz on the Left Ear D4P1 Day 4 (04/10/23) 20/30 20/15 Student Pilot Well rested All Frequencies at 30dB D4P2 Day 4 (04/10/23) 20/13 20/15 None Well rested All Frequencies at 30dB D5P1 Day 5 (07/10/23) 20/13 20/13 None Neutral All Frequencies at 30dB D5P2 Day 5 (07/10/23) 20/13 20/20 None Neutral All Frequencies at 30dB D5P3 Day 5 (07/10/23) 20/15 20/13 None Neutral All Frequencies at 30dB D6P1 Day 6 (07/20/23) 20/30 20/13 None Neutral All Frequencies at 30dB except 8 kHz on the Left Ear D6P2 Day 6 (07/20/23) 20/20 20/20 None Not rested All Frequencies at 30dB D7P1 Day 7 (07/31/23) 20/25 20/20 None Well rested All Frequencies at 30dB except 0.25kHz on the Left Ear D7P2 Day 7 (07/31/23) 20/13 20/20 None Not rested All Frequencies at 35dB except 0.25kHz on the Left Ear D7P3 Day 7 (07/31/23) 20/13 20/20 None Neutral All Frequencies at 35dB D8P1 Day 8 (08/11/23) 20/13 20/13 None Well rested All Frequencies at 35dB D8P2 Day 8 (08/11/23) 20/13 20/30 None Well rested All Frequencies at 35dB except 4 and 8 kHz on the Left Ear 47 3.4.3 Location of VO Stations and UAS Reference Path The VOs were located at predetermined stations. A minimum of two and a maximum of three operating stations were active on a given test day. Researchers gave the operating stations (also referred to as VO stations) the following IDs: P1, P2, and P3. The three VO stations were located approximately 200 ft apart from each other. Figure 34 shows the locations of the operating stations along with the reference UA flight path. The UA flight path was a box pattern defined using four waypoints, as shown in Figure 34. The box pattern's center was about 1.25 miles north of the VO stations. The simulated UA operated at a constant speed of 45 knots and a constant altitude of 400 ft AGL throughout its mission flight path. Figure 34. Location of Operating Stations and UA Reference Flight Path for the KSU Flight Test Encounters. 3.4.4 KSU Flight Test Results 3.4.4.1 Dependent and Independent Variables Table 8 lists the dependent and independent variables of interest for this study. The primary dependent variable of interest was the intruder detection distance. This variable was used to determine the detection performance of VOs. The dependent variables, including (1) the VO- suggested maneuvers and (2) the Mitigated and Unmitigated CPAs, were used to determine the effectiveness of VOs and RPICs in maintaining the separation between the UA and the intruder aircraft. Other dependent variables, including (1) the response times to initiate the UA avoidance maneuvers and (2) to complete the avo