Flight Test Series 3: Flight Test Report

FLIGHT TEST SERIES 3

Flight Test Report Mike Marston, NASA AFRC Operations Engineer Daniel Sternberg, NASA AFRC Operations Engineer Steffi Valkov, NASA AFRC Operations Engineer NASA Armstrong Flight Research Center UAS-NAS IT&E Subproject October 2015 RELEASE: REVISION B

Overview

This document is a flight test report from the Operational perspective for Flight Test Series 3, a subpart of the Unmanned Aircraft System (UAS) Integration in the National Airspace System (NAS) project. Flight Test Series 3 testing began on June 15, 2015, and concluded on August 12, 2015.

Participants included NASA Ames Research Center, NASA Armstrong Flight Research Center, NASA Glenn Research Center, NASA Langley Research center, General Atomics Aeronautical Systems, Inc., and Honeywell.

Key stakeholders analyzed their System Under Test (SUT) in two distinct Configurations.

Configuration 1, known as Pairwise Encounters, was subdivided into two parts: 1a, involving a low-speed UAS ownship and intruder(s), and 1b, involving a high-speed surrogate ownship and intruder. Configuration 2, known as Full Mission, involved a surrogate ownship, live intruder(s), and integrated virtual traffic.

Table 1 is a summary of flights for each Configuration, with data collection flights highlighted in green .

Section 2 and 3 of this report give an in-depth description of the Flight Test Period, Aircraft involved, Flight Crew, and Mission Team.

Overall, Flight Test 3 gathered excellent data for each SUT. We attribute this successful outcome in large part from the experience that was acquired from the ACAS Xu SS flight test flown in December 2014. Configuration 1 was a tremendous success, thanks to the training, member participation, integration/testing, and in-depth analysis of the flight points. Although Configuration 2 flights were cancelled after 3 data collection flights due to various problems, the lessons learned from this will help the UAS in the NAS project move forward successfully in future flight phases.

ii Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Table 1. Overview of Flights from Flight Test Series 3 for UAS in the NAS.

Configuration 1 Flight Date Hours Encounters Aircraft 0 June 15, 2015 6.9 0 NASA 870 1 June 17, 2015 5.0 15 NASA 870, N3GC 2 June 18, 2015 4.9 23 NASA 870, N3GC 3 June 22, 2015 4.5 20 NASA 870, N3GC 4 June 24, 2015 4.7 20 NASA 870, N3GC 5 June 26, 2015 4.6 16 NASA 870, N3GC, NASA 865 6 July 7, 2015 4.8 22 NASA 870, N3GC 7 July 9, 2015 4.8 23 NASA 870, N3GC 8 July 10, 2015 4.6 20 NASA 870, N3GC, NASA 865 9 July 21, 2015 4.8 20 NASA 870, NASA 850, NASA 865 10 July 22, 2015 3.4 17 NASA 870, NASA 865 11 July 24, 2015 3.2 16 NASA 870, NASA 865 56.2 212 Configuration 2 Flight Date Hours Encounters Aircraft 1 July 13, 2015 3.1 0 NASA 608 2 July 16, 2015 1.3 0 NASA 608 3 July 28, 2015 3.1 2 NASA 608, NASA 865 4 July 29, 2015 2.0 0 NASA 608 5 July 29, 2015 1.9 0 NASA 608 6 July 30, 2015 1.1 0 NASA 608 7 August 3, 2015 3.3 4 NASA 608, NASA 7, N3GC 8 August 4, 2015 2.8 0 NASA 608 9 August 4, 2015 2.9 2 NASA 608, N3GC 10 August 5, 2015 3.0 5 NASA 608, N3GC 11 August 6, 2015 1.1 0 NASA 608, NASA 865, N3GC 12 August 7, 2015 2.8 1 NASA 608, NASA 865, N3GC 13 August 10, 2015 2.8 8 NASA 608, NASA 865, N3GC 14 August 11, 2015 2.7 8 NASA 608, NASA 865, N3GC 15 August 12, 2015 2.5 8 NASA 608, NASA 865, N3GC 36.35 38 iii Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Table of Contents OVERVIEW ............................................................................................................................... II iv Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 8.1.1 An intruder was within 1 NM and less than 500 ft vertical separation without being visual 8.1.2 Full mission Configuration 2 flights only completed 3 of 10 planned data collection flights 8.2.7 A separate truth source for positional data, TSPI, from each aircraft was not available for 8.3.2 Both ARC and LaRC requested that all test aircraft keep heading and airspeed more stable v Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 12.2 F LIGHT 2 R EDLINED F LIGHT C ARDS ................................................................................................... 1 19 12.3 F LIGHT 3 R EDLINED F LIGHT C ARDS ................................................................................................... 1 42 12.4 F LIGHT 4 R EDLINED F LIGHT C ARDS ................................................................................................... 1 63 12.5 F LIGHT 5 R EDLINED F LIGHT C ARDS ................................................................................................... 1 80 vi Flight Test Series 3: Flight Test Report, October 2015 – REVISION B List of Tables Table 1. Overview of Flights from Flight Test Series 3 for UAS in the NAS. ................................................iii vii Flight Test Series 3: Flight Test Report, October 2015 – REVISION B List of Figures viii Flight Test Series 3: Flight Test Report, October 2015 – REVISION B

1 Introduction

The desire and ability to fly Unmanned Aircraft Systems (UAS) in the National Airspace System (NAS) is of increasing urgency. The application of unmanned aircraft to perform national security, defense, scientific, and emergency management are driving the critical need for less restrictive access by UAS to the NAS. UAS represent a new capability that will provide a variety of services in the government (public) and commercial (civil) aviation sectors. The growth of this potential industry has not yet been realized due to the lack of a common understanding of what is required to safely operate UAS in the NAS.

NASA’s UAS Integration into the NAS Project is conducting research in the areas of Separation Assurance/Sense and Avoid Interoperability, Human Systems Integration (HSI), and Communication to support reducing the barriers of UAS access to the NAS.

This research is broken into two research themes namely, UAS Integration and Test Infrastructure. UAS Integration focuses on airspace integration procedures and performance standards to enable UAS integration in the air transportation system, covering Sense and Avoid (SAA) performance standards, command and control performance standards, and human systems integration. The focus of Test Infrastructure is to enable development and validation of airspace integration procedures and performance standards, including the integrated test and evaluation. In support of the integrated test and evaluation efforts, the Project will develop an adaptable, scalable, and schedulable relevant test environment capable of evaluating concepts and technologies for unmanned aircraft systems to safely operate in the NAS.

To accomplish this task, the Project will conduct a series of Human-in-the-Loop and Flight Test activities that integrate key concepts, technologies and/or procedures in a relevant air traffic environment. Each of the integrated events will build on the technical achievements, fidelity and complexity of the previous tests and technical simulations, resulting in research findings that support the development of regulations governing the access of UAS into the NAS.

1.1 Scope The integrated Flight Test 3 (FT3) test period was divided into two distinct configurations.

Configuration 1 test execution began on 15 June, completed 27 July and comprised 12 flights and more than 200 test points. Configuration 2 began concurrently on 16 July and was concluded early on 12 August. The Configuration 2 phase conducted three test sorties and numerous systems integration sorties but was unable to achieve the desired performance needed by the final authority and was therefore concluded early. Further details about each configuration are describe in the main body of this test report.

Per ITE-FT3-01 Section 5 “Test Reporting” this test report will include details on any deficiencies with either the system under test or the test equipment and methods as well as preliminary test results and analysis. This report may also be considered a general progress report to the Integrated Test and Evaluation (IT&E) sub-project.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 1.2 Purpose FT3 gathered data for UAS researchers for their development and evaluation of a Communication system, Sense and Avoid (referred to as Detect and Avoid (DAA) in the RTCA SC 228 ToR) algorithms and pilot displays for candidate UAS systems in a relevant environment. The technical goals of FT3 were to: 1) perform end to end traffic encounter test of pilot guidance generated by Self Separation algorithms (aircraft sensor to wind, Traffic Collision and Avoidance System (TCAS) II, and latency uncertainties to Ground Control Station (GCS) display); and 2) conduct flight test of prototype Communication system as part of an integrated DAA system; 3) collect data to inform the preliminary draft of the Minimum Operational Performance Standards (MOPS) for DAA and Command and Control (C2), to include display and human performance standards in both MOPS.

Furthermore, FT3 increased the team’s capabilities and reduce the risks associated with building a pertinent flight test environment moving towards the final flight tests FT4 and Capstone.

The UAS-NAS project support and participation on the 2014 flight test of ACAS Xu and Self Separation (SS) significantly contributed to building up infrastructure and developing procedures for FT3. The FT3 experiment was divided into two distinct test configurations, each focusing on different aspects of the primary technical goals. The first (described as Pairwise Encounters) looked at the SS and Collision Avoidance algorithms to support the definition of well-clear and TCAS integration onto the ownship platform. The second (described as Full Mission, or FM, flights) focused on UAS pilot response times to, and acceptability of, the same SAA alerts, resolutions, and GCS displays under real world uncertainties.

The two test planned baseline configurations were conducted in two phases. The Pairwise Encounters (Configuration 1) were conducted at NASA Armstrong from 17 June until 27 July. NASA Armstrong provided the primary “ownship” test aircraft for this configuration. The Full Mission flights (Configuration 2), likewise conducted from NASA Armstrong, started integration efforts on 16 July and concluded on 12 August. NASA Glenn provided the primary “ownship” test aircraft for this configuration as well as the Communication system under test. The test period did not exceed original planned schedule estimates, even though Configuration 1 took longer than expected and the Configuration 2 was concluded early.

Testing facilities are Government owned, managed, leased, or under agreement and fall into two categories: 1. Development Facilities: • Distributed System Research Laboratory (DSRL) at NASA Ames • Flight Deck Display Research Laboratory (FDDRL) at NASA Ames • Research Aircraft Integration Facility (RAIF) at NASA Armstrong • UAS Sense and Avoid Research Lab at Stinger Gaffarian Technologies (SGT, outside of NASA Langley) Flight Test Series 3: Flight Test Report, October 2015 – REVISION B • Aircraft Operations Research Hangar at NASA GRC • Communication Laboratory at NASA GRC • GA-ASI Grey Butte Flight Test Facility • GA-ASI System Integration Lab 2. Test Facilities: • Crew Vehicle Simulation Research Facility (CVSRF) at NASA Ames • DSRL at NASA Ames • RAIF at NASA Armstrong • Dryden Aeronautical Test Range (DATR) at NASA Armstrong • Stand Alone Facility (SAF) at NASA Armstrong • The Radio Frequency (RF) Communications facility at NASA Armstrong • Edwards R-2508 Complex 1.3 Stakeholders, Participants, and Responsibilities NASA Integrated Aviation Systems Program (IASP) provides direction for the UAS in the NAS project. The project office had the overall responsibility for FT3 flight test. NASA Ames, NASA Armstrong, NASA Glenn, NASA Langley, GA-ASI and Honeywell supported the project and were participants in the FT3 activity. The following is a brief description of responsibilities: • NASA Ames Research Center (ARC) : NASA Ames provided the HSI research requirements for subject pilot evaluation based on performance during scenario events. Subject pilots performed scenario tests from the Research Ground Control Station (RGCS) located at NASA Armstrong. ARC also provided AutoResolver/Java Architecture for DAA Extensibility and Modeling (JADEM), one of the Self Separation algorithms used during Configuration 1.

• NASA Armstrong Flight Research Center (AFRC) : NASA Armstrong was the responsible test organization for all test missions flown from AFRC. AFRC provided the RGCS for subject pilot evaluation. Further, AFRC hosted the Live Virtual Constructive (LVC) infrastructure for data distribution between NASA Ames, Glenn, and Langley. AFRC also provided the live unmanned aircraft used as intruders for both configurations. NASA 870 (“Ikhana”) was the unmanned aircraft ownship platform for Configuration 1 encounters within the R-2515 airspace.

• NASA Glenn Research Center (GRC) : NASA Glenn was the participating test organization for all test missions flown with the NASA 608 T-34C. GRC provided the communication and control system interface and the UAS Surrogate ownship aircraft for use during Full Mission flights. Although initially planned, the NASA Glenn S- 3B Viking aircraft was not available for use as a “high” speed ownship during Configuration 1 testing.

• NASA Langley Research Center (LaRC) : NASA Langley provided a Self- Separation algorithm (Stratway+) that was displayed and evaluated by subject pilots during flight encounters.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B • General Atomics Aeronautical Systems Inc. (GA-ASI) : Provided hardware, software and integration support on the Ikhana UAS and specifically the Due Regard Radar (DRR). GA-ASI also provided pairwise encounter requirements for autonomous aircraft response maneuvers to TCAS alerting, as well as a Self- Separation Algorithm, CPDS (Conflict Prediction and Display System), for evaluation.

• Honeywell : Honeywell provided the software for the Surveillance Tracking Module (STM) prototype that contained the Honeywell Fusion Tracker. Honeywell also provided a second TCAS II equipped intruder aircraft (N3GC) in support of both configurations. N3GC had onboard TCAS recording capability, and that recorded data was made available to the rest of the FT3 test team to support their data analysis.

1.4 Operations Working Group The Operations Working Group (OWG) was a collaborative meeting between all stakeholders and participants for FT3. The working group, which met weekly on Tuesdays and bi-weekly on Fridays prior to testing, discussed all FT3 ground and flight operations topics. The working group was responsible for flight planning and coordination, assigning actions items, safety concerns which would feed into the System Safety Working Groups (SSWG), hardware integration and testing discussion, training, and readiness. The OWG was new for FT3 and was responsible for a large part of the successes in FT3.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B

2 Flight Test Period

The flight test period spanned from 15 June 2015 to 12 August 2015. The flight days are detailed in Table 2 .

Table 2. Flight Test Series 3 Flight Dates.

Flight Flight Date SUT Types Configuration 1 0 15 June 2015 Ikhana System c heck - out 1 17 June 2015 AutoResolver Fly - throughs 2 18 June 2015 AutoResolver Fly - throughs 3 22 June 2015 AutoResolver Follow display 4 24 June 2015 CPDS TCAS , Radar 5 26 June 2015 CPDS Low - altitude, Multi - ship, TCAS sequential, TCAS 6 7 July 2015 Stratway+ Follow display 7 9 July 2015 Stratway+ Follow display 8 10 July 2015 Stratway+/CPDS Follow display, Multi - ship / Radar 9 21 July 2015 Stratway+ High - speed intruder, Multi - ship 10 22 July 2015 AutoResolver Fly - throughs 11 24 July 2015 CPDS TCAS, Zig - Zag, Radar Configuration 2 GRC1 13 July 2015 CNPC Characterization flight (GRC only) CST1 16 July 2015 CNPC Check - out CST2 28 July 2015 CNPC Check - out GRC2 29 July 2015 INS Check - out (GRC only) CST3 29 July 2015 CNPC Check - out GRC3 30 July 2015 CNPC Check - out (GRC only) REH1 3 August 2015 CNPC Full mission rehearsal GRC4 4 August 2015 CNPC Check - out (GRC only) CST4 4 August 2015 CNPC Check - out CST5 5 August 2015 CNPC Check - out REH2 6 August 2015 RGCS/HSI Cancelled – weather REH3 7 August 2015 RGCS/HSI Full mission rehearsal 1 10 August 2015 RGCS/HSI Data collect 1 2 11 August 2015 RGCS/HSI Data collect 2 3 12 August 2015 RGCS/HSI Data collect 3 Flight Test Series 3: Flight Test Report, October 2015 – REVISION B

3 Flight Crew and Mission Team

NASA 870, known as Ikhana, served as the ownship for the Configuration 1 flight test series. NASA 608, the GRC T-34C, served as the ownship for the Configuration 2 flight test series. Flight test encounter setups included a single ownship vs. a single intruder or single ownship vs. multiple intruders. The intruder role was supported by multiple aircraft due to availability and crew rest considerations. The aircraft and flight crew required to complete the test series are identified in Table 3 .

Table 3. Flight Test Series 3 Aircraft and Flight Crew.

Aircraft Role Position Flight Crew NASA Pilots Howe, Posada, Less Air National Guard LtCol Reiss, Maj Deveroux, Hinton Pilots NASA 870 MQ - 9 Config. 1 Maj Rhodes, Maj Baughman, SSgt (Ikhana) Ownship DET3 Cade NASA Mission Buoni, Howell, Valeri Director Honeywell Pilots Dubbury, McMahon, Walker Config. 1/2 N3GC C90 King Air Intruder Honeywell FTEs Singh, Dougherty Config. 1/2 Purifoy, Miller, Newton, Howe, NASA 865 T - 34C NASA Pilots Intruder Broce Config. 1 High - NASA 850 F - 18A NASA Pilots Larson Speed Intruder Config. 2 NASA 7 King Air NASA Pilots Howe, Williams Intruder Config. 2 GRC Pilots Demers, Micklewright NASA 608 T - 34C Surrogate GRC FTEs Griner Ownship Air National Guard LtCol Shaw, Maj Brooks Pilots RGCS Config. 2 GCS Global Vigilance CTF LtCol Allen Along with the aircraft and flight crew assets, an operations mission control team was utilized to manage the overall test effort. The test conductor was responsible for overall mission success and the coordination of all test assets. The test director provided flight safety oversight and supported the test conductor by performing all back channel and engineering channel coordination. The test coordinator acted as the scribe and performed control room supporting tasks. The mission control team was located in the SAF for all test missions performed during FT3. Table 4 provides a listing of the mission team.

Furthermore, engineering coordinators in the LVC, Ikhana GCS, and Ames virtual Air Traffic Control (ATC)/pilot coordination facility supported the operation.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Table 4. Mission Team.

Mission Team NASA Test Conductor Marston (Config. 1), Sternberg (Config. 2) NASA Test Directors Sternberg (Config. 1), Marston (Config. 2) NASA Test Coordinator Valkov (Config. 1, 2) LVC Engineering Coordinator Kim, Willhite GCS Engineering Coordinator Loera Ames Ghost Controller Bridges

4 System Configuration

All aircraft that participated in this flight test were equipped with navigation systems that use a Global Positioning System (GPS). All manned intruder aircraft were equipped with TCAS, and the Honeywell C90 King Air was equipped with TCAS II version 7.1.

A high level summary of the equipage installed on each aircraft is found in Table 5 .

Table 5. Aircraft Equipage.

CFG 1 CFG 2 CFG 1 CFG 1 , 2 CFG 1 , 2 CFG 2 Flight Test Series 3: Flight Test Report, October 2015 – REVISION B

5 Flight Test Administrative Information

5.1 Operating Area The operating area for Configuration 1 flight test occurred in the Restricted Airspace, R- 2515, located at Edwards Air Force Base (EAFB), along with the Buckhorn Military Operating Area (MOA), with operations scheduled and coordinated through the Air Force Test Center (AFTC). Specific airspace scheduled each day during these flight tests included the Four Corners Area, Mercury Spin Area, overflight of the Precision Impact Range Area (PIRA) East/West, and the Buckhorn MOA. These areas within R-2515 are depicted within the yellow shaded area shown in Figure 1 .

This operating area was adequate for the majority of the Configuration 1 encounters.

However, there were some encounters that required either or both the intruder and ownship to extend north or west, remaining within R-2515, of the airspace. The extensions were required to either start or complete these encounters. For those encounters where an extension was required to accomplish the test encounter, approval from the controlling agency (SPORT) was required. In some cases the extension was not permitted and the encounter was either terminated early or skipped. The Buckhorn MOA was used by the manned intruder aircraft only for many of the test encounters.

Figure 1. Configuration 1 Flight Test Area.

The operating area for Configuration 2 flight test required both R-2508 and R-2515 due to the length of the mission plan, and therefore, included additional organizational cross talk between High Desert TRACON and SPORT controlling agencies. The Isabella MOA block 10,000 ft MSL to 16,000 ft MSL and the western portion of R-2515 were used for this configuration. The primary working area for this configuration is shown in Figure 2 .

The ‘fire mission’ route (depicted in green on Figure 2 ) was the ownship route of flight.

Given that ownship transited between the Isabella MOA and R-2515, prior coordination Flight Test Series 3: Flight Test Report, October 2015 – REVISION B was required to ensure a seamless communication flow from one agency to the next.

Although on some of the sorties extra coordination was required, for the majority of these flights, no significant issues were noted.

It is important to note that NASA operations within R-2515 are subject to the priority of United States Air Force (USAF) programs under the AFTC. Generally speaking, NASA is given a general priority that is superseded by most other AFTC operations. Configuration 2 flights were impacted by this low prioritization and required several mitigations that were in one case acceptable, and in two other cases, were inadequate. On one occasion the sortie start time was changed to avoid an airspace conflict. This was acceptable to the test team. On two other occasions the ownship route of flight and Intruder’s route needed modification to avoid R-2515 altogether. This was not adequate for the test team.

Figure 2. Configuration 2 Flight Test Area.

5.2 Aircraft Staging and Locations During FT3 both intruders and ownship aircraft operated from three different locations.

The Honeywell N3GC aircraft operated from Van Nuys, KVNY, for both configurations, and NASA 608 operated from both Palmdale, KPMD (during systems check flights) and Bakersfield, KBFL, for the three Configuration 2 mission flights. All other aircraft, Ikhana and intruders, operated from Armstrong Flight Research Center, KEDW. The geographical separation resulted in some operational challenges that were overcome with mission planning. Further details on the challenges are listed in later sections of this report. Figure 3 depicts the staging locations for each aircraft.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B KBFL NASA 608 KEDW NASA 870 NASA 865 NASA 7 NASA 850 KVNY N3GC Figure 3. Aircraft Staging Areas.

5.3 Weather FT3 mission rules, FT3-1 and FT3-2, plus Ikhana Standard-8, required all tests to be conducted in Visual Meteorological Conditions (VMC) conditions and 3 statute miles of visibility with at least 1,000 feet of cloud clearance above and below planned block altitude. Additionally, Ikhana Standard mission rules 9 and 10 prohibited flight into known icing conditions as well as transit through visible moisture for repositioning operations. All other participating aircraft complied with their operating limitations as defined by be their respective flight manuals for repositioning operations.

No Configuration 1 flights were cancelled due to weather and all flights completed as originally scheduled. On Flight 5 the visibility was reduced (due to smoke/haze generated by the Lake fire), although still likely greater than 3 miles; this condition was considered a contributing factor for a mission rule violation when an encounter occurred without the intruder having visual on Ikhana.

Configuration 2 was impacted by weather with one cancellation on 6 August 2015 when broken cloud layers impacted the northern half of the desired operating airspace. All other Configuration 2 flights were completed as scheduled; however, high winds aloft, Flight Test Series 3: Flight Test Report, October 2015 – REVISION B greater than 30 kts, were noted on several occasions and needed to be accounted for when controlling each individual intercept. High winds aloft resulting in incorrect compensations were a contributing factor to two or possibly more individual encounters not achieving desired alerting results.

5.4 Stand Alone Facility Mission Control Room The SAF, located at the NASA AFRC main building, was used by the operations team to coordinate, manage, and execute the flight test. The room was configured with three workstations and multiple support stations; one of the three workstations was dedicated to UAS-NAS operations while the other two were used to support other programs (though not concurrently). Each work station was configured with a DICES III voice communication system and several display monitors, shown in Figure 4 . Zeus, Test and Evaluation Command and Control System (TECCS), Ikhana video camera, and Vigilant Spirit Control Station (VSCS) traffic displays provided SA and two-way voice capability to the control room team for test execution.

Figure 4. SAF Control Room Displays.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 5.4.1 Control Displays The Raytheon Solipsys Zeus system acted as the primary control display for both Test Conductor (TC) and Test Director (TD) positions. The Zeus system used surveillance radar information to provide Time-Space-Position Information (TSPI) to the test team. The imbedded range and bearing tool were used to maintain range Situational Awareness (SA) in order to ensure mission rule FT3-9 visual required compliance when applicable.

The use of Zeus as the SA and mission control tool for FT3 was considered a must-have system and the test was completed with no safety incidents and one mission rule violation.

The upper right two displays (see Figure 4 ) were repeaters of the SUT data sourced from the LVC. Using these displays the LVC would pipe JADEM, Stratway+, CPDS, and Radar information to the SAF. The upper left two displays were used as test support displays and included a repeater of the Ikhana Heads-Up Display (HUD) video, which provided TCAS advisory information as well as state information for Ikhana such as altitude, heading and airspeed. Additional information was available on the upper left displays and this included Long Range Optics (LRO) (if active) and access to commercial weather sources.

5.4.2 Test Coordinator Position Throughout the FT3 event an additional position was matured to support data collection.

This Test Coordinator position (located to the right of the image in Figure 4 ) was a simple work station that included a desk, large monitor, and computer docking station. The work station was not restricted to the secure intranet that the TC and TD position required, and enabled access to the external network. The team was able to use public access tools such as Flightradar24®, AviationWeather.gov, SkyVector®, FalconView®, and standard Microsoft Office® products not available on the TC/TD computers.

The primary role for Test Coordinator was to collect test data for use during debrief as well as this report. However, being able to access the aforementioned websites proved to be invaluable as it provided information Zeus could not, such as: take off and land TSPI for KVNY, KBFI and KPMD, screen captures, simple encounter replay, weather information, access to all planning material, and other administrative data. The operations team found this workstation, mission control room position, and external tools as invaluable to supporting the overall flight test execution.

5.5 Mission Information Executing FT3 flights required a large amount of coordination. Configuration 1 flights were planned at a rate of three flights per week due to the duration of each sortie and the amount of test cards executed per sortie. Furthermore, flying the encounters was user work load intensive, with 10 minutes allocated to each encounter, and setup required in between runs. Further discussion on user workload is found in later sections.

Configuration 2 work load was considerably less intensive for the airborne participants as the subject pilot under test was now located in the RGCS and the overall design of the Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Full Mission test provided a more controlled environment than is available to a pilot under test operating from an airborne aircraft.

As a pre-requisite to executing Configuration 1 flight tests, a T-1 crew briefing was accomplished the day prior to the event. The T-1 briefing covered, in detail, the following aspects related to the upcoming flight:  Roll Call  Mission Summary  Mission Timeline  Weather / NOTAMs  UAS Status  Mission Information  GCS Status  Airspace / Airfield  Support Assets  Contingencies  Miscellaneous  Flight Card Review A flight could be delayed or postponed based on information discussed during the T-1 briefing. All team members were required to participate in the briefing either in-person or remotely.

Configuration 2 flights did not require a T-1 for each event. Given the static nature of the Fireline route and intruder routes a T-1 for each flight was considered excessive and any changes were briefed at the pre-flight briefing before conducted 2.5 hours prior to takeoff.

Changes were minimal and largely constituted discussion of contingency plans for loss of airspace or weather effects.

All FT3 Configuration 1 flights started at 0600 (local) Pacific Time. Subsequently, the morning brief was held at 0415L. The morning briefings covered at a higher level the same information with emphasis on any changes from the previous T-1 briefing. The intent was for this briefing to be about 15 minutes in length. A final go/no-go decision was made at this briefing. After the brief, the team was dismissed to prepare for the flight and in some cases additional crew familiarity training was conducted for each display SUT.

The SAF was manned at approximately 0545L to support any systems troubleshooting or coordination efforts required by the supporting aircraft teams.

In general, flights were planned for approximately 5 hours. One hour of that total flight time was allocated to transit, altitude calibration, and systems startup procedures. The limiting factor for the Ikhana was frequency coordination and typically required an OFF time of 1100L for both Satcom and Line of Sight (LOS) frequencies. For the intruders, fuel Flight Test Series 3: Flight Test Report, October 2015 – REVISION B available was the limiting factor. TheT-34C aircraft could only support up to 3.5 hours of flight time while the King Air aircraft could support 4.0 hours although on a few occasions N3GC dropped into Victorville for fuel and then returned to KVNY thereafter, which provided the test team an additional 20 minutes of play time. Moreover, the longer transit times required by N3GC due to operating out of KVNY had to be considered for fuel planning.

A flight debrief was mandatory in order to discuss the day’s flight event s, identify any aircraft discrepancies, and discuss test inefficiencies that may have decreased the number of encounters and test objectives achieved. Action items were assigned for issues and lessons learned that needed to be closed out prior to the next flight. A post- flight test card review and high level data analysis was conducted as well. If the next flight occurred on the following day, a T-1 was then conducted to review test objectives for that next flight, otherwise the T-1, as appropriate, prior to the next test opportunity.

The following figures show the basic Configuration 1 and 2 timelines previously discussed. Start of test day for Configuration 1 flights was typically 0415L and completed around 1415L. While the Configuration 2 flights started at 1200L and end of test day was approximately 1945L.

Figure 5. Configuration 1 Timeline.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Figure 6. Configuration 2 Timeline.

5.6 Training and Qualifications All visiting flight crew team members were required to participate in local area familiarization briefing and conduct a local area familiarization flight. In addition, all flight crew and mission team members were required to have obtained current Crew Resource Management (CRM) training. The TC and TD were required to obtain a formal approval from the NASA Armstrong Director of Flight Operations in order to serve in that capacity.

The requirements for the FT3 test conductor were derived from NASA Armstrong DCP- O-003, Mission Control Procedure. The requirements were tailored from the mission controller section.

5.6.1 Pre-Test Coordination and Training A training event was conducted approximately one month prior to the start of test.

Representatives from all aircraft participants, SUT stakeholders, and IT&E operations were present. The following is a list of training and coordination conducted.

Admin / Motherhood  R-2508 annual refresher training – Conducted by R-2508 representatives Flight Test Series 3: Flight Test Report, October 2015 – REVISION B  R-2515 detailed training – Conducted by R-2515 Sport representatives  International Traffic in Arms Regulations (ITAR) constraints  Basic flight admin expectations Test Admin  Mission Rules and Go/No-Go criteria  Roles and responsibilities  Terminology  General flight day timeline  Communication plan  Altimeter check  Success criteria  Contingencies, aborts and lost links Test Execution  Specific encounters and test card overview At the completion of this training event all stakeholders and aircrew were considered prepared and ready to support the FT3 events in the planned airspace, by all oversight organizations.

5.6.2 System Under Test Training SUT training was conducted closer to the actual flight date. Aircrew availability and Subject Matter Expert (SME) availability were the two primary reasons that the training was conducted either as part of the T-1 briefing or following the flight briefing at 0415L.

This training was used to inform the aircrew who would be executing the flight what each SME was expecting from them. The following questions were addressed during these exchanges  Test configuration. What does the SME/Researcher want; ON, OFF or de- energized.

 Maneuver type. Maneuvering (mitigated) or Non-maneuvering (unmitigated).

 Guidance type. Will the SUT provide directive or descriptive guidance?

 Display under test familiarity. What display will the aircrew be using to gain SA and make a maneuver decision?

 Miscellaneous expectations. Is there anything specific to a particular SUT that the aircrew need to know?

Given that there were multiple SUT being evaluated during FT3, variations were observed in the quantity and quality of training that the aircrew received. Training conducted by the SME on the CPDS system was considered the most thorough, timely, and informative.

The CPDS training was conducted the day prior to the flight in a separate presentation Flight Test Series 3: Flight Test Report, October 2015 – REVISION B with just the operations team present. Representations of CPDS in both picture format as well as a replay of previous simulated events were presented to the crew. In some cases, training and explanation of the SUT were required during the conduct of the flight. Had the training sessions been more thorough, using the CPDS training as an example, more efficiencies would have been gained by the test team and potentially better data collected.

This is due to the flight crew would have better understood the SUT and expectations on its use during flight.

5.7 Altimeter Calibration An altimeter calibration was required for all encounters where the vertical separation between intruders and ownship was less than 500 ft (MR FT3-20). The mission rule was enforced for Configuration 1 flights; however during Configuration 2, it was waived and not required for reasons explained in Section 5.7.2 .

5.7.1 Configuration 1 The altimeter calibration was designed to take out the standard errors found within the pitot-static systems in order to ensure the planned vertical separation was as close to planned as possible. According to FAR 91.411 and Appendix E of Part 43 aircraft pitot systems must be within 75 ft of field elevation when dialed into the local altimeter setting.

Additional errors come with changes in altitude and airspeed. Since some of the planned encounters were with a 200 ft vertical separation, it was possible to be much closer with the errors identified above if they were not mitigated with the calibration.

The calibration was conducted at a flight condition that closely approximated all the planned encounters; 140 KIAS and 13,000 ft MSL. The Ikhana platform set standard 29.92 inHg and the other aircraft adjusted their altimeter settings to show a 13,000 ft altitude. At those conditions each participating aircraft observed the same difference from Ikhana. N3GC consistently showed 60 ft high and NASA865 100 ft high.

The altimeter calibration was performed prior to each flight that required it using the flight card shown in Figure 7 .

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B

FT3 ALTIMETER CALIBRATION

Figure 7. Altimeter Calibration Flight Card.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 5.7.2 Configuration 2 Although the Configuration 2 encounters were planned at 400 ft altitude separation, the test team received approval to waive the altimeter calibration as the conduct of such would require approximately 30 min to complete, with limited technical and no safety value. For the following reasons the team elected to exclude the altimeter calibration:  All aircraft set local altimeter settings.

 All participating aircraft are both properly maintained (24 month certification) and perform a ground altimeter check daily based off of known field elevation with local altimeter set (±75 ft) prior to takeoff. (AIM 7-2-3, Part 43 Appendix E)  All aircraft are TCAS equipped.

 Ownship will be maneuvering (lateral or vertical) away from conflict.

 Configuration 2 encounters are self-separation alerting encounters where the alerting threshold is 120 seconds prior to Closest Point of Approach (CPA).

 All participating aircraft are manned and all encounters require Visual Identification (VID) by 1 nautical mile (NM) between aircraft based off TCAS (MR FT3-9).

 Altimeter calibration data was not required for user interface SUT.

The decision to not conduct an altimeter calibration during Configuration 2 flights did have an impact on safe operations. Positively, the team was able to conduct two Fireline route data collection runs per test day. Had an altimeter calibration been conducted, the ability to conduct two runs may have been limited or not possible. However, deviations closer than 400 ft of separation were prohibited, and in multiple instances, that separation was too great and failed to trigger the appropriate SUT alerting. Had the runs been conducted at 300 or 200 ft of separation, the alerting would likely have been what was desired by the researchers. The team elected to not attempt any encounters less than 400 ft without conducting a calibration.

5.8 Safety and Mission Rules All operations were conducted in accordance with NASA AFRC safety policies outlined in DCP-S-001 and 002. A safety representative was present for all operations planning and was responsible for chairing the SSWG. All encounters and configurations were concurred with by the safety representative. The following Mission Rules were developed in a coordinated effort with the need to maintain approved levels of safety paramount.

Table 6 Mission Rules were used during Configuration 1 flights. They comprised two sections: the first section is the standard to Ikhana project rules and independent of the supported project. The second being specific to FT3.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Table 6. Configuration 1 Mission Rules.

Rationale/Hazard Rule # Rule Description – Flight Notes Report Ikhana Standard Any team member may call an abort for pilots to SOP, FT3 - 01, FT3 - 02, abort the flight at any time for safety reasons FT3 - 07 Only authorized AFRC employees and AFRC Restrict pilots /operators 2 approved pilots are permitted to operate the to the authorized group aircraft Only authorized AFRC employees and AFRC Restrict System Monitors 3a approved System Monitors (Sys Mons) are (Sys Mons) to the permitted to operate the GCS authorized group Only authorized AFRC employee and AFRC Restrict Flight Systems approved Flight Systems Engineers (FSEs) are 3b Engineers (FSEs) to the permitted to assist the PIC and operate Payloads authorized group from PSO2 SOP, AFRC Aircraft Ikhana will be operated according to FAA, AFRC, Checklist for Hosting; 4 USAF, CBP, and General Atomics sta ndard and Space Act Agreement, emergency procedures FT3 - 01, FT3 - 03 Flight will remain within R - 2515 and in accordance 5 with any applicable COA restrictions for Predator SOP B 6 No envelope expansion tests will be performed FT3 - 05, FT3 - 06, FT3 - 13 Flights will be conducted in VMC conditions of no SOP more than moderate turbulence. COA requirement - VMC Standard Hazard 9 No flight in known icing conditions Mitigation No de - icing capability Standard Hazard No flight through opaque clouds, nor sustained Mitigation flight through translucent clouds No de - icing capability Laminar flow wing Standard Hazard 11 Monitor weather forecasts for icing conditions Mitigation No de - icing capability Standard Hazard 12 No flight over densely populated areas Mitigation, Dryden Range Mission Rule DEEC envelope 13 No flights above 45,000 ft clearance has only been conducted to 45,000 ft Inside R - 2515, the AFRC/EAFB RSO has real - time directive authority, including vehicle destruction, to the Ikhana pilot. Note: RSO 14 responsibility and authority may be delegated to Range Safety Plan the Ikhana PIC as a result of RSO recommendation and Tech Brief Committee concurrence.

If returning with a controllable, but compromised AFRC process for Range aircraft, to the best degree possible, the flight path Safety Requirement to will avoid over flight of populated areas and major avoid population ar eas, roads to the maximum extent possible FT3 - 14 Provides at least >=20 If generator out, land at EDW 22/04 with battery minutes of battery power 16 bus voltage at least 23.6V, otherwise, make a to land, taxi, and lakebed landing.

shutdown aircraft Flight Test Series 3: Flight Test Report, October 2015 – REVISION B FT3 Specific Test runs will be conducted in VMC with inflight Standard Hazard FT3 - 1 visibility at least 3 statute miles. Mitigation Test runs will be conducted clear of clouds and with at least 1000’ cloud clearance above and FTP, Standard Hazard FT3 - 2 below the planned test block, including abort Mitigation maneuvers.

At the beginning of each test run, pilots will check their navigational system accuracy; runs with less FTP, Mission Rule, FT3 - Acceptable error is FT3 - 4 than 500’ vertical separation will be aborted if the 01, FT3 - 02 0.1 nm predicted error exceeds 0.1 nm.

All participating aircraft will maintain at least 1000’ vertical separation from other participating aircraft FTP, Standard Hazard FT3 - 5 between test runs unless visual. Maintain Mitigation, FT3 - 03 deconfliction altitude noted on each test card or as instruct ed.

Intruder aircraft will not climb/descend to scenario FTP, Mission Rule, FT3 - FT3 - 6 altitude until Ikhana has reestablished its deconfliction altitude.

This MR is not The test run will be aborted if a UAS loses LOS FT3 - 7 Hazard Mitigation FT3 - 03 applicable during low Link.

altitude test runs.

Number of seconds is The test run will be reset if timing constraints of ±5 test run/point specific to ±10 seconds (as defined for each test Some test runs/points FT3 - 8 FTP, Hazard Mitigation encounter) cannot be met by a minimum of 120 will be greater than seconds prior t o CPA. 120 secs of steady state variables.

For all test encounters where vertical separation is less than 500’, the test run will be aborted if all FT3 - 9 manned aircraft do not have visual on all FTP, Hazard Mitigation participating aircraft at any point inside 1 nm separation.

During auto TCAS runs, the test run will be aborted if Ikhana begins an automatic maneuver Hazard Mitigation, FT3 - FT3 - 10 in the opposite direction than expected for that 01 test encounter.

The test run will be aborted if any aircraft is off - FTP, Mission Rule, FT3 - FT3 - 11 track by more than 0.1NM inside 1 minute to CPA. 01 The manned intruder pilot will not f ollow a TCAS contrary to the pre - briefed abort procedures FT3 - 12 unless they have reason to believe the RA is FTP, Hazard Mitigation generated by non - participating traffic and they have SA on ownship position.

Update of the appropriate Ikhana Lost Link Lost link mission M ission variables (Entry Waypoint, ILLH, ILLA) FTP, Hazard Mitigation, FT3 - 13 identified on each test will be verified, by both aircrew and the Mission FT3 - 01, FT3 - 03 card.

Director, prior to commencing each test run.

When not on a test run, Ikhana crew will ensure FT3 - 14 SAAP Maneuver Mode is set to ADVISORY or Mission Rule, FT3 - 02 OFF.

Anytime below 5000’ AGL, SAAP auto Mission Rule, FT3 - 02, FT3 - 15 maneuvering modes will be ADVISORY or OFF. FT3 - 19 Confirm each aircraft’s nav system time matches FT3 - 16 Mission Rule the UTC time hack given in the pre - flight brief .

The test run will be aborted if any aircraft is off FTP, Standard Hazard FT3 - 17 altitude by more than 50’ toward other aircraft Mitigation, FT3 - 01 inside 1 minute to CPA.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B A 200’ foot min vertical separation shall be FT3 - 18 Mission Rule maintained for all test geometries.

Flight operations outside of the approved mission FT3 - 19 flight envelope for Ikhana are prohibited during FT3 - 14 test encounters A side - by - s ide altitude calibration will be Applicable to performed between aircraft for any mission that FT3 - 20 Mission Rule Configuration 1a and includes an encounter less than 500 ft vertical 1b separation.

Table 7 lists the Mission Rules applicable to Configuration 2 flights. Since Ikhana was not participating the list is significantly reduced and supplanted with platform specific flight manual operational limitations.

Table 7. Configuration 2 Mission Rules.

Rationale/Hazard Rule # Rule Description – Flight Notes Report FT3 Configuration 2 Specific Test runs will be conducted in VMC with inflight Standard Hazard FT3 - 1 visibility at least 3 statute miles. Mitigation Test runs will be conducted clear of clouds and with at least 1000’ cloud clearance above and FTP, Standard Hazard FT3 - 2 below the planned test block, including abort Mitigation maneuvers.

At the beginning of each test run, pilots will check their navigational system accuracy; runs with less FTP, Mission Rule, FT3 - Acceptable error is FT3 - 4 than 500’ vertical separation will be aborted if the 01, FT3 - 02 0.1 nm predicted error exceeds 0.1 nm.

For al l test encounters where vertical separation is Lateral and Vertical less than 500’, the test run will be aborted if all vectors may be FT3 - 9 manned aircraft do not have visual on all FTP, Hazard Mitigation applied prior to an encounter aircraft at any point inside 1 nm abort to maintain separation. applicable separation.

The manned intruder pilot will not follow a TCAS contrary to the pre - briefed abort procedures FT3 - 12 unless they have reason to believe the RA is FTP, Hazard Mitigation generated by non - pa rticipating traffic and they have SA on ownship position.

Confirm each aircraft’s nav system time matches FT3 - 16 Mission Rule the UTC time hack given in the pre - flight brief .

A 200’ foot min vertical separation shall be FT3 - 18 Mission Rule maintained for all test geometries.

Intentional ownship vertical maneuvers towards FT3 - 21 intruder aircraft shall not be made within 60 Mission Rule seconds of CPA during live intruder encounters.

Additionally, during all FT3 flights a Senior Operations Representative (SOR) was present in the SAF. The SOR acted as a spokesperson for the NASA AFRC Director of Flight Operations and their responsibility was to monitor general conduct of the flight test operations, monitor the team’s real -time decisions, and initiate the Aircraft Incident Response Procedure (DCP-S-001) in the case of an aircraft mishap.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B

6 Flight Execution

The Flight Test Series 3 was split into two distinct phases: Pairwise Encounters (Configuration 1) and Full Mission Encounters (Configuration 2). The following sections describe the flight execution activities related to each phase.

6.1 Configuration 1: Pairwise Encounters This test configuration evaluated the advisories generated by the Self Separation and Collision Avoidance Algorithms fed by data from live aircraft during flight. Flight Test Configuration 1 was further defined into two distinct groups, Configuration 1a and 1b.

Configuration 1a involved flight test encounters using NASA’s Ikhana aircraft as the low speed ownship. Configuration 1b was planned for use of a high-speed ownship aircraft.

However, the aircraft planned for Configuration 1b ultimately could not support the FT3 event.

In these tests the Ikhana ownship aircraft was flown against either one or two manned intruder aircraft. Both Self Separation and Collision Avoidance algorithms were evaluated.

The SS algorithms were evaluated using both mitigated and unmitigated encounters.

Unmitigated, also known as fly-through and non-maneuvering encounters, were designed for each aircraft to fly the route as planned all the way to CPA regardless of alerting displayed. These encounters evaluated each SUT’s ability to maintain the correct alerting thresholds. The mitigated encounters were designed for the test aircrew to maneuver the ownship aircraft away from the Collision Avoidance Threshold (CAT) or Near Mid-Air Collision (NMAC) thresholds and maintain a well clear distance between intruder and ownship. Table 8 categorizes each SUT.

Table 8. Configuration 1 SUT Summary.

SUT Researcher/Developer SS CA Mitigated Un mitigated JADEM Ames Research Center Primary Secondary Yes Yes Stratway+ Langley Research Center Primary Secondary Yes Yes CPDS GA – ASI & TU Delft Primary Secondary Yes No Radar GA – ASI Primary Secondary No Yes TCAS GA – ASI Secondary Primary Yes No 6.1.1 Configuration 1 Nomenclature Development Based on requirements from researchers, a nomenclature was developed to capture the needs of each scenario. Each encounter’s name included four parts: 1. Type of encounter. Low speed players, high speed players, or multi-ship encounter 2. Altitude offset between the ownship and intruder(s).

3. Vertical profile.

4. Angle of the intruder flying relative to ownship’s path. For some letter indices, two angles were defined to accommodate multi-ship encounters.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B The full set of nomenclature definition is shown in Figure 8 .

Figure 8. Configuration 1 Pairwise Encounters Nomenclature.

This nomenclature would help define and distinguish encounters, and give a quick, simple assessment of the type of encounter being performed.

6.1.2 Configuration 1a Low Speed Ownship Configuration 1a encounters used a low speed ownship (<210 KGS). Some of these encounters (noted) consisted of a high speed intruder (≥210 KGS ).

6.1.2.1 Ames (AutoResolver) Pairwise Encounter Geometries Ames’ low -speed ownship Pairwise Encounters were designed to test the AutoResolver/JADEM algorithm. The algorithm was further divided into two, AutoResolver 1 and 2, with different alerting thresholds. Ames’ encounters were divided in three types: fly-through/maneuvering climbs/descents/level ( Figure 9 ), fly-through alerting TCAS ( Figure 10 ), and high-speed intruder ( Figure 12 ). The vertical profiles for the fly-through/maneuvering encounters is shown in Figure 11 . Below is an explanation of each.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Aircraft in the fly-through scenarios fly towards a target CPA of 0 NM horizontal offset.

Actual CPA is not critical since any CPA can be compared to the CPA predictions. It is desirable, however, to fly close enough to trigger a self-separation alert. The maneuver encounters are scenarios in which the pilot maneuvers the aircraft as directed by the specific self-separation display.

These fly-through/maneuvering encounters tested various angles into: 0°, 45°, 90°, 110°, 180°, and a 45° and 90° blunder into. All encounters had a minimum 1,000 ft vertical separation that was offset artificially within the algorithm, so as to make the ownship and intruder appear co-altitude.

Figure 9. ARC Pairwise Encounter Angles 1.

The TCAS/Self Assurance/Sense and Avoid Interoperability (SSI) encounters were designed to evaluate interoperability between TCAS and self-separation systems. Self- separation systems are expected to keep the ownship well clear of an intruder. Although well clear is not specifically defined to avoid alerting the intruder’s TCAS, alerting TCAS can generally be considered not well clear. Ideally, the self-separation alert would trigger long before the TCAS alert.

These encounters were designed simply as a fly-through to gather data. Because the vertical profile was planned to go as close as 300 ft, a “build - up” approach was used ; the encounters were flown first at 1,000 ft, then 500 ft, and finally, 300 ft. To stay consistent, Flight Test Series 3: Flight Test Report, October 2015 – REVISION B all of these encounters employed the 3,000 ft lateral separation (although only the 300 ft encounter required the lateral offset). These encounters had an angle into of 0°, 45°, and 90°.

Figure 10. ARC Pairwise Encounter Angles 2.

The vertical profiles for all of Ames’ low -speed intruder encounters is shown in Figure 11 .

For the fly-through/maneuver encounters, angles 0°, 45°, and 90° required climb/descent airspeed as opposed to groundspeed, and are explained in detail in Section 6.1.2.1.1 .

Angles 110°, 180°, turning 45°, and turning 90° had 1,000 ft vertical separation. Fly- through alerting TCAS had 1,000 ft, 500 ft, and 300 ft separation.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Figure 11. ARC Pairwise Encounter Vertical Profiles.

As mentioned, Ames required cases with a high-speed intruder to mimic jet aircraft conditions. The angles for these encounters were 0°, 45°, 90°, and overtaking 180°.

These encounters were done both as a fly-through and a maneuver performed with required 1,000 ft vertical separation, shown in Figure 12 .

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Figure 12. ARC High Speed Intruder Pairwise Encounter Angles.

6.1.2.1.1 ARC Indicated Airspeed Encounter Wind Matrix To validate ARC’s simulation results for AutoResolver/JADEM, a slightly different approach was required for their climbing and descending encounters. Because their simulations were built using indicated airspeed (KIAS) as opposed to knots groundspeed (KGS), the researcher requested that all of these climb/descent encounters be flown with a constant airspeed.

However, this posed a problem for CPA timing. All other encounters were designed with groundspeed in mind, so that daily variable winds aloft would not affect the timing of the encounter. T o mitigate this problem for ARC’s climb/descent encounters, a wind adjust matrix was built to help with timing and variable winds.

As depicted in Table 9 , the Wind Adjust Matrix used a spreadsheet format to calculate relative timing from the Initial Point (IP). Each series of climb/descent encounters (L13, L14, L15, and L16) were designed to start at a specific altitude. Using this altitude, a true airspeed (KTAS) was calculated for the desired airspeed. Once current winds were input (green boxes, wind speed and wind direction) simple trigonometry was used to show the headwind the aircraft would experience for that encounter based on encounter angle.

Distance adjust value was then calculated (more or less distance traveled for the encounter) based on headwind and the leg length. Although the researcher originally Flight Test Series 3: Flight Test Report, October 2015 – REVISION B wished to communicate this distance adjust to the pilots, operationally it made more sense to communicate a time adjust. Therefore, two additional values were calculated: the groundspeed the aircraft would experience based on the headwind and true airspeed at altitude, and finally a time adjust was calculated using the simple formula time = distance / rate, using the distance adjust and groundspeed.

For each of the encounters of this type, the TC would ask the winds aloft for the climbing or descending aircraft. The values were input into the matrix and the spreadsheet would automatically calculate a time adjust from the IP. The TC would then communicate this value back to the airborne players. A positive value indicated to the aircraft to begin their descent that many seconds after IP crossing. A negative value indicated to the aircraft to begin their descent that many seconds before the IP crossing.

During later flights and for simplicity, values were adjusted to be called out in increments of 5 seconds since the aircrew found this solution to be easier to implement while performing the dynamic encounters.

This compromise led to successful climb/descent encounters and the researcher received good data where the matrix was used correctly.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Table 9. ARC Wind Adjust Matrix.

TIME BASED WIND DRIFT ADJUSTMENT FOR KIAS CLIMB/DESCENT ENCOUNTERS Version: 22Jun2015 Wind Speed (kt) 10 + later IP crossing Wind Direction (deg) 258 - earlier IP crossing Leg Length (min) 3 L13 Series: Climbing Intruder, 140 KIAS, Initial Alt = 12500 ft KTAS at Initial Alt = 169 Aircraft Distance Time Ground Encounter Angle Heading Headwind (kt) Adjust Adjust Speed (kt) (deg) (nmi) (sec) A - 0 deg 78 -10.0 0.5 179.0 10.1 C - 45 deg 33 -7.1 0.4 176.1 7.2 D - 90 deg 258 10.0 -0.5 159.0 -11.3 L14 Series: Descending Intruder, 140 KIAS, Initial Alt = 16000 ft KTAS at Initial Alt = 178.6 Aircraft Distance Time Ground Encounter Angle Heading Headwind (kt) Adjust Adjust Speed (kt) (deg) (nmi) (sec) A - 0 deg 78 -10.0 0.5 188.6 9.5 C - 45 deg 33 -7.1 0.4 185.7 6.9 D - 90 deg 258 10.0 -0.5 168.6 -10.7 L15 Series: Climbing Ownship, 120 KIAS, Initial Alt = 12000 ft KTAS at Initial Alt = 143.8 Aircraft Distance Time Ground Encounter Angle Heading Headwind (kt) Adjust Adjust Speed (kt) (deg) (nmi) (sec) A - 0 deg 258 10.0 -0.5 133.8 -13.5 C - 45 deg 258 10.0 -0.5 133.8 -13.5 D - 90 deg 348 0.0 0.0 143.8 0.0 L16 Series: Descending Ownship, 120 KIAS, Initial Alt = 16000 ft KTAS at Initial Alt = 153.3 Aircraft Distance Time Ground Encounter Angle Heading Headwind (kt) Adjust Adjust Speed (kt) (deg) (nmi) (sec) A - 0 deg 258 10.0 -0.5 143.3 -12.6 C - 45 deg 258 10.0 -0.5 143.3 -12.6 D - 90 deg 348 0.0 0.0 153.3 0.0 Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 6.1.2.2 Langley (Stratway+) Pairwise Encounter Geometries NASA Langley designed self-separation encounters to test and collect data on their self- separation algorithm Stratway+ (now called Detect & AvoID Alerting Logic for Uncrewed Systems (DAIDALUS)) using maneuvering flight test encounters. This series of scenarios was designed to collect data to validate CPA predictions and validate the Stratway+ solution well clear band data during live flight test conditions. The encounters also operated on the edge of the TCAS Resolution Advisory (RA) envelope and ensured Stratway+ guidance provided maneuver bands to operate outside the RA envelope of TCAS II. Most encounters were set at 3,000 ft lateral planned CPA with 400 or 500 ft vertical offset.

All of Langley’s encounters required a lateral maneuver and were divided into low -speed ownship level/climb/descent/double blunder ( Figure 13 , Figure 14 , and Figure 15 ), low- speed intruder multi-ship ( Figure 16 , Figure 17 , and Figure 18 ), high-speed intruder ( Figure 19 ), and high-speed intruder multi-ship ( Figure 20 and Figure 21 ).

Encounter geometries were flown at 0°, 20°, 45°, 90°, and 135° angle into. The 135° geometry was of particular interest to evaluate the effectiveness of Stratway+ in a late intruder discovery scenario where radar is operating at the edge of its azimuth. Multiple runs were conducted with varying sensor selection.

Figure 13. LaRC Pairwise Encounter Angles.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Vertical maneuvers represented further evaluation of the Stratway+ algorithm performance and also engaged the TCAS II RA envelope. Additionally, a double blunder encounter was added for FT3, to gauge the solution space of Stratway+. Climb/descent encounters had a 500 ft vertical separation and required a 1,000 fpm climb or descent.

Figure 14. LaRC Pairwise Encounter Vertical Profiles 1.

Figure 15. LaRC Pairwise Encounter Vertical Profiles 2.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B The multiple intruder series of scenarios were designed to constrain the solution space presented to the pilot and to evaluate the Stratway+ solution well clear band data.

Stratway+ was designed to present well clear maneuver space as the union of all threats and a solution space which provides guidance well clear of all intruders. These scenarios increased the complexity of the solution band data presented to the pilot as there were solutions which were constrained to either side of the aircraft’s course. For an SAA system to operate effectively in the NAS, it must be able to solve a multiple intruder scenario even though this may be a very low probability scenario.

All multi-ship encounters had a 500 ft vertical separation between ownship and intruder and all were level flight/level maneuvers to introduce the first stage of this complexity which is planned to be continued in FT4.

The multi-ship encounters included several permutations: 0°/0°, 20°/-20°, 0°/45°, 45°/90°, 0°/90°, 0°/135°, and 90°/135° angles into, as depicted in Figure 16 , Figure 17 , and Figure 18 .

Figure 16. LaRC Pairwise Multi-ship Encounters 1.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Figure 17. LaRC Pairwise Multi-ship Encounters 2.

Figure 18. LaRC Pairwise Multi-ship Encounters 3.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B The high speed intruder encounter series of scenarios were designed to evaluate the effectiveness of the Stratway+ algorithm when engaging intruders operating at speeds typically encountered with commercial jet transport aircraft transiting below Class A airspace. The increased intruder speed shortened the available pilot reaction time and provided faster closure while the ownship started to execute the maneuver to remain well clear. It was also of interest to evaluate if alerting times effective at lower closure rates with slower intruders would remain sufficient with higher closure speeds.

For Stratway+, high-speed intruder encounters were performed at 400 ft vertical separation and at angles into of 0°, 45°, 90°, and 135° ( Figure 19 ).

Figure 19. LaRC High Speed Intruder Pairwise Encounter Angles.

Additionally, multi-ship encounters were performed with a high-speed intruder with 500 ft vertical separation. These encounters’ permutations include: 0°/45°, 0°/90°, and 0°/135°.

Figure 20 and Figure 21 detail the high-speed and low-speed intruder for each.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Figure 20. LaRC High-Speed Pairwise Multi-ship Encounters 1.

Figure 21. LaRC High-Speed Pairwise Multi-ship Encounters 2.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 6.1.2.3 General Atomics CPDS Pairwise Encounter Geometries The first group of encounters that was provided from General Atomics was to test the self- separation algorithm CPDS.

Five single intruder encounters were provided for CPDS: one at 400 ft vertical separation and four at 500 ft vertical separation. Each of these encounters, all low-speed, had the intruder performing a 45° standard rate turn blunder into the ownship ’s path .

CPDS also required three multi-ship encounters. Each featured an intruder performing a 45° standard rate turn blunder into the ownship ’s path , as well as a second intruder constraining the solution space. These encounters were designed with the first intruder at 400 ft above ownship and the second at 500 ft below ownship.

Below is a detailed description of the encounters and their main objectives, as well as a table ( Table 10 ) to summarize/ CPDS Objectives 1. Test the current system / algorithms beyond the situation in which an intruder traverses the various self-separation alert states in the way that would happen if the conflict geometry already exists outside of the temporal limit defining Self-Separation Proximate Traffic (SSPT).

a. Test the system for situations in which Well Clear is resolved by intruder maneuver while having status CSSA (Corrective Self-Separation Alert) (before Self-Separation Warning Alert (SSWA) occurs).

i. Loss of well clear is predicted and after the intruder alert status has become Self- Separation Corrective Alert (SSCA), the intruder maneuvers in such a way that well clear will not be violated.

b. Test the system for situations in which an intruder becomes CSSA due to a maneuver within the 75 -25 seconds to the well clear boundary.

i. In these situations the predictability in terms of time remaining until SSWA cannot be deduced from the time the yellow band intersected ownship track or the time traffic became SSCA.

c. Test the system for situations in which the intruder self-separation alert state due to a maneuver cycles from normal to CSSA to normal.

2. Test the current system / algorithms beyond the situation in which an intruder traverses the various self-separation alert states in the way that would happen if the conflict geometry already exists outside of the temporal limit defining SSPT and an additional constraint on the solution space.

a. Test the system for situations in which well clear is resolved by intruder maneuver while having status CSSA (before SSWA occurs).

i. Loss of well clear is predicted and after the intruder alert status has become SSCA, the intruder maneuvers in such a way that well clear will not be violated.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B b. Test the system for Situations in which an intruder becomes CSSA due to a maneuver within the 75 -25 seconds to the well clear boundary.

i. In these situations the predictability in terms of time remaining until SSWA cannot be deduced from the time the yellow band intersected ownship track or the time traffic became SSCA.

c. Test the system for situations in which the intruder self-separation alert state due to a maneuver transitions from normal to CSSA to normal.

3. Test the conflict probe function for the most opposite impacts of wind on the same conflict geometry.

Table 10. CPDS Objective and Encounter Overview.

Desired UAS Pilot Performance The desired UAS pilot performance in the task of remaining well clear comprises two aspects: 1.Timely detection of all conflicts (future loss of well clear) that will require a maneuver to prevent them from occurring unless the intruder resolves it in time, and appropriate execution of the maneuver (timing and magnitude) that prevents the otherwise occurring well clear violation.

2. A minimum of unnecessary maneuvering. This comprises the prevention of: a. Situations in which the pilot initiates a maneuver to remain well clear whereas the continuation of the current direction and velocity would not have resulted in a loss of well clear.

b. Situations in which ownship maneuvers due to a temporary predicted loss of well clear outside the 85 second threshold use for the SSPT.

c. Situations in which the maneuver performed by the pilot to remain well clear is far more severe than necessary.

Requirements for CPDS Conflict Geometries ( Figure 22 ) Given the objectives, the following three types of encounters are needed: Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 1. To meet objective 1a and 2a: Encounters that are predicted to result in a loss of well clear at a time to CPA larger than 120 seconds, but are resolved by the intruder maneuvering between 75 and (25+TBD margin) seconds to well clear.

2. To meet objective 1b and 2b: Encounters in which the intruder maneuvers within 110 seconds to CPA in such a way that the predicted distance at CPA crosses the well clear dmod threshold.

3. To meet objective 1c and 2c: Encounters that only during the maneuver of the intruder cause a predicted loss of well clear with a time to CPA that always remains above 60 seconds.

Figure 22. GA-ASI CPDS Pairwise Encounters.

Requirements for Second Intruder ( Figure 23 ) To meet objective 2, the trajectory for the second intruder must meet the following requirements: 4. The second intruder is not used to generate a geometry which causes a predicted loss of well clear.

5. The second intruder is not intended to maneuver, unless necessitated by an (unplanned) maneuver of Ikhana.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 6. The second intruder is to be positioned in such a way that during the encounter with intruder 1 (with ‘during’ defined as the perio d Ikhana being at least from 120 seconds to moment until the predicted loss of well clear with intruder 1 occurs) the intruder will be Preventative Self-Separation Alert (PSSA) (using the proposed update to the PSSA specification).

7. The second intruder is to be positioned in such a way that within 10 seconds of the start of a rate-one turn to the left of Ikhana, the PSSA becomes CSSA.

Figure 23. GA-ASI CPDS Multi-ship Pairwise Encounters.

6.1.2.4 General Atomics Radar Pairwise Encounter Geometries The next set of encounters provided by General Atomics were designed to test the Engineering Development Module (EDM) DRR.

Test encounter geometries provided by GA-ASI collected data on the performance of the company provided EDM radar system and to help inform the SC-228 radar working group MOPS. The EDM radar performance operating at low altitudes was unknown, thus during FT3, test encounters were planned to explore how the radar performs at low altitude with ground clutter effecting target resolution. Figure 24 depicts the planned low altitude radar flight test geometries. The minimum test altitude was 1,000 ft AGL based off the highest ground feature located along the flight path of the encounter. Both the ownship (Ikhana) and the intruder performed 1,000 ft AGL runs but at no time did an encounter participant operate below 1,000 ft. Eight low altitude radar runs were planned.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B All low-altitude encounters performed had a vertical separation of 1,000 ft, 3,000 ft lateral offset, and were all flown head-on.

Figure 24. GA-ASI Low-Altitude Pairwise Radar Encounters.

Further, GA-ASI required performance testing of the EDM radar to determine targeting capabilities at the azimuth limits of the radar system ( Figure 25 ), performance of the system of the radar when the intruder is persisting on the beam ( Figure 26 ), as well as, system performance of the radar during intruder acceleration maneuvering called “Zig - Zag” encounter ( Figure 27 ).

The Constant Bearing, Decreasing Range (CBDR) encounters for radar held the intruder at a relative angle of either 110° or 90° bearing to the ownship. These encounters were longer than most, being flown for 5 or 6 minutes to collect further radar data. The encounters included climbs and descents with 500 ft vertical separation, and were performed in level flight with a minimum of 300 ft vertical separation.

The “Zig - Zig” encounter, although depicted with defined angles in the figure, could change during the flight; as long as the constant acceleration was in place, researchers would be receiving the data they required. This encounter was flown with 1,000 ft vertical separation.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Figure 25. GA-ASI Radar CBDR Pairwise Encounters (110°).

Figure 26. GA-ASI Radar CBDR Pairwise Encounters (90°) Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Figure 27. GA-ASI Radar Zig-Zag Pairwise Encounter.

6.1.2.5 General Atomics TCAS Pairwise Encounter Geometries The Collision Avoidance (CA) performance encounters were designed to test the full range of TCAS Resolution Advisories (i.e., preventive and corrective) and when executed automatically, to test the performance of the vehicle response in a real world environment.

Climbing/descending ownship and intruders were included to capture realistic encounter dynamics of the Phase I DAA MOPS definition of "transition". These encounters also served to capture Radar performance data all the way through a CA maneuver.

Figure 28 and Figure 29 depict the mitigated single intruder TCAS runs that were designed to further investigate the threshold between collision avoidance and self- separation boundaries. Runs were planned in a variety of geometries and used a buildup approach starting with 500 ft vertical separation and building up to 300 ft vertical separation encounters, running in advisory and then AUTO (automatic) mode. Vertical blunder type maneuvers were planned with ownship maneuvers, intruder maneuvers, and some encounters where both ownship and intruder perform vertical maneuvering toward each other with a minimum of 500 ft separation at the completion of the encounter.

Angles into for these encounters included 0°, 20°, 45°, 90°, 135°, and 160°.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Figure 28. GA-ASI TCAS Mitigated Pairwise Encounters 1.

Figure 29. GA-ASI TCAS Mitigated Pairwise Encounters 2.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Several of the CA performance encounters included multiple threat aircraft. While not traditional "multi-threat" encounters as defined by TCAS, these encounters were designed to generate TCAS RA's one at a time or sequentially. These did not directly test the TCAS multi-threat logic, but were designed to test the dynamics of multiple TCAS RAs generated in different directions. The encounter was timed to induce one TCAS RA, followed by a "clear of conflict", followed by another RA in the opposite direction to the first. These encounters were the most complex to be tested during FT3. A buildup approach was used for these encounters starting at 300 ft vertical separation with Ikhana operating in advisory mode. Once the 300 ft encounter had been cleared in advisory, the encounter would be performed in AUTO mode at 300 ft vertical separation. Once 300 ft encounters were cleared, 200 ft vertical separation would be tested using the same buildup approach. As mentioned later in this report, researchers stated on the flight day that advisory 300 ft encounters were enough for their data collection (thus only M67Q and M68Q were performed, and in advisory mode).

The figures below show the planned top view ( Figure 31 ) of these TCAS multi-ship sequential encounters as well as the vertical profile views ( Figure 32 , Figure 33 ).

Figure 30. GA-ASI TCAS Sequential Pairwise Encounters.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Figure 31. GA-ASI TCAS Sequential Pairwise Encounters Vertical Profile 1.

Figure 32. GA-ASI TCAS Sequential Pairwise Encounters Vertical Profile 2.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 6.1.3 Configuration 1b High Speed Ownship Configuration 1b encounters required a high speed ownship (≥210 KGS). Unfortunately due to time constraints and unavailability of the high speed ownship, these encounters were not completed.

6.1.3.1 Ames (AutoResolver) Pairwise Encounter Geometries Ames required some encounters where a high-speed ownship (210 and 250 KGS) was necessary to test the limits of their algorithm, with a low-speed (130 KGS) intruder. The planned angles into for this configuration were 0°, 45°, 90°, and an overtaking 180°, seen in Figure 33 . The high-speed encounters would test the fly- through (“unmitigated”) case, as well as maneuvering based on algorithm directive (“display maneuver”). All high -speed ownship scenarios had planned 1,000 ft vertical separation for safety.

Figure 33. ARC High Speed Ownship Pairwise Encounter Angles.

6.1.4 Matrix Development A comprehensive flight test matrix was built for FT3 based on researcher requirements and the geometries described in Section 6.1 . The encounters were grouped into sections by their encounter angle or type: head-on, 45°, 90°, 135°, high-speed participant, radar, TCAS, and CPDS.

Originally, researchers from Langley Research Center, Ames Research Center, and General Atomics each provided their own set of maneuvers and requirements in Flight Test Series 3: Flight Test Report, October 2015 – REVISION B spreadsheet form. These requirements were consolidated into one matrix by personnel at Langley, and this spreadsheet was inherited by Armstrong Flight Research Center and extensively expanded upon. Flight planning by Armstrong began with this version – gathering requirements for the type of maneuver being performed, altitudes, groundspeeds, and sensors.

The purpose of the matrix was to provide a centralized spreadsheet to input all required research geometries, plan their flight in the approved airspace, and populate the flight cards with information and requirements for each encounter.

The matrix was built in Microsoft Excel® and used Visual Basic for Applications (VBA) for calculating pertinent values, such as GPS coordinates in multiple formats. Look-up tables were used on several occasions and especially to populate Ikhana Lost Link mission for each encounter based on CPA. Additionally, Excel was useful for calculating IP to Maneuver Point (MP) and CPA using dead-reckoning equations. Here is a breakdown of the parameters, some of which ( bolded ) are shown in the snapshot Table 11 .

 Scenario Number (S/N) : The Scenario Number was the most critical number for each encounter. This number served as an identification number for the unique geometry, flight altitude, and other relevant information. Although encounters could have the same scenario name, the number was unique. S/N was used throughout the entire flight test, from the matrix for building the encounter, to encounter prioritization, and finally, for creating the flight test cards.

 Scenario Name: Scenario names were based from the Pairwise Encounters Nomenclature (see Section 6.1 ). The Scenario Name was a quick reference to gain SA on what type of encounter was being performed. Scenario Number could be the same for two or more encounters.

 Scenario Name (Old Nomenclature): Previously, encounters had a naming convention developed from researchers at Ames. Later, this was refined and modified to the version currently being used. The old nomenclature was kept on the matrix as a trace to the original required encounter.

 OWN True Course: True Course of the ownship. This value was used to calculate GPS coordinates (magnetic course was later calculated on the flight cards themselves).

 Leg time: Time for the encounter from Commence Exercise (COMEX). Leg time included some buffer for setting up the encounter geometry. Most encounters had 3 minute legs, with some maneuvering encounters (3.25, 3.5), low-altitude radar (3.5), and CBDR radar (5, 6).

 Min. Vertical Separation: Smallest vertical separation between ownship and intruder(s) for the encounter at CPA. If the vertical separation necessary for an encounter was ≤500 ft, a lateral offset was required for safety.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B  Angle Into: Relative angle of the intruder(s) into the ownship for that geometry.

More can be seen about the angles into in Section 6.1 . This value was used to calculate GPS coordinates.

 Lateral Offset: A lateral offset of half a nautical mile (~3,000 ft) was calculated into the geometry for encounters with a vertical separation of ≤500 ft. This was to ensure that if visual was not acquired according to mission rule, there would still be a safety buffer.

 GS OWN: Groundspeed of the ownship. Depending on what the researcher wanted to see, the groundspeed varied between encounters. Most low-speed encounters had a GS of 130 or 150 KGS. High-speed encounters required an ownship GS of ≥210 KGS. Groundspeed was preferred for calculations since it did not have to take wind into account. For some encounters, an airspeed of 120 KIAS was required. GPS coordinates were provided for these encounters from the researcher. More discussion can be seen in Section 6.1.2.1.1 on how these unique encounters were performed.

 GS INT1: Most encounters required the intruder(s) to fly at 150 or 180 KGS. For high- speed encounters, intruders were required to fly ≥210 KGS. For some encounters, an airspeed of 140 KIAS was required.

 GS INT2: Multi-ship encounters only had low-speed intruders for Flight Test Series 3. Thus, all intruder 2 groundspeeds were 150 or 180 KGS.

 Ownship Initial Altitude: Altitudes chosen for each encounter took Ikhana and intruder flight performance into consideration, as well as airspace. Encounters began 10K-20K ft MSL. Low-altitude radar encounters took the highest point on the terrain (3,200 ft MSL) and added 1,000 ft for the flight level (thus 4,200 ft MSL).

 Ownship Vertical Velocity: For some encounters, a climb or descent was required by the ownship. Rates required were either 1,000 fpm (climb) or -1,000 fpm (descent).

 Ownship Final Altitude: Once more, the final altitude was within the block of 10K- 20K ft MSL or 1,000 ft above the highest terrain point.

 Intruder 1 Initial Altitude: Identical to ownship.

 Intruder 1 Vertical Velocity: Identical to ownship.

 Intruder 1 Final Altitude: Identical to ownship.

 Intruder 2 Initial Altitude: Identical to ownship.

 Intruder 2 Vertical Velocity: Identical to ownship.

 Intruder 2 Final Altitude: Identical to ownship.

 CPA OWN: The CPA of the ownship was one of its most important parameters.

The CPA was the point where the ownship and intruder(s) would be nearest in space for each encounter. Within R-2515, 11 different ownship CPAs were chosen for all Pairwise Encounters. CPAs were chosen to accommodate for the 3 minute legs in the airspace (as well as the longer radar legs), plan sun angles for manned Flight Test Series 3: Flight Test Report, October 2015 – REVISION B intruders, and were used to build the Ikhana Lost Link mission. Additionally, the CPAs made it easier to group encounters based on matching CPA when building these geometries in Zeus for SAF SA. Finally, the CPAs were used in a lookup table to build GPS coordinates for all geometries.

 CPA OWN Lat/Lon: Chosen latitude and longitude for each CPA in Decimal Degrees (DD) format. The CPA latitude/longitude was found using FalconView.

 IP OWN: The IP of the ownship was chosen to fit within the airspace and to accommodate for the 3 min (or more) legs. The IP served as the point where the encounter would start and where the aircraft needed to be at the COMEX. Each IP had an identification number based on its coordinates, and for encounters that used the same IP, an identical IP ID was used. The IP was also used on the flight cards for reference on the top view (see Figure 34 ) and coordinates.

 IP OWN Lat/Lon: Calculated latitude and longitude of the ownship IP from the CPA using dead reckoning equations, in DD format.

 IP OWN DME: Calculated distance in NM from the CPA to the IP for ownship.

 IP INT: The same procedure was used for intruder IP as for ownship.

 IP INT Lat/Lon: Calculated latitude and longitude of the initial point for intruder from the CPA in DD format.

 IP INT DME: Calculated distance in NM from the CPA to the IP for intruder.

 CPA INT: Similar to the ownship, the CPAs for the intruder were also grouped based on GPS coordinates. However, since the geometries for the intruders were built around those for the ownship, there were many more CPAs for intruders than for the ownship (due to various angles into, groundspeeds, etc.).

 CPA INT Lat/Lon: Calculated latitude and longitude of the intruder CPA in DD. The CPA for the intruder was either the same as the owns hip (˃500 ft vertical separation) or calculated to be 3,000 ft away (≤500 ft vertical separation) from ownship CPA using the relative angle into.

 MP INT: For some encounters, a maneuver was required in the middle of the encounter for the intruder to create a “blunder” type scenario. Maneuver points once again held the same ID if they had the same GPS coordinates.

 MP INT Lat/Lon: Calculated latitude and longitude in DD that the intruder was expected to begin their standard rate turn to the CPA.

 On Condition: Each encounter required that the aircraft be on condition a certain number of seconds from CPA. This was to ensure that the researcher’s algorithm would have enough time to pick up the aircraft in the encounter for their required conditions (speeds, altitudes, vertical speed, etc.). Most encounters had a requirement of being on condition for 2.5 min (0.5 min for setting up) for the total 3 min legs required.

 Tolerance: A carryover from ACAS Xu, the tolerance for an encounter was the number of seconds that each player could be away from the CPA and still achieve Flight Test Series 3: Flight Test Report, October 2015 – REVISION B the correct alerting for that encounter. Most encounters had a timing of ±5, 8, or 10 seconds. This value was determined from simulation by the researchers or from previous experience. As time went on, it became apparent that tolerance was not as critical for FT3 (especially for the maneuvering encounters) as much as achieving stable conditions.

 Ikhana Lost Link : In the case that Ikhana would lose link, a Lost Link mission was programmed into its flight computer. The Lost Link mission was based on the CPA the ownship would be heading to for that encounter. For this reason, it was critical for the Ikhana team to have all CPAs prior to flight testing so they could build this mission. The Lost Link mission was input in the flight matrix using a lookup table based on CPA.

It’s important to note that once all the geometries were built, they were geo -referenced in FalconView to ensure they fit in the assigned airspace or gave enough maneuverability space for all aircraft participants. If they did not, the CPA and GPS coordinates were iterated until a suitable geometry was achieved.

The Flight Test Matrix also included a section showing Scenario Number, priority, and what sensors would be required for that encounter. Encounters for a particular flight day were chosen in this way.

Thus the matrix provided the basic requirements for execution, safety mitigation, and prioritization for planned FT3 scenarios.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Table 11. Pairwise Encounters Scenario Matrix.

Min Vertical Lateral Ownship Ownship Ownship Intruder Intruder Intruder Intruder Intruder Intruder CPA On Scenario Scenario Leg Time Angle Ikhana Separation Offset GS OWN GS INT1 GS INT2 Initial Vertical Final 1 Initial 1 Vertical 1 Final 2 Initial 2 Vertical 2 Final Tolerance Condition Number Name (minutes) Into Lost Link (ft) (ft) Altitude Velocity Altitude Altitude Velocity Altitude Altitude Velocity Altitude (sec) (min) 1 L42A 3 400 0 3000 150 180 NA 12000 0 12000 12400 0 12400 NA NA NA ± 10 2.5 LL WPT 9 2 L42B 3 400 20 3000 150 180 NA 12000 0 12000 12400 0 12400 NA NA NA ± 10 2.5 LL WPT 9 3 L55A 3 500 0 3000 130 180 NA 11000 1000 14000 14500 0 14500 NA NA NA ± 10 2.5 LL WPT 9 4 L55B 3 500 20 3000 130 180 NA 11000 1000 14000 14500 0 14500 NA NA NA ± 10 2.5 LL WPT 9 5 L56A 3 500 0 3000 130 180 NA 16000 -1000 13000 12500 0 12500 NA NA NA ± 10 2.5 LL WPT 9 6 L56B 3 500 20 3000 130 180 NA 16000 -1000 13000 12500 0 12500 NA NA NA ± 10 2.5 LL WPT 9 7 L57A 3 500 0 3000 130 180 NA 11000 1000 14000 17500 -1000 14500 NA NA NA ± 10 2.5 LL WPT 9 8 M59Q 3 500 0 3000 150 180 150 13000 0 13000 13500 0 13500 12500 0 12500 ± 10 2.5 LL WPT 2 9 M59U 3 500 20/-20 3000 150 180 150 13000 0 13000 13500 0 13500 12500 0 12500 ± 10 2.5 LL WPT 5 10 L13A 3 1000 0 0 150 140 NA 16500 0 16500 12500 1000 15500 NA NA NA ± 10 2.5 LL WPT 9 11 L13A 3 1000 0 0 150 140 NA 16500 0 16500 12500 1000 15500 NA NA NA ± 10 2.5 LL WPT 9 12 L14A 3 1000 0 0 150 140 NA 12000 0 12000 16000 -1000 13000 NA NA NA ± 10 2.5 LL WPT 9 13 L14A 3 1000 0 0 150 140 NA 12000 0 12000 16000 -1000 13000 NA NA NA ± 10 2.5 LL WPT 9 14 L15A 3 1000 0 0 120 150 NA 12000 1000 15000 16000 0 16000 NA NA NA ± 10 2.5 LL WPT 9 15 L15A 3 1000 0 0 120 150 NA 12000 1000 15000 16000 0 16000 NA NA NA ± 10 2.5 LL WPT 9 16 L16A 3 1000 0 0 120 150 NA 16000 -1000 13000 12000 0 12000 NA NA NA ± 10 2.5 LL WPT 9 17 L16A 3 1000 0 0 120 150 NA 16000 -1000 13000 12000 0 12000 NA NA NA ± 10 2.5 LL WPT 9 Head-On Level/Ascending/Descending 18 L12A 3 1000 0 3000 150 180 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 9 19 L52A 3 500 0 3000 150 180 NA 12000 0 12000 12500 0 12500 NA NA NA ± 10 2.5 LL WPT 9 20 L32A 3 300 0 3000 150 180 NA 12000 0 12000 12300 0 12300 NA NA NA ± 10 2.5 LL WPT 9 21 L42C 3 400 45 3000 150 180 NA 12000 0 12000 12400 0 12400 NA NA NA ± 10 2.5 LL WPT 9 22 L55C 3 500 45 3000 130 180 NA 11000 1000 14000 14500 0 14500 NA NA NA ± 10 2.5 LL WPT 9 23 L56C 3 500 45 3000 130 180 NA 16000 -1000 13000 12500 0 12500 NA NA NA ± 10 2.5 LL WPT 9 24 L57C 3 500 45 3000 130 180 NA 11000 1000 14000 17500 -1000 14500 NA NA NA ± 10 2.5 LL WPT 9 25 L53C 3 500 45 3000 150 180 NA 14500 0 14500 11000 1000 14000 NA NA NA ± 10 2.5 LL WPT 9 26 L54C 3 500 45 3000 150 180 NA 12000 0 12000 15500 -1000 12500 NA NA NA ± 10 2.5 LL WPT 9 27 M59R 3 500 0/45 3000 150 180 150 13000 0 13000 13500 0 13500 12500 0 12500 ± 10 2.5 LL WPT 8 28 M59V 3 500 45/90 3000 150 180 150 13000 0 13000 13500 0 13500 12500 0 12500 ± 10 2.5 LL WPT 10 29 L12C 3 1000 45 3000 150 180 NA 12000 0 12000 13000 0 13000 NA NA NA ± 8 2.5 LL WPT 9 30 L52C 3 500 45 3000 150 180 NA 12000 0 12000 12500 0 12500 NA NA NA ± 8 2.5 LL WPT 9 31 L32C 3 300 45 3000 150 180 NA 12000 0 12000 12300 0 12300 NA NA NA ± 8 2.5 LL WPT 9 Right 32 L13C 3 1000 45 0 150 140 NA 16500 0 16500 12500 1000 15500 NA NA NA ± 10 2.5 LL WPT 9 33 L13C 3 1000 45 0 150 140 NA 16500 0 16500 12500 1000 15500 NA NA NA ± 10 2.5 LL WPT 9 34 L14C 3 1000 45 0 150 140 NA 12000 0 12000 16000 -1000 13000 NA NA NA ± 10 2.5 LL WPT 9 35 L14C 3 1000 45 0 150 140 NA 12000 0 12000 16000 -1000 13000 NA NA NA ± 10 2.5 LL WPT 9 36 L15C 3 1000 45 0 120 150 NA 12000 1000 15000 16000 0 16000 NA NA NA ± 10 2.5 LL WPT 9 37 L15C 3 1000 45 0 120 150 NA 12000 1000 15000 16000 0 16000 NA NA NA ± 10 2.5 LL WPT 9 38 L16C 3 1000 45 0 120 150 NA 16000 -1000 13000 12000 0 12000 NA NA NA ± 10 2.5 LL WPT 9 39 L16C 3 1000 45 0 120 150 NA 16000 -1000 13000 12000 0 12000 NA NA NA ± 10 2.5 LL WPT 9 40 L12M 3.25 1000 45 0 150 180 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 9 45° Crossing Level/Ascending /Descending Left-to- 41 L12M 3.25 1000 45 0 150 180 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 9 42 L42D 3 400 90 3000 150 180 NA 12000 0 12000 12400 0 12400 NA NA NA ± 10 2.5 LL WPT 10 43 L55D 3 500 90 3000 130 180 NA 11000 1000 14000 14500 0 14500 NA NA NA ± 10 2.5 LL WPT 10 44 L56D 3 500 90 3000 130 180 NA 16000 -1000 13000 12500 0 12500 NA NA NA ± 10 2.5 LL WPT 10 45 L57D 3 500 90 3000 130 180 NA 11000 1000 14000 17500 -1000 14500 NA NA NA ± 10 2.5 LL WPT 10 46 L53D 3 500 90 3000 150 180 NA 14500 0 14500 11000 1000 14000 NA NA NA ± 10 2.5 LL WPT 10 47 L54D 3 500 90 3000 150 180 NA 12000 0 12000 15500 -1000 12500 NA NA NA ± 10 2.5 LL WPT 10 48 M59S 3 500 0/90 3000 150 180 150 13000 0 13000 13500 0 13500 12500 0 12500 ± 10 2.5 LL WPT 9 49 L12D 3 1000 90 3000 150 180 NA 12000 0 12000 13000 0 13000 NA NA NA ± 5 2.5 LL WPT 10 50 L52D 3 500 90 3000 150 180 NA 12000 0 12000 12500 0 12500 NA NA NA ± 5 2.5 LL WPT 10 51 L32D 3 300 90 3000 150 180 NA 12000 0 12000 12300 0 12300 NA NA NA ± 5 2.5 LL WPT 10 52 L13D 3 1000 90 0 150 140 NA 16500 0 16500 12500 1000 15500 NA NA NA ± 10 2.5 LL WPT 10 53 L13D 3 1000 90 0 150 140 NA 16500 0 16500 12500 1000 15500 NA NA NA ± 10 2.5 LL WPT 10 to-Right 54 L14D 3 1000 90 0 150 140 NA 12000 0 12000 16000 -1000 13000 NA NA NA ± 10 2.5 LL WPT 10 55 L14D 3 1000 90 0 150 140 NA 12000 0 12000 16000 -1000 13000 NA NA NA ± 10 2.5 LL WPT 10 56 L15D 3 1000 90 0 120 150 NA 12000 1000 15000 16000 0 16000 NA NA NA ± 10 2.5 LL WPT 10 57 L15D 3 1000 90 0 120 150 NA 12000 1000 15000 16000 0 16000 NA NA NA ± 10 2.5 LL WPT 10 58 L16D 3 1000 90 0 120 150 NA 16000 -1000 13000 12000 0 12000 NA NA NA ± 10 2.5 LL WPT 10 59 L16D 3 1000 90 0 120 150 NA 16000 -1000 13000 12000 0 12000 NA NA NA ± 10 2.5 LL WPT 10 60 L12E 3 1000 110 0 150 180 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 10 61 L12E 3 1000 110 0 150 180 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 10 62 L12N 3.5 1000 90 0 150 180 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 9 90° / 110° Crossing Level/Ascending/Descending Left- 63 L12N 3.5 1000 90 0 150 180 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 9 64 L42F 3 400 135 3000 150 180 NA 12000 0 12000 12400 0 12400 NA NA NA ± 10 2.5 LL WPT 1 65 L55F 3 500 135 3000 130 180 NA 11000 1000 14000 14500 0 14500 NA NA NA ± 10 2.5 LL WPT 1 66 L56F 3 500 135 3000 130 180 NA 16000 -1000 13000 12500 0 12500 NA NA NA ± 10 2.5 LL WPT 1 67 L57F 3 500 135 3000 130 180 NA 11000 1000 14000 17500 -1000 14500 NA NA NA ± 10 2.5 LL WPT 1 68 L53F 3 500 135 3000 150 180 NA 14500 0 14500 11000 1000 14000 NA NA NA ± 10 2.5 LL WPT 1 69 L54F 3 500 135 3000 150 180 NA 12000 0 12000 15500 -1000 12500 NA NA NA ± 10 2.5 LL WPT 1 70 M59T 3 500 0/135 3000 150 180 150 13000 0 13000 13500 0 13500 12500 0 12500 ± 10 2.5 LL WPT 6 135° Overtaking Level/Ascending /Descending Left- 71 M59W 3 500 90/135 3000 150 180 150 13000 0 13000 13500 0 13500 12500 0 12500 ± 10 2.5 LL WPT 10 72 H42A 3 400 0 3000 150 300 NA 12000 0 12000 12400 0 12400 NA NA NA ± 10 2.5 LL WPT 2 73 H42C 3 400 45 3000 150 300 NA 12000 0 12000 12400 0 12400 NA NA NA ± 10 2.5 LL WPT 10 74 H42D 3 400 90 3000 150 300 NA 12000 0 12000 12400 0 12400 NA NA NA ± 10 2.5 LL WPT 10 75 H42F 3 400 135 3000 150 300 NA 12000 0 12000 12400 0 12400 NA NA NA ± 10 2.5 LL WPT 1 76 M59R 3 500 0/45 3000 150 300 150 13000 0 13000 13500 0 13500 12500 0 12500 ± 10 2.5 LL WPT 6 77 M59S 3 500 0/90 3000 150 300 150 13000 0 13000 13500 0 13500 12500 0 12500 ± 10 2.5 LL WPT 6 78 M59T 3 500 0/135 3000 150 300 150 13000 0 13000 13500 0 13500 12500 0 12500 ± 10 2.5 LL WPT 6 79 H12A 3 1000 0 0 130 210 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 6 80 H12A 3 1000 0 0 130 210 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 6 81 H12A 3 1000 0 0 130 250 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 6 0/45/90/135/180 82 H12A 3 1000 0 0 130 250 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 6 83 H12A 3 1000 0 0 210 130 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 9 84 H12A 3 1000 0 0 210 130 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 9 High Speed Level Encounters 85 H12A 3 1000 0 0 250 130 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 9 Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 86 H12A 3 1000 0 0 250 130 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 9 87 H12C 3 1000 45 0 130 210 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 9 88 H12C 3 1000 45 0 130 210 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 9 89 H12C 3 1000 45 0 130 250 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 9 90 H12C 3 1000 45 0 130 250 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 9 91 H12C 3 1000 45 0 210 130 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 9 92 H12C 3 1000 45 0 210 130 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 9 93 H12C 3 1000 45 0 250 130 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 9 94 H12C 3 1000 45 0 250 130 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 9 95 H12D 3 1000 90 0 130 210 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 10 96 H12D 3 1000 90 0 130 210 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 10 97 H12D 3 1000 90 0 130 250 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 10 98 H12D 3 1000 90 0 130 250 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 10 99 H12D 3 1000 90 0 210 130 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 9 100 H12D 3 1000 90 0 210 130 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 9 101 H12D 3 1000 90 0 250 130 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 9 102 H12D 3 1000 90 0 250 130 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 9 103 H12H 3 1000 180 0 210 130 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 9 104 (Deleted) 3 1000 180 0 210 130 NA 12000 0 12000 13000 0 13000 NA NA NA LL WPT 9 105 H12H 3 1000 180 0 250 130 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 2.5 LL WPT 9 High Speed Level Encounters 0/45/90/135/180 106 3 1000 0 250 130 NA NA (Deleted) 180 12000 0 12000 13000 0 13000 NA NA LL WPT 9 107 L12A (1) 3.5 1000 0 3000 150 180 NA 4200 0 4200 5200 0 5200 NA NA NA ± 10 3 LL WPT 7 108 L12A (2) 3.5 1000 0 3000 150 180 NA 5200 0 5200 6200 0 6200 NA NA NA ± 10 3 LL WPT 3 109 L12A (3) 3.5 1000 0 3000 150 180 NA 6200 0 6200 7200 0 7200 NA NA NA ± 10 3 LL WPT 7 110 L12A (4) 3.5 1000 0 3000 150 180 NA 7200 0 7200 8200 0 8200 NA NA NA ± 10 3 LL WPT 3 111 L11A (1) 3.5 1000 0 3000 150 180 NA 5200 0 5200 4200 0 4200 NA NA NA ± 10 3 LL WPT 7 112 L11A (2) 3.5 1000 0 3000 150 180 NA 6200 0 6200 5200 0 5200 NA NA NA ± 10 3 LL WPT 3 113 L11A (3) 3.5 1000 0 3000 150 180 NA 7200 0 7200 6200 0 6200 NA NA NA ± 10 3 LL WPT 7 114 L11A (4) 3.5 1000 0 3000 150 180 NA 8200 0 8200 7200 0 7200 NA NA NA ± 10 3 LL WPT 3 115 L32G (110) 5 300 160 3000 150 180 NA 12000 0 12000 12300 0 12300 NA NA NA ± 10 3.5 LL WPT 1 116 L31G (110) 5 300 160 3000 150 180 NA 12300 0 12300 12000 0 12000 NA NA NA ± 10 3.5 LL WPT 1 117 L53G (110) 5 500 160 3000 150 180 NA 16500 0 16500 11000 1000 16000 NA NA NA ± 10 3.5 LL WPT 1 118 L55G (110) 5 500 160 3000 150 180 NA 11000 1000 16000 16500 0 16500 NA NA NA ± 10 3.5 LL WPT 1 119 L54G (110) 5 500 160 3000 150 180 NA 11000 0 11000 16500 -1000 11500 NA NA NA ± 10 3.5 LL WPT 1 120 L56G (110) 5 500 160 3000 150 180 NA 16500 -1000 11500 11000 0 11000 NA NA NA ± 10 3.5 LL WPT 1 Radar Runs for GA 121 L32G (90) 6 300 160 3000 150 160 NA 12000 0 12000 12300 0 12300 NA NA NA ± 10 3.5 LL WPT 1 122 L31G (90) 6 300 160 3000 150 160 NA 12300 0 12300 12000 0 12000 NA NA NA ± 10 3.5 LL WPT 1 123 L53G (90) 6 500 160 3000 150 160 NA 17500 0 17500 11000 1000 17000 NA NA NA ± 10 3.5 LL WPT 1 124 L55G (90) 6 500 160 3000 150 160 NA 11000 1000 17000 17500 0 17500 NA NA NA ± 10 3.5 LL WPT 1 125 L54G (90) 6 500 160 3000 150 160 NA 11000 0 11000 17500 -1000 11500 NA NA NA ± 10 3.5 LL WPT 1 126 L56G (90) 6 500 160 3000 150 160 NA 17500 -1000 11500 11000 0 11000 NA NA NA ± 10 3.5 LL WPT 1 127 L12P 5 1000 65/115 0 150 150 NA 12000 0 12000 13000 0 13000 NA NA NA ± 10 3.5 LL WPT 8 128 L32A 3 300 0 3000 150 180 NA 12000 0 12000 12300 0 12300 NA NA NA ± 10 2.5 LL WPT 9 129 L32C 3 300 45 3000 150 180 NA 12000 0 12000 12300 0 12300 NA NA NA ± 10 2.5 LL WPT 9 130 L32D 3 300 90 3000 150 180 NA 12000 0 12000 12300 0 12300 NA NA NA ± 10 2.5 LL WPT 10 131 L32F 3 300 135 3000 150 180 NA 12000 0 12000 12300 0 12300 NA NA NA ± 10 2.5 LL WPT 1 132 L31A 3 300 0 3000 150 180 NA 12300 0 12300 12000 0 12000 NA NA NA ± 10 2.5 LL WPT 9 133 L31C 3 300 45 3000 150 180 NA 12300 0 12300 12000 0 12000 NA NA NA ± 10 2.5 LL WPT 9 134 L31D 3 300 90 3000 150 180 NA 12300 0 12300 12000 0 12000 NA NA NA ± 10 2.5 LL WPT 10 135 L31F 3 300 135 3000 150 180 NA 12300 0 12300 12000 0 12000 NA NA NA ± 10 2.5 LL WPT 1 136 L53A 3 500 0 3000 150 180 NA 14500 0 14500 11000 1000 14000 NA NA NA ± 10 2.5 LL WPT 9 137 L53C 3 500 45 3000 150 180 NA 14500 0 14500 11000 1000 14000 NA NA NA ± 10 2.5 LL WPT 9 138 L53D 3 500 90 3000 150 180 NA 14500 0 14500 11000 1000 14000 NA NA NA ± 10 2.5 LL WPT 10 139 L53F 3 500 135 3000 150 180 NA 14500 0 14500 11000 1000 14000 NA NA NA ± 10 2.5 LL WPT 1 140 L55A 3 500 0 3000 150 180 NA 11000 1000 14000 14500 0 14500 NA NA NA ± 10 2.5 LL WPT 9 141 L55C 3 500 45 3000 150 180 NA 11000 1000 14000 14500 0 14500 NA NA NA ± 10 2.5 LL WPT 9 142 L55D 3 500 90 3000 150 180 NA 11000 1000 14000 14500 0 14500 NA NA NA ± 10 2.5 LL WPT 10 143 L55F 3 500 135 3000 150 180 NA 11000 1000 14000 14500 0 14500 NA NA NA ± 10 2.5 LL WPT 1 144 L54A 3 500 0 3000 150 180 NA 12000 0 12000 15500 -1000 12500 NA NA NA ± 10 2.5 LL WPT 9 145 L54C 3 500 45 3000 150 180 NA 12000 0 12000 15500 -1000 12500 NA NA NA ± 10 2.5 LL WPT 9 146 L54D 3 500 90 3000 150 180 NA 12000 0 12000 15500 -1000 12500 NA NA NA ± 10 2.5 LL WPT 10 147 L54F 3 500 135 3000 150 180 NA 12000 0 12000 15500 -1000 12500 NA NA NA ± 10 2.5 LL WPT 1 148 L56A 3 500 0 3000 150 180 NA 15500 -1000 12500 12000 0 12000 NA NA NA ± 10 2.5 LL WPT 9 149 L56C 3 500 45 3000 150 180 NA 15500 -1000 12500 12000 0 12000 NA NA NA ± 10 2.5 LL WPT 9 150 L56D 3 500 90 3000 150 180 NA 15500 -1000 12500 12000 0 12000 NA NA NA ± 10 2.5 LL WPT 10 TCAS Mitigated Runs for GA 151 L56F 3 500 135 3000 150 180 NA 15500 -1000 12500 12000 0 12000 NA NA NA ± 10 2.5 LL WPT 1 152 L32B 3 300 20 3000 150 180 NA 12000 0 12000 12300 0 12300 NA NA NA ± 10 2.5 LL WPT 9 153 L32G 3 300 160 3000 150 180 NA 12000 0 12000 12300 0 12300 NA NA NA ± 10 2.5 LL WPT 9 154 L32H 3 300 180 3000 150 180 NA 12000 0 12000 12300 0 12300 NA NA NA ± 10 2.5 LL WPT 9 155 L31B 3 300 20 3000 150 180 NA 12300 0 12300 12000 0 12000 NA NA NA ± 10 2.5 LL WPT 9 156 L31G 3 300 160 3000 150 180 NA 12300 0 12300 12000 0 12000 NA NA NA ± 10 2.5 LL WPT 9 157 L31H 3 300 180 3000 150 180 NA 12300 0 12300 12000 0 12000 NA NA NA ± 10 2.5 LL WPT 9 158 L57A 3 500 0 3000 150 180 NA 11000 1000 14000 17500 -1000 14500 NA NA NA ± 10 2.5 LL WPT 9 159 L57D 3 500 90 3000 150 180 NA 11000 1000 14000 17500 -1000 14500 NA NA NA ± 10 2.5 LL WPT 10 160 M67Q 3.1 300/700 0 3000 150 180 180 13300/13700 1000/-1000 13700/13000 14000 0 14000 13000 0 13000 ± 10 2.5 LL WPT 6 161 M68Q 3.1 300/700 0 3000 150 180 180 13700/13300 -1000/1000 13300/14000 14000 0 14000 13000 0 13000 ± 10 2.5 LL WPT 6 162 M27Q 3.2 200/700 0 3000 150 180 180 13200/13700 1000/-1000 13700/13000 13900 0 13900 13000 0 13000 ± 10 2.5 LL WPT 6 163 M28Q 3.2 200/700 0 3000 150 180 180 13700/13200 -1000/1000 13200/13900 13900 0 13900 13000 0 13000 ± 10 2.5 LL WPT 6 164 L42M 3 400 45 3000 150 160 NA 12000 0 12000 12400 0 12400 NA NA NA ± 10 2.5 LL WPT 8 165 L52M (1) 3 500 45 14000 150 180 NA 12000 0 12000 12500 0 12500 NA NA NA ± 10 2.5 LL WPT 8 166 L52M (2) 3 500 45 14000 150 180 NA 12000 0 12000 12500 0 12500 NA NA NA ± 10 2.5 LL WPT 2 167 L52M (3) 3 500 45 12150 150 180 NA 12000 0 12000 12500 0 12500 NA NA NA ± 10 2.5 LL WPT 8 168 L52M (4) 3 500 45 3000 150 160 NA 12000 0 12000 12500 0 12500 NA NA NA ± 10 2.5 LL WPT 8 169 M79X (1) 3 400/500 45 3000 150 160 180 13000 0 13000 13400 0 13400 12500 0 12500 ± 10 2.5 LL WPT 6 170 M79X (2) 3 400/500 45 14000 150 180 180 13000 0 13000 13400 0 13400 12500 0 12500 ± 10 2.5 LL WPT 6 CPDS Runs for GA 171 M79X (3) 3 400/500 45 12150 150 180 180 13000 0 13000 13400 0 13400 12500 0 12500 ± 10 2.5 LL WPT 6 Algorithm Type Special Cases JADEM Multiship KIAS Stratway+ Low Altitude CPDS High-Speed O/S Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 6.1.5 Flight Card Description Flight cards for Configuration 1 were developed based on cards created during ACAS Xu by personnel from Massachusetts Institute of Technology (MIT) Lincoln Laboratory.

These flight cards, which were atypical, were used for similar type encounters during that flight test.

Due to the success of that program and card clarity, a similar format was used for Flight Test Series 3. With the collaborative effort of FT3 Ikhana Operations, Armstrong IT&E Operations, and researcher input, the product was designed to provide a simple, easy- to-use, and easily modifiable card that met researcher requirements for mission success.

The cards also presented a familiar format to that of an instrument approach plate which enabled the aircrew to quickly determine test parameters and critical flight information.

An additional factor that was taken into the design was that of Human Factors: throughout all scenario cards for Configuration 1, a standard color format was selected:  Ownship and middle altitude aircraft was green  Intruder 1 and higher altitude aircraft was red  Intruder 2 and lower altitude aircraft was blue This color scheme was chosen for its quick SA for the customer and easy cockpit use.

The issue of color-blindness was considered, but since all of the test pilots had been through physical examination that checks for color blindness, it was a non-factor.

The flight cards were a tremendous success, both visually and for being highly informative and practical. Kudos were received from Ikhana, intruder aircrew, and industry stakeholders alike.

6.1.5.1 Production Flight Cards were built in Excel and were directly linked to the Flight Test Matrix. The matrix had the capability of auto-populating much of the information for the card based on look-up tables from the scenario number: IP/CPA names and coordinates, altitudes, headings, velocities, distances, groundspeeds, lost link mission for Ikhana, on-condition timing, and CPA tolerances. Manual input was required for the sensor selection, deconfliction altitudes, notes, and abort procedures.

The top down view is a geo-referenced image that was built in FalconView, and furthermore auto-generated by a program called Excel2FV. Excel2FV plotted and created FalconView files automatically by taking user-grouped identical geometries from Excel as an input. Once the geometries/files were created, IP/CPA icons were manually added onto the FalconView files and a normal image file was created. These image files were moved to Microsoft PowerPoint® and aircraft icons added. The top-down views were then pasted onto the flight cards.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Vertical profiles were built in PowerPoint. Groups of similar encounters were easily pasted onto the flight cards.

The cards were designed to fit on an 8.5”x11” sheet of paper, with one half dedicated to ownship and the other to intruder. This allowed users to either cut the deck in half or fold their card to the one of interest. For multi-ship, an additional sheet of standard size paper was required.

Since there were over 200 test points, over 200 unique flight cards were created in this fashion for Configuration 1. Each flight card had its own spreadsheet and the cards were later converted into PDF, packaged into a document for that particular flight day, and distributed in soft- and hard-copy form to all FT3 participants.

6.1.5.2 Breakdown Figure 34 and Figure 35 below show an example of the Flight Cards used for Configuration 1. The following is a breakdown of card sections.

6.1.5.2.1 Ownship Ownship cards showed the aircraft route of flight as green. Here are the main cards sections, listed alphabetically, and a brief explanation for each:  Abort Alt: This altitude is defined as the safe escape altitude the aircraft were expected to immediately hold and maintain if an abort is called. For the ownship, abort altitudes normally required no maneuver other than to remain level, or during climbs and descents to level off at the identified level off altitude. The encounter would end if an abort was called. The abort altitude provided each participating aircraft at least 1,000 ft of vertical separation.

 Abort Heading: Heading to maintain or fly to in the case of an abort. The encounter would end if an abort was called.

 Aircraft Role: This section specified whether the aircraft was Ownship, Intruder 1, or Intruder 2.

 Aircraft: Callsign of the aircraft for that test card.

 Card Number: The card numbers were chosen the day of flight and represented the flight order.

 Card Type Version: Cards went through several iterations before reaching their final version (9). The reason for versions was to keep track of all format updates made during the production of cards. Changes from version to version included additional information added to top view, notes section added, updated encounter instructions, etc.

 CPA: The CPA was the predicted point where the ownship and intruder(s) would be closest vertically and laterally. For encounters less than 500 ft in separation, this CPA had a 0.5 NM lateral offset. Each CPA had a unique number (i.e. CPA7) and if GPS coordinates for a CPA repeated, the same CPA number would be used.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B The coordinates listed were in two forms for use by the Ikhana operators and intruders. CPAs were denoted by a triangle symbol.

 COMEX: Each encounter had a COMEX – a time where all participants needed to be positioned at the IP or ready to commence the run.

 Configuration: Flight Test Series Configuration. For Configuration 1, two types existed: 1a (low-speed ownship) and 1b (high-speed ownship).

 CPA Tolerance: Tolerance was based on requirements from the researchers’ simulations. The tolerance required the aircraft to be at the CPA within that number of seconds from each other. Typically the value was ±5, 8, or 10 seconds.

 Deconfliction Alt: Altitude aircraft would fly in between test points.

 Encounter Instructions: These instructions were radio instructions as well as what to expect from the TC. The instructions also showed at least how much time the aircraft needed to be on condition for that run (step 3). On condition was typically 2.5, 3, or 3.5 min depending on the encounter length.

 Final Alt: Expected altitude at the CPA.

 Groundspeed: Encounter horizontal velocity parameter for all participating aircraft. For Flight Test Series 3, all speeds were constant (no acceleration). As mentioned, some encounters required airspeed – this was specified on those unique cards and highlighted.

 Ikhana Lost Link: Each Ikhana CPA had a single waypoint on a Lost Link Mission that the aircraft would go to in the event of this condition.

 IP: The IP was the point in space where the aircraft would need to be for COMEX.

Each IP had a unique number (i.e. IP16) and if GPS coordinates for an IP repeated, the same IP number would be used. The coordinates listed were in two forms for use by the Ikhana operators and intruders. IPs were denoted by a square symbol.

 IP to CPA in NM: The distance between IP and CPA in nautical miles, for reference.

 Magnetic Course: Expected Magnetic Course (MC) between the IP to CPA.

 O/S Pilot Instructions: Instructions specific to that geometry/SUT that the pilot had to execute.

 Profile View: Side view of the encounters, showing initial and final altitudes at IP and CPA respectively, abort procedures (dashed lines), and vertical separation.

 Scenario Name: This name was a quick reference to the type of encounter being executed, using the Scenario Nomenclature. For more information, see Figure 8 .

 Scenario Number: This number was a unique to each encounter. In fact, it carried through from the Flight Test Matrix and was critical to building the cards based on lookup tables. For a full list of scenario numbers, see Table 11 .

 Sensor Selected: The Ikhana had several sensors that could be selected or deselected based on researcher requirements for that encounter. “Selected” Flight Test Series 3: Flight Test Report, October 2015 – REVISION B meant that track data was being fed into the SUT. ADS-B, Radar, or TCAS data could be selected, as well as “Tracker”, which would fuse all three.

 Start Alt: Aircraft starting altitude at the IP.

 SUT: Display for that particular encounter. The choices were AutoResolver 1 or 2, Stratway+, or CPDS.

 TCAS Mode: The TCAS mode would be selected depending on if the Ikhana should perform a maneuver based on TCAS alerts. “OFF” meant that no TCAS alerts would be received, “Advisory” would show the alerts to the pilot and let them decide whether to maneuver b ased on the guidance, and “AUTO” would enable the flight control computer to automatically take control and maneuver based on TCAS alerts.

 Time Adjust: The time adjust was useful for airspeed encounters. See Section 6.1.2.1.1 for more information.

 Time Hack: A time hack (based off of UTC as displayed in the SAF) was performed prior to starting encounters. The time hack allowed all participants to sync up their clocks prior to COMEX. Time hack was called by the TC, and although each card had a section for this, it was only performed once for each flight day.

 Top View: Geo-referenced top view of the encounter (based off of FalconView) showing what the ideal case would look like. CPA and IP are shown for the aircraft for that particular card.

 Vertical Velocity : Climb or descent rate of the aircraft in feet per minute.

 VID Notice: If an encounter was <500 ft in vertical separation, a notice was displayed on the top view to warn participants that a VID was required by 1 NM lateral separation or an abort would be called.

6.1.5.2.2 Intruder Intruder cards showed the aircraft route of flight as either red (Intruder 1) or blue (Intruder 2). However, the cards were almost identical to the ownship cards:  No Sensor Select : The intruder aircraft did not have the same sensor selection capabilities for inputting to the SUT; thus, this section was not required.

 Intruder Pilot Instructions : These instructions varied from the ownship and gave SA to the pilot and how the ownship would be performing.

Not depicted on example card:  Maneuver Point : The MP was the point the intruder aircraft was expected to fly to and then perform a standard rate turn to another set of coordinates. The MP was denoted by a circle symbol.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Figure 34. Pairwise Encounters Ownship Example Test Card.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Figure 35. Pairwise Encounters Intruder Example Test Card.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 6.2 Configuration 2: Full Mission Encounters Full Mission flight encounters, also identified as Configuration 2, followed a pre-planned flight plan that represents a fictitious Fireline route mission flown in Oakland Center Class E airspace (ZOA) that has been previously used for Integrated Human in the Loop (IHITL) and Full Mission simulation exercises. FT3 Full Mission would gather real flight data to improve simulation.

These missions involved a single ownship aircraft (UAS Surrogate T-34C) navigating a flight plan, two live intruder aircraft performing flight encounters that were generally scripted but had flexibility in execution to accommodate real-time changes during the test runs, and finally, multiple virtual intruders that were not displayed to the airborne aircrew but the ownship aircraft maneuvered to avoid based off inputs executed by the subject pilot located in the RGCS. Each live intruder encounter with UAS Surrogate ownship were 1v1 encounters. Figure 36 shows the ownship Fireline route and the expected paths and intercepts of the two live intruders (Intruder 1 – red, Intruder 2 – blue). Figure 37 shows the ownship, Intruder 1, and Intruder 2 routes overlaid, as well as expected live and virtual encounters.

Figure 36. Fireline Ownship Route with Live Intruder Intercepts.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Figure 37. Fireline Routing, Ownship and Live/Virtual Intruders.

NASA 608 acted as a surrogate UAS for this configuration. The aircraft was directly controlled laterally by the RGCS pilot, and NASA 608’s pilot would perform other maneuvers such as airspeed and altitude changes, received by the research computer from VSCS, and relayed to an on-board tablet. To the RGCS pilot, the route appeared to be in ZOA airspace on the VSCS display.

6.2.1 Fireline Route Development The following stakeholders helped develop the Configuration 2 event:  ARC – IT&E team members. These members represented the virtual ATC team and the ARC HSI interests. They were responsible for integrating the route depicted above into ZOA airspace ( Figure 38 ), and with the restrictions required for flight in R-2508.

 ARC – HSI team members were the primary research team. The encounter intercepts they developed would put the test pilot under heavy working conditions in order to evaluate their display.

 AFRC – IT&E team members were responsible for local R-2508 and R-2515 coordination as well as overall flight execution. AFRC was responsible for relaying Flight Test Series 3: Flight Test Report, October 2015 – REVISION B the information to ARC HSI/IT&E to create a Temporary Flight Restriction (TFR) and slightly modify the Fireline route on waypoint 6.

 R-2508 CCB – The Complex Control Board (CCB) is the governing body for the R- 2508 complex and as such represented High Desert TRACON as well as USAF R-2515 interests.

 GRC – Communication – The Glenn team was responsible for the T-34 NASA 608 during Configuration 2.

Figure 38. Full Mission Fireline Route and Encounters.

6.2.2 Constraints and Limitations The Configuration 2 ownship was controlled by the RGCS, whose subject pilot under test was immersed in a virtual ATC, ZOA, and environment. Where aircrew in Configuration 1 flights maneuvered according to the display, aircrew in Configuration 2 coordinated with ATC as if they were actually operating an aircraft in Oakland Center’s airspace.

Furthermore, ownship maneuvers were less predictable in Configuration 2 and resulted in missed encounters as discussed in Section 8 of this report.

Transiting from High Desert TRACON (Joshua) control to SPORT control needed prior coordination and in some cases imposed minor delays, on the order of minutes, while the controllers conducted hand overs. Although entry and use of the Isabella MOA was Flight Test Series 3: Flight Test Report, October 2015 – REVISION B essentially guaranteed for participating aircraft with a Sage or Pancho clearance, entry into R-2515 was not guaranteed as NASA aircraft are assigned a lower priority than many USAF programs.

6.2.3 Flight Card Description Due to its unique nature, Full Mission flight cards were completely developed by Armstrong IT&E Operations for Flight Test Series 3. Because of the distinctive and repeating route for each flight day, one set of cards was created that was used every day: Ownship, Intruder 1, and Intruder 2.

These cards were produced to be easy to use by the airborne users and other ground participants, not designed for the pilot using the SUT (RGCS). The cards were made to clearly show where live encounters were to occur, from what waypoints and holding patterns the intruder pilots should maneuver, and from where the ownship should expect fused reality virtual encounters.

Using the same color scheme but reversed altitudes from Configuration 1:  Ownship and middle altitude aircraft was green  Intruder 1 and lower altitude aircraft was red  Intruder 2 and higher altitude aircraft was blue 6.2.3.1 Production As in Configuration 1, Configuration 2 cards were built in Microsoft Excel®. However, since all coordinates and geometries were provided by Ames IT&E Operations (and there was only one Fireline route), the cards were created manually. The geo-referenced top view images were again created in FalconView, but manually.

6.2.3.2 Breakdown The following is a breakdown and description of unique elements on each of the Configuration 2 flight cards. Figure 39, Figure 40, and Figure 41 depict the flight cards used for Configuration 2.

6.2.3.2.1 Ownship Ownship card depicted the aircraft route of flight as green. Here are the main cards sections, listed alphabetically, and a brief explanation for each:  Abort Procedure: For Configuration 2, the word “abort” did not have the same meaning as in Configuration 1. Since the tracks for Configuration 2 were much longer (~40min), an abort simply meant that the intruder aircraft would “increase vertical separation” (to obtain a separation of at least 500 ft between aircraft) and continue the encounter. If visual was not acquired within the 1 NM range, the pilots would call “ blind ” and the TC would instruct them to hold level, climb, or descend, as required.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B  Aircraft Role: This section specified whether the aircraft was Ownship, Intruder 1, or Intruder 2.

 Aircraft: Callsign of the aircraft for that test card.

 COMEX: Although there are several sections to input COMEX, the Fireline route only had one COMEX at the beginning of the mission. This section was therefore used to show at what time an intruder would push to CPA instead.

 Configuration: Flight Test Series Configuration. Configuration 2 had one Full Mission.

 Deconfliction Alt: Altitude aircraft would fly in between test points. A carry-over from Configuration 1, ultimately participants stayed at their encounter altitude in between runs for efficiency.

 Distance: The distance between IP WP and encounter WP in nautical miles, for reference.

 Fly-to WP: The Fly-to WP is the point the aircraft is expected to fly-to if there is no encounter. For a couple of encounters (3, 4) this is the case since the Live Encounter WP is not on the flight plan for ownship.

 Holding Pattern: The ownship’s holding pattern before COMEX and the start of the Fireline route.

 Live Encounter #: Denotes which Live Encounter on the Fireline is being shown.

The route had 4 live encounters (1, 3 – intruder 1 and 2, 4 – intruder 2).

 Live Encounter WP: The WP at which the intruder is heading to, and the live encounter, will occur.

 Live Encounter: Live encounters occur with one ownship and one intruder (no multi-ship for Configuration 2).

 Magnetic Course: Expected MC at waypoint.

 O/S Enc. Alt: Expected altitude of the ownship at the live encounter.

 O/S Enc. Mag. Course: Expected magnetic course of the ownship at the live encounter.

 Ownship Airspeed: Expected airspeed of the ownship at the live encounter.

 Profile View: Side view of the encounter showing expected altitudes and vertical separation for ownship and intruder.

 TFR: A virtual TFR was added to the Fireline route to keep all aircraft west of specific test areas (per a request from airspace coordination).

 Top View: Geo-referenced top view of the Fireline route showing what the ideal case would look like. The top view shows both the Live and Virtual Encounters, as well as the TFR and holding patterns.

 VID Notice: Since all live encounters were <500 ft for Configuration 2, a notice was displayed on the top view to warn participants that a VID was required by 1 NM lateral separation or an abort would be called.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B  Virtual Encounter: These encounters were only visible to the RGCS pilot. They were displayed to the aircraft pilots to show where a maneuver would be expected by ownship from RGCS.

 Waypoints: List of waypoints for the aircraft to follow for the Fireline.

6.2.3.2.2 Intruder 1  Waypoint IP: The waypoint the intruder aircraft was expected to push out from.

This was denoted as a square symbol and labeled so that the intruder could have quick SA on where to push from.

 Holding Pattern 1: Intruder 1 had two separate holding patterns. This one is before encounter 1 and is located in the southwest.

 Holding Pattern 2: Holding pattern before encounter 3 on the Fireline (intruder 1’s second encounter) located in the northeast.

6.2.3.2.3 Intruder 2  Holding Pattern 1: Holding pattern before encounter 2 on the Fireline (intruder 2’s first encounter) located in the southeast.

 Holding Pattern 2: Holding pattern before encounter 4 on the Fireline (intruder 2’s second encounter) located in the north.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Figure 39. Full Mission Ownship Test Card.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Figure 40. Full Mission Intruder 1 Test Card.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Figure 41. Full Mission Intruder 2 Test Card.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B

7 Flight Summary

The following is a summary of each flight day for Flight Test Series 3.

7.1 Configuration 1 Configuration 1 flight points were conducted from June 17, 2015, to July 24, 2015. Figure 42 shows the prioritization of flights and test points. Scenarios were prioritized in the following ways:  Numbering scheme based on researcher input (Priority 1, 2, 3, 4).

 Build-up approach (higher to lower vertical separation, advisory to AUTO, etc.).

 Ease of flight (airspace transition from one encounter to next).

 Repeating encounters grouped.

 Sun angle consideration.

Test points not completed are grayed out. Test points that are crossed out were removed from the flight schedule by the researcher (deemed unnecessary for mission success).

Figure 42. Prioritization of and Flown Encounters.

The subjective analy sis is according to Armstrong Ops and may differ from researchers’ opinions. Test points are shown in order flown. Flight duration is based on the ownship’s time in the air. COMEX is written in local time. Traffic Advisories (TAs) and RAs are noted.

Altitudes are flight level MSL and at CPA or maneuver start. Sensors selected for that Flight Test Series 3: Flight Test Report, October 2015 – REVISION B encounter are shown (ADS-B, Radar, TCAS, Tracker). The last three columns are timing from CPA (+ is to arrive after, - is to arrive before) or maneuver suggested by display (if type is “Follow”). Boxes with a “ - ” denote missing data. [R] is repeat.

7.1.1 Flight 1: June 17, 2015 Table 12. Config 1 Flight 1 Data.

Flight 1 SUT AutoResolver Duration 5 hours Intruder(s) N3GC # Scenario Type COMEX O/S Alt. TA/RA Int1 Alt. TA/RA Int2 Alt. TA/RA ADS-B RDR TCAS TRC O/S Int1 Int2 1 18-L12A Fly-through, AR1 0758 120 130 TA N/A N/A X X X X -10 -10 N/A 2 19-L52A Fly-through, AR1 0810 120 125 N/A N/A X X X 0 -2 N/A 3 20-L32A Fly-through, AR1 0820 120 RA 123 N/A N/A X X X X +10 +8 N/A 4 29-L12C Fly-through, AR1 0834 120 130 N/A N/A X X X X +2 +10 N/A 5 49-L12D Fly-through, AR1 0843 120 130 N/A N/A X X X X 0 +6 N/A 6 60-L12E Fly-through, AR1 0855 120 130 N/A N/A X X X X +10 +10 N/A 7 40-L12M Fly-through, AR1 0910 140 150 N/A N/A X X X X +15 +23 N/A Data 8 62-L12N Fly-through, AR1 0923 140 150 N/A N/A X X X X +0 +1 N/A 9 12-L14A Fly-through, AR1 0938 120 TA 130 N/A N/A X X X X +8 -7 N/A 10 34-L14C Fly-through, AR1 0950 120 130 N/A N/A X X X X +6 -20 N/A 11 54-L14D Fly-through, AR1 1000 120 130 N/A N/A X X X X -1 +5 N/A 12 30-L52C Fly-through, AR1 1013 120 125 N/A N/A X X X X - - N/A 13 30-L52C [R] Fly-through, AR1 1023 120 125 N/A N/A X X X X -2 -5 N/A 14 50-L52D Fly-through, AR1 1033 120 125 N/A N/A X X X X -20 +9 N/A 15 50-L52D [R] Fly-through, AR1 1043 120 125 N/A N/A X X -10 +6 N/A Notes : Altimeter calibration performed (N3GC +50 ft). Level acceleration performed.

Artificial offset was applied for 1,000 ft runs. No 300 ft non-head-on encounters allowed due to no ADS-B in SAF. Used wind matrix for climb/descent encounters.

Encounters : 1 – good, 2 – good, 3 – O/S RA, good, 4 – int1 start off angle ~30deg MC, good, 5 – good, 6 – good, 7 – run 3 min instead of 3.25, int1 turn late, bad, 8 – good, 9 – wind adjust -10s, TA to climb rather than descend (O/S), good, 10 – wind adjust +7s, int1 late, bad, 11 – wind adjust -12s, good, 12 – ABORT lost VID, 13 – good, 14 – O/S late, bad, 15 - good Airspace : Requested Buckhorn, SPORT raised flight test to FL140 then FL150 (temporary) due to MQ-9 lasing in West Range. Affected runs 7, 8.

Baro/Vis : 29.92, clear Wind : 1 – O/S 251/20, int1 220/10, 8 – O/S 205/18, int1 195/20, 10 – int1 256/10, 11 – int1 210/17 Bottom Line : Overall, test points were conducted well. The wind matrix had some errors and went through its first iteration. Although ADS-B was not functional in the SAF, the test was still able to continue for 300 ft “head - on” (0 degree angle into) encounters.

Additionally, Fusion (Tracker) was seeing problems with tracks. Nonetheless the researcher was pleased with data sinc e there was no “real world” data previous to this. A decision was made to audibly announce TA/RA alerts for proceeding flights (all platforms).

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 7.1.2 Flight 2: June 18, 2015 Table 13. Config. 1 Flight 2 Data.

Flight AutoResolver SUT Duration 4.9 hours Intruder(s) N3GC # Scenario Type COMEX O/S Alt. TA/RA Int1 Alt. TA/RA Int2 Alt. TA/RA ADS-B RDR TCAS TRC O/S Int1 Int2 1 31-L32C Fly-through, AR1 0643 120 TA/RA 123 TA N/A N/A X -2 -2 N/A 2 51-L32D Fly-through, AR1 0653 120 TA/RA 123 TA/RA N/A N/A X +8 +6 N/A 3 10-L13A Fly-through, AR1 0704 165 155 N/A N/A X X 0 -10 N/A 4 32-L13C Fly-through, AR1 0715 165 155 N/A N/A X +8 -30 N/A 5 52-L13D Fly-through, AR1 0725 165 155 N/A N/A X X +4 +10 N/A 6 16-L16A Fly-through, AR1 0735 136 120 N/A N/A X 0 +10 N/A 7 38-L16C Fly-through, AR1 0745 130 120 N/A N/A X X X X -2 +1 N/A 8 58-L16D Fly-through, AR1 0755 130 120 N/A N/A X X X X +7 +2 N/A 9 14-L15A Fly-through, AR1 0805 150 160 N/A N/A X X X X +18 -2 N/A 10 36-L15C Fly-through, AR1 0817 150 160 N/A N/A X X X X +5 +4 N/A 11 56-L15D Fly-through, AR1 0826 150 160 N/A N/A X X X X +5 +7 N/A Data 12 10-L13A Fly-through, AR2 0835 165 155 N/A N/A X X X X +9 -7 N/A 13 32-L13C Fly-through, AR2 0845 165 155 N/A N/A X X X X +5 -15 N/A 14 52-L13D Fly-through, AR2 0855 165 155 N/A N/A X X X X 0 +28 N/A 15 16-L16A Fly-through, AR2 0905 130 120 N/A N/A X X X X +6 -1 N/A 16 38-L16C Fly-through, AR2 0916 130 120 N/A N/A X X X X +3 +3 N/A 17 58-L16D Fly-through, AR2 0925 130 120 N/A N/A X X X X -15 +8 N/A 18 12-L14A Fly-through, AR2 0935 120 135 N/A N/A X X X X +1 -20 N/A 19 34-L14C Fly-through, AR2 0945 120 TA 130 TA N/A N/A X X X X +4 -10 N/A 20 54-L14D Fly-through, AR2 0955 120 TA 130 N/A N/A X X X X +5 +17 N/A 21 60-L12E Fly-through, AR2 1005 120 130 N/A N/A X X X X - - N/A 22 60-L12E [R] Fly-through, AR2 1011 120 130 N/A N/A X X X X +1 +4 N/A 23 40-L12M Fly-through, AR2 1020 120 130 N/A N/A X X X -3 -4 N/A Notes : Altimeter calibration performed (N3GC +60 ft). Artificial offset was applied for 1,000 ft runs. Used wind matrix climb/descent encounters. O/S collected additional ADS- B data after run 23.

Encounters : 1 – O/S RA, good, 2 – O/S RA descend, int1 RA climb, good, 3 – wind adjust +13s, int1 off course ~130 KIAS, good, 4 – no wind adjust comm., relax VSI, bad, 5 – raised int1 start alt FL130, wind adjust -20s, relax VSI, good, 6 – wind adjust +15s, good, 7 – wind adjust +8s, wind died to 16kts by end, set 1,100 VSI for Ikhana = 1,000 VSI, good, 8 – wind adjust -19s, good, 9 – wind adjust +20s, O/S sped up KIAS, good, 10 – wind adjust +5s, O/S sped up KIAS, good, 11 – wind adjust -10s, sped up KIAS, good, 12 – no wind adjust, good, 13 – wind adjust +12s, good, 14 – wind adjust -26s, climb too early (should have reset), bad, 15 – wind adjust +20s, good, 16 – wind adjust +8s, good, 17 – decision: no wind adjust, O/S early/int1 late (whole run), bad, 18 – wind adjust +12s, late start descent w/overshoot, bad, 19 – wind adjust +5s, good, 20 – wind adjust +18s (opposite of wind matrix), good, 21 – reset, int1 too slow to make CPA, 22 – good, 23 - good Airspace : No Buckhorn below FL130 and above FL200 (temporary). Did not affect runs.

Baro/Vis : 29.92, clear Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Wind : 1 – O/S 274/23, int1 274/27, 3 – O/S 282/15, int1 296/17, 4 – int1 289/20, 5 – int1 265/18, 6 – O/S 282/15, 7 – O/S 285/25, 8 – O/S 270/15, 9 – O/S 285/20, 10 – O/S 290/19, 11 – O/S 300/10, 12 – int1 252/21, 13 – O/S 270/20, int1 252/16, 14 – int1 281/23, 15 – O/S 280/22, 16 – O/S 285/22, 17 – O/S 260/11, 18 – int1 264/13, 19 – int1 288/21, 20 – int1 256/16 Bottom Line : Overall, test points were conducted well. Additional errors were present in the wind matrix that were corrected after this flight. Team was getting into a flow that helped with obtaining more test runs this day.

7.1.3 Flight 3: June 22, 2015 Table 14. Config. 1 Flight 3 Data.

Flight 3 AutoResolver SUT 4.5 hours Duration Intruder(s) N3GC # Scenario Type COMEX O/S Alt. TA/RA Int1 Alt. TA/RA Int2 Alt. TA/RA ADS-B RDR TCAS TRC O/S Int1 Int2 1 61-L12E Follow AR1 0650 120 130 N/A N/A X R007 0 N/A 2 41-L12M Follow AR1 0659 120 130 N/A N/A X R007 +5 N/A 3 63-L12N Follow AR1 0710 120 130 N/A N/A X R007 +10 N/A 4 13-L14A Follow AR1 0722 120 135 N/A N/A X R007 +4 N/A 5 35-L14C Follow AR1 0731 120 140 N/A N/A X R277 - N/A 6 55-L14D Follow AR1 0742 120 150 N/A N/A X R287 +3 N/A 7 15-L15A Follow AR1 0753 135 160 N/A N/A X L227 0 N/A 8 37-L15C Follow AR1 0803 140 160 N/A N/A X L217 -2 N/A 9 57-L15D Follow AR1 0812 150 160 N/A N/A X None +2 N/A 10 11-L13A Follow AR1 0822 165 145 TA N/A N/A X R277 +16 N/A Data 11 33-L13C Follow AR1 0833 165 135 TA N/A N/A X R277 -7 N/A 12 53-L13D Follow AR1 0844 165 150 N/A N/A X R277 +14 N/A 13 17-L16A Follow AR1 0854 145 120 N/A N/A X R287 0 N/A 14 39-L16C Follow AR1 0905 148 120 N/A N/A X R287 -5 N/A 15 59-L16D Follow AR1 0917 148 120 N/A N/A X None 0 N/A 16 61-L12E Follow AR2 0928 120 130 N/A N/A X L317 -90 N/A 17 41-L12M Follow AR2 0940 120 130 N/A N/A X R277 +15 N/A 18 63-L12N Follow AR2 0950 120 130 N/A N/A X R277 +12 N/A 19 13-L14A Follow AR2 1002 120 145 N/A N/A X R277 -20 N/A 20 35-L14C Follow AR2 1013 120 143 N/A N/A X R267 -15 N/A Notes : Altimeter calibration performed (N3GC +60 ft). Artificial offset was applied for 1,000 ft runs. Used wind matrix climb/descent encounters. All runs were ADS-B only (due to poor Fusion performance).

Encounters : 1 – int1 ground speed 10-15 low, good, 2 – seems excessive turn, bad, 3 – good, 4 – wind adjust +5s, good, 5 – wind adjust +5s, good, 6 – wind adjust -5s, good, 7 – wind adjust -25s, good, 8 – 29.97 (ground), wind adjust -20s, good, 9 – wind adjust +5s, O/S slow and late, bad, 10 – wind adjust +10s, good, 11 – wind adjust +10s, good, 12 – wind adjust -20s, int1 request reset, good, 13 – wind adjust -5s, maneuver and level off, good, 14 – wind adjust -5s, maneuver and level off, good, 15 – wind adjust +5s, maneuver and level off, O/S request reset (too fast), bad, 16 – CBDR, merged, bad, 17 – O/S request reset, good, 18 – CBDR, merged, bad, 19 – wind adjust 0s, int1 request reset, good, 20 – wind adjust +10s, int1 request reset, good Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Airspace : R-2515 FL200 and below, Buckhorn FL100 and above, R-2515 SPORT stay FL120 and above. Did not affect runs.

Baro/Vis : 29.95, clear Wind : 3 – int1 221/19, 4 – int1 209/7, 5 – int1 207/5, 6 – int1 294/8, 7 – O/S 257/18, int1 204/13, 8 – O/S 237/15, int1 160/3, 9 – O/S 217/8, int1 217/10, 10 – O/S 203/6, int1 194/22, 11 – O/S 224/10, int1 232/13, 12 – O/S 167/6, int1 225/20, 13 – O/S 220/8, int1 223/21, 14 – O/S 217/8, int1 256/12, 15 – O/S 195/7, int1 232/18, 19 – O/S 237/18, int1 180/10, 20 – O/S 244/15, int1 232/14 Bottom Line : Overall, test points were conducted well, but maneuvers were only of “acceptable” quality. Display of VSCS (seen in SAF) was showing identical maneuvers for different encounters. This was thought odd as each encounter was of a new geometry.

Wind matrix worked well for this flight day. This flight was the first case where it seemed CPA timing was not as important as intent for maneuvering encounters – algorithm only needed to be alerted, followed by O/S maneuver.

7.1.4 Flight 4: June 24, 2015 Table 15. Config. 1 Flight 4 Data.

Flight 4 CPDS SUT Duration 4.7 hours N3GC Intruder(s) # Scenario Type COMEX O/S Alt. TA/RA Int1 Alt. TA/RA Int2 Alt. TA/RA ADS-B RDR TCAS TRC O/S Int1 Int2 1 137-L53C TCAS Advisory 0640 145 137 TA N/A N/A X X X -2 +5 N/A 2 137-L53C [R] TCAS Advisory 0650 145 TA/RA 140 TA/RA N/A N/A X X X -5 -6 N/A 3 137-L53C TCAS AUTO 0702 145 TA/RA 140 TA/RA N/A N/A X X X +25 0 N/A 4 146-L54D TCAS Advisory 0720 120 TA 130 TA N/A N/A X X X -12 -8 N/A 5 146-L54D [R] TCAS Advisory 0730 120 TA/RA 125 TA/RA N/A N/A X X X 0 +5 N/A 6 146-L54D TCAS AUTO 0741 120 TA/RA 125 TA/RA N/A N/A X X X 0 0 N/A 7 140-L55A TCAS Advisory 0753 135 145 N/A N/A X X X +5 +3 N/A 8 140-L55A [R] TCAS Advisory 0804 140 TA/RA 145 TA/RA N/A N/A X X X +13 +5 N/A 9 140-L55A TCAS AUTO 0813 140 TA/RA 145 TA/RA N/A N/A X X X -2 +3 N/A 10 151-L56F TCAS Advisory 0823 125 TA/RA 120 TA/RA N/A N/A X X X -7 -2 N/A Data 11 151-L56F TCAS AUTO 0833 125 TA/RA 120 TA/RA N/A N/A X X X - 0 N/A 12 115-L32G Radar CBDR (110) 0841 120 TA/RA 123 TA N/A N/A X -40 -30 N/A 13 121-L32G Radar CBDR (90) 0855 120 TA/RA 123 TA/RA N/A N/A X X - -30 N/A 14 115-L32G [R] Radar CBDR (110) 0909 120 TA/RA 124 TA/RA N/A N/A X X -30 -60 N/A 15 117-L53G Radar CBDR (110) 0923 165 TA/RA 160 TA/RA N/A N/A X X -30 -30 N/A 16 124-L55G Radar CBDR (90) 0936 170 TA/RA 175 TA/RA N/A N/A X X -30 -40 N/A 17 128-L32A TCAS Advisory 0955 120 TA/RA 123 TA/RA N/A N/A X X X +7 -1 N/A 18 128-L32A TCAS AUTO 1005 120 TA/RA 123 TA/RA N/A N/A X X X +6 +5 N/A 19 129-L32C TCAS Advisory 1015 120 TA/RA 123 TA/RA N/A N/A X X X -7 -4 N/A 20 129-L32C TCAS AUTO 1025 120 TA/RA 123 TA/RA N/A N/A X X X -3 -7 N/A Notes : Altimeter calibration performed (N3GC +80 ft).

Encounters : 1 – no RA, bad, 2 – raised int1 start alt FL115, O/S RA climb, good, 3 – raised int1 start alt FL115, O/S RA climb, good, 4 – multiple rolex, no RA, bad, 5 – lowered int1 start alt FL150, O/S RA descend, good, 6 – lowered int1 start alt FL150, O/S RA descend, good, 7 – no RA, bad, 8 – raised O/S start alt FL115, O/S RA descend, good, 9 – raised O/S start alt FL115, O/S RA do not climb, good, 10 – O/S RA do not descend, Flight Test Series 3: Flight Test Report, October 2015 – REVISION B good, 11 – O/S RA do not descend, good, 12 – O/S RA do not climb, int1 fast, bad, 13 – O/S RA descend, int1 slow, bad, 14 – angle ~106°, good, 15 – raised int1 start alt FL120, int1 descend 1100 fpm, good, 16 – raised O/S alt FL120, O/S RA do not climb, good, 17 – O/S RA do not climb, good, 18 – O/S RA descend, good, 19 – O/S RA descend, did not descend, bad, 20 – O/S RA descend, good Airspace : Buckhorn active FL200 and below, later cleared FL120-200. Did not affect runs.

Baro/Vis : 29.91, clear Wind : 1 – O/S 167/17, int1 160/16, 2 – O/S 152/17, int1 141/20, 3 – O/S 168/9, int1 156/19, 4 – O/S 147/13, int1 195/8, 5 – O/S 134/13, int1 231/6, 6 – O/S 134/13, int1 278/7, 7 – O/S 184/12, int1 188/12, 8 – O/S 185/10, int1 192/10, 9 – O/S 190/10, int1 calm, 10 – O/S 200/10, int1 240/19, 11 – O/S 280/6, int1 calm, 13 – O/S 188/12, int1 188/17, 14 – O/S 192/12, int1 178/18, 15 – O/S 222/13, 219/15, 16 – O/S 169/16, int1 188/11, 17 – O/S 205/16, int1 155/13, 18 – O/S 208/17, int1 calm, 19 – O/S 205/15, int1 161/11, 20 – O/S 209/15, int1 176/21 Bottom Line : Some starting altitudes needed to be raised or lowered (500 ft) real-time in order for aircraft to achieve desired performance and trigger RAs. Once this occurred, the TCAS system was successfully tested and Ikhana performed maneuvers that were expected for the particular encounter, and in the milestone AUTO mode. For radar CBDR encounters, N3GC attempted to use a bearing tool on board their aircraft – although the tool itself was effective, the encounter angle itself was not completely understood by the crew. Thus, several of the radar encounters ’ relative angle was incorrect. For the encounters where the angle was correct, the radar data was deemed good by the researcher. Altitude redlines were made to proceeding flights cards to meet aircraft performance based on this flight’s outcome. CBDR cards were modified for proceeding flights to better highlight how to run this type of encounter.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 7.1.5 Flight 5: June 26, 2015 Table 16. Config. 1 Flight 5 Data.

Flight CPDS SUT Duration 4.6 hours N3GC, NASA865 Intruder(s) # Scenario Type COMEX O/S Alt. TA/RA Int1 Alt. TA/RA Int2 Alt. TA/RA ADS-B RDR TCAS TRC O/S Int1 Int2 1 108-L12A Radar low alt. 0637 052 062 N/A N/A X -2 +2 N/A 2 107-L12A Radar low alt. 0647 042 052 N/A N/A X X -8 +1 N/A 3 112-L11A Radar low alt. 0657 062 052 N/A N/A X X 0 +2 N/A 4 111-L11A Radar low alt. 0708 052 042 N/A N/A X X +9 0 N/A 5 169-M79X Fly-through, CPDS 0822 130 TA/RA 134 TA/RA 125 X X RT +10 +5 6 170-M79X Fly-through, CPDS 0834 130 134 125 X X - - - 7 171-M79X Fly-through, CPDS 0846 130 TA 134 TA 125 X X +2 0 -4 8 160-M67Q TCAS Advisory 0858 133 TA/RA 14 TA/RA 13 TA X X X - - - Data 9 161-M68Q TCAS Advisory 0909 137 TA/RA 14 TA/RA 13 TA X X X 0 - +8 10 165-L52M Fly-through, CPDS 0924 120 TA 125 TA N/A N/A X X +10 +15 N/A 11 166-L52M Fly-through, CPDS 0937 120 TA 125 TA N/A N/A X X +10 0 N/A 12 167-L52M Fly-through, CPDS 0947 120 TA 125 TA N/A N/A X X -9 0 N/A 13 168-L52M Follow CPDS 0957 120 TA/RA 125 TA/RA N/A N/A X X RT -3 N/A 14 164-L42M Follow CPDS 1007 120 TA/RA 125 RA N/A N/A X X LT 0 N/A 15 132-L31A TCAS Advisory 1018 123 120 N/A N/A X X X - - N/A 16 132-L31A TCAS AUTO 1028 123 120 N/A N/A X X X 0 +5 N/A Notes : Altimeter calibration performed (N3GC +60 ft, NASA865 +190 ft). Additional 200 ft encounters were planned but not performed (TCAS sequential) – alerting achieved with 300 ft separation (runs 8, 9).

Encounters : 1 – good, 2 – good, 3 – good, 4 – good, 5 – O/S was not supposed to maneuver (+not called out), int1 RA climb, good, 6 – int1 heading wrong on maneuver, good, 7 – good, 8 – O/S RA climb, about 20 seconds then O/S RA descend, NO VID int2 on O/S, good, 9 – O/S RA descend, about 8 seconds then O/S RA climb, good, 10 – good, 11 – good, 12 – good, 13 – RA int1 monitor vertical speed, good, 14 – int1 RA adjust vertical speed, good, 15 – O/S RA climb, int1 RA adjust vertical speed, good, 16 – O/S RA climb, int1 RA descend, good Airspace : Received Buckhorn early FL100-200.

Baro/Vis : 29.99, hazy (due to Lake wildfire) Wind : 12 – O/S 160/13, int1 158/19, 14 – O/S 169/14, int1 150/11 Bottom Line : Considering the challenging geometries and conditions (weather), this flight collection day was excellent. The first multi-ship live UAS encounter in flight test history was performed (runs 5, 6, 7). First low-altitude radar runs were performed to test DRR on Ikhana (runs 1, 2, 3, 4). The TCAS multi-ship sequential encounters (runs 8, 9) ran smoothly and safely, all triggering expected alerting in advisory mode. Although a mission rule was violated (run 8, no VID), at no point did the pilots or any other team member feel unsafe or that the flight could not continue. Additional, directive guidance was given to pilots concerning VID after run 8 and in proceeding flights.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 7.1.6 Flight 6: July 7, 2015 Table 17. Config. 1 Flight 6 Data.

Flight 6 SUT Stratw ay+ Duration 4.8 hours Intruder(s) N3GC # Scenario Type COMEX O/S Alt. TA/RA Int1 Alt. TA/RA Int2 Alt. TA/RA ADS-B RDR TCAS TRC O/S Int1 Int2 1 25-L53C Follow Stratw ay+ 0645 145 TA 130 TA N/A N/A X RT 0 N/A 2 46-L53D Follow Stratw ay+ 0655 145 TA/RA 135 TA/RA N/A N/A X X RT 0 N/A 3 68-L53F Follow Stratw ay+ 0707 145 120 N/A N/A X X LT 0 N/A 4 26-L54C Follow Stratw ay+ 0717 120 TA 145 TA N/A N/A X X RT 0 N/A 5 47-L54D Follow Stratw ay+ 0727 120 TA 135 TA N/A N/A X X RT +3 N/A 6 69-L54F Follow Stratw ay+ 0736 120 137 N/A N/A X X LT +7 N/A 7 5-L56A Follow Stratw ay+ 0746 142 TA 125 TA N/A N/A X X RT -2 N/A 8 6-L56B Follow Stratw ay+ 0756 141 TA/RA 125 TA/RA N/A N/A X X LT 0 N/A 9 23-L56C Follow Stratw ay+ 0808 - TA/RA 125 TA/RA N/A N/A X X X None -10 N/A 10 44-L56D Follow Stratw ay+ 0817 150 TA 125 TA N/A N/A X X X RT +3 N/A 11 66-L56F Follow Stratw ay+ 0825 130 125 N/A N/A X None +7 N/A Data 12 66-L56F [R] Follow Stratw ay+ 0842 140 125 N/A N/A X LT early N/A 13 1-L42A Follow Stratw ay+ 0852 120 TA/RA 124 TA/RA N/A N/A X RT 0 N/A 14 2-L42B Follow Stratw ay+ 0902 120 TA/RA 124 RA N/A N/A X X X X LT -13 N/A 15 21-L42C Follow Stratw ay+ 0912 120 TA/RA 124 TA/RA N/A N/A X X X X LT 0 N/A 16 42-L42D Follow Stratw ay+ 0922 120 TA 124 TA N/A N/A X X X X LT 0 N/A 17 64-L42F Follow Stratw ay+ 0940 120 124 N/A N/A X X X X LT 0 N/A 18 20-L32A Follow Stratw ay+ 0950 120 TA 124 TA N/A N/A X X X X RT +2 N/A 19 31-L32C Follow Stratw ay+ 1005 130 TA/RA 133 TA/RA N/A N/A X LT +2 N/A 20 51-L32D Follow Stratw ay+ 1015 130 TA 133 TA N/A N/A X LT 0 N/A 21 68-L53F Follow Stratw ay+ 1027 145 125 N/A N/A X X LT -1 N/A 22 69-L54F Follow Stratw ay+ 1037 120 145 N/A N/A X X X LT early N/A Notes : Altimeter calibration performed (N3GC +60 ft). What looked like a level acceleration was performed (should not have been). All climb/descent leg altitudes were redlined prior to flight to achieve “run - in” type encounters for Stratway+.

Encounters : 1 – good, 2 – odd run (O/S maneuver into int1), good, 3 – good, 4 – VSCS split track, band issues (almost terminate run), bad, 5 – vertical velocity noise in Stratway+, good, 6 – good, 7 – split track, good, 8 – int1 RA descend, good, 9 – maintained heading, O/S RA do not descend, int1 RA descend, bad, 10 – split track, good, 11 – maintained heading, O/S fast, bad, 12 – TC push int1 faster (210 KGS), terminate early due to north airspace activity, good, 13 – O/S RA descend, int1 RA climb, good, 14 – O/S RA descend, int1 RA climb, good, 15 – O/S RA do not climb, int1 RA monitor vertical speed, good, 16 – good, 17 – changed Stratway+ sensitivity, laptop problem and multiple rolex, good, 18 – int1 ended at higher altitude (+100 ft), good, 19 – good, 20 – good, 21 – good, 22 - good Airspace : Multiple calls to SPORT that were unanswered. Stay above FL105 (did not affect runs). SPORT called combat laser, request stay above FL130 (affected runs 19, 20). Later stay above FL100 (did not affect runs).

Baro/Vis : 29.97, clear Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Wind : 4 – O/S 170/25, int1 180/19, 7 – O/S 189/7, int1 175/18, 10 – O/S 193/6.5, int1 207/24, 13 – O/S 160/17, int1 176/20, 18 – O/S 178/13, int1 169/20, 19 – O/S 191/19, int1 134/19, 21 – O/S 176/10, int1 191/13 Bottom Line : The bulk of the data was good but the “split tracks” that kept occurring on VSCS were thought too distracting/incorrect for subsequent days. Thus, in proceeding flights the native Stratway+ display was used in the Ikhana GCS instead of the algorithm being fed through VSCS.

7.1.7 Flight 7: July 9, 2015 Table 18. Config. 1 Flight 7 Data.

Flight SUT Stratw ay+ 4.8 hours Duration Intruder(s) N3GC # Scenario Type COMEX O/S Alt. TA/RA Int1 Alt. TA/RA Int2 Alt. TA/RA ADS-B RDR TCAS TRC O/S Int1 Int2 1 22-L55C Follow Stratw ay+ 0653 132 TA/RA 145 TA/RA N/A N/A X R275 - N/A 2 43-L55D Follow Stratw ay+ 0703 125 145 N/A N/A X R050 +2 N/A 3 65-L55F Follow Stratw ay+ 0713 127 145 N/A N/A X L060 +20 N/A 4 24-L57C Follow Stratw ay+ 0725 122 TA 167 TA N/A N/A X R265 0 N/A 5 45-L57D Follow Stratw ay+ 0735 118 TA 162 TA N/A N/A X L330 +5 N/A 6 67-L57F Follow Stratw ay+ 0743 119 16 N/A N/A X L060 +10 N/A 7 7-L57A Follow Stratw ay+ 0755 120 TA 158 TA N/A N/A X X X R270 - N/A 8 3-L55A Follow Stratw ay+ 0807 131 145 N/A N/A X X X L200 0 N/A 9 4-L55B Follow Stratw ay+ 0819 120 TA 145 TA N/A N/A X X L225 +1 N/A 10 21-L42C Follow Stratw ay+ 0830 120 TA/RA 124 TA/RA N/A N/A X X X X R260 0 N/A 11 42-L42D Follow Stratw ay+ 0840 120 TA 124 TA N/A N/A X X X X L330 +5 N/A Data 12 64-L42F Follow Stratw ay+ 0848 120 TA 124 TA N/A N/A X X X X R130 0 N/A 13 26-L54C Follow Stratw ay+ 0900 120 132 N/A N/A X X X X R285 0 N/A 14 47-L54D Follow Stratw ay+ 0909 120 TA/RA 126 TA/RA N/A N/A X L230 - N/A 15 47-L54D Follow Stratw ay+ 0920 120 TA 140 TA N/A N/A X X X X R030 +4 N/A 16 22-L55C Follow Stratw ay+ 0930 131 TA 145 TA N/A N/A X X X X R270 0 N/A 17 43-L55D Follow Stratw ay+ 0940 127 145 N/A N/A X X X X R060 0 N/A 18 65-L55F Follow Stratw ay+ 0948 - 145 N/A N/A X L050 early N/A 19 24-L57C Follow Stratw ay+ 0958 124 TA 168 TA N/A N/A X X X X R270 0 N/A 20 45-L57D Follow Stratw ay+ 1007 - TA/RA - TA N/A N/A X X X X R055 +5 N/A 21 67-L57F Follow Stratw ay+ 1016 140 TA 145 TA N/A N/A X None - N/A 22 25-L53C Follow Stratw ay+ 1026 145 TA 130 TA N/A N/A X R270 -9 N/A 23 46-L53D Follow Stratw ay+ 1036 145 TA/RA 125 TA N/A N/A X R045 0 N/A Notes : Altimeter calibration performed (N3GC +60 ft). All climb/descent leg altitudes were redlined prior to flight to achieve “ merge intent ” type encounters for Stratway+. Used native Stratway+ display in Ikhana GCS.

Encounters : 1 – O/S RA descend, int1 RA climb, TC slowed int1 10kts, good, 2 – O/S large deviation from CPA, good, 3 – TC slowed int1 10kts, good, 4 – performed maneuver but did not turn back, bad, 5 – good, 6 – TC slowed int1 10kts, resumed speed at turn, good, 7 – good, 8 – good, 9 – good, 10 – O/S RA do not climb, int1 RA monitor speed, 11 – good, 12 – good, 13 – good, 14 – O/S RA do not climb, int1 RA maintain vertical speed, good, 15 – late turn back on course, good, 16 – good, 17 – good, 18 – terminate early due to no airspace north, good, 19 – good, 20 – 200 ft excursion, O/S RA descend, good, 21 – no banding, bad, 22 – good, 23 – O/S RA climb, good Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Airspace : SPORT stay FL105-200 (did not affect runs), C-17 FL060 and below (did not affect runs). FT3 multiple spill-outs north of airspace (coordinated minutes prior with SPORT).

Baro/Vis : 29.81, haze layer, good vis. at altitude Wind : 1 – O/S 220/15.5, int1 235/12, 4 – O/S 191/17, int1 228/24, 7 – O/S 193/16, int1 233/26, 8 – O/S 206/15, int1 238/17, 11 – O/S 174/11, int1 209/9, 13 – O/S 180/19, int1 250/16, 15 – O/S 200/12, int1 234/16, 16 – O/S 205/15, int1 195/16, 18 – O/S 190/11, int1 218/7, 19 – O/S 200/14, int1 237/19, 22 – O/S 220/20, int1 180/21 Bottom Line : Overall, encounters were good and using the native Stratway+ display helped the O/S pilots better understand the banding which they needed to fly. Intruder aircraft had timing issues this day (caused several rolex calls) but did not affect runs.

7.1.8 Flight 8: July 10, 2015 Table 19. Config. 1 Flight 8 Data.

Flight SUT Stratw ay+, CPDS 4.6 hours Duration N3GC, NASA865 Intruder(s) # Scenario Type COMEX O/S Alt. TA/RA Int1 Alt. TA/RA Int2 Alt. TA/RA ADS-B RDR TCAS TRC O/S Int1 Int2 1 8-M59Q Follow Stratw ay+ 0646 130 TA 135 TA 125 X L330 +1 +3 2 8-M59Q Follow Stratw ay+ 0656 130 TA 135 TA 125 X X X X L040 +2 +4 3 28-M59V Follow Stratw ay+ 0705 130 TA 135 TA 125 TA X R290 0 +8 4 28-M59V Follow Stratw ay+ 0715 130 TA 135 TA 125 TA X X X X R290 +5 +4 5 71-M59W Follow Stratw ay+ 0725 130 TA 135 TA 125 TA X L370 +4 +11 6 71-M59W Follow Stratw ay+ 0735 130 TA 135 125 TA X X X X L300 +3 +5 7 9-M59U Follow Stratw ay+ 0745 130 135 125 TA X R370 +1 0 8 27-M59R Follow Stratw ay+ 0755 130 TA/RA 135 TA/RA 125 X R310 +6 0 9 48-M59S Follow Stratw ay+ 0805 130 TA 135 TA 125 TA X R330 +1 0 Data 10 70-M59T Follow Stratw ay+ 0815 130 TA/RA 135 TA/RA 125 TA X L270 +10 0 11 9-M59U Follow Stratw ay+ 0827 130 TA/RA 135 125 TA X X X X R300 0 +5 12 27-M59R Follow Stratw ay+ 0854 130 TA 135 TA 125 X X X X R270 +1 -4 13 48-M59S Follow Stratw ay+ 0905 130 TA 135 TA 125 TA X X X X R285 +2 0 14 70-M59T Follow Stratw ay+ 0915 130 TA 135 TA 125 TA X X X X R303 +4 +5 15 63-L12N Follow Stratw ay+ 0927 120 130 N/A N/A X R330 -20 N/A 16 63-L12N Follow Stratw ay+ 0935 120 130 N/A N/A X X X X R302 -15 N/A 17 63-L12N Follow Stratw ay+ 0943 120 130 N/A N/A X R300 -15 N/A 18 122-L31G Radar CBDR (90) 0953 120 TA/RA 123 TA/RA N/A N/A X X -40 -50 N/A 19 122-L31G [R] Radar CBDR (90) 1010 120 TA/RA 123 TA/RA N/A N/A X X - - N/A 20 125-L54G Radar CBDR (90) 1022 110 TA/RA 115 TA/RA N/A N/A X X - - N/A Notes : Altimeter calibration performed (N3GC +60 ft, NASA865 +100 ft). All climb/descent leg altitudes were redlined prior to flight to achieve “merge intent” type encounters for Stratway+. Used native Stratway+ display in Ikhana GCS. TCAS on NASA865 showed Ikhana 300 ft high. Runs 18, 19, 20 radar runs with CPDS display.

Encounters : 1 – good, 2 – good, 3 – good, 4 – good, 5 – good, 6 – good, 7 – good, int2 late TA from int1, 8 – int1 RA monitor vertical speed, good, 9 – good, int2 late TA from int1, 10 – O/S RA maintain level, int1 monitor/adjust vertical speed, int2 TA from int1, good, 11 – O/S RA do not descend, int2 TA from int1, 12 – good, 13 – good, int2 TA from int1, 14 – good, int2 TA from int1, 15 – good, 16 – good, 17 – good, 18 – int1 RA descend, Flight Test Series 3: Flight Test Report, October 2015 – REVISION B int1 (fast) ahead of O/S, bad, 19 – angle about 80° instead of 90°, O/S RA climb, int1 RA descend, good, 20 – angle 95° most of run and 110° at the end, O/S RA descend, int1 RA monitor vertical speed, good Airspace : Stay below FL230 Buckhorn (did not affect runs). Spin aircraft FL110 and above 45min (did not affect runs).

Baro/Vis : 29.86, clear Wind : 1 – O/S 270/3, 9 – O/S 265/11, int1 223/5, int2 calm, 15 – O/S 145/10, int1 194/15 Bottom Line : Overall data was good. Second day of multi-ship encounters went smoothly as the first. Adjusting the vertical profile caused a lot more alerting for Stratway+ (good).

Runs 18, 19, 20 gathered good radar data due to change in altitude and better understanding of angle requirements from pilots.

7.1.9 Flight 9: July 21, 2015 Table 20. Config. 1 Flight 9 Data.

Flight SUT Stratw ay+ 4.8 hours Duration NASA850, NASA865 Intruder(s) # Scenario Type COMEX O/S Alt. TA/RA Int1 Alt. TA/RA Int2 Alt. TA/RA ADS-B RDR TCAS TRC O/S Int1 Int2 1 72-H42A Follow Stratw ay+ 0705 120 TA 124 N/A N/A X X X L055 +20 N/A 2 72-H42A Follow Stratw ay+ 0715 120 TA/RA 124 N/A N/A X X L045 - N/A 3 73-H42C Follow Stratw ay+ 0723 120 TA/RA 124 TA N/A N/A X X X R270 +25 N/A 4 73-H42C Follow Stratw ay+ 0733 120 TA/RA 124 N/A N/A X X L215 -6 N/A 5 74-H42D Follow Stratw ay+ 0741 120 TA 124 N/A N/A X X X L330 - N/A 6 74-H42D Follow Stratw ay+ 0751 120 TA 124 TA N/A N/A X X L320 - N/A 7 75-H42F Follow Stratw ay+ 0800 120 124 N/A N/A X X X L045 -5 N/A 8 75-H42F Follow Stratw ay+ 0808 120 TA/RA 124 N/A N/A X X L010 0 N/A 9 23-L56C Follow Stratw ay+ 0835 145 TA/RA 125 N/A N/A X X X X R270 - N/A 10 44-L56D Follow Stratw ay+ 0843 142 TA 125 TA N/A N/A X X X X L335 -10 N/A Data 11 66-L56F Follow Stratw ay+ 0851 152 125 TA N/A N/A X X X X L060 - N/A 12 31-L32C Follow Stratw ay+ 0901 120 TA/RA 123 TA N/A N/A X X X X L220 -5 N/A 13 51-L32D Follow Stratw ay+ 0910 120 TA 123 TA N/A N/A X X X X R020 0 N/A 14 23-L56C Follow Stratw ay+ 0926 145 125 TA N/A N/A X L200 -20 N/A 15 44-L56D Follow Stratw ay+ 0935 152 125 TA N/A N/A X L330 +5 N/A 16 66-L56F Follow Stratw ay+ 0944 145 TA/RA 125 TA N/A N/A X L045 - N/A 17 66-L56F Follow Stratw ay+ 0955 152 125 TA N/A N/A X X L030 -20 N/A 18 76-M59R Follow Stratw ay+ 1015 130 TA/RA 135 TA 125 TA X X X L220 - - 19 76-M59R Follow Stratw ay+ 1025 130 TA 135 125 TA X X R295 - - 20 77-M59S Follow Stratw ay+ 1035 130 TA 135 125 TA X X X R340 +10 0 Notes : Altimeter calibration performed (NASA850 +140 ft, NASA865 +100 ft). All climb/descent leg altitudes were redlined prior to flight to achieve “merge intent” type encounters for Stratway+. Used native Stratway+ display in Ikhana GCS. Runs 1-12 int1 NASA850, runs 9-17 int1 NASA865, runs18-20 int1 NASA850, int2 NASA865.

Encounters : 1 – good, 2 – good, 3 – good, 4 – O/S RA do not climb, good, 5 – slow to develop and int1 lagging, bad, 6 – good, 7 – became tail chase run, good, 8 – good, 9 – TC increase int1 10 kts, int1 started FL120 instead of 125, good, 10 – O/S fast, int1 started FL123 instead of 125, bad, 11 – int1 off course on IP, bad, 12 – O/S RA do not climb, good, 13 – good, 14 – good, 15 – int1 fast TC request go card speed, good, 16 – O/S RA Flight Test Series 3: Flight Test Report, October 2015 – REVISION B do not descend, O/S fast, int1 increased speed, good, 17 – good, 18 – good, 19 – good, 20 – good Airspace : Fly below FL200 (did not affect runs), received Buckhorn FL100-200.

Baro/Vis : 29.97, clear Wind : 12 – O/S 212/5 Bottom Line : This was another successful flight day for Stratway+ and the first ever high- speed multi-ship encounters. As the previous multi-ship encounters, these went surprisingly smoothly and gave good data for the researcher. Although int1 was late/early to the CPA for many of the encounters, numerous of these runs were successful due to the “intent” of the intruder – Stratway+ displayed good alerting for the O/S pilot.

7.1.10 Flight 10: July 22, 2015 Table 21. Config. 1 Flight 10 Data.

Flight AutoResolver SUT Duration 3.4 hours NASA865 Intruder(s) # Scenario Type COMEX O/S Alt. TA/RA Int1 Alt. TA/RA Int2 Alt. TA/RA ADS-B RDR TCAS TRC O/S Int1 Int2 1 41-L12M Follow AR1 0635 120 130 TA N/A N/A X 280 -15 N/A 2 63-L12N Follow AR1 0645 120 TA 130 TA N/A N/A X 280 -20 N/A 3 61-L12E Follow AR1 0655 120 TA 130 TA N/A N/A X R010 +5 N/A 4 13-L14A Follow AR1 0705 120 TA 144 TA N/A N/A X R280 -8 N/A 5 35-L14C Follow AR1 0715 120 TA 147 TA N/A N/A X R280 -3 N/A 6 55-L14D Follow AR1 0725 120 TA 147 TA N/A N/A X L320 - N/A 7 15-L15A Follow AR1 0735 138 160 N/A N/A X L230 0 N/A 8 37-L15C Follow AR1 0745 135 160 N/A N/A X R270 -25 N/A Data 9 57-L15D Follow AR1 0755 132 160 N/A N/A X L330 +8 N/A 10 11-L13A Follow AR1 0805 165 TA 140 TA N/A N/A X R270 0 N/A 11 33-L13C Follow AR1 0815 165 138 N/A N/A X R280 +3 N/A 12 53-L13D Follow AR1 0825 165 TA 140 TA N/A N/A X L330 +12 N/A 13 18-L12A Fly-through, AR1 0835 120 130 TA N/A N/A X X Some - N/A 14 29-L12C Fly-through, AR1 0845 120 130 TA N/A N/A X X None -4 N/A 15 49-L12D Fly-through, AR1 0855 120 130 TA N/A N/A X X -30 +9 N/A 16 18-L12A Fly-through, AR1 0905 120 130 N/A N/A X X X X +12 +10 N/A 17 29-L12C Fly-through, AR1 0915 120 130 TA N/A N/A X X X X +5 -4 N/A Notes : No altimeter calibration (all runs >500 ft). Artificial offset was applied for 1,000 ft runs.

Encounters : 1 – good, 2 – good, 3 – good, 4 – wind adjust 0s, good, 5 – wind adjust +20s, good, 6 – wind adjust -10s, good, 7 – wind adjust -10s, O/S kept climbing after maneuver, good, 8 – wind adjust -20s, O/S kept climbing after maneuver, good, 9 – wind adjust +20s, good, 10 – wind adjust -5s, int1 max performance on climb, good, 11 – wind adjust +20s, good, 12 – wind adjust -10s, good, 13 – intermittent headings, fly-through, bad, 14 – no headings, fly-through, bad, 15 – both aircraft too fast, O/S arrive early, int1 corrected, bad, 16 – O/S airspeed low at start of run, good, 17 – good Airspace : Buckhorn received FL100-200, F-35 at FL220 (did not affect runs). Later in day stay FL110-170 (did not affect runs).

Baro/Vis : 29.86, clear Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Wind : 4 – O/S 215/15, int1 168/38, 5 – O/S 220/14, int1 180/26, 6 – O/S 175/13, int1 187/30, 7 – O/S 210/15, int1 180/20, 8 – O/S 210/20, int1 180/25, 9 – O/S 170/15, int1 230/24, 10 – O/S 210/20, int1 160/30, 11 – in1 180/25, 12 – O/S 200/20, int1 190/22 Bottom Line : Overall it was a good, smooth day of data collection. This day had no aborts, rolex calls, or resets – a first for FT3. The wind matrix worked well for the climb/descent encounters.

7.1.11 Flight 11: July 24, 2015 Table 22. Config. 1 Flight 11 Data.

Flight SUT CPDS 3.2 hours Duration NASA865 Intruder(s) # Scenario Type COMEX O/S Alt. TA/RA Int1 Alt. TA/RA Int2 Alt. TA/RA ADS-B RDR TCAS TRC O/S Int1 Int2 1 168-L52M Follow CPDS 0633 120 TA 125 TA N/A N/A X R275 -2 N/A 2 164-L42M Follow CPDS 0643 120 TA/RA 124 TA N/A N/A X L250 +13 N/A 3 159-L57D TCAS Advisory 0652 140 TA/RA 145 TA N/A N/A X X X +7 +7 N/A 4 130-L32D TCAS Advisory 0702 120 TA/RA 123 TA N/A N/A X X X +10 +16 N/A 5 131-L32F TCAS Advisory 0710 120 TA/RA 123 TA N/A N/A X X X +7 -3 N/A 6 152-L32B TCAS Advisory 0720 120 TA/RA 123 TA N/A N/A X X X +10 +7 N/A 7 153-L32G TCAS Advisory 0730 120 TA/RA 123 TA N/A N/A X X X +2 +5 N/A Data 8 154-L32H TCAS Advisory 0738 120 TA/RA 123 TA N/A N/A X X X +15 +15 N/A 9 155-L31B TCAS Advisory 0747 123 TA/RA 120 TA N/A N/A X X X - - N/A 10 156-L31G TCAS Advisory 0756 123 TA/RA 120 TA N/A N/A X X X +5 +1 N/A 11 157-L31H TCAS Advisory 0805 123 TA/RA 120 N/A N/A X X X +5 +9 N/A 12 127-L12P Radar Zig Zag 0815 120 130 TA N/A N/A X X +3 +10 N/A 13 116-L31G Radar CBDR (110) 0824 123 RA 120 TA N/A N/A X X - - N/A 14 135-L31F TCAS Advisory 0832 123 TA/RA 120 TA N/A N/A X X X +8 +1 N/A 15 126-L56G Radar CBDR (90) 0842 115 TA/RA 110 TA N/A N/A X X -30 -32 N/A 16 120-L56G Radar CBDR (110) 0856 115 RA 110 N/A N/A X X -20 -30 N/A Notes : Altimeter calibration performed (NASA865 +100 ft).

Encounters : 1 – virtual offset 500 ft good, 2 – O/S RA descend, good, 3 – O/S RA do not climb, good, 4 – O/S RA do not climb, good, 5 – O/S RA do not climb, good, 6 – O/S RA do not climb, good, 7 – O/S RA descend (before run start), O/S RA descend, int1 fly 190kts per TC, 8 – O/S RA do not climb, good, 9 – O/S RA climb to FL130, good, 10 – O/S RA climb FL130, int1 on O/S camera, good, 11 – O/S RA do not descend (before run start), O/S RA climb to FL130, good, 12 – lagged slightly, good, 13 – O/S RA climb, outside radar field of view, bad, 14 – O/S RA climb, good, 15 – O/S RA do not descend, good, 16 – O/S RA do not descend, int1 hold 200kts per TC, good Airspace : Buckhorn received FL100-200.

Baro/Vis : 29.92, slight haze Wind : Alt Cal – O/S 230/25 Bottom Line : An additional day of data collection of CPDS and radar encounters was well-received. Due to the previous practice of doing these types of encounters, the aircrews and ops team were well-prepared and the data collected good. Although the “Zig - Zag” (run 12) encounter was expected to be more of an S-turn, the researchers were still pleased with the data received. The extra TCAS encounters also allowed all Flight Test Series 3: Flight Test Report, October 2015 – REVISION B geometries shown on the “pinwheel” ( Figure 28 and Figure 29 ) to be flown. This particular day also used the “wing flash” (run 13) technique to acquire VID, something that was considered for future testing.

7.2 Configuration 2 Configuration 2 flights were conducted from July 13, 2015, to August 12, 2015. Table 23 shows a summary of these flight days. A total of 12 preparation flights (Combined Systems Test (CST), Inertial Navigation System (INS), Rehearsal type flights) and 3 data collection flights were performed. As mentioned, the system was not ready for test and did not meet perceived requirements after these 3 runs. Thus, additional data runs were cancelled for Configuration 2 only.

For the rehearsal runs, a truncated route was flown on some flight days. Data runs flew the entire route. Additionally, an altimeter calibration was not performed for Configuration 2 since the aircraft were simulating a normal flight environment.

GRC flights are Glenn only as the participant (NASA608); CSTs, Rehearsals, and Data Flights involved Glenn, Armstrong, and Ames.

Table 23. Configuration 2 Flights.

Day of System Under Planned LIVE Flown LIVE LIVE Points Virtual Points Flight Date Encounter Types Week Test Encounters Encounters Achieved Flown GRC 1 13-Jul-15 M CNPC (None) 0 0 0 0 CST 1 16-Jul-15 Th CNPC Targets of Opportunity 0 0 0 0 CST 2 28-Jul-15 Tu CNPC Live 4 2 0 0 GRC 2 29-Jul-15 W INS (None) 0 0 0 0 CST 3 29-Jul-15 W CNPC (None) 0 0 0 0 GRC 3 30-Jul-15 Th CNPC (None) 0 0 0 0 Rehearsal 1 3-Aug-15 M CNPC Live 8 4 0 0 GRC 4 4-Aug-15 Tu CNPC (None) 0 0 0 0 CST 4 4-Aug-15 Tu CNPC Live, Virtual 4 2 0 2 CST 5 5-Aug-15 W CNPC Live, Virtual 6 5 0 9 Rehearsal 2 6-Aug-15 Th CNPC (None) 8 0 0 0 Rehearsal 3 7-Aug-15 F CNPC Live 8 1 0 0 Data 1 10-Aug-15 M HSI/RGCS Live, Virtual 8 8 7 11 Data 2 11-Aug-15 Tu HSI/RGCS Live, Virtual 8 8 6 9 HSI/RGCS Data 3 12-Aug-15 W Live, Virtual 8 8 5 10 CANCELLED Data 4 13-Aug-15 Th HSI/RGCS Data 5 17-Aug-15 M HSI/RGCS CANCELLED Data 6 18-Aug-15 Tu HSI/RGCS CANCELLED CANCELLED Data 7 19-Aug-15 W HSI/RGCS Data 8 20-Aug-15 Th HSI/RGCS CANCELLED Data 9 21-Aug-15 F HSI/RGCS CANCELLED HSI/RGCS CANCELLED Data 10 24-Aug-15 M Totals 62 38 18 41 Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 7.2.1 Flight 1: July 13, 2015 Table 24. Config 2 Flight 1 Data.

GRC 1 Flight CNPC SUT Duration 3.1 hours None Intruder(s) Notes : RF Characterization Flight, NASA608 flew the Fireline route once as planned to gather data on the Control and Non-Payload Communication (CNPC) RF radiation pattern. The Fireline route was flown a second time with a 10-mile buffer to account for expected deviations during the Configuration 2 Flight Test.

7.2.2 Flight 2: July 16, 2015 Table 25. Config 2 Flight 2 Data.

CST 1 Flight SUT CNPC 1.3 hours Duration None Intruder(s) Notes : Initial system checkout flight to verify command and control function with RGCS and establish valid data flow to the LVC environment from the ADS-B system using targets of opportunity. Issues were observed: 1. Up to 15 second delay from RGCS command to NASA608 response. 2. ADS-B targets were incorrectly displayed on the VSCS and LVC systems. ADS-B traffic was stacked in columns.

7.2.3 Flight 3: July 28, 2015 Table 26. Config 2 Flight 3 Data.

CST 2 Flight SUT CNPC 3.1 hours Duration Intruder(s) NASA865 Notes : Flight to verify ADS-B issues were fixed. Issues observed: 1. Remote GRC team unavailable to start Vehicle Specific Module (VSM) scripts. 2. INS state information stale.

3. Research computer C2 script crashed multiple times. 4. ADS-B traffic data exhibits incorrect conversion to lat/lon. 5. Airspeed not shown on VSCS. 6. LVC Gateway crashed after 1 minute of initializing the GRC VSM script.

Encounters : NASA865 practiced first leg of route for int2 at FL144 while troubleshooting issues for NASA608. COMEX 0820 performed encounter for WP2 as int1. Performed encounter for WP15 as int1.

Airspace : Some activity, moved NASA865 to int1 holding and route.

Baro/Vis : 29.92, hazy but workable Wind : WP2 encounter int1 – 320/11 Bottom Line : For the intruder, it was easier to skip through to waypoints and save time instead of following the pattern on the flight cards. Calls were being made in minutes and Flight Test Series 3: Flight Test Report, October 2015 – REVISION B seconds to CPA, but it was mentioned by the pilots they preferred whole minutes (which was difficult to implement). From an operational perspective, all participants need to be ready at flight time. This was mitigated for future flights.

7.2.4 Flight 4: July 29, 2015 Table 27. Config 2 Flight 4 Data.

Flight GRC 2 INS SUT 2 hours Duration Intruder(s) None Notes : GRC led flight test to troubleshoot INS issues observed during last flight.

Encounters : None.

7.2.5 Flight 5: July 29, 2015 Table 28. Config 2 Flight 5 Data.

Flight CST 3 CNPC SUT Duration 1.85 hours None Intruder(s) Notes : RGCS/LVC systems connected for 30 minutes at the end of the flight. Issues observed: 1. LVC Gateway software crashed multiple times at around 1 minute after GRC VSM traffic script was started. 2. ARC Multi Aircraft Control System (MACS) observer station never received O/S data.

Encounters : None.

7.2.6 Flight 6: July 30, 2015 Table 29. Config 2 Flight 6 Data.

Flight GRC 3 CNPC SUT 1.1 hours Duration Intruder(s) None Notes : GRC led flight test to troubleshoot LVC issues. Afternoon system checkout flight cancelled due to weather.

Encounters : None.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 7.2.7 Flight 7: August 3, 2015 Table 30. Config 2 Flight 7 Data.

Flight Rehearsal 1 CNPC SUT 3.3 hours Duration Intruder(s) NASA7, N3GC Result # Scenario Type Push O/S Alt. TA/RA Int1 Alt. TA/RA Int2 Alt. TA/RA Fireline 1 - COMEX 1637 1 ENC1 Live 16:38:58 140 136 TA N/A N/A - Data 2 ENC2 Live 16:45:36 140 N/A N/A 144 TA/RA 16:46:15 Left turn, int2 RA climb 100ft 3 ENC3 Live 17:06:00 140 136 TA N/A N/A 16:57:29 Left turn 4 ENC4 Live 17:07:30 140 N/A N/A 144 TA 17:10:40 Left turn Notes : No virtual encounters, LVC gateway problems at beginning of flight day. Multiple system restarts. NASA608 originally could not go into nav. mode for any runs.

Encounters : 1 – no lateral offset, bad, 2 – no VID, bad, 3 – good, 4 - good Airspace : No data.

Baro/Vis : 29.88, changed to 29.85 at 1630 Wind : O/S 220/20, int1 250/13, int2 197/11 Bottom Line : Although it was useful to practice the live encounters, to do a full rehearsal, virtual traffic would have been required as well.

7.2.8 Flight 8: August 4, 2015 Table 31. Config 2 Flight 8 Data.

GRC 4 Flight SUT CNPC Duration 2.8 hours Intruder(s) None Notes : Good O/S data and traffic from the VSM. Heading/altitude/speed changes looked good.

Encounters : None.

7.2.9 Flight 9: August 4, 2015 Table 32. Config 2 Flight 9 Data.

Flight CST 4 SUT CNPC Duration 2.9 hours Intruder(s) N3GC # Scenario Type Push O/S Alt. TA/RA Int1 Alt. TA/RA Int2 Alt. TA/RA Result Fireline 1 - COMEX 1707 1 ENC2 Live 17:13:45 140 N/A N/A 144 TA/RA 17:12:43 Left turn, int2 RA climb Data 2 ENC4 Live 17:36:00 140 N/A N/A 144 TA 17:31:59 Left turn Fireline 2 - COMEX 1808 1 VE1 Virtual 140 (140) N/A N/A All virtual encounters Notes : Good live traffic, bad virtual. Inertial Measurement Unit (IMU) data was stale on multiple occasions. NASA608 was able to go into nav. mode.

Encounters : Fireline1: 1 – good, 2 – good; Fireline 2: all virtual, only for testing scripts Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Airspace : Predator FL100-210, F-18s, did not affect runs.

Baro/Vis : 29.89, ENC4 29.90, hazy Wind : int2 237/27, ENC4 int2 226/30 Bottom Line : Actual performance for NASA608 was approximately 500 fpm (as opposed to required 1,000 fpm). On this flight day, it became absolutely clear there was disconnect between the systems here at AFRC and those at/near GRC. What should have been the th second day of data collection was the 9 check flight.

7.2.10 Flight 10: August 5, 2015 Table 33. Config 2 Flight 10 Data.

Flight CST 5 SUT CNPC Duration 3 hours Intruder(s) N3GC # Scenario Type Push O/S Alt. TA/RA Int1 Alt. TA/RA Int2 Alt. TA/RA Result Fireline 1 - COMEX 1528 1 VE1 Virtual 140 - N/A N/A Virtual intruder passed too early.

2 ENC1 Live 15:31:14 140 136 TA N/A N/A 15:33:34 Left turn (due to WP or command?)

3 ENC2 (virt) Virtual 140 (144) N/A N/A 15:38:30 Right turn 4 ENC3 Live 15:42:05 140 136 TA N/A N/A 15:39:18 Left turn, climb to 145 (later) 5 VE2 Virtual 140 - N/A N/A Fly-through WP7 to WP9 Fireline 2 - COMEX 1603 1 VE1 Virtual 140 (145) N/A N/A Fly-through WP1 to WP2 Data 2 ENC1 Live 16:05:58 140 136 TA N/A N/A 16:07:14 Right turn and climb 3 ENC2 (virt) Virtual 148 (144) N/A N/A Descend 140, turn left 4 VE2 Virtual 140 - N/A N/A Fly-through Fireline 3 - COMEX 1636 1 VE1 Virtual 140 (145) N/A N/A Fly-through 2 ENC1 Live 16:40:02 141 136 TA N/A N/A 16:40:46 Expect climb 3 VE2 Virtual 138 (144) N/A N/A 16:45:50 Left turn, climb to 140 4 VE3 Virtual 140 (144) N/A N/A Right turn 5 ENC4 Live 16:53:00 140 N/A N/A 154 16:57:01 Right turn Notes : Stale tracks on LVC system. Even after reboot, VSCS stale data. NASA608 was receiving commands from RGCS late or did not see them. N3GC acted as int1 for Fireline1, in1 for Fireline2, and both int1/int2 for Fireline3. At end of flight, NASA608 was asked to fly a cardinal direction (due west) but this did not look correct in any of the ground displays.

Encounters : Fireline1: 1 – stale data, bad, 2 – int1 late, bad, 3 – good, 4 – good, 5 – RGCS did not have control of NASA608, bad, Fireline2: 1 – good, 2 – good, 3 – good, 4 – good, Fireline3: 1 – good, 2 – good, 3 – good, 4 – good, 5 – int2 climbed to check vertical velocity Airspace : F-35s with tanker coming in to R-2515, may not use airspace for up to 2 hours (did not affect runs).

Baro/Vis : 29.91, good Wind : int1 227/29, Fireline2 ENC1 int1 199/22, Fireline3 ENC1 int1 225/24 Bottom Line : Due to the fuel capacity of the O/S, 3 full runs would not be possible for any of the data runs as planned. Either the Fireline had to be truncated (as was done this flight day) or less runs could be performed per day.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 7.2.11 Flight 11: August 6, 2015 Table 34. Config 2 Flight 11 Data.

Rehearsal 2 Flight CNPC SUT Duration 1.1 hours NASA865, N3GC Intruder(s) Notes : Encounters were cancelled due to weather. All aircraft flew to altitude but could not establish VMC. At end of flight, NASA608 flew on its own to troubleshoot magnetic course problem.

Encounters : None.

Airspace : No data.

Baro/Vis : 29.95, virga/rain, clouds at FL145 Wind : No data.

Bottom Line : The weather was too poor to perform encounters this day. This was unfortunate since it seemed that this would be the first day where all systems are functional.

7.2.12 Flight 12: August 7, 2015 Table 35. Config 2 Flight 12 Data.

Flight Rehearsal 3 CNPC SUT Duration 2.8 hours NASA865, N3GC Intruder(s) Notes : Began as a rehearsal flight but turned more into a CST. Ames was showing traffic somewhere close to the North Pole.

Encounters : N3GC RTB 1641. Only flew a partial encounter due to problems with scripts.

COMEX was set to 1731 and a 500 fpm descent (FL140 to 136) by NASA865 at ENC2 was performed. TA received.

Airspace : F-35 NOTAM, but cancelled (did not affect runs). Affected takeoff time: 1530 as opposed to 1430.

Baro/Vis : 29.84 Wind : 265/37 Bottom Line : Continuing troubleshooting entire system since software changes were made on NASA608 computer without informing other team members. During the encounter, TC/TD were uncertain if visual had been picked up since a “negative visual” was called. A visual was picked up after by the pilots, but there was still uncertainty on the ground. Thus it was communicated to only call positive visuals during flights.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 7.2.13 Flight 13: August 10, 2015 – Data Collection 1 Table 36. Config 2 Flight 13 Data.

Data Collect 1 Flight HSI SUT Duration 2.8 hours NASA865, N3GC Intruder(s) # Scenario Type Push O/S Alt. TA/RA Int1 Alt. TA/RA Int2 Alt. TA/RA Result Fireline 1 - COMEX 1510 1 VE1 Virtual 140 (145) N/A N/A Clear of conflict 15:12:56 2 ENC1 Live 15:45:15 141 136 TA N/A N/A 15:16:03 O/S turned right 3 ENC2 Live 15:17:15 141 N/A N/A 145 TA 15:19:39 turning left 4 VE2 Virutal 141 - N/A N/A Left turn 5 VE3 Virtual 141 (145) N/A N/A - 6 VE3.5 Virtual 141 - N/A N/A Fly-through 7 VE4 Virtual 141 (143) N/A N/A 15:36:07 L273 8 VE5 Virtual 141 (135) N/A N/A Fly-through 9 ENC3 Live 15:41:37 141 137 TA N/A N/A 17:04:01 RT Data 10 ENC4 Live 15:42:39 141 N/A N/A 145 TA/RA 15:50:42 L248, int2 O/S RA climb 100ft Fireline 2 - COMEX 1610 1 VE1 Virtual 141 (145) N/A N/A Fly-through 2 ENC1 Live 16:12:48 141 TA 137 TA N/A N/A 16:15:06 R168 3 ENC2 Live 16:17:12 141 TA N/A N/A 144 TA 16:21:43 R129 4 VE2 Virtual 141 (145) N/A N/A 16:29:32 L107 5 VE3 Virtual 141 (144) N/A N/A 16:34:33 L263 6 VE4 Virtual 141 (145) N/A N/A Fly-through 7 VE5 Virtual 141 (135) N/A N/A Fly-through 8 ENC3 Live 16:38:14 141 (137) TA N/A N/A 16:41:46 R326, South 319 9 ENC4 Live 16:40:49 141 N/A N/A 145 TA 16:47:13 R344 Notes : NASA865 was seeing NASA608 200 ft above.

Encounters : Fireline1: 1 – good, 2 – course correction, bad, 3 – good, 4 – good, 5 – good, 6 – good, 7 – good, 8 – good, 9 – good, 10 – some stale data on RGCS, int2 RA climb 100 ft, good; Fireline2: 1 – good, 2 – good, 3 – good, 4 – good, 5 – good, 6 – good, 7 – good, 8 – good, 9 – good Airspace : F-16 south of HWY58, could not use WP5 for short time (did not affect runs).

Baro/Vis : 29.86, clear at FL140, haze at FL120 and below Wind : int2 152/19, Fireline2 int2 ENC2 201/28 Bottom Line : Overall, it was a good flight day. A few problems were discovered, like turning prior to a WP 5 versus of flying into it – what the researchers expected and what was done did not coincide. Additionally, ENC4 was expected to be mor e of a “beak to beak” enc ounter. A correction was made to the flight cards for a different intruder heading to create this geometry.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 7.2.14 Flight 14: August 11, 2015 – Data Collection 2 Table 37. Config 2 Flight 14 Data.

Data Collect 2 Flight HSI SUT Duration 2.7 hours NASA865, N3GC Intruder(s) # Scenario Type Push O/S Alt. TA/RA Int1 Alt. TA/RA Int2 Alt. TA/RA Result Fireline 1 - COMEX 1510 1 VE1 Virtual 141 (145) N/A N/A Fly-through 2 ENC1 Live 15:12:47 141 136 N/A N/A 15:14:59 L111 3 ENC2 Live 15:18:55 141 N/A N/A 144 TA No maneuver 4 VE2 Virutal 141 (139) N/A N/A 15:25:38 R148 5 VE3 Virtual 141 (145) N/A N/A Fly-through 6 VE4 Virtual 141 (135) N/A N/A Fly-through 7 ENC3 Live 15:38:34 141 136 TA N/A N/A 15:41:39 R301 8 ENC4 Live 15:43:00 141 TA N/A N/A 145 TA/RA 15:47:39 L272 Data Fireline 2 - COMEX 1606 1 VE1 Virtual 141 (145) N/A N/A 16:08:28 L111 2 ENC1 Live 16:08:48 141 TA 136 TA N/A N/A 16:11:06 L115 3 ENC2 Live 16:14:34 141 TA N/A N/A 145 TA 16:17:56 R113, R133 4 VE2 Virtual 141 (140) N/A N/A Left 5 VE3 Virtual 141 (145) N/A N/A 16:25:10 L076 6 VE4 Virtual 141 - N/A N/A Fly-through 7 VE5 Virtual 141 (135) N/A N/A 16:33:29 R334 8 ENC3 Live 16:34:50 141 136 TA N/A N/A No maneuver 9 ENC4 Live 16:37:47 141 TA N/A N/A 145 TA/RA 16:42:27 L262 Notes : N3GC was showing 100 ft high on Zeus at one point.

Encounters : Fireline1: 1 – good, 2 – good, 3 – bad, 4 – good, 5 – good, 6 – good, 7 – O/S seemed to head too much south, int1 RA descend 100 ft, good, 8 – good; Fireline2: 1 – good, 2 – good, 3 – good, 4 – good, 5 – good, 6 – good, 7 – good, 8 – int1 not enough run-in, bad, 9 – int2 RA climb Airspace : Temporarily restricted from using WP 5 (did not affect runs).

Baro/Vis : Fireline1: 29.87, Fireline2: 29.86, clear Wind : int1 190/25, int2 182/36, Fireline2: ENC1 int1 158/25, ENC2 int2 189/29 Bottom Line : Another seemingly good flight day. Although the researchers mentioned that only 6 of the 8 encounters were usable, this was still real flight data and was still considered beneficial. The heading changed employed a day prior for one of the encounters was also more successful.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 7.2.15 Flight 15: August 12, 2015 – Data Collection 3 Table 38. Config 2 Flight 15 Data.

Data Collect 3 Flight HSI SUT Duration 2.5 hours NASA865, N3GC Intruder(s) # Scenario Type Push O/S Alt. TA/RA Int1 Alt. TA/RA Int2 Alt. TA/RA Result Fireline 1 - COMEX 1455 1 VE1 Virtual 139 (145) N/A N/A - 2 ENC1 Live 14:58:18 139 136 N/A N/A 14:59:57 R179 3 ENC2 Live 15:03:02 139 TA N/A N/A 143 TA/RA 15:07:07 R124 4 VE2 Virutal 139 (147) N/A N/A Right 5 VE3 Virtual 139 (132) N/A N/A 15:12:18 L078 6 VE4 Virtual 139 (142) N/A N/A 15:21:23 L322 7 VE5 Virtual 139 (135) N/A N/A - 8 ENC3 Live 15:24:11 139 TA 134 TA N/A N/A No alert on VSCS Data 9 ENC4 Live 15:26:05 139 TA N/A N/A 143 TA No alert on VSCS Fireline 2 - COMEX 1555 1 VE1 Virtual 139 (145) N/A N/A Fly-through 2 ENC1 Live 15:58:12 139 TA 136 TA N/A N/A 16:00:50 L173 3 ENC2 Live 16:01:40 139 N/A N/A 144 TA 16:06:14 R124 4 VE2 Virtual 139 (143) N/A N/A Fly-through 5 VE3 Virtual 139 (145) N/A N/A Fly-through 6 VE4 Virtual 139 (147) N/A N/A 16:18:20 L289 7 VE5 Virtual 139 (135) N/A N/A 16:22:27 R308 8 ENC3 Live 16:22:54 139 136 N/A N/A 16:26:21 L243 9 ENC4 Live 16:25:01 139 TA N/A N/A 143 TA 16:31:18 L300 Notes : Problem observed with RGCS only sending a WP command once when off autopilot. This day marked the highest barometric pressure seen during Flight Test Series 3, topping out at 30.02 inHg. This was showing NASA608 100 ft low on Zeus. During the phase of no alerting during Fireline1, ADS-B targets were showing frozen for Ghost controller.

Encounters : Fireline1: 1 – good, 2 – good, 3 - int2 RA climb, good, 4 – good, 5 – good, 6 – good, 7 – good, 8 – no alerting, bad, 9 – no alerting, bad; Fireline2: 1 – good, 2 – target freeze on NASA608, int1 RA descend 100 ft, bad, 3 – good, 4 – good, 5 – good, 6 – good, 7 – good, 8 – good, 9 – altitude seemed incorrect, bad Airspace : No data.

Baro/Vis : Fireline1: 30.02, Fireline2: 30.00, clear but slight haze Wind : int1 185/20, int2 185/20, Fireline2: int1 180/28, int2 227/22 Bottom Line : This day did not seem successful as others, due to only achieving 5 of the 8 planned live intruder points. Problems with the RGCS/VSCS and also barometric pressure affected system performance. Flight 15 was also the final flight day before additional Configuration 2 flights were cancelled. The researchers recognized that they were not getting the type of data and alerting they required, and also, the system was not acting the way it was thought to (VSGC, etc.). The decision was made to end test and gather lessons learned to make improvements to the Fireline and further flight testing of the Full Mission.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B

8 Observations and Issues

The many months of planning and coordination was a key factor in the success of the Flight Test Series 3 Configuration 1. Although successful, several lessons learned can be drawn and built into future similar flight testing, especially the lessons learned from the decision to forgo Configuration 1b and additional Configuration 2 flights. Those lessons learned are described below.

8.1 Major Impacts The following issues significantly impacted the flights or resulted in mission rule violations.

8.1.1 An intruder was within 1 NM and less than 500 ft vertical separation without being visual on Ikhana.

Background Data: During Flight 5 of Configuration 1, while conducting a challenging multi- intruder sequential TCAS encounter, the first intruder did not acquire a visual acquisition (or callout) of Ikhana and reported this after the encounter had been completed. Further, personnel within the SAF did not recognize the missed call or query the intruder aircrew to callout the visual. The flight was completed without further incident. This violated FT3- 9 mission rule.

Analysis: After post flight discussion it was determined that hazy conditions, test point complexity and an inside cockpit focus were significant contributors to the mission rule violation. It was also determined that pre-planned routing which maintained a 0.5 NM lateral and 300 ft vertical separation was able to maintain an adequate level of safety, although should not be relied upon as a the sole mitigation.

Mission Relation: Should any manned aircraft fail to acquire visual during FT4 or subsequent tests, the consequence may lead to a closer than planned encounter (either laterally or vertically) and may increase the chances of a midair collision.

Recommendation/Conclusion: The test team developed CRM procedures to help aircrew determine when the visual boundary, 1 NM, was approaching:  The new procedures became a mandatory briefing item  The TC was responsible for notifying all participants that the next encounter would require a visual  The TC would provide range calls over the radio and;  At approximately 1.5 NM separation a “Check Visual” call was made giving the aircrew ~5 seconds to acquire a visual or abort the encounter.

No further violations occurred after Flight 5 and these additional procedures should be carried forward into future flight test events.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 8.1.2 Full mission Configuration 2 flights only completed 3 of 10 planned data collection flights before being cancelled.

Background Data: On 12 August Configuration 2 flight test was terminated with only 3 of 10 data collection flights completed. Although 15 flights in total had been completed, 12 of them were system check and troubleshooting events. Additional details on the effectiveness and suitability of the Configuration 2 system are subject to ongoing analysis and not available at the time of drafting this report.

Analysis: ARC-HSI determined that the flight test data would not adequately suit their objectives and recommended termination of the remaining flights.

Recommendation/Conclusion: Ensuring adequate development time prior to FT4 or subsequent flight tests is paramount. The addition of system check periods with a long enough time to allow system enhancements, changes, or fixes, should be mandatory.

8.1.3 Configuration 1b was not attempted.

Background Data/Analysis: The aircraft planned for this test event, the GRC S-3B, was not able to meet the development time constraints and was not in the desired flight test configuration for FT3.

Recommendation/Conclusion: The other research teams felt that the data acquired with the Ikhana ownship was adequate and their Configuration 1 objectives were met.

8.2 Minor Impacts and Lessons Learned The following observations resulted in minor impacts to FT3 or are lessons learned to impart upon FT4 and subsequent flight test events.

8.2.1 Configuration 1 and Configuration 2 flight tests are distinct and separate.

The two flight test configurations were considered as a single test event. A limited amount of time was available between the two configurations to complete all necessary planning, integrating, and approval briefings. The following highlights the distinctions between the two configurations: The workload was significantly different. Configuration 1 flights were workload intensive.

As many as 30 encounters were planned during a flight and each encounter was allotted only 10 minutes to complete including setup. Aircrew needed to navigate to new coordinates, understand the upcoming encounter geometry, and setup to make the IP within 5 to 10 seconds of the COMEX time. Each encounter was distinct and required the above mentioned steps to be re-executed each time. Additionally, up to 3 early morning flights were performed per week, adding to crew rest considerations.

Configuration 2 flights were significantly less intensive for the airborne participants. The majority of the test decisions were made by the RGCS, a ground based resource. The Flight Test Series 3: Flight Test Report, October 2015 – REVISION B airborne participants simply ensured the aircraft was on the planned parameters while the ground element directed the ownship maneuvers. Furthermore, from one run to the next the encounters did not change. Each intruder executed the same two encounters while the ownship expected the same four encounters changed only by the ground element’s response and basic environmental factors such as winds aloft. Configuration 2 flights were afternoon flights flow daily.

That being said, the ownship maneuvers were less predictable in this configuration. The pilot in the RGCS changed from one flight to the next which resulted in a variation of responses to the presented encounters. On one occasion a pilot skipped an encounter altogether having requested and been granted approval by the virtual ATC controller.

Recommendation/Conclusion: Decrease the number of flights per week for early morning sorties (or increase each sortie’s duration). Reduce the planned number of test encounters per sortie in order to provide more time to execute each run. Increase the number of test sorties in order to complete the desired test points.

Allocate more time between configurations. The additional time will help ensure success criteria is clearly understood. Clearly communicate the differences between the two configurations more as test phases than simply configurations.

8.2.2 Multiple operating/staging locations decreased efficiency in test execution.

Operating from KEDW, KVNY, KPMD, and KBFL was a challenge to ensure efficient test execution. On multiple occasions, supporting aircraft were held at their staging locations for ATC clearances. Additionally, the offsite aircraft needed a higher bingo fuel in order to return to their staging location. During Configuration 1 events, one or possibly two more encounters per flight could have been completed if all aircraft were co-located.

During Configuration 2 flights the remote staging location of NASA 608 was planned to rd add the ability for a 3 fire line run; however, the T-34 was too fuel limited and could not rd attempt a 3 run regardless of staging location, and therefore negated the primary benefit of being located off site. Furthermore, the amount of troubleshooting conducted with NASA 608, LVC, and RGCS would have benefitted significantly from being collocated.

Recommendation: Co-locate at AFRC for FT4 and subsequent flight test events.

8.2.3 Low priority within R-2515 resulted in missed flight test opportunities.

Both test configurations were impacted by low prioritization. Configuration 2 operations were more significantly impacted since the routes of flight for both ownship and intruder aircraft utilized more of the airspace than Configuration 1 required and were therefore more in conflict with other airspace users. Additionally, Configuration 2 flights started at 1500L which was an impacted time of day (although this operation time was chosen at the recommendation of USAF airspace management personnel).

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Configuration 1 operations within the Mercury Spin and Four Corners work areas were more contained within known UAS working areas and were therefore less impacted by other users. However, on some occasions the FT3 participating aircraft were restricted to certain altitude blocks or lateral bounds which decreased the amount of encounters completed while the team waited for the traffic to clear. Configuration 1 flights started at 0600L and, in general, operations occurring between 0600L to 0800L tend to experience little to no conflicts with other traffic.

Recommendation: For complex routing like the Configuration 2 route, it would increase the mission success rate if the route of flight remains clear of R-2515. Additionally, planning for early morning operations (from 0600L to 0800L) will likely increase the success rate for executing and completing the test as planned.

8.2.4 Planning for nominal and off-nominal conditions was not clear or distinct enough.

Some discussion in brief and de-brief was spent on the differences between Lost Link Altitude and Deconfliction Altitude during the briefing and de-brief. This resulted in some confusion on which altitude the aircraft should fly after the encounter. Here are the expectations and recommendations from the operations group: Deconfliction Altitude : This is the planned altitude listed on the card that all aircraft should be at after completing the encounter and with TC coordination. This is nominal condition.

Lost Link Altitude : This is the altitude Ikhana will be at in a contingency event where the aircraft loses link. It is designed to keep the aircraft predictable for the other intruder aircraft for a short period of time prior to Ikhana proceeding to lost link holding points. This is an off-nominal condition and the altitude does not need to mirror that of nominal conditions.

8.2.5 Haze, clouds and winds aloft.

Environmental factors impacted FT3 in the following ways: Haze due to smoke from southern California wildfires reduced the visibility at the aircraft operating altitude enough that it was a contributing factor to the mission rule violation. It is of note that later in the flight visibility at higher altitudes was significantly better.

Clouds, broken layers, caused the cancellation of one Configuration 2 mission rehearsal flight. No other flights were significantly impacted.

Winds aloft were sometimes greater than 30kts. Station keeping was affected for Configuration 1 flights and made arriving at the IP on time challenging. For Configuration 2 flights, the intercept was significantly impacted by high winds and in some cases caused missed encounters.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B Recommendation: It is important for aircrew to assess the environmental conditions and make recommendations to the TC. Should a haze or cloud layer decrease the visuals the TC needs to be informed and all participants can make accurate assessments and decisions. In most cases a shift in test altitude would mitigate the haze layer problem. A mitigation for high winds aloft is to develop a tool for the TC/TD to determine push times that accounts for winds aloft.

8.2.6 Understanding success criteria and training operators was critical to mission success.

Over the course of FT3, the test team seemed to struggle to understand what the exact success criteria was for each SUT. Since there were multiple SUTs, the success criteria was not the same from one to another, and this situation was not always clear to the team. For Configuration 1 the following were SUT: JADEM, Stratway+, Radar, TCAS, and CPDS. Each system utilized a different display. Training was conducted by each SUT researcher, and with the researcher present during their respective flights, mission success was maximized. It is noteworthy to point out that the training conducted by the CPDS researcher was the most effective and required the least amount of intervention during the flight itself. In the other system cases, the researcher had to provide instruction in real-time to ensure correct data was being generated.

Recommendation: In order to emulate the desired training, the operations team will provide a template for future test events that identifies what training is required, a format for presenting the material and a schedule of when it will be accomplished. Additionally, a clear understanding of what the researcher expects out of the SUT will be explained to the test team. An example of why proper training in required is in one case during FT3 a system was de-energized by the test team when the team, in fact, wanted to de-select a component of the system.

8.2.7 A separate truth source for positional data, TSPI, from each aircraft was not available for post flight analysis.

Post flight analysis of TSPI information was not a requirement for FT3. However, for higher fidelity evaluation of the Radar, TCAS, and CPDS, a TSPI truth source such as Differential Global Positioning System (DGPS) would have been beneficial. Additionally, the time and geo-location sync from one data source to another was not easily and clearly determined resulting in significant post processing to sync all data sources.

Recommendation: A truth data source should be considered standard equipment for any flight test operation. Incorporate DGPS or suitable TSPI data source on each intruder and ownship aircraft. Ensure all data being collected is time synced.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 8.3 Researcher Observations The following observations were noted and discussed with each research team. The findings below are preliminary and may become outdated once FT3 data has been fully analyzed .

8.3.1 FT3 Configuration 1 successfully completed the major objectives for all SUT.

All stakeholders considered this phase a success and in many cases a first for their SUT.

ARC team members collected good data that will be used to update their simulation model and support future test efforts including PT6 (Part Task 6, V&V of MOPS) and FT4. The data supports operational concept developments for aircraft in the cruise phase of flight.

LaRC team members collected more data in one flight test event than had been collected in past simulated events. The data will be used to update their simulation model and help develop Phase I MOPS for SC-228 and eventually the FAA. Additionally, FT3 was the first time a multi-intruder encounter was conducted for these purposes. For LaRC, many successful scenarios were completed and all objectives were met, and they will continue to find areas for improvement.

CPDS teams consider FT3 a success at collecting several corner case scenarios that challenge both the algorithm and aircrew judgment and decision making based off the CPDS displays. All data analyzed to date appears to correlate with simulation.

The TCAS and Radar stakeholders from GA-ASI both report good data collected for their systems and intend to implement enhancements based off the data collected. The TCAS alerts presented to the crews were within TCAS specifications, but crews recommended some user interface changes that better help them get instant SA once a TCAS message is displayed.

The HSI teams were not able to use the data from Configuration 2 flights for technical reasons still being analyzed.

8.3.2 Both ARC and LaRC requested that all test aircraft keep heading and airspeed more stable during future test events.

Both teams noted in preliminary data analysis that in some cases the aircraft would change airspeed or heading in order to arrive at CPA at the planned time. Although this was intended and permitted, up to 90 seconds from CPA the changes in aircraft state perturbed the researcher’s desired data to a minimal extent. In follow -on test events the research teams request aircrew accept the aircraft conditions at the IP as long as they are within timing tolerances. One way to mitigate this is to add real-time monitoring of the ownship and intruder speeds, heading, altitude, predicted CPA, etc. Further planning is required for FT4 to ensure accurate understanding of the success criteria and on- condition parameters will be constrained to help mitigate perturbations.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 8.3.3 VSCS displays did not function for ARC and LaRC.

For both teams the planned SUT display did not function correctly. The LaRC team determined that there was not enough granularity in the display for crews to make accurate heading change decisions. They reverted to the Stratway+ native displays on subsequent flight days.

The ARC team used VSCS for all of their planned encounters, but for the encounters that provided directive guidance, the display did not function correctly. Integration issues between AutoResolver and the VSCS display gave inaccurate headings and turn directions to the crews, which resulted in skewed data for those runs that required the aircrew to maneuver per the guidance. The ARC team did not have an alternative display available. For FT4, resources should be assigned to ensure proper integration between Autoresolver and VSCS.

8.3.4 Radar vertical speed indications were not filtered.

The SUT stakeholders that used the radar as a contributing senor were expecting filtered radar data. Their algorithms noted significant shifts in vertical speed that would have been dampened out if filtered. However, the GA-ASI radar team understood that filtering to be inherent to the system that used the data, as opposed to imbedded with the radar itself.

Since there are no clear requirements for what a certified radar system should do, GA- ASI intends to implement a filtered data stream for FT4.

8.3.5 An FT4 data collection plan is desired.

SC-228 representatives as well as the FT3 research teams noted that the data collection plan and implementation for FT3 resulted in some inconsistencies that may be reduced with a more detailed collection effort. As noted in Section 8.2.7 , multiple data sources with different time syncs needed to be post processed. Specifically, the data from N3GC was a relative time sync as opposed to a GPS time. Planning for a common time sync and installation of a truth source data system for test aircraft should help mitigate this issue.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B

9 Acronyms

ACAS Airborne Collision Avoidance System AFRC Armstrong Flight Research Center AFTC Air Force Test Center AGL Above Ground Level AIM Aeronautical Information Manual ARC Ames Research Center ATC Air Traffic Control AUTO Automatic C2 Command and Control CA Collision Avoidance CAT Collision Avoidance Threshold CBDR Constant Bearing Decreasing Range CCB Complex Control Board CFG Configuration CNPC Control and Non - Payload Communication COA Certificates of Authorization COMEX Commence Exercise CPA Closest Point of Approach CPDS Conflict Prediction and Display System CRM Crew Resource Management CSSA Corrective Self - Separation Alert CST Combined Systems Test CTF Combined Test Force CVSRF Crew Vehicle Simulation Research Facility DAA Detect And Avoid DAIDALUS Detect and AvoID Alerting Logic for Uncrewed Systems DATR Dryden Aeronautical Test Range DCP Dryden Centerwide Procedure DD Decimal Degrees DEEC Digital Electronic Engine Control DET3 Detachment 3 DGPS Differential Global Position ing System DICES Type of Radio Panel used at NASA Armstrong DME Distance Measuring Equipment DRR Due Regard Radar DSRL Distributed System Research Laboratory EAFB Edwards Air Force Base EDM Engineering Development Module ENC Encounter Flight Test Series 3: Flight Test Report, October 2015 – REVISION B FAR Federal Aviation Regulation FDDRL Flight Deck Display Research Laboratory FINEX Finish Exercise FL Flight Level FM Full Mission FSE Flight Systems Engineer FT3 Flight Test 3 FT4 Flight Test 4 FTE Flight Test Engineer FV FalconView GA - ASI General Atomics Aeronautical Systems, Inc.

GCS Ground Control Station GPS Global Positioning System GRC Glenn Research Center GS Groundspeed HSI Human Systems Integration HUD Heads - Up Display HWY Highway IASP Integrated Aviation Systems Program ID Identification IFR Instrument Flight Rules IHITL Integrated Human In The Loop ILLA Ikhana Lost Link Altitude ILLH Ikhana Lost Link Heading IMU Inertial Measurement Unit INS Inertial Navigation System INT Intruder IP Initial Point IT&E Integrated Test and Evaluation ITAR International Traffic in Arms Regulations JADEM Java Architecture for DAA Extensibility and Modeling KBFI ICAO airfield code. King Field in Seattle KBFL ICAO airfield code. Meadows Field in Bakersfield KBFL ICAO airfield code. Meadows Field in Bakersfield KEDW ICAO airfield code. Edwards AFB KGS Knots Groundspeed KIAS Knots Indicated Airspeed KPMD ICAO airfield code. Palmdale airfield KTAS Knots True Airspeed KVCV ICAO airfield code. Victorville KVNY ICAO airfield code. Van N uys LaRC Langley Research Center LOS Line Of Sight Flight Test Series 3: Flight Test Report, October 2015 – REVISION B LRO Long Range Optics LVC Live Virtual Constructive MACS Multi Aircraft Control System MC Magnetic Course MIT Massachusetts Institute of Technology MOA Military Operating Area MOPS Minimum Operational Performance Standards MP Maneuver Point MR Mission Rule MSL Mean Sea Level NAS National Airspace System NASA National Aeronautics and Space Administration NM Nautical Miles NMAC Near Mid - Air Collision NOTAM Notice To Airmen OWG Operations Working Group OWN Ownship PIC Pilot In Command PIRA Precision Impact Range Area PSSA Preventive Self - Separation Alert PT6 Part Task 6 RA Resolution Advisory RAIF Research Aircraft Integration Facility REH Rehearsal RF Radio Frequency RGCS Research Ground Control Station RSO Range Safety Officer RTB Return To Base RTCA Radio Technical Commission for Aeronautics S/N Scenario Number SA Situational Awareness SAA Sense And Avoid SAAP Sens e And Avoid Processor SAF Stand Alone Facility SC - 228 ToR Special Committee - 228 Terms of Reference SGT Stinger Gaffarian Technologies SME Subject Matter Expert SOP Standard Operating Procedures SOR Senior Operations Representative SPORT Call Sign for AFTC Radar Control Facility SS Self - Separation SSCA Self - Separation Corrective Alert (CSSA) SSI Self Assurance/Sense and Avoid Interoperability Flight Test Series 3: Flight Test Report, October 2015 – REVISION B SSPT Self - Separation Proximate Traffic SSWA Self - Separation Warning Alert SSWG System Safety Working Group STM Surveillance Tracking Module SUT System Under Test TA Traffic Advisory TBD To Be Determined TC Test Conductor TCAS Traffic Collision and Avoidance System TD Test Director TECCS Test and Evaluation Command and Control System TFR Temporary Flight Restriction TRACON Terminal Radar Approach Control Facility TSPI Time - Space - Position Information UAS Un manned Aircraft System USAF United States Air Force UTC Coordinated Universal Time V&V Verification and Validation VBA Visual Basic for Applications VFR Visual Flight Rules VID Visual Identification VMC Visual Meteorological Conditions VSCS Vigilant Spirit Control Station VSM Vehicle Specific Module WP Waypoint ZOA Oakland Air Route Traffic Control Center Flight Test Series 3: Flight Test Report, October 2015 – REVISION B

10 References

Document Number Document Title FT3 - FTP - 01 FT3 Flight Test Plan DCP - O - 025 NASA Armstrong Aircrew Flight Operations Manual DCP - O - 003 Mission Control Procedure EAFBI 13 - 100 Edward AFB Instruction Flying and Airfield Operations FT3 IT&E DMP - 001 Flight Test 3 Data Management Plan NPR 7900.3 Aircraft Management Operations OIEP SRD - 01 Ownship and Intruder Equipage & Performance SRD R - 2508 R - 2508 Complex Users Handbook Title 14 CFR Part 91 General Operating and Flight Rules Flight Test Series 3: Flight Test Report, October 2015 – REVISION B

11 Appendix A: Definition of Terms

Blunder A planned vertical or horizontal maneuver performed by the intruder, ownship or both aircraft that occurs at some point during the flight test encounter. The blunder maneuver is a technique by which the researcher uses to obtain data required to refine algorithm parametric logic.

Configuration 1 This test configuration investigates the advisories generated by the Self Separation and Collision Avoidance Algorithm displays provided by NASA Ames, NASA Langley , or GA - ASI and fed by data from live aircraft during flight. The configuration further investigates TCAS and DRR alerting. Flight Test Configuration 1 is defined into two distinct groups (Configuration 1a and 1b ). Configuration 1a involves fli ght test encounters using a low - speed, unmanned ownship aircraft.

Configuration 1b involves flight test encounters using a high - speed manned ownship aircraft.

Configuration 2 The Full Mission test configuration is designed to connect virtual ATC and const ructive aircraft processes running at NASA Ames with a live manned intruder aircraft and a UAS surrogate ownship aircraft controlled by the research GCS located at NASA Armstrong. The UAS Surrogate aircraft is flown on a V isual F light R ules (VFR) flight pl an with scenarios containing a mix of two live and several virtual manned I nstrument F light R ules (IFR) and VFR (squawking) aircraft.

Intruder Intruder aircraft (when properly equipped) provide a target solution for the self - separation algorithm under tes t. Both low speed, high speed, and multi - ship encounters are planned using intruder aircraft .

Ownship Ownship aircraft provide the self - separation algorithm host solution for testing airborne geospatial encounters with target (intruder) aircraft.

The ownship may be a UAS or UAS surrogate aircraft. Self - separation alerting solutions are presented to the ground control station pilot who determines the best course of action based on display alerting evaluation.

Mitigated Flight test encounters that are designed for the controlling UAS pilot to either manually respond to a self - separation or RA alert or monitor the aircraft response during an automatic RA alert. Mitigated test encounters are typically planned with vertical, lateral, and timing flight safety margins designed into the flight test encounters to help minimize the potential for an inflight collision.

Unmitigated Flight test encounters that due to adequate vertical offsets do not require an associated lateral off set for flight safety. Unmitigated encounters are non - maneuvering.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B

12 Appendix B: Redlined Flight Cards

Flight cards (divided into sections by day) that were flown for Configuration 1 are shown in this Appendix. For Configuration 2, flights cards for each flight day are identical, and can therefore be referenced earlier in this document in Figure 39, Figure 40, and Figure 41.

Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 12.1 Flight 1 Redlined Flight Cards Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 12.2 Flight 2 Redlined Flight Cards 1 19 Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 12.3 Flight 3 Redlined Flight Cards 1 42 Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 12.4 Flight 4 Redlined Flight Cards 1 63 Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 12.5 Flight 5 Redlined Flight Cards 1 80 Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 12.6 Flight 6 Redlined Flight Cards Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 12.7 Flight 7 Redlined Flight Cards Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 12.8 Flight 8 Redlined Flight Cards Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 12.9 Flight 9 Redlined Flight Cards Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 12.10 Flight 10 Redlined Flight Cards Flight Test Series 3: Flight Test Report, October 2015 – REVISION B 12.11 Flight 11 Redlined Flight Cards L12P Flight Test Series 3: Flight Test Report, October 2015 – REVISION B