Development, Implementation, and Pilot Evaluation of a Model-Driven Envelope Protection System to Mitigate the Hazard of In-Flight Ice Contamination on a Twin-Engine Commuter Aircraft

Appendix A.—Abbreviations and Symbols List

Abbreviations AOA angle of attack D-ICES Dynamic Inversion Control Evaluation System ERAU Embry-Riddle Aeronautical University FCD Flight Control Display H null hypothesis ICEFTD Ice Contamination Effects Flight Training Device ICEPro Icing Contamination Envelope Protection system IPS Ice Protection System ISP Icing Severity Parameter J cost function LSE least-squared error PCE Pilot Coupling Event RT-PID Real Time Parameter Identification PFD Primary Flight Display RMS root-mean square SLD Super-cooled Large Droplet Symbols A stability matrix B control matrix C stability matrix C rolling moment due to sideslip angle derivative, E w w C l l E C rolling moment due to aileron deflection derivative, C G w w l a l G a C rolling moment due to rudder deflection derivative, C G w w l G r r l C rolling moment due to nondimensional roll rate derivative, V pb C 2 / w w l lp C rolling moment due to nondimensional yaw rate derivative, V rb C 2 / w w lr l C pitching moment due to AOA derivative, D w w C m m D C pitching moment due to elevator deflection derivative, C G w w m e m G e C pitching moment due to nondimensional pitch rate derivative, V c q C 2 / w w m mq C normal force due to AOA derivative, D w w C N N D C normal force due to elevator deflection derivative, C G w w N G e e N C normal force due to nondimensional pitch rate derivative, V c q C 2 / w w Nq N C yawing moment due to sideslip angle derivative, E w w C n n E C yawing moment due to aileron deflection derivative, C G w w a n n G a C yawing moment due to rudder deflection derivative, C G w w r n n G r NASA/CR—2014-218320 43 C yawing moment due to nondimensional roll rate derivative, V pb C 2 / w w n np C yawing moment due to nondimensional yaw rate derivative, V rb C 2 / w w nr n C side force due to sideslip angle derivative, E w w C Y Y E C side force due to aileron deflection derivative, C G w w Y a Y G a C side force due to rudder deflection derivative, r C G w w Y Y G r C side force due to nondimensional roll rate derivative, V pb C 2 / w w Y Yp C side force due to nondimensional yaw rate derivative, V rb C 2 / w w Yr Y U Theil inequality coefficient th a k row of matrix A k th b k row of matrix B k b reference span c mean aerodynamic chord t time u control vector  u control vector predicted from inversion routine ~ u control k V true velocity, ft/s x state vector x 0 state vector initial condition ~ x state k y , y ˆ measured and predicted output Į alternate hypothesis Ȧ frequency, rad/s ș parameter vector ˆ T estimated Parameter V ˆ confidence bound NASA/CR—2014-218320 44

Appendix B.—Post-Test Survey Questionnaire

Part I. Demographics Name: _______________________ Male Female Pilot Number: __________ Total Flight Hours: __________ Multi – Engine Hours: _________ FAA Ratings: Commercial Multi CFI CFII Aircraft Flown and Hours in Type: _____________________________________________________________________________________ _____________________________________ Icing related training and experience: 1. In my all my prior flying experience, I encountered in-flight icing Never Rarely Sometimes Very often Always 2. I felt that my prior icing related knowledge and experience would have adequately prepared me to perform this test scenario without the familiarization training I received on aircraft handling characteristics.

Strongly Disagree Strongly Agree Disagree Undecided Agree 3. The NASA videos and web based training materials provided me with information about icing that I had never known about before.

Strongly Disagree Disagree Undecided Agree Strongly Agree 4. Before this test, my icing related flight training was mostly focused on how to operate the ice protection system.

Strongly Disagree Disagree Undecided Agree Strongly Agree NASA/CR—2014-218320 45

Part II. Situation Awareness 1. My flight displays were adequate for me to determine when airframe icing was having an effect on aircraft characteristics.

Never Rarely Sometimes Very often Always 2. I felt as though I knew the minimum safe speed I could fly for a given wing flap setting within 5 kt.

Never Rarely Sometimes Very often Always 3. I knew how to adjust my pitch attitude to avoid a wing stall or a tail stall upset.

Never Rarely Sometimes Very Often Always 4. I wasn’t always sure which wing flap settings would allow a safe rate of climb in the event of an engine failure.

Strongly Disagree Disagree Undecided Agree Strongly Agree 5. I relied solely on aircraft control response to determine how icing affected pitch, yaw, or roll characteristics.

Strongly Disagree Disagree Undecided Agree Strongly Agree 6. I was confident that the final approach airspeed I chose to fly after I departed the final approach fix would prevent me from stalling.

Strongly Disagree Disagree Undecided Agree Strongly Agree 7. While I was putting the flaps down during the approach and landing, I felt I could effectively manage my control inputs and airspeed to avoid aircraft handling problems or tail stall upsets.

Strongly Disagree Disagree Undecided Agree Strongly Agree 8. The various colored bands on the airspeed tape were useful for me to safely fly the aircraft during the entire flight.

Never Infrequently Sometimes Frequently Always NASA/CR—2014-218320 46

9. I relied upon the stick shaker to prevent me from inadvertently stalling.

Never Rarely Sometimes Very Often Always 10. I could always tell when I was approaching a wing or tail stall condition.

Strongly Disagree Disagree Undecided Agree Strongly Agree Part III. Implementation of Envelope Protection Cueing (Experimental ICEPro Group Only) 1. I felt that it was important to combine the visual cues from ICEPro with aural alerts, and tactile feedback from the stick shaker.

Strongly Disagree Disagree Undecided Agree Strongly Agree 2. The ROL DGRD, PTCH DGRD, or YAW DGRD messages on the PFD did not provide useful information, and only cluttered the display.

Strongly Disagree Disagree Undecided Agree Strongly Agree 3. When flying at slow speeds, I could easily use the AOA bars as a good pitch control reference for safely flying the airplane.

Strongly Disagree Disagree Undecided Agree Strongly Agree 4. I had difficulty understanding the relationship between the AOA bars and the high and low speed carets on the airspeed tape.

Strongly Disagree Disagree Undecided Agree Strongly Agree 5. The color coded flight control surfaces on the flight control synoptic were more useful than the messages for assessing degraded control state.

Strongly Disagree Disagree Undecided Agree Strongly Agree 6. I favored using the airspeed carets rather than the AOA bars to remain within a safe operating condition.

Never Infrequently Sometimes Frequently Always NASA/CR—2014-218320 47 7. When I unintentionally got into an upset condition, the disappearance of all ICEPro messages except the AOA bars on the PFD did not affect my ability to recover the airplane. (Answer only if a wing stall or tail stall upset occurred).

Strongly Disagree Disagree Undecided Agree Strongly Agree 8. When I extended the wing flaps and the red CLIMB LIM and FLAP LIM messages came on together, I immediately knew if an engine failed I would have to reduce my flap setting in order to climb.

Strongly Disagree Disagree Undecided Agree Strongly Agree 9. The occasional “buzzing” of the flight controls by ICEPro made flying the aircraft very difficult.

Strongly Disagree Disagree Undecided Agree Strongly Agree 10. I tended to confuse baseline airspeed limits with the airspeed limits (carets) that were posted by ICEPro.

Strongly Disagree Disagree Undecided Agree Strongly Agree 11. I tended to fly so as to keep my pitch attitude in the middle of the AOA bars when they were presented by ICEPro Never Infrequently Sometimes Frequently Always 12. When the AOA bars started flashing, they immediately captured my attention.

Never Infrequently Sometimes Frequently Always 13. I felt that the airspeed carets from ICEPro were useful for helping me determine safe flight speeds when maneuvering during approach and landing.

Never Infrequently Sometimes Frequently Always 14. The low AOA cue on the PFD enabled me to avoid a tail stall upset.

Never Infrequently Sometimes Frequently Always NASA/CR—2014-218320 48

15. I immediately noticed when the CLIMB LIM and FLAP LIM messages were on at the same time.

Strongly Disagree Disagree Undecided Agree Strongly Agree Part IV. Workload 1. I found it difficult to control vertical speed when on glide slope.

Never Infrequently Sometimes Frequently Always 2. Operating the flight controls to avoid an upset condition when on final approach was a physically demanding effort.

Never Infrequently Sometimes Frequently Always 3. I felt it took a considerable amount of concentration and effort to safely execute the missed approach procedure.

Strongly Disagree Disagree Undecided Agree Strongly Agree 4. Keeping the aircraft on the glide slope without experiencing a pitch upset was a very demanding task.

Strongly Disagree Disagree Undecided Agree Strongly Agree 5. On final approach I spent so much time trying to fly the glide slope that I was unable to maintain good localizer course control.

Never Infrequently Sometimes Frequently Always NASA/CR—2014-218320 49 NASA/CR—2014-218320 50

Appendix C.—Results of Survey Questions from Parts I, II, and IV

Control Group (n=14) 10 Experimental Group (n=15) Response Frequency 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 I-1 I-2 I-3 I-4 Figure C.1.—Pilot demographics—Part I Survey questions 1 to 4 Control Group (n=14) Experimental Group (n=15) Response Frequency 1 2 3 4 5* 1 2 3 4 5* 1 2 3 4 5* 1* 2 3 4 5 1* 2 3 4 5 II-1 II-2 II-3 II-4 II-5 *Direction of responses that indicated better situational awareness Figure C.2.—Situational Awareness—Part II Survey questions 1 to 5 NASA/CR—2014-218320 51 Control Pilot Group (n=14) 10 Experimental Group (n=15) Response Frequency 1 2 3 4 5* 1 2 3 4 5* 1 2 3 4 5* 1* 2 3 4 5 1 2 3 4 5* II-6 II-7 II-8 II-9 II- 10 * Direction of responses that indicated better situational awareness Figure C.3.—Situational Awareness—Part II Survey questions 6 to 10 Control Group (n=14) Experimental Group (n=15) Response Frequency 1* 2 3 4 5 1* 2 3 4 5 1* 2 3 4 5 1* 2 3 4 5 1* 2 3 4 5 IV-1 IV-2 IV-3 IV-4 IV-5 * Direction of responses that indicated lower workload Figure C.4.—Workload—Part IV Survey questions 1 to 5 NASA/CR—2014-218320 52

Appendix D.—One Year Atmospheric Turbulence Extension Final

Report—Effect of Turbulence and Sensor Noise On ICEPro

Progress Report: Year 5 Borja Martos, Principal Investigator University of Tennessee Space Institute (UTSI) 411 B. H. Goethert Pkwy Tullahoma, TN 37388-9700 and Ryan Oltman, David Gingras, and Billy Barnhart Bihrle Applied Research (BAR) 81 Research Drive, Hampton VA 23666 Cooperative Agreement: NNX07AD58A NASA/CR—2014-218320 53

List of Symbols and Abbreviations

Abbreviations AIAA American Institute of Aeronautics and Astronautics AIMMS-20 Atmospheric Integrated Meteorological Measurement System-20 AOA angle of attack ATP Airline Transport Pilot BAR Bihrle Applied Research COTS commercial off the shelf D-ICES Dynamic Inversion Control Evaluation System D-Six Bihrle Applied Research’s Simulation Environment Software I0 no ice condition I2 20 min ice accumulation condition ICEFTD Ice Contamination Effects Flight Training Device ICEPro Icing Contamination Envelope Protection System ID identification ISP icing severity parameter IVHM Integrated Vehicle Health Monitoring KIAS knots indicated airspeed LIDAR light detection and ranging MIL military handbook MOS measure of success NASA National Aeronautics and Space Administration GRC NASA Glenn Research Center LaRC NASA Langley Research Center NASA TLX NASA Task Load Index OBES Onboard Excitation System PFD Primary Flight Display PFLF power for level flight RMS root mean square ROC rate of climb RT-PID Real Time Parameter Identification SFTE Society of Flight Test Engineers T0 no turbulence condition T1 light turbulence condition T2 moderate turbulence condition T3 severe turbulence condition TMS Turbulence Model Structure TOGA Take Off - Go Around UTSI University of Tennessee Space Institute VIAS velocity indicated airspeed VMC visual meteorological conditions VSST Vehicle Systems Safety Technology Symbols A longitudinal acceleration X A lateral acceleration Y A vertical acceleration Z NASA/CR—2014-218320 54 c mean aerodynamic chord C dihedral effect stability coefficient l E C aileron effectiveness stability coefficient l G a C gust longitudinal static stability coefficient m D G C rigid body longitudinal static stability coefficient m D S C longitudinal static stability coefficient m D C pitch stability coefficient mO C pitch damping stability coefficient mq C elevator effectiveness stability coefficient m G e C normal lift coefficient N D C rudder effectiveness stability coefficient N G r deU Theil inequality coefficient g gravitational acceleration constant H pressure altitude P G V aircraft velocity relative to the earth g G V true airspeed t G V true wind velocity vector w V velocity down D V velocity east E W wind velocity, down D W wind velocity, east E W wind velocity, north N V velocity north N V stalling airspeed S V true airspeed T p roll rate q dynamic pressure q pitch rate r yaw rate u longitudinal velocity v lateral velocity w vertical velocity ǻ change Į angle of attack Į measured angle of attack m ȕ angle of sideslip ȕ flow sideslip angle m G flap deflection F G elevator deflection e T pitch angle I roll angle ȥ yaw angle NASA/CR—2014-218320 55 D.1 Abstract A team comprised of personnel from The University of Tennessee Space Institute, Bihrle Applied Research, Inc., NASA Glenn Research Center (GRC), and NASA Langley Research Center (LaRC), has successfully developed a prototype Icing Contamination Envelope Protection System (ICEPro). The ICEPro system facilitates flight envelope protection by making continuous real time vehicle state assessments, which are synthesized into flyable pilot cueing along with visual and aural alerts during in-flight icing conditions. The simulation could not duplicate all of the real-world conditions that could have first-order effects on system performance, such as those due to real atmospheric turbulence. Therefore, in order to assess system performance under these conditions and minimize the technical risk of an eventual flight-test validation program, the effects of atmospheric turbulence on ICEPro are studied in this extension.

This extension focused on the impact turbulence has on the ICEPro system. Turbulence will degrade the quality of air data sensing. This study explored the accuracy of ICEPro’s ice detection and parameter estimation with perfect sensing and degraded sensing to determine an acceptable amount of degradation that would allow the ICEPro system to function correctly. Finally, after an acceptable error in air data collection was defined, it would be possible to determine whether the tolerance was obtainable in a commercially available air data sensor.

D.2 Introduction D.2.1 Research Background D.2.1.1 Previous Studies and Results A team comprised of personnel from The University of Tennessee Space Institute (UTSI), Tullahoma, Tennessee, Bihrle Applied Research Inc (BAR), Hampton, Virginia, NASA Glenn Research Center (GRC), Cleveland, Ohio, and NASA Langley Research Center (LaRC), Hampton, Virginia, has successfully developed a prototype Ice Contamination Envelope Protection System (ICEPro). ICEPro was developed using a DH-6 Twin Otter simulation for development and evaluation. The aerodynamic math models of the Twin Otter were derived from a series of wind tunnel tests for both the clean and iced airplanes. The math models were then validated against NASA flight data for both the clean aircraft and the test aircraft flown with artificial ice shapes. The Twin Otter was chosen because of the availability of the flight test data.

The ICEPro system facilitates flight envelope protection by making continuous real time vehicle state assessments, which are synthesized into flyable pilot cueing along with visual and aural alerts during in-flight icing conditions. Detection of degraded aircraft stability and control and performance due to icing is carried out by a dynamic inversion control evaluation system (D-ICES) that compares expected aircraft response from a-priori knowledge base with the current response. When differences reach defined thresholds, real time parameter identification (RT-PID) methods are invoked to estimate current state parameters, which continuously support pilot cueing and alerts. The development effort included simulation-based design, testing, and verification. A pilot in-the-loop study was conducted to gather pilot performance data and opinions of the utility of ICEPro during simulated icing encounters. Results of the study indicated that the system performed as expected and pilot performance benefited from the envelope protection cues.

Figure D.1 presents a flow chart of the ICEPro system. (Some aspects of the system such as mode latching/delatching and logic for displaying messages, logic to force flushing the D-ICES and RT-PID buffers, etc. are not shown for clarity). The system starts in Monitor mode where D-ICES calculates, via an inversion technique using an a priori clean aircraft aerodynamic model, what the elevator deflection should be for the flight condition. This is compared with the actual elevator deflection using a statistical parameter called the Theil inequality coefficient. If this coefficient exceeds its threshold the system transitions to the ID mode. In the ID mode, the Real Time Parameter Identification (RT-PID) module is started and it predicts, based on flight state parameters, the linearized aerodynamic coefficients. These coefficients in conjunction with comparable clean and iced coefficients provided by D-ICES are used to calculate an Icing Severity Parameter (ISP). If the ISP becomes larger than its threshold, the system transitions to the Report mode where pilot cues and messages are displayed indicating the degraded aircraft’s condition and providing limits for safe operation of the aircraft.

NASA/CR—2014-218320 56 Flight Data Monitor Mode IMOD=0 No į e deviation > D-ICES Data IMODE = 0 threshold (controls) Stay in Monitor Mode Yes IMODE = 1 Go to ID Mode Start RTPID a priori aero PID aerodynamics coefficients ID Mode IMOD=1 Calculate ISP No No ISP > į deviation) > e Report threshold Threshold IMODE = 0 Go to Monitor Mode Yes Yes IMODE = 2 Go to Report Mode IMODE = 1 Stay in ID Mode Report Mode Check individual Stab & IMOD=2 Control values + ROC Yes No Deviation > ISP >Report Update ISP Threshold Threshold No Yes IMODE = 1 Update Display Go to ID Mode Cautions /Warnings Figure D.1.—ICEPro System Flow Chart D.2.1.2 Current Year Five Study The simulation used for the development and evaluation did not duplicate all of the real-world conditions that could have first-order effects on system performance, such as those due to real atmospheric turbulence. Therefore, in order to assess system performance under these conditions and minimize the technical risk of an eventual flight-test validation program, the effects of atmospheric turbulence on ICEPro are studied in this extension.

This extension focused on the impact turbulence has on the ICEPro system. Turbulence will degrade the quality of air data sensing. This study explored the accuracy of ICEPro’s ice detection and parameter NASA/CR—2014-218320 57 estimation with perfect sensing and degraded sensing to determine an acceptable amount of degradation that would allow the ICEPro system to function correctly. Finally, after an acceptable error in air data collection was defined, it would be possible to determine whether the tolerance was obtainable in a commercially available air data sensor.

D.2.2 Research Objectives Four tasks and their measures of success (MOS) have been defined in the ICEPro atmospheric turbulence test plan (see Appendix E). This summary will describe the questions asked in each of the first three tasks and a brief introduction into the software tools developed to accomplish this work. The effort on Tasks 1 to 3 exhausted the allowed time and remaining funds, therefore no Task 4 results were included.

D.2.2.1 Task 1: Simulating Real World RT-PID Performance in Turbulence Determine whether the scatter in the turbulence estimates arise from, 1) colored noise in the regressor(s), and if so what is the source of the colored noise and in which regressor(s), 2) insufficient model structure, and if so how should the model structure be adjusted or, 3) some other means. In the system identification process, regressor(s) are defined as the independent variables.

D.2.2.2 Task 2: Effect of Sensor Noise and Turbulence Modify, test, and validate the use of RT-PID and D-ICES algorithms with atmospheric turbulence as an input for various levels of atmospheric turbulence, and to identify an acceptable error in the wind measurement during turbulence for RT-PID and D-ICES algorithms. Additionally, verify MOS during straight and level flight for varying atmospheric turbulence levels and icing conditions.

D.2.2.3 Task 3: Effect of Turbulence for Different Flight Phases at Selected Noise Level Modify, test, and validate the use of RT-PID and D-ICES algorithms with atmospheric turbulence as an input. Select an acceptable atmospheric turbulence measurement error based on findings from Task 2 and commercially available atmospheric turbulence sensors. Verify MOS during each phase of flight for varying atmospheric turbulence levels and icing conditions.

D.2.2.4 Changes to ICEPro Logic Several specific features were added to the ICEPro module for the current effort. The new features included adding several input parameters within the DSix variable list. These parameters permitted overriding ICEPro’s logic in the Monitor mode forcing the system into identification mode. In addition, user specified surface buzzing times were created to override normal ICEPro buzzing logic. Buzzing refers to exercising the controls through a set of orthogonal multi-sine inputs to provide additional control content for RT-PID operation. Finally, the use of these new features was optional allowing ICEPro to operate in an unmodified logic set.

Utilizing DSix scripting capabilities, a series of scripts were developed to facilitate batch testing. These scripts interfaced an Excel file containing the test matrix, set up the simulation initial conditions, stepped through the simulation using control inputs provided by virtual pilot logic contained in the scripts, recorded selected time history variables stopped the simulation, and, exported the recorded data to MATLAB. The script virtual pilot logic used the differences between the desired flight path angle, heading, and airspeed and the current actual values as inputs to the elevator, aileron, and throttles as control inputs. Sideslip feedback to the rudder was also used to coordinate turns. Equations (D.1) show the basic control laws used by the virtual pilot. The gains were set to low values that worked satisfactorily for the no turbulence cases, but were less than optimal with turbulence. These values were used for all cases to minimize excessive control movements during turbulent conditions. For some cases, the throttle commands were set to zero to eliminate excessive throttle movements encountered with turbulent conditions.

  dt K K e J  J  J  J  J G  (D.1a) p p desired desired command

³

NASA/CR—2014-218320 58 M  \  \ G K K a (D.1b) I \ desired command E G K r (D.1c) E command  V K V V K    G V desired Throttle (D.1d)  V D.3 Discussion of Study and Results There are at least two ways to account for turbulence in system identification algorithms, 1) model the turbulence mathematically and estimate the corresponding parameters and 2) measure the turbulence. In the first approach, the identification of dynamic models containing turbulence is classified as a system with state and measurement noise. In the second approach, favored in this work, turbulence is measured indirectly since air turbulence is imbedded in the true wind velocity vector. This approach is favored because it makes it possible to use commercial off the shelf sensors to measure the atmospheric turbulence and treat it as a G measured explanatory variable in the system identification problem. The true wind velocity vector, V of w ’ atmospheric air relative to the surface of the Earth is found from the vector subtraction: G G G V V V  (D.2) w t e G G Where the true aircraft velocity relative to the air mass and V is the velocity of the aircraft with respect V t e G G G V to the Earth. Since is the difference between two large quantities it is essential that and are V V w t e G measured as accurately as possible in order to minimize errors in V . The details of these calculations w can be found in flight test engineering handbooks (Refs. 1 to 3) The most obvious difficulty in measuring air turbulence from an aircraft is the removal of the aircraft’s velocity with respect to the Earth.

Additional difficulties arise since the aircraft distorts the airflow around itself. This requires precise G G measurement of and V V . Any inaccuracies in the measurements or calibrations will affect the derived t e wind, and consequently the aerodynamic estimates. It is only recently that COTS technology has been developed with sufficient measurement accuracy.

A search was conducted to identify candidate COTS atmospheric turbulence sensors. Several sensors were identified using state of the art LIDAR technology with an airflow angle accuracy of 1.0°. In comparison, flight test air data booms with traditional air data technology are calibrated to an accuracy of 0.1°. The cutoff accuracy for this work was set at 0.1°. Unfortunately, the accuracy of LIDAR based sensors requires an order of magnitude improvement to be useful for the present work. Several additional sensors were identified using traditional air data technology. The AIMMS-20 probe by Aventech Research was chosen because 1) It provides the most accurate wind determination out of the candidate probes and 2) it is already part of GRC’s inventory. Table D.1 summarizes the AIMMS-20 performance based on an error analysis.

Turbulence was tested as calm, light, moderate, and severe. Turbulence implementation and levels were defined per MIL-F-8785C and were based on the Dryden Turbulence Spectrum. According to MIL- –1 –2 F-8785C, the probability of exceeding light turbulence levels is between 10 and 10 , for moderate –3 –5 turbulence, the probability is approximately 10 , and for severe turbulence it is approximately 10 , as shown in Figure D.2 (Ref. 4) derived from MIL-F-8785C. Therefore, severe turbulence levels are not very likely to be encountered. The RMS turbulence wind speed values are also shown in Figure D.2 as a function of altitude. The turbulence velocity components were generated randomly within the guidelines of the Dryden spectrum and added to the aircraft velocity components to generate the total velocity components. A random number was used to generate the velocity components from the turbulence spectrum. By specifying a constant initial random number seed, the sequence of random numbers become reproducible allowing an “apples to apples” comparison of results. This method is referred to as a “fixed” NASA/CR—2014-218320 59 seed. Likewise, using a varying random number seed results in unique sequences of random numbers from test to test. The three velocity components from the turbulence calculations were added to their appropriate wind frame velocity components ( u , v , w ) that were generated from the aircraft dynamics so that the velocity components used for the vehicle motion calculations included the components from the turbulence model.

TABLE D.1.—AIMMS-20 PROBE PERFORMANCE Parameter 1 Sigma Error Longitudinal acceleration, A , (g) ....................................... 0.003 X Lateral acceleration, A , (g) ................................................. 0.003 Y Normal acceleration, A , (g) .................................................. 0.05 Z Angle of a WWDFNĮ GHJ ........................................................ 0.10 Angle of s LGHVOLSȕ GHJ ..................................................... 0.10 Pitch rate, q , (dps) .................................................................. 0.01 Roll rate, p , (dps) ................................................................... 0.07 Yaw rate, r , (dps)................................................................... 0.01 3LWFKDWWLWXGHș GHJ ............................................................ 0.20 Roll attitude, ࢥ , (deg)............................................................. 0.20 <DZDWWLWXGHȥ GHJ ............................................................ 0.20 GPS velocity down, V , (mps) .............................................. 0.10 D GPS velocity north, V , (mps) ............................................... 0.10 N GPS velocity east, V , (mps) ................................................. 0.10 E True airspeed, V , (mps) ........................................................ 0.20 T Wind velocity down, W , (mps) ............................................ 0.30 D Wind velocity north, W , (mps) ............................................ 0.30 N Wind velocity east, W , (mps) ............................................... 0.30 E Figure D.2.—Turbulence Definition (derived from MIL-F-8785C).

NASA/CR—2014-218320 60 D.3.1 Task 1 The following results are representative of the effect of turbulence on aircraft system identification.

Data were collected using NASA’s AIRSTAR T2 model during atmospheric turbulence conditions.

NASA’s T2 model is a 5.5 percent dynamically-scaled generic transport model. It is outfitted with typical flight test instrumentation and air data booms with flow vanes on each wingtip. In Figure D.3, each color represents a different run, and only the last estimate of each real-time parameter estimation run is shown.

A total of nine runs are represented; six in high turbulence and three in low turbulence.

Simulating the correct cause and effect due to atmospheric turbulence on aircraft parameter identification is an ongoing research topic. It is known that colored measurement noise causes scatter and bias (Refs. 5 and 6) in the parameter estimates, which is observed in Figure D.3 during repeated flight test maneuvers in atmospheric turbulence. However, it is unknown whether the scatter in the turbulence estimates arise from 1) colored noise in the regressor, and if so what is the source of the colored noise and in which regressor 2) inadequate model structure, and if so how should the model structure be adjusted or 3) some other means.

D.3.1.1 Hypothesis Colored AOA regressor noise and/or inadequate model structure is the cause of scatter in aircraft system identifcaiton estimates in atmospheric turbulence.

D.3.1.2 Sources of Scatter in Estimates Based on the hypothesis a literature search was carried out to examine previous related work. The results are summarized in the following sections.

D.3.1.2.1 Colored Regressor Noise Based on the literature search, the most likely causes of colored regressor noise are the wing aerodynamic response, the frequency response of long air data booms, and the frequency response of airflow angle vanes in atmospheric turbulence.

T-2 FLT 15 C21 WT04 T-2 FLT 41 C18 WT02 ( High Turbulence) (Low Turbulence) ˆ ˆ C C l l D D ˆ ˆ C C m m D D Figure D.3.—Effect of Turbulence on Modeling Results (Dr. Eugene A. Morelli, personal communication, April 24, 2011) NASA/CR—2014-218320 61 D.3.1.2.1.1 Flexure of AirData Boom During low-level flight tests over the terrestial and marine boundary layer, a Long EZ research aircraft was used to measure atmospheric turbulence (Ref. 7). Details of the aircraft and its instrumentation system are found in the cited reference. The boom motion with respect to the center of gravity (accelerometers were mounted at each location) is summarized in the following figure (Ref. 7) during light atmospheric turbulence. The phase trace compares fuselage and probe accelerometer sensors, whereas the magnitude traces represent each individual accelerometer sensor.

Figure D.4 illustrates the complexity of motion for this particular boom-airframe combination.

Significant difference is apparent in Figure D.4 between the vertical motion of the probe and the CG.

Below 0.6 Hz the coherence between the two accelerations is 1.0 with zero phase shift. Between 0.6 Hz and 2.0 Hz the coherence decreases and the phase changers from zero to –45°. Beyond 6 Hz there is no significant relationship between the two accelerations. In other words, in the frequency range of interest for rigid body aircraft dynamics (0.1 to 2.0 Hz), the drop in coherence and phase indicates that the AOA LVGLIIHUHQWEHWZHHQORFDWLRQV7KLVLQWXUQUHVXOWVLQFRORUHGQRLVHLQWKHĮUHJUHVVRU1RGRXEWWKHUH would be similarities for other boom-airframe combinations but with different frequency responses.

In order to investigate the in-flight structural response of the air data boom, airflow angle flight data collected in a DHC-6 Twin Otter during multiple parameter etstimation maneuvers was analyzed in the frequecy domain using Fourier coefficients. Figure D.5 shows the frequency response data for the entire frequency range, up to the Nyquist frequency at 25 Hz. The longitudinal and lateral natural frequencies for the dominant structural mode of the air data boom are approximately 8 and 9 Hz. These natural frequencies are associated with the AOA and angle of sideslip measurements respectively. In Figure D.5, the Fourier coeffcients for the AOA near 8 Hz are at the noise floor. Therefore, the structural response of the nose boom on which the AOA sensor is mounted, and from fuselage bending, is not a source of colored noise in the AOA. Similar results were calculated for angle of sideslip but the results are not shown.

Figure D.4.—Comparison between vertical accelerations of the boom and airplane center of gravity NASA/CR—2014-218320 62 Figure D.5.—Fourier coefficients for measured AOA D.3.1.2.1.2 Wing Response to Turbulence During flight tests in the marine boundary layer, a Twin Otter research aircraft was used to study the aerodynamic effects on wind turbulence measurements(Ref. 8). Details of the aircraft and its instrumentation system are found in the cited reference. Although mounted on the wing the airdata boom was not used as the primary source of air data measurements. Instead a flush air data system was mounted on the nose of the Twin Otter. The flush air data system avoids the vibration and frequency response problems of a long boom mounted on the nose of the aircraft (Ref. 9). The wing response to atmospheric turbulence can be summarized in Figure D.6 during a 30 km flight in light atmospheric turbulence. Figure D.6 represents the estimated AOA scale upwash correction versus frequency.

The frequency dependant upwash correction is caused by chordwise and spanwise upwash varations as a result of the wing vortex system response to atmospheric turbulence (Refs. 8 and 9). The frequency scale is the reduced frequency k based on the half wing span where Z is the angular frequency, b is the b –1 wingspan, and U is the airspeed. The AOA upwash correction (1 + k ) is explained by examining a u sensor model: 1 k D  D  D (D.3) M u T B where D = AOA bias B D = measured AOA M D = true AOA T –1 In Figure D.6, the vertical axis (1 + k ) represents the correction required to calculate the true AOA u from measured data.

1  1 k D  D  D (D.4) T u M B At low frequencies (representing flight in a calm atmosphere) the scale factor is approximately 0.75 and increases to 0.90 as k approaches 2.0. A k value of 2.0 (using the DHC-6 Twin Otter wing span and b b 110 kt cruise airspeed) corresponds to approximately 2.0 Hz. Therefore, over the frequency range of NASA/CR—2014-218320 63 interest for rigid body aircraft dynamics (0.1 to 2.0 Hz), the scale factor changes from 0.75 to 0.90 due to atmospheric turbulence. It was also surmised that the frequency-dependent sidewash was a major source of colored noise but this was not tested.

Figure D.7 depicts the time delay as a function of frequency. The time delay is caused by the separation between the nose boom mounted airflow vanes (measurement position) and the wing tip (lift and upwash generation), along which turbulence eddies travel.

The magnitude of the time delay represents the correction required to calculate the true AOA from measured data. In the frequency range of interest for rigid body aircraft dynamics (0.1 to 2.0 Hz), the time delay is approximately 0.1 sec.

–1 ) u k (1+ k = Z *( b /2)/ U b Figure D.6.—Angle of attack upwash versus frequency Figure D.7.—Time delay versus frequency NASA/CR—2014-218320 64 D.3.1.2.1.3 Frequency Response of Airdata Vanes Although References 10 to 12 were found to determine the frequency response of airdata vanes via natural frequency and damping ratio, no analysis was found confirming that the frequency response of airdata vanes was adequate in atmospheric turbulence.

D.3.1.2.2 Model Structure It is conceivable that the aerodynamic coefficients are different in atmospheric turbulence that in smooth air although such differences have not been documented in flight (Ref. 13). The new model structure—referred to as the Turbulence Model Structure (TMS)—includes new coefficients C and m D S C .These parameters represent the nondimensional SLWFKLQJPRPHQWGXHWRĮ (rigid body dynamics) m D S G and the nondimensional SLWFKLQJPRPHQWGXHWRĮ (atmospheric turbulence) respectively. Similar G coefficients were implemented for the normal force. These coefficients represent different contributions due to the effect of atmospheric turbulence on the aircraft. Stated differently, knowing the effect of atmospheric WXUEXOHQFHRQĮVHQVRUGRHVQRWLPSO\WKDWWKHHIIHFWRI atmospheric turbulence on the aircraft response can be calculated by simply m XOWLSO\LQJĮE\WKHXVXDO ĮFRHIILFLHQWV,WLVDVVXPHGWKDW WKHHIIHFWRIWXUEXOHQFHRQDLUFUDIWUHVSRQVHLVSURSRUWLRQDOWRWKHHIIHFWRIWXUEXOHQFHRQWKHĮVHQVRU therefore, turbulenc HFRHIILFLHQWVDUHHVWLPDWHGXVLQJWXUEXOHQWĮDVDUHJUHVVRUWRFDSWXUHLWVHIIHFW6R LW¶VQRWUHDOO\DWXUEXOHQWĮFRHIILFLHQWDVPXFKDVLWLVDQHIIHFWRIWXUEXOHQFHXVLQJWKHEHVWPHDVXUHRI turbulence. Both aerodynamic model structures are presented in Section D.3.1.3.1.

D.3.1.3 Modeling For this study, a simplified Twin Otter aircraft simulation was generated outside the DSIX software environment using the MATHWORKS SIMULINK software. The simulation simplified the nonlinear equations of motion (Ref. 14) to 3 degrees of freedom (3-DOF) longitudinal dynamics and a linear aerodynamics model. The simulation rate was 50 Hz and the pertinent output variables were corrupted with gaussian noise with magnitudes that resulted in a signal to noise ratio of 50 for the measured outputs.

The use of this simplified simulation was warranted to avoid changes to the high fidelity DSIX database as model structure changes were investigated.

D.3.1.3.1 Model Equations For reference purposes, a detailed listing of the 3-DOF longitudinal equations of motion (Ref. 14) for a symmetric rigid aircraft is included.

Force Equations:  sin ( ) / u qw g X T m  T   (D.5a) X  cos ( ) / w qu g Z T m  T   (D.5b) Z Moment Equation: I q M M   (D.6) Y T Kinematic Equation:  q T (D.7) Navigation Equations: cos sin p u w T  T  (D.8a) N NASA/CR—2014-218320 65  sin cos h u w T  T (D.8b) Atmospheric Turbulence Contributions: u u u  (D.9a) aerodynamics turbulence w w w  (D.9b) aerodynamics turbulence Where the atmospheric turbulence velocity components, u and w , were calculated using turbulence turbulence MIL-F-8785C specification and the Dryden Turbulence SIMULINK block for calm, light, moderate, and severe turbulence. Nondimensional aerodynamic force and moment coefficients for an aircraft can be computed from flight measurements as follows: C ma T qS  (D.10a) X X X C ma T qS  (D.10b) Z Z Z / C I q M qSc   (D.10c)

> @

m Y T The nondimensional aerodynamic force coefficients are transformed from the body axis to the wind axis as follows: cos sin C C C  D  D (D.11) L Z X For local modeling over a short time period, the force and moment coefficients computed from Equations (D.10) can be modeled using linear expansions in the aircraft states and controls: C C C e C 'D  'G  (D.12a) L L S L L e O D G qc ' (D.12b) C C C C e C 'D   'G  m m S m m m q e O D G 2 V Atmospheric Turbulence Model Structure C C C C e C 'D  'D  'G  (D.12c) L L S L G L L e O D D G G qc ' (D.12d) C C C C C e C 'D  'D   'G  m m S m G m m m q e O D D G G 2 V 7KHǻQRWDWLRQLQGLFDWHVSHUWXUEDWLRQIURPD reference condition. In Equations (D.12b) and (D.12d), C represents the nondimensional pitching moment at a reference condition, and similarly for the other m O expansions.

D.3.1.3.2 Wing Response to Turbulence In order to investigate the effects of upwash correction as a function of frequency and time delay as a function of frequency the recursive Fourier transform in the RT-PID algorithm was modified. The recursive Fourier transform is defined as: 2 1 j f n t  S  ' k   X X x e  (D.13) 1 1 n n n   NASA/CR—2014-218320 66 Where the frequency vector is defined as f = [0.1:0.04:2.0], the time delay is modeled as: 2 *time delay j  S f e I (D.14) 2 1 j f n t  S  ' k   X X x e  I 1 1 n n n   Where time delay, is a vector of time delays for each frequency f = [0.1:0.04:2.0]. The upwash correction 1 + k ( f )is a vector of upwash corrections for each frequency f = [0.1:0.04:2.0]. The updated recursive u Fourier transform is defined as: 2 1 j f n t  S  ' k   1 X X x k f f e  ª  º ªI º (D.15) 1 1 n n n u   ¬ ¼ ¬ ¼ D.3.1.3.3 Vane Sensor Model In order to investigate the frequency response of airflow angle vanes, a second-order airflow vane sensor dynamic model was studied. A natural frequency of Z = 9 Hz and a damping ratio ] = 0.35 were n chosen as representative values for a dynamic model of the airflow vanes (Ref. 10 to 12). Typical natural frequency and damping ratio values range from 5 to 20 Hz and 0.2 to 0.6, respectively. Equation (D.6) represents a second order continous-time transfer function: Z n G s (D.16) 2 2 2 s s  ]Z  Z n n The transfer function representation does not support non-zero initial conditions; therefore, the vane sensor model was converted to a state space representation which supported a trim D initial condition.

D.3.1.4 Testing The hypothesis was tested using the SIMULINK model described and the following test conditions: 1. Baseline run with a perfect AOA measurement and the aerodynamic model structure defined in Equation (D.12).

2. Based on findings in Section D.3.1.3.3, repeat test condition 1 with a representative frequency response model for the airflow angle vanes.

3. Repeat test condition 2 with varying AOA magnitude and time delay as a function of frequency.

4. Based on findings in Section D.3.1.2.2, add turbulence coefficients to the aerodynamic model structure and DSHUIHFWĮPHDVXUHPHQW D.3.1.5 Results and Analysis The following parameters are common to all four test conditions.

1. Aircraft trimmed at 10,000 ft and 140 kt 2. 20 sec runs 3. Orthogonal multi-sine inputs (Ref. 14) invoked from 1 to 11 sec 4. Model parameter estimates taken at the end of each 20 sec run 5. 200 runs per atmospheric turbulence intensity (calm, light, moderate, and severe) 6. Fourier transform frequency spacing 0.04 Hz 7. Fourier transform frequency range [0.1 2.0] Hz 8. Signal-to-noise ratio of 50 9. 50 Hz data rate NASA/CR—2014-218320 67 The following comments are common to all of the figures associated with test conditions one to four: 1. Green line—truth value of the parameter 2. Red dashed line—confidence interval using ±10 percent of truth value of the parameter 3. Red diamond—parameter estimate 4. Blue bars—estimated two sigma error bounds In the dynamic modeling process, no attempt was made at estimating the drag coefficients, because the aircraft excitations did not appreciably changed airspeed, resulting in a low signal-to-noise ratio for the axial force coefficient parameter estimation. Additionally, contributions from the lift damping coefficient C were small and omitted from the model and the estimation process. In total, five Lq coefficients were estimated, C , C , C , C , and C .

L L m mq m D D G e G e D.3.1.5.1 Test Condition 1 As shown in Figure D.8, the baseline model estimates for C resulted in improved estimates with Z D increasing atmospheric turbulence. This is not representative of actual flight data and is attributed to SHUIHFWĮ measurement given the additional excitation caused by atmospheric turbulence.

As shown in Figure D.9, the baseline model estimates for C resulted in greater scatter as m G e turbulence intensity increased. This is caused by noisy explanatory variables (Ref. 15) from the atmospheric turbulence implementation in Equation (D.9). Similar patterns were observed with the remaining coefficients C , C , C , and C , although the results are not shown. These simulations Zq Z m mq D G e demonstrate that all if the AOA measurements are perfect, all model parameters can be estimated accurately for any level of turbulence.

D D Z Z C C D Z D Z C C Figure D.8.—Test Condition 1—Effect on C Z D NASA/CR—2014-218320 68 e G e G m m C C e G e G m m C C Figure D.9.—Test Condition 1—Effect on C m G e D.3.1.5.2 Test Condition 2 Estimates for C using the second-order vane sensor model of Equation (D.6) ( Z = 9 Hz and Z n D ] = 0.35) caused degradations and biasing in the estimates with increasing turbulence intensities, as shown in Figure D.10. This is representative of actual flight data.

As shown in Figure D.11, the baseline model estimates for C resulted in greater scatter as m G e atmospheric turbulence intensity increased. This is caused by noisy explanatory variables (Ref. 15) from the atmospheric turbulence implementation in Equation (D.9), and the vane sensor model. Similar patterns were observed with the remaining coefficients C , C , C , and C , although the results are Zq Z m mq D G e not shown.

NASA/CR—2014-218320 69 D D Z Z C C D D Z Z C C Figure D.10.—Test Condition 2—Effect on C Z D e G e G m m C C e G e G m m C C Figure D.11.—Test Condition 2—Effect on C m G e NASA/CR—2014-218320 70 D.3.1.5.3 Test Condition 2—Expanded Testing Although no quantitative analysis was done, qualitatively, small differences were observed between a perfect and typical vane frequency response in test conditions 1 and 2. This is expected because the vane dynamics do not substantially contribute to the scatter in the estimates, since all of the added dynamics are outside the range of frequencies analyzed.

To further validate this claim, additional testing was carried out using the same simulation. A total of 121 different combinations of flow angle vane natural frequency and damping ratios were tested. The natural frequency values tested were 5 through 15 in 1 Hz increments. The damping ratio values tested were 0.01, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, and 0.6. Since the AIMMS-20 probe approaches a gain of one and zero phase shift, it is characterized by the results of test condition 1. The AIMMS-20 probe offers superior frequency response performance to mechanical vanes by using 1) a 5 hole pressure probe , 2) short tube lengths (air data tubes are typically less than 5 in. long), and 3) the resonance frequency of the silicon pressure diaphragm is in the kilohertz region. The following figures depict the results of 121 combinations, times 4 turbulence levels, times 200 runs per case. This totaled 96,800 runs.

Figure D.12 depicts the results of 96,800 runs for C . As an example, the bottom left corner block in Z D severe turbulence is light blue. The color represents the number of exceedances of C from ±10 percent Z D of truth. The remaining 120 blocks are dark blue representing less than 10 exceedances. Figure D.12 suggests that any combination of natural frequency and damping ratio tested provides adequate performance. Similar results were obtained for C and C results are not shown.

m mq D Figure D.12.—Number of exceedances for C Z D NASA/CR—2014-218320 71 Figure D.13 shows that independent of the natural frequency and damping ratio tested, accurate estimates of C are feasible in atmospheric turbulence. This is expected due to strong elevator control m G e authority.

Results for C , not shown, demonstrate that independent of the natural frequency and damping ratio Zq tested, exceedances less than 20 out of 200 or ±10 percent of truth are not feasible in any level of atmospheric turbulence except calm. For the remaining stability and control estimates, the results show that most combinations of natural frequency and damping ratio tested provide adequate performance in calm, light, moderate, and severe atmospheric turbulence.

D.3.1.5.4 Test Condition 3 As shown in Figure D.14, estimates for C resulted in degraded estimates with increasing Z D atmospheric turbulence for varying AOA magnitude and time delay as a function of frequency implemented using the data in Figure D.6 and Figure D.7. The degradation is seen as biasing in the estimates and increased scatter and uncertainty estimate error bounds. This is representative of actual flight data.

As shown in Figure D.15, the estimates for C also resulted in degraded estimates with increasing Z G e atmospheric turbulence. The degradation is seen as biasing in the estimates and increased scatter and uncertainty estimates. The increase in biasing and uncertainty compared to C estimates is caused by the Z D relatively small contribution of the elevator term to the normal force coefficient.

Figure D.13.—Number of exceedances for C m G e NASA/CR—2014-218320 72 D D Z Z C C D D Z Z C C Figure D.14.—Test condition 3—Effect on C Z D e e G G Z Z C C e e G G Z Z C C Figure D.15.—Test condition 3—Effect on C Z G e NASA/CR—2014-218320 73 Figure D.16 shows that the estimates for C degraded with increasing atmospheric turbulence. The m D degradation is seen as biasing in the estimates and increased scatter and uncertainty estimates. This is representative of actual flight data.

As shown in Figure D.17, the estimates for C degraded with increasing atmospheric turbulence. The mq degradation is seen as biasing in the estimates and increased scatter and uncertainty estimates. The estimates for C resulted in the largest biasing of all the aerodynamic estimates considered because mq angular velocity model parameters are most susceptible to time delays (Ref. 16).

Figure D.18 shows that the estimates for C degraded with increasing atmospheric turbulence. The m G e degradation is seen as a scatter in the estimates and increased scatter and uncertainty estimates. This is representative of actual flight data and correlates with the flight results presented in Figure D.3.

Corrupting the AOA measurement by a frequency-dependent magnitude and time delay affected all of the estimated stability and control coefficients. The estimates of C and C exhibited moderate biasing m m D G e and increased scatter and uncertainty, C and C estimates had large biases and increased scatter and Z mq D uncertainty, and C estimates showed the largest biases and increases in scatter and uncertainty.

Z G e Previous work on the effects of time-shifted data on flight determined stability and control estimates (Ref. 16) using the output-error method in the time domain demonstrated similar trends in the biasing and increased uncertainty, compared to the results shown here due to atmospheric turbulence.

D D m m C C D D m m C C Figure D.16.—Test condition 3—Effect on C m D NASA/CR—2014-218320 74 mq mq C C mq mq C C Figure D.17.—Test condition 3—Effect on C mq e e G G m m C C e e G G m m C C Figure D.18.—Test condition 3—Effect on C m G e NASA/CR—2014-218320 75 D.3.1.5.5 Test Condition 4 Based on the findings in Sections D.3.1.2.2, additional turbulence coefficients were added to the model structure. All of the turbulence estimates for C except for the estimates in calm atmosphere L D resulted in large degradations. In fact, the degradations were so large that they were not captured by the standard plot range. A similar pattern was observed with the estimates of C . This is not representative m D of actual flight data. The estimates for C resulted in greater scatter and an overall bias as turbulence m G e intensity increased. The overall biasing effect is not representative of actual flight data. Similar patterns were observed with the remaining coefficients results are not shown.

D.3.1.6 Task 1 Conclusions Several practical issues related to real-time parameter estimation for a linear longitudinal dynamics model in atmospheric turbulence using indirect turbulence measurements were examined and discussed.

The frequency response of the airflow vanes, wing response to atmospheric turbulence, and the structural response of the air data boom were identified as sources of colored noise. The frequency response of the airflow vanes for a wide variety of combinations of natural frequency and damping ratio did not adversely impact parameter estimation results in turbulence, except for C and C . This was attributed to noisy Zq Z G e explanatory variables in the atmospheric turbulence implementation and relatively low aerodynamic contributions as compared to C . Airflow vane dynamics do not substantially contribute to the scatter in Z D the estimates, because all of the added dynamics are outside the range of frequencies analyzed. The structural modes of the boom lead to similar conclusions, because the natural frequencies of the boom structural modes are outside the range of frequencies used for dynamic modeling. A major source of colored noise was identified as the frequency-dependent upwash and time delay induced by the wing- bound vortex system and the longitudinal separation between the AOA measurement and the wing. It was also surmised that the frequency-dependent sidewash was a major source of colored noise. The frequency-dependent upwash and time delay appear to be significant contributors to biasing and increased scatter and uncertainty for parameter estimates in atmospheric turbulence.

Practical issues were examined using data from a Twin Otter DHC-6 longitudinal linear simulation, with realistic noise sequences added to the computed aircraft responses. This allowed a clear view of the effect of each source of colored noise to the modeling problem, because the true values of the model parameters were known. This approach was used initially to show that real-time parameter estimation can be done accurately in all atmospheric turbulence conditions if the AOA measurement is accurate. Flight test data from the GRC DHC-6 Twin Otter aircraft was used to validate the effect of the identified colored noise sources.

Based on these findings, several practical recommendations for flight testing in atmospheric turbulence are suggested, as well as areas for future study. First, mount the air data boom at the nose of the aircraft to minimize the effect of the wing-bound and wing-tip vortices. Second, if the air data boom is mounted on the nose of the aircraft, an upwash and time delay calibration as a function of frequency is required.

The following Task 2 and Task 3 results were carried out assuming that these precautions and calibrations had been taken care of. This assumed that the frequency response was adequate (good assumption based on the results of this section for a well calibrated, stiff, nose mounted sensor). This left the sensor’s static performance to be evaluated in Task 2 and Task 3. This eliminated the need to update the D-Six Twin Otter simulation from previous studies. In other words, if these precautions are taken, the RT-PID estimates will behave similarly inside and outside of atmospheric turbulence.

D.3.2 Task 2 Through initial testing it was found that the Twin Otter at a typical cruise condition was not capable of controlled flight in severe turbulence with the virtual pilot used for this study. During runs in severe turbulence, the aircraft would consistently enter a wing stall. The data from stalled runs was not NASA/CR—2014-218320 76 considered useful, because stalled flight does not produce reliable data. As a result, all severe turbulence test conditions have been omitted from Task 2 analysis. Additionally, non-ice test conditions were run, but in the context of RT-PID, were considered outside of the scope and omitted from the analysis. Both ice and no ice runs were considered when evaluating D-ICES performance.

The purpose of Task 2 was to modify, test, and validate the use of RT-PID and D-ICES algorithms with atmospheric turbulence as an input for various levels of atmospheric turbulence, and to identify an acceptable error in the wind measurement during turbulence for RT-PID and D-ICES algorithms. Wind measurement error was varied from 0.0 to 1.0 kt. Eighty test runs were conducted with and without ice, turbulence, and wind measurement errors.

D.3.2.1 D-ICES Study Results It was found that the application of turbulence and sensor noise on D-ICES had a minimal impact on performance without filtering. With the filtering scheme of state variables described in Section D.3.2.1.1, D-ICES performance allowed accurate detection of possible icing situations in the presence of turbulence and sensor noise. D-ICES used a Theil inequality coefficient, comparing the a-priori elevator deflection for a clean aircraft to that of the current flight condition, to statistically estimate the aircraft’s current state. The estimated clean elevator deflections were calculated using a flight model inversion technique to find a corresponding control deflection for the given flight condition. The Theil inequality coefficient provided a measure of how well the actual elevator deflection matched the calculated elevator deflection, where a value of 0 indicated a total match, and 1 indicated a total mismatch. If the coefficient exceeds a threshold of 0.18 with a 3 sec delay to latch true, D-ICES activates the IVHM ID mode.

As a result, a metric for success with sensor noise and turbulence is determined by their effect on creating false positive or false negative mode switches. A false positive condition would be one where D- ICES latches the ICEPro ID mode in a non-iced flight condition. While the impact of an occasional false positive is a human factors exercise, the impact of a false negative where the pilot is told the aircraft is clean while accreting ice would have serious implications.

As shown in Figure D.19 D-ICES allows one false positive indication without filtering. Adding sensor noise to selected parameters used in the ICEPro logic, as discussed in Section D.2 causes the Theil inequality coefficient to increase, growing larger as the severity of the sensor noise increases. When the largest sensor noise was used the Theil inequality coefficient exceeded the ID mode threshold, which would cause an IVHM mode switch to ID mode in normal operation.

Looking at the iced aircraft results without filtering in Figure D.20, D-ICES fails to indicate iced conditions in severe turbulence. This system failure is due to the severity of the turbulence as the aircraft stalls and full elevator deflection has been reached by the virtual pilot. The virtual pilot, described in Equation (D.1), was unable to maintain airspeed or a level attitude. This is seen in Figure D.21. Due to the lack of aircraft control and severity of the turbulence in the severe turbulence scenarios these conditions will not be evaluated. As discussed in Section D.2.2, low virtual pilot gains were used for all conditions which worked well without turbulence, but were less than optimal in turbulence. The low virtual pilot gains most likely contributed to the aircraft stalling in severe turbulence. For Phase I tests, a fixed throttle setting was used. However, during Phase II testing, aircraft configuration changes required the throttles to be actively controlled. The deactivation of the throttles during Phase I was done to eliminate the excessive and frequent throttle movements during flight in turbulence where the control logic was attempting to hold velocity in dynamic conditions.

NASA/CR—2014-218320 77 D-ICES Unfiltered State Vector Clean Aircraft No Turbulence 0.9 Light Turbulence Med Turbulence 0.8 Sev Turbulence 0.7 0.6 0.5 ID Mode Threshold 0.4 (Flight vs Inversion) e G 0.3 Theil Inequality Coefficient 0.2 0.1 0 2 4 6 8 10 12 14 16 18 20 Time, sec Figure D.19.—Unfiltered D-ICES Performance with Clean Aircraft D-ICES Unfiltered State Vector Iced Aircraft No Turbulence 0.9 Light Turbulence Med Turbulence 0.8 Sev Turbulence 0.7 0.6 0.5 ID Mode Threshold 0.4 (Flight vs Inversion) e G 0.3 Theil Inequality Coefficient 0.2 0.1 0 2 4 6 8 10 12 14 16 18 20 Time, sec Figure D.20.—Unfiltered D-ICES Performance with Iced Aircraft NASA/CR—2014-218320 78 Severe Turbulence Flight Condtions -10 , deg e G -20 Elevator Deflection -30 0 5 10 15 20 , deg D Angle of Attack -10 0 5 10 15 20 ft/sec Downward Wind Component -20 0 5 10 15 20 Time, sec Figure D.21.—Severe Turbulence Flight Conditions D.3.2.1.1 Results of 80 Runs With Fixed Random Seed The flight model inversion process in D-ICES is dependent on an input state vector derived from the current flight conditions. This state vector includes the following eight elements: x Angle of attack x Angle of sideslip x Roll rate x Pitch rate x Yaw rate x Elevator deflection x Aileron deflections x Rudder deflection Since D-ICES was only used in initial mode switching, it was proposed that filtering these inputs for the flight model inversion calculation could be done with a minimal delay on system mode switching. A second order low pass filter was chosen with a cut-off frequency of 2.5 Hz.

The four turbulence scenarios were tested, no turbulence, light turbulence, medium turbulence and severe turbulence. As previously explained, the severe turbulence results were omitted from the analysis.

In each turbulence scenario, sensor noise was added, set to either a nominal condition or off. Finally, a fixed random seed, or a reusable series of random numbers, was used to generate turbulence to allow direct comparison between the runs.

Figure D.22 shows the filtered conditions for the no-ice or clean aircraft with a fixed random noise seed. Here the noise influence shown in Figure D.19 resulting in a false positive has been addressed. In addition the effect of atmospheric turbulence on the D-ICES algorithm is minimal.

NASA/CR—2014-218320 79 In the iced aircraft scenario, shown in Figure D.23, there were no false negatives with D-ICES. The sensor noise effects have been addressed as the scenarios with and without noise are almost over-plots. There is a banding effect due to the level of turbulence (i.e., all of the results for a common level of turbulence grouped tightly together but at different Theil inequality coefficient values); however, all conditions are well above the selected ID Mode switch threshold and result in mode switching at similar times.

D-ICES Filtered State Vector Clean Aircraft No Turbulence 0.9 Light Turbulence Med Turbulence 0.8 0.7 0.6 0.5 ID Mode Threshold 0.4 (Flight vs Inversion) e G 0.3 Theil Inequality Coefficient 0.2 0.1 0 2 4 6 8 10 12 14 16 18 20 Time, sec Figure D.22.—Filtered Clean Aircraft with Fixed Random Seed D-ICES Filtered State Vector Iced Aircraft No Turbulence 0.9 Light Turbulence Med Turbulence 0.8 0.7 0.6 0.5 ID Mode Threshold 0.4 (Flight vs Inversion) e G 0.3 Theil Inequality Coefficient 0.2 0.1 0 2 4 6 8 10 12 14 16 18 20 Time, sec Figure D.23.—Filtered Iced Aircraft with Fixed Random Seed NASA/CR—2014-218320 80 D.3.2.1.2 Results of 60 Runs Random Seed A total of 60 scenarios were tested (the 20 severe turbulence cases were dropped, as mentioned earlier), again at three turbulence levels with one condition without sensor noise and nine additional scenarios with sensor noise. In addition, the seed for the random number generation was selected randomly creating a unique set of random number for every run.

In Figure D.24, the clean aircraft conditions continue to exhibit a very tight banding with only three turbulence conditions potentially exhibiting a false positive. Further examination revealed that in the immediate trim AOA region, small changes in AOA produced very little variation in pitching moment due to a specific elevator deflection. If the AOA went to higher values where there was a variation with AOA, the Theil inequality coefficient remained well below the threshold value. It was surmised that the D-ICES inversion process had some troubles in determining the correct elevator deflection near trim, in the presence of light turbulence with these pitch due to elevator characteristics. A real pilot would probably not chase the variations in speed and flight path in the same manner as the virtual pilot did in these runs, and the results could differ. Looking at only the initial 6 sec after buffer fill ( t = 6 to t = 12) provides a snapshot of the typical D-ICES operation.

Figure D.25 shows 60 separate runs in D-ICES performance under three turbulence conditions. Under light and medium turbulence D-ICES successfully detected the presence of icing conditions without degradation or delaying IVHM mode shifting.

D-ICES Filtered State Vector Clean Aircraft No Turbulence 0.9 Light Turbulence Med Turbulence 0.8 0.7 0.6 0.5 ID Mode Threshold 0.4 (Flight vs Inversion) e G 0.3 Theil Inequality Coefficient 0.2 0.1 0 2 4 6 8 10 12 14 16 18 20 Time, sec Figure D.24.—Filtered Clean Aircraft with Random Seed NASA/CR—2014-218320 81 D-ICES Filtered State Vector Iced Aircraft No Turbulence 0.9 Light Turbulence Med Turbulence 0.8 0.7 0.6 0.5 ID Mode Threshold 0.4 (Flight vs Inversion) e G 0.3 Theil Inequality Coefficient 0.2 0.1 0 2 4 6 8 10 12 14 16 18 20 Time, sec Figure D.25.—Filtered Iced Aircraft with Random Seed D.3.2.2 RT-PID Study Results It was found that the application of turbulence and sensor noise on RT-PID had a significant impact on performance. As previously mentioned, non-ice test conditions were run, but in the context of RT-PID were considered outside of the scope and omitted from the analysis. Therefore, only the 40 iced runs will be reported. Results from the 40 iced runs with fixed turbulence seed are depicted in Figure D.26. The error in flow angles was varied from 0.0° to 1.0° in 0.1° increments. All other errors were fixed using the probe specifications in Table D.1.

The results of Figure D.27 illustrate the effect of flow angle error on the coefficient C . As m D expected, increased flow angle error results in increasing error in the estimate and increased error bars throughout each turbulence level. The severe turbulence cases are discarded from the analysis since the majority of the flight resulted in a stalled condition.

Next, the MOS message criterion was applied with white Gaussian measurement noise. The following results for C , representative of the pitch degrade message, are depicted in Figure D.27 with fixed m G e turbulence seed. As the turbulence level increases so does the error in the estimate and the error bounds.

NASA/CR—2014-218320 82 Iced airplane C m D Figure D.26.—Effect of flow angle error on C m D NASA/CR—2014-218320 83 Iced airplane C m G e Figure D.27.—Effect of flow angle error on C m e G D.3.2.3 Task 2 Measures of Success (MOS) The MOS message criterion is summarized below for convenience (Appendix E).

1) Message accuracy Message error will be determined by the error bounds and accuracy of its associated stability and control coefficient for all ICEPro messages. Coefficients must meet or exceed the non- atmospheric turbulence error bound and accuracy in atmospheric turbulence.

In general, the MOS for message accuracy is not achieved. As seen in Figure D.27, the estimates move away from the truth and error bounds increase with each increase in turbulence level. The same NASA/CR—2014-218320 84 runs that were used to disqualify the MOS message criterion were used to evaluate the ISP MOS criterion summarized below for convenience (Appendix E).

2) ISP accuracy ISP error will be determined by comparing the true ISP value with the estimated ISP value. ISP is a function of five stability and control coefficients: Longitudinal static stability, elevator control power, dihedral effect, aileron control power, and normal lift coefficient. The ISP error must meet or exceed its non-atmospheric turbulence error bound and accuracy in atmospheric turbulence.

The following results for ISP, representative of all 40 ice runs with fixed turbulence seed, are depicted in Figure D.28.

Figure D.28 depicts the ISP time history for each simulated flow angle error. The ISP value is initially set to zero at the start of each run and ramps up as ice is detected. The D-ICES threshold for determining an ice condition is an ISP value of 0.5. This is depicted by dashed red lines in each subplot. Visual inspection confirms that the ISP is different in and out of turbulence. In order to satisfy the original MOS criteria, the ISP lines in and out of turbulence must be the same. As shown in Figure D.28 this is not the case. Additionally the ISP deteriorated greatly with increased flow angle error. In general, the MOS criterions for both messages and ISP, as originally defined, were not achieved. A new MOS criterion was formulated based on the results depicted in Figure D.27 and Figure D.28. The new criterion takes into account the degradation of RT-PID estimates and error bounds in turbulence. Errors in the estimates and in the error bounds with turbulence were tolerated as long as the ICEPro messages and AOA brackets are equivalently displayed inside and outside of turbulence. The criterion is summarized below.

Figure D.28.—ISP for all turbulence levels and flow angle errors ( D and E ) NASA/CR—2014-218320 85 1. Compare high and low AOA brackets across turbulence levels. Angle of attack brackets are strongly dependent on AOA and flap deflection and weekly dependent on ISP.

ż Equal or conservative bracket location – Successful envelope protection ż Unconservative bracket location – Unsuccessful envelope protection 2. Compare message state across turbulence levels as well as time to latch and time to delatch.

ż Same state annunciation – Successful envelope protection ż Varying state annunciation – Unsuccessful envelope protection ż Latch / delatch within 10 sec of no turbulence results – Successful envelope protection The following results for message accuracy, representative of AIMMS-20 measurement errors for all turbulence levels, are depicted in Figure D.29 using the new MOS criterion. Only the results of the Roll degrade message are plotted since the remaining messages did not result in false positives or false negatives and were considered successful. In theory, the same messages are expected for all turbulence levels since the aircraft is flown about the same reference condition. In Figure D.29, this is represented by the average AOA and angle of sideslip during each run. In practice, for zero, light, and moderate turbulence the average flow angles are similar. In severe turbulence, the average flow angles are significantly different due to wing stall, but do not affect the message accuracy. As stated previously the severe turbulence runs are omitted from the analyses and all runs used a fixed turbulence seed for comparison purposes.

In the zero turbulence case, the roll degrade message results in a false positive warning message. This is explained by the aileron control power estimate subplot. The estimate of the aileron control power quickly diverges from the truth. This difference causes the false positive indication. It should be noted that although a false positive condition occurs due to poor estimates the ICEPro logics latches at the correct levels. As a side note, this condition was repeated with higher buzzing amplitudes and no false positives occurred. In each of the turbulence cases, the additional aileron inputs provided by the virtual pilot improved the RT-PID estimates so that no false positives were annunciated. In general the new MOS message criterion is considered to be successful.

Additional testing was conducted to access the new MOS AOA bracket criterion. The following results for AOA bracket accuracy, representative of AIMMS-20 measurement errors with conservative flow angle errors for all turbulence levels, are depicted in Figure D.30. A conservative flow angle error of 0.3° versus the stated 0.1° was chosen because experience has proven the difficulty of calibrating up-wash and side-wash corrections on air data probes.

As in the message MOS criterion, the same AOA bracket locations are expected for all turbulence levels since the aircraft is flown about the same reference condition. In Figure D.30, these are represented by the upper and lower bar delta plots using zero turbulence as the reference. For light and moderate turbulence the average deltas are similar. In severe turbulence, the average delta positions are significantly different due to wing stall and hence significantly affect the angle of attach bracket location and accuracy. The severe turbulence results are omitted from the analysis.

As shown in Figure D.30, after 25 sec from start-up the ISP oscillates between 0.8 and 1.0. Similarly, after 25 sec the AOA brackets oscillate about the zero turbulence AOA brackets. Additional filtering was implemented to smooth the AOA brackets in turbulence. The filtered results are shown in Figure D.31.

The results presented in Figure D.31 are filtered by a low pass fourth order Butterworth filter with a cut off frequency of 0.1 radians per second. Similar plots were generated to determine the effect of flow NASA/CR—2014-218320 86 angle error versus ISP. It was determined that the AOA brackets are strongly dependant on AOA and weekly dependant on ISP. Even with perfect instruments the ISP was very similar to the ISP in Figure D.31 with 0.3° flow angle error. The trend in Figure D.31 is that the AOA bracket location in turbulence oscillates around the zero turbulence case. This agrees with the analysis in Figure D.29 were the average AOA and sideslip was approximately the same for all runs except severe turbulence. Closer inspection shows that the AOA brackets are conservative and unconservative throughout the runs. Looking at the period after 30 sec, where the ISP has stabilized after start-up, the oscillations are approximately 1° for light turbulence and 2° for moderate turbulence. These small changes are not considered to be discernible by the pilot. Additionally, since the oscillations are due to the variation in AOA throughout the run and not due to the ISP, the new MOS AOA bracket criterion is considered to be successful for this task.

a G l C Figure D.29.—Roll degrade indication analyses NASA/CR—2014-218320 87 AOA limit indication (0.3° flow vane error) Figure D.30.—Angle of attack bracket indication analyses NASA/CR—2014-218320 88 AOA limit indication (0.3° flow vane error) Figure D.31.—Filtered AOA bracket indication analyses NASA/CR—2014-218320 89 D.3.2.4 Task 2 Conclusions RT-PID x The original RT-PID MOS (Appendix E) for message accuracy were not achieved. A new MOS criterion was formulated due to results depicted in Figure D.27 and Figure D.28.

x A new MOS was defined were errors in the estimates and in the error bounds with turbulence are tolerated as long as the ICEPro messages and AOA brackets are equivalently displayed inside and outside of turbulence.

x The average flow angles are similar for zero, light, and moderate turbulence. In severe turbulence, the average flow angles are significantly different due to wing stall, but do not affect the message accuracy.

x With the exception of the Roll degrade message, there were no false positives or false negatives.

The roll degrade message was caused during a period of low control activity. In each of the turbulence cases, the additional aileron inputs provided by the virtual pilot improved the RT-PID estimates so that no false positives were annunciated.

x In general the new MOS message criterion is considered to be successful.

D-ICES x It was found that the application of turbulence and sensor noise on D-ICES had minimal impact on performance without filtering.

ż Only one condition was found that yielded a false positive that would have caused the system to enter ID Mode when no icing was present.

ż There was one condition where the elevator deflection Theil inequality coefficient for an iced airplane failed to exceed the threshold and two where the value originally exceeded the threshold but with time decreased until it was below the threshold.

x When the state vectors going to the D-ICES inversion routine were filtered to remove higher frequency disturbances due to turbulence and noise, D-ICES accurately detected all icing conditions in the presence of turbulence and sensor noise and thus operated correctly.

x Three potential false positive cases were found that showed the elevator Theil Coefficient built up over time to exceed the threshold when no ice was present. An examination of the data in this region led to supposition that the D-ICES inversion process had some troubles in determining the correct elevator deflection with the pitch due to elevator characteristics at those flight conditions.

Overall it was found that the objective to identify an acceptable error in the wind measurement was not required and that the performance of the AIMMS-20 probe was satisfactory for Task 2.

D.3.3 Task 3 The purpose of Task 3 was to evaluate four representative phases of flight (Appendix E) with the performance results of Task 2. Twenty four test runs were conducted with and without ice, turbulence, and wind measurement errors over four phases of flights. Turbulence was tested as calm, light, moderate, and severe. Turbulence levels are defined per MIL-F-8785C. All measurement errors were biased in a random time history with limits defined by the condition matrix and sensor specifications using the probe specifications in Table D.1 and the same conservative flow angle errors.

Whereas Task 2 focused on straight and level flight in the cruise configuration, Task 3 involved additional flight phases including, for a reference, a repeat of flaps up straight and level flight, straight and level flight flap transition, straight and level flight heading capture, and –3° flight path angle with a flap transition. This last phase is representative of the instrument approach task studied with the ICEFTD and thirty subject pilots at Embry-Riddle Aeronautical University during the summer of 2009.

Through initial testing it was found that the Twin Otter in approach configuration (20° flap) was not capable of controlled flight in moderate and severe turbulence. During some runs in moderate and all runs in severe turbulence, the aircraft entered a wing stall with the virtual pilot. The data from stalled runs was NASA/CR—2014-218320 90 not considered useful. As a result, all moderate and severe turbulence test conditions have been omitted from the RT-PID Task 3 analysis. The non-ice test conditions were run, but in the context of RT-PID, were considered outside of the scope and omitted from analysis. Additionally, the severe turbulence test conditions have been omitted from the D-ICES Task 3 analyses.

D.3.3.1 D-ICES Study Results D.3.3.1.1 Results of 24 Runs Fixed Seed The first phase matched the Task 2 conditions. The aircraft was level at 2500 ft, 110 kt with the flaps retracted. As expected, results in Figure D.32 match Figure D.22 and Figure D.23 in Task 2.

In the second phase of flight, the aircraft was slowed from an initial velocity of 110 to 80 kt and the flaps were then extended from 0° to 20°. D-ICES experienced no false positives during this scenario as seen in Figure D.33. The clean aircraft tests experienced little change due to the dynamic flight conditions.

The next phase of flight examined a heading change from 090° to 045°. Here a false negative was seen for the iced airplane with no turbulence, as shown in Figure D.34. The value of the Theil inequality coefficient for elevator deflection is a measure of how well the actual elevator time history matches the predicted time history for an uniced airplane. For an uniced airplane, the elevator deflection time history should be virtually identical to the D-ICES predicted time history. This will produce very low values of the deU Theil inequality coefficient. A sample of the elevator and corresponding Theil inequality coefficients for the uniced airplane with light turbulence is shown in Figure D.35. In the lower plot, INV_PitchCmd is the predicted elevator deflection, while G is the actual deflection.

e Task 2 Level flight, 2500 ft, 110 kt, G : 0 ° f Clean/No Turb 0.9 Clean/Light Turb Clean/Med Turb 0.8 Ice/No turb 0.7 Ice/Light Turb Ice/Med Turb 0.6 0.5 ID Mode Threshold 0.4 (Flight vs Inversion) e G 0.3 Theil Inequality Coefficient 0.2 0.1 0 2 4 6 8 10 12 14 16 18 20 Time, sec Figure D.32.—D-ICES Level Flight Scenario NASA/CR—2014-218320 91 Task 2 Level flight, 110 kt o 80 kt, G : 0 ° o 20 ° f Clean/No Turb 0.9 Clean/Light Turb Clean/Med Turb 0.8 Ice/No turb 0.7 Ice/Light Turb Ice/Med Turb 0.6 Flap 0.5 Transition ID Mode Threshold 0.4 (Flight vs Inversion) e G 0.3 Theil Inequality Coefficient 0.2 0.1 0 5 10 15 20 25 30 35 Time, sec Figure D.33.—D-ICES Flap Transition Scenario Task 2 Level flight, left turn 090 ° o 045 ° 80 kt, G : 20 ° f Clean/No Turb 0.9 Clean/Light Turb Clean/Med Turb 0.8 Ice/No turb 0.7 Ice/Light Turb Ice/Med Turb 0.6 Heading 0.5 Change ID Mode Threshold 0.4 (Flight vs Inversion) e G 0.3 Theil Inequality Coefficient 0.2 0.1 0 5 10 15 20 25 30 Time, sec Figure D.34.—D-ICES Heading Change Scenario For an iced airplane, there should be some differences between the expected (uniced) elevator deflection and the actual deflection. The closer the iced and uniced C values are, the less this will be.

m G e In Figure D.36, one can see that the actual and predicted elevator deflection were starting to separate NASA/CR—2014-218320 92 resulting in the larger Theil inequality coefficient values. However, because the iced C was not very m G e different from the uniced values at this flight condition, the differences without turbulence were not enough to exceed the threshold. With the addition of turbulence, the elevator deflection separation became greater and resulted in Theil inequality coefficient values that exceeded the threshold. This is probably because the turbulence oscillations caused changes in alphas where there was greater difference between the iced and uniced C .

m ’s G e In the fourth phase of flight, the aircraft was flown down a 3° glide slope from 3000 ft at 75 kt. As the descent was established, the airspeed was reduced from 75 to 70 kt and flaps were then extended to 30°.

While there were no false indications, Figure D.37 shows that the D-ICES results remained close to the singular condition seen in the level turning flight case condition. However, the Theil inequality coefficient values are still sufficient to exceed the threshold for transitioning to ID mode for all of the iced conditions.

Figure D.35.—Uniced Elevator Deflection Theil inequality Coefficient Analysis Figure D.36.—Iced Elevator Deflection Theil inequality Coefficient Analysis NASA/CR—2014-218320 93 Task 2 3 ° descent from 3000 ft, 75 kt o 70 kt, G : 20 ° o 30 ° f Clean/No Turb 0.9 Clean/Light Turb Clean/Med Turb 0.8 Ice/No turb 0.7 Ice/Light Turb Ice/Med Turb 0.6 Airspeed Flap 0.5 Change Transition ID Mode Threshold 0.4 (Flight vs Inversion) e G 0.3 Theil Inequality Coefficient 0.2 0.1 0 5 10 15 20 25 30 Time, sec Figure D.37.—D-ICES Descent with Flap Change Scenario Clean/No Turb 0.9 Clean/Light Turb Ice/No turb 0.8 Ice/Light Turb 0.7 0.6 0.5 ID Mode ID Mode ID Mode ID Mode 0.4 Threshold Threshold Threshold Threshold 0.3 0.2 0.1 0 5 10 15 20 25 Figure D.38.—D-ICES Task 2 with Random Seed D.3.3.1.2 Results of 24 Runs Random Seed Task 3 was rerun using a random seed. Figure D.38, summarizes the results with all four scenarios plotted. Due to occasional low speed stall issues with medium turbulence with the random seed, these conditions were excluded. As expected, the condition that caused the iced aircraft with no turbulence to exhibit a false positive during the heading capture scenario still produced similar results here. Also, with NASA/CR—2014-218320 94 the random seed, the same scenario light turbulence condition was delayed by 10 sec before latching into ID Mode.

D.3.3.2 RT-PID Study Results In general, the original MOS criterion for both messages and ISP are not achieved in Task 3 and results are similar to Figure D.27 and Figure D.28 in Task 2. As in Task 2 the new MOS criterion based on AOA brackets and messaging was applied to each flight phase. The results for this task are broken up by respective flight phase.

D.3.3.2.1 Phase 1—Flaps Up Straight and Level Flight As expected from Task 2 results, Phase 1 runs had no pitch degrade, climb limit, or flap limit messages for zero, light, and moderate turbulence levels.

As shown in Figure D.39, the light turbulence roll degrade message appears before the zero turbulence roll degrade message. Since this is within the 10 sec defined in the MOS this is considered successful. Like Figure D.29 in Task 2, Figure D.39 shows false positives for light and no turbulence cases. This condition was repeated with higher buzzing amplitudes and no false positives occurred. The MOS criterion for AOA bracket was applied to Phase 1 with similar results as shown in Task 2 Figure D.31 and considered to be successful.

a G l C Figure D.39.—Phase 1 Roll degrade indication analyses NASA/CR—2014-218320 95 D.3.3.2.2 Phase 2—Straight and Level Flight Flap Transition Phase 2 runs had no pitch degrade, yaw degrade, climb limit, or flap limit messages for zero, light, and moderate turbulence levels. As shown in Figure D.40 the roll degrade message performance in turbulence was better than outside of turbulence. In light and moderate turbulence the system did not annunciate any caution or warning messages. The increase in virtual pilot activity in turbulence during the flap transition provided improved RT-PID performance to that shown in Figure D.40. In Figure D.40 the bottom plot shows poor RT-PID performance in zero turbulence with respect to the truth values due to lack of information content.

As in Task 2, the same AOA bracket locations are expected for all turbulence levels since the aircraft is flown about the same reference condition. In Figure D.41, these are represented by the upper and lower bar delta plots using calm atmosphere as a reference. After approximately 35 sec the flap transition from zero to 20° flap is complete. Incidentally it takes approximately the same amount of time for the ISP to stabilize from start-up as shown at the bottom of Figure D.41. It should be noted that the AOA brackets exhibit similar behavior to that found in Task 2 from 35 to 60 sec and therefore considered to be successful. There are no pitch degrade messages generated during this time because the current ICEPro logic for pitch degrade messaging is based on the elevator effectiveness, C , which is well within the m G e thresholds such that no messages are generated. Data after 60 sec should be discarded because the virtual pilot is unable to adapt near the tail stall boundary and enters into a pilot coupling event between repeated incipient tail stalls and wing stalls causing the large oscillations shown. Along these lines it should be noted that the virtual pilot gains were set as low as practical while obtain reasonable performance without excessive control activity. A human pilot who is familiar with the aircraft and its flying characteristics may be better suited to control these situations. It is recommended that additional piloted runs be conducted to verify this conclusion by the virtual pilot.

a G l C Figure D.40.—Phase 2 Roll degrade indication analyses NASA/CR—2014-218320 96 Figure D.41.—Phase 2 AOA bracket analyses D.3.3.2.3 Phase 3—Straight and Level Flight Heading Capture Phase 3 runs had no pitch degrade, climb limit, or flap limit messages for zero, light and moderate turbulence levels. As shown in Figure D.42 the roll degrade message performance in turbulence was better than out of turbulence. In light turbulence the system annunciates a caution message and in moderate turbulence the system did not annunciate any caution or warning messages. The increase in virtual pilot activity during the heading capture due to turbulence provided improved RT-PID performance. In Figure D.42 the bottom plot shows poor RT-PID performance in zero turbulence with respect to the truth values due to lack of information content.

Figure D.43 shows a typical run for Phase 3, heading capture. Ice did not greatly affect the rudder effectiveness for this airplane, so it was anticipated that there would be few, if any yaw degrade messages as is shown to be the case in Figure D.43 no messages were generated for any of the heading capture conditions investigated.

NASA/CR—2014-218320 97 a G l C Figure D.42.—Phase 3 Roll degrade indication analyses G N C Figure D.43.—Phase 3 Yaw degrade indication analyses NASA/CR—2014-218320 98 The MOS criterion for AOA bracket was applied to Phase 3 with similar results as shown in Task 2, Figure D.31 and Task 3, Figure D.41 and considered to be successful. Similar to Task 3 Figure D.41 the last 20 sec should be discarded because the virtual pilot is unable to adapt near the tail stall boundary and enters into a pilot coupling event between repeated incipient tail stalls and wing stalls causing large oscillations.

D.3.3.2.4 Phase 4—Descending Flight Flap Transition—3° Glide Slope Phase 4 runs had no pitch degrade or yaw degrade messages for zero, light or moderate turbulence levels. As shown in Figure D.44 the roll degrade message performance in turbulence was better that out of turbulence. The increase in virtual pilot activity during the heading capture due to turbulence provided improved RT-PID performance. In Figure D.44 the bottom plot shows poor RT-PID performance in zero turbulence with respect to the truth values due to lack of information content.

As shown in Figure D.44 the zero turbulence roll degrade caution and warning messages annunciate before the light turbulence messages. This is considered to be conservative and successful based on MOS criterion. The toggle in the warning message in moderate turbulence is discarded because as before the virtual pilot is unable to adapt near the tail stall boundary and enters into a pilot coupling event between repeated incipient tail stalls and wing stalls. Because the elevator effectiveness during this maneuver stays within the thresholds for caution and warning messages, no pitch degrade messages are generated.

Figure D.44.—Phase 4 Roll degrade indication analyses NASA/CR—2014-218320 99 As shown in Figure D.45 the zero turbulence climb limit warning message annunciated at the same time as in light turbulence. This is considered to be successful based on the MOS criterion. In moderate turbulence the warning message came on for approximately 7 sec. The ICEPro message logic requires messages to persist for 10 sec before they can unlatch, however, when a stall is detected by ICEPro, it declutters the PFD by removing all other messaging for 20 sec to allow the pilot to regain normal flight.

After that time, ICEPro returns to normal messaging. In the situation shown in Figure D.45, a stall condition occurred which terminated the messaging.

As shown in Figure D.46 the zero turbulence flap limit warning message annunciates at the same time as in light turbulence. This is considered to be successful based on the MOS criterion. As in the climb limit message, in the moderate turbulence case the warning message came on for approximately 7 sec.

The ICEPro message logic requires messages to persist for 10 sec before they can unlatch. The stall condition resulted in decluttering the PFD and removed the message after 7 sec, as discussed above.

The MOS criterion for AOA bracket was applied to Phase 4 and shown in Figure D.47. After approximately 30 sec the flap transition from 20 to 30° flap is complete. Incidentally it takes approximately the same amount of time for the ISP to stabilize from start-up as show at the bottom of Figure D.47. It is believed that the increased oscillations in Phase 4 AOA brackets over previous phases is because of the narrow envelope between wing stall and tail stall at flaps 30°. The inability to adapt by the virtual pilot in this reduced stability regime causes larger oscillations. Applying the MOS criterion would be unsuccessful in this phase since a pilot is able to discern the unconservative variation in the AOA brackets. It is recommended that additional piloted runs be conducted to verify this conclusion by the virtual pilot.

Figure D.45.—Phase 4 Climb Limit indication analyses NASA/CR—2014-218320 100 Figure D.46.—Phase 4 Flap Limit indication analyses NASA/CR—2014-218320 101 Figure D.47.—Phase 4 AOA bracket analyses D.3.3.3 Task 3 Conclusions D-ICES x The first scenario was a repeat of Task 2 conditions and the results were similar.

x For the level flight flap extension scenario, the aircraft was slowed from an initial velocity of 110 to 80 kt and the flaps were then extended from 0° to 20°. D-ICES experienced no false positives during this scenario as seen in Figure D.33 x The next scenario examined was a heading change from 090° to 045°. Here a false negative was seen for the iced airplane with no turbulence but correctly predicted positives for the low and mid NASA/CR—2014-218320 102 turbulence levels. The false positive was believed to be due to a singular condition where there was little difference between the iced and clean airplane’s control effectiveness at the chosen flight condition.

x For the 3° glide slope descent scenario there were no false indications, even though the results remained close to a singular condition seen in the level turning flight case. However, the Theil inequality coefficient values are still sufficient to exceed the threshold for transitioning to ID mode for all of the iced conditions.

x In most cases, D-ICES performed as designed in the presence of turbulence and noise. However, the exceptions noted above, especially the false negatives, do indicate that some further development of the system is warranted. One possible area to examine would be utilizing other terms related to the Theil inequality coefficient, namely the bias, variance and co-variance terms in some way to develop a more robust detection scheme.

RT-PID x Phase 1 runs had no pitch degrade, climb limit, or flap limit messages for zero, light, and moderate turbulence levels. The MOS criterion for AOA bracket was applied to Phase 1 and considered to be successful.

x Phase 2 runs had no pitch degrade, yaw degrade, climb limit, or flap limit messages for zero, light, and moderate turbulence levels. The MOS criterion for AOA bracket was applied to Phase 2 and therefore considered to be successful.

x Phase 3 runs had no pitch degrade, climb limit, or flap limit messages for zero, light, and moderate turbulence levels. The MOS criterion for AOA bracket was applied to Phase 3 and considered to be successful.

x Phase 4 runs had no pitch degrade or yaw degrade messages for zero, light, and moderate turbulence levels. As shown in Figure D.46 the calm atmosphere flap limit warning message annunciates at the same time as in light turbulence. This is considered to be successful based on the MOS criterion. The MOS criterion for AOA bracket was applied to Phase 4: Applying the MOS criterion would be unsuccessful in this phase since a pilot is able to discern the unconservative variation in the AOA brackets.

x It is recommended that additional piloted runs be conducted to verify this conclusion by the virtual pilot.

In general, it was found that the performance of the AIMMS-20 probe was satisfactory for Task 3.

D.4 Conclusions and Recommendations Summaries are provided of the results of each task. For Task 2 and Task 3 the summaries are divided into D-ICES and RT-PID.

D.4.1 Summary of Results Task 1 Several practical issues related to real-time parameter estimation for a linear longitudinal dynamics model in atmospheric turbulence using indirect turbulence measurements were examined and discussed.

The frequency response of the airflow vanes, wing response to atmospheric turbulence, and the structural response of the air data boom were identified as sources of colored noise. The frequency response of the airflow vanes for a wide variety of combinations of natural frequency and damping ratio did not adversely impact parameter estimation results in turbulence, except for C and C . This was attributed to noisy Zq Z G e explanatory variables in the atmospheric turbulence implementation and relatively low aerodynamic contributions as compared to C . Airflow vane dynamics do not substantially contribute to the scatter in Z D the estimates, because all of the added dynamics are outside the range of frequencies analyzed. The NASA/CR—2014-218320 103 structural modes of the boom lead to similar conclusions, because the natural frequencies of the boom structural modes are outside the range of frequencies used for dynamic modeling. A major source of colored noise was identified as the frequency-dependent upwash and time delay induced by the wing- bound vortex system and the longitudinal separation between the AOA measurement and the wing. It was also surmised that the frequency-dependent sidewash was a major source of colored noise. The frequency-dependent upwash and time delay appear to be significant contributors to biasing and increased scatter and uncertainty for parameter estimates in atmospheric turbulence.

Practical issues were examined using data from a Twin Otter DHC-6 longitudinal linear simulation, with realistic noise sequences added to the computed aircraft responses. This allowed a clear view of the effect of each source of colored noise to the modeling problem, because the true values of the model parameters were known. This approach was used initially to show that real-time parameter estimation can be done accurately in all atmospheric turbulence conditions if the AOA measurement is accurate. Flight test data from the GRC DHC-6 Twin Otter aircraft was used to validate the effect of the identified colored noise sources.

Based on these findings, several practical recommendations for flight testing in atmospheric turbulence are suggested, as well as areas for future study. First, mount the air data boom at the nose of the aircraft to minimize the effect of the wing-bound and wing-tip vortices. Second, if the air data boom is mounted on the nose of the aircraft, an upwash and time delay calibration as a function of frequency is required.

Task 2 D-ICES x It was found that the application of turbulence and sensor noise on D-ICES generally had a minimal impact on performance without filtering.

ż Only one condition was found that yielded a false positive that would have caused the system to enter ID Mode when no icing was present.

ż There was one condition where the elevator deflection Theil inequality coefficient for an iced airplane failed to exceed the threshold and two where the value originally exceeded the threshold but with time decreased until it was below the threshold.

x When the state vectors going to the D-ICES inversion routine were filtered to remove higher frequency disturbances due to turbulence and noise, D-ICES accurately detected all icing conditions in the presence of turbulence and sensor noise and thus operated correctly.

x Three potential false positive cases were found that showed the elevator Theil Coefficient built up over time to exceed the threshold when no ice was present. An examination of the data in this region led to supposition that the D-ICES inversion process had some troubles in determining the correct elevator deflection with the pitch due to elevator characteristics at those flight conditions.

RT-PID x In general, the (Old) RT-PID MOS for message accuracy was not achieved. A new MOS criterion was formulated due to results depicted in Figure D.27 and Figure D.28.

x Errors in the estimates and in the error bounds with turbulence are tolerated as long as the ICEPro messages and AOA brackets are equivalently displayed inside and outside of turbulence (New MOS).

x In practice, for zero, light, and moderate turbulence the average flow angles are similar. In severe turbulence, the average flow angles are significantly different due to wing stall, but do not affect the message accuracy.

x With the exception of the Roll degrade message, there were no false positives or false negatives.

x In each of the turbulence cases, the additional aileron inputs provided by the virtual pilot improved the RT-PID estimates so that no false positives were annunciated.

NASA/CR—2014-218320 104 x In general the new MOS message criterion is considered to be successful.

Overall it was found that the objective to identify an acceptable error in the wind measurement was not required and that the performance of the AIMMS-20 probe was satisfactory for Task 2.

Task 3 D-ICES x For the level flight flap extension scenario, the aircraft was slowed from an initial velocity of 110 to 80 kt and the flaps were then extended from 0° to 20°. D-ICES experienced no false positives during this scenario as seen in Figure D.33.

x The next scenario examined was a heading change from 090° to 045°. Here a false negative was seen for the iced airplane with no turbulence but correctly predicted positives for the low and mid turbulence levels. The false positive was believed to be due to a singular condition where there was little difference between the iced and clean airplane’s control effectiveness at the chosen flight condition.

x For the 3° glide slope descent scenario there were no false indications, even though the results remained close to a singular condition seen in the level turning flight case. However, the Theil inequality coefficient values are still sufficient to exceed the threshold for transitioning to ID mode for all of the iced conditions.

RT-PID x Phase 1 runs had no pitch degrade, climb limit, or flap limit messages for zero, light, and moderate turbulence levels. The MOS criterion for AOA bracket was applied to Phase 1 and considered to be successful.

x Phase 2 runs had no pitch degrade, yaw degrade, climb limit, or flap limit messages for zero, light, and moderate turbulence levels. The MOS criterion for AOA bracket was applied to Phase 2 and therefore considered to be successful.

x Phase 3 runs had no pitch degrade, climb limit, or flap limit messages for zero, light, and moderate turbulence levels. The MOS criterion for AOA bracket was applied to Phase 3 and considered to be successful.

x Phase 4 runs had no pitch degrade or yaw degrade messages for zero, light or moderate turbulence levels. As shown in Figure D.46 the calm atmosphere flap limit warning message annunciates at the same time as in light turbulence. This is considered to be successful based on the MOS criterion. The MOS criterion for AOA bracket was applied to Phase 4: Applying the MOS criterion would be unsuccessful in this phase since a pilot is able to discern the unconservative variation in the AOA brackets.

x It is recommended that additional piloted runs be conducted to verify this conclusion by the virtual pilot.

In general, it was found that the performance of the AIMMS-20 probe was satisfactory for Tasks 2 and 3. Acceptable error is therefore defined by current specification of AIMMS-20 probe. As long as booms are not used and the calibration for wing response to turbulence has been taken care of the system will work.

D.4.2 Additional Questions Raised by Research 1. What is the calibration accuracy required of the wing response to turbulence for RT-PID estimates to be within ± 10 percent?

2. What is the boom stiffness required for RT-PID estimates to be within ± 10 percent 3. How can the other terms related to the Theil inequality coefficient, namely the bias, variance and co- variance terms be used to develop a more robust detection scheme.

NASA/CR—2014-218320 105 D.4.3 Recommended Future Studies 1. Execute Task 4. This task evaluates the effectiveness of manual pilot inputs alone for making high confidence vehicle state estimates when performing typical maneuvering flight tasks under varying levels of atmospheric turbulence.

2. Flight test to validate RT-PID findings based on colored AOA regressor findings.

3. Flight test to validate ICEPro.

D.5 References 1. Lawless, Al, “Inertial-Based Instrumentation for Performance Flight Testing,” SFTE Symposium XX Proceedings , 2008.

2. Olson, Wayne, “Aircraft Performance Flight Testing,” AFFTC-TIH-99-01.

3. Advisory Group for Aeronautical Research and Development, “Flying Qualities Flight Testing of Digital Flight Control Systems,” Flight Test Techniques Series – RTO AG-300 , Volume 21, pp. 58- 60, Dec, 2001.

4. Mathworks, “Dryden Wind Turbulence Model (Discrete)” [http://www.mathworks.com/help/toolbox/aeroblks/drydenwindturbulencemodeldiscrete.html.

Accessed 1/20/12.]

5. Maine, R. E. and Iliff, K. W., “Formulation and Implementation of a Practical Algorithm for Parameter Estimation with Process and Measurement Noise,” SIAM Journal of Applied Mathematics, Vol. 41, No. 3, 1981, pp. 558-579.

6. Maine, R.E. and Iliff, K.W., “Identification of Dynamic Systems,” AGARD AG-300 Vol. 2, Jan.

1985.

7. Crawford, T.L., and R. J. Dobosy, 1992: A sensitive fast-response probe to measure turbulence and heat flux from any airplane. J. Boundary Layer Meteorology, 59, 257–278.

8. Kalogiros, John A., Qing Wang, 2002: Aerodynamic Effects on Wind Turbulence Measurements with Research Aircraft. J. Atmos. Oceanic Technol., 19, 1567–1576.

9. Kalogiros, J. A., and Q. Wang, 2002: Calibration of a radome-differential GPS system on a Twin Otter research aircraft for turbulence measurements. J. Atmos. Oceanic Technol., 19, 159–171.

10. Lenschow, D.H., 1971: Vanes for sensing incidence angles of the air from an aircraft. Journal of Applied Meteorology 1989-2005 (After 2005 - Journal of Applied Meteorology and Climatology), 10 , pp. 1339–1343.

11. Sakamoto, G.M., “Aerodynamic Characteristics of a Vane Flow Angularity Sensor System Capable of Measuring Flight Path Accelerations for the Mach Number Range from 0.40 to 2.54,” NASA TN- D8242, 1976.

12. Foster, G.W, “The Identification of Aircraft Stability and Control Parameters in Turbulence,” Ph.D Thesis, Cranfield Institute of Technology, 1982.

13. Iliff, K.W., Maine, R.E., and Montgomery, T.D., “Important Factors in the Maximum Likelihood Analysis of Flight Test Maneuvers,” NASA TM-1459, 1979.

14. Klein, V and Morelli, E.A., Aircraft System Identification Theory and Practice, AIAA Education Series, Reston, VA, 2006.

15. Morelli, E.A., “Practical Aspects of the Equation-Error Method for Aircraft Parameter Estimation,” AIAA-2006-6144.

16. Steers, S.T. and Iliff, K.W., “Effects of Time-Shifted Data on Flight-Determined Stability and Control Derivatives,” NASA TN D-7830, 1975.

NASA/CR—2014-218320 106

Appendix E.—ICEPro Atmospheric Turbulence Test Plan

VSST – 1 Year Extension

Borja Martos

Aviation Systems and Flight Research

University of Tennessee Space Institute

MAY 2011

NASA/CR—2014-218320 107 E.1 Abstract The effects of atmospheric turbulence on the RT-PID and D-ICES algorithms are studied by modeling representative atmospheric turbulence. Atmospheric turbulence is measured directly as an input to both methods. The performance of these algorithms is evaluated using reasonable assumptions for measurements from currently-available turbulence sensors in operational flight. Methods for improving the performance of these algorithms in these conditions are implemented and studied.

E.2 Introduction E.2.1 Background A team comprised of personnel from The University of Tennessee Space Institute (UTSI), Tullahoma, Tennessee, Bihrle Applied Research Inc (BAR), Hampton, Virginia, NASA Glenn Research Center (GRC), Cleveland, Ohio, and NASA Langley Research Center (LaRC), Hampton, Virginia, has successfully developed a prototype Ice Contamination Envelope Protection System (ICEPro). The ICEPro system facilitates flight envelope protection by making continuous real time vehicle state assessments, which are synthesized into flyable pilot cueing along with visual and aural alerts during in-flight icing conditions. Detection of degraded aircraft stability and control and performance due to icing is carried out by a dynamic inversion control evaluation system (D-ICES) that compares expected aircraft behavior from a-priori knowledge base with current measures of those behaviors. When differences reach defined thresholds, real time parameter identification (RT-PID) methods are invoked to estimate current state parameters, which continuously support pilot cueing and alerts. The development effort included simulation-based design, testing, and verification. A pilot in-the-loop study was conducted to gather pilot performance data and opinions of the utility of ICEPro during simulated icing encounters. Results of the study indicated that the system performed as expected and pilot performance benefited from the envelope protection cues. However, the simulation could not duplicate all of the real-world conditions that could have first-order effects on system performance, such as those due to real atmospheric turbulence.

Therefore, in order to assess system performance under these conditions and minimize the technical risk of an eventual flight-test validation program, the effects of atmospheric turbulence on ICEPro are studied.

E.2.2 Participating Organizations The following organizations will provide simulator flight test support and facilities.

UTSI: Is one of the primary organizations and facilities for simulator flight testing and algorithm development of the ICEPro system. The basis for the simulator flight tests will be a COTS desktop simulator running BAR’s D-six software and representative flight controls, visuals, and virtual aircraft instruments.

BAR: Is one of the primary organizations for algorithm and software development of the ICEPro system. The basis for algorithm and software development will be a COTS desktop simulator running BAR’s D-Six software and representative flight controls, visuals, and virtual aircraft instruments.

GRC: Will provide technical monitoring and serve as the NASA focal point for this grant.

LaRC: Will provide technical assistant with real-time parameter identification algorithms.

E.2.3 Overview of Test Concepts Simulator Flight Test The ICEPro systems software will be tested in Birhle’s D-six software environment. Software will be tested both at UTSI and BAR. In support of these tests, BAR will provide the following: 1. Update the ICEPro software so that desktop simulations at BAR and UTSI match the ICEFTD.

NASA/CR—2014-218320 108 2. Update the ICEPro system logic diagram.

3. Synchronize system software between UTSI and BAR as software changes are made.

4. Author release notes documents with each software revision and update the system logic diagram were appropriate.

E.2.4 Outline of Document The remainder of this document is divided into two parts. Section E.3 contains the simulator flight test plan and Section E.4 contains the simulator flight test tasks.

E.3 Simulator Flight Tests E.3.1 Test Objectives The primary objectives of the simulator flight tests are to assess the performance of RT-PID and D- ICES algorithms in the presence of atmospheric turbulence and determine methods for improving algorithm performance in these conditions. The simulations will be based on events that cannot be repeatedly duplicated during flight testing and benefit from a controlled environment.

The secondary objective is to access the performance of RT-PID and D-ICES algorithms in the absence of an onboard excitation system (OBES) with and without atmospheric turbulence.

E.3.2 Test Guidelines E.3.2.1 Primary Objectives E.3.2.1.1 Performance Testing in Turbulence The performance of the RT-PID and D-ICES algorithms will be assessed by flying a low gain and a high gain pilot-in-the-loop task in the presence of atmospheric turbulence. The low gain task is straight and level unaccelerated flight with the flaps retracted. The high gain task is the final segment of an instrument approach. This represents a descent from the final approach fix to the inner marker and requires tight control of the glide slope and localizer. Both tasks will be performed under Visual Meteorological Conditions (VMC) conditions. Prior to these tasks, a UTSI pilot will train to ensure proficiency in basic instrument flying skills and can perform a precision approach within Airline Transport Pilot (ATP) standards.

E.3.2.1.2 Development of Algorithms for Atmospheric Turbulence To develop RT-PID and D-ICES algorithms in the presence of atmospheric turbulence this study will utilize the previous pilot in-the-loop test scenario. A copy of this scenario is found in Figure E.1.

However, to coordinate testing and algorithm and software development between UTSI and BAR, the test scenario will be broken up into a series of segments or phases of flight. These will consist of straight and level unaccelerated flight, flap transition, heading capture, descent, and glide slope and localizer tracking.

The use of autopilots will provide a standardized pilot performance model for batch testing at UTSI and BAR.

E.3.2.2 Secondary Objectives E.3.2.2.1 Preliminary Feasibility Study Without OBES Finally, the updated RT-PID and D-ICES algorithms will be assessed in the in the presence of atmospheric turbulence, with a pilot-in-the-loop, with and without an on board excitation system. The algorithms will be assessed by the same pilot-in-the-loop low gain and high gain tasks as in the initial task. Additionally, a UTSI pilot will be under the same training and flying standards as previously mentioned.

NASA/CR—2014-218320 109 E.3.3 Performance Parameters to be Measured All flight data will be recorded at 50 Hz. The following parameters will be recorded and calculated.

1. OBES frequency and duration 2. ISP value 3. Error in ISP parameters a. C Longitudinal Static Stability m Į b. C Elevator Control Power m į e c. C Dihedral Effect l ȕ d. C Aileron Control Power l į a e. C Normal Lift Coefficient N Į 4. Error in Message parameters 5. Glide slope and localizer performance 6. Airspeed performance 7. Number of AOA exceedances a. Stick shaker and stick pusher firings b. STALL message postings c. Tail stall events E.3.4 Measures of Success (MOS) To ensure that test objectives are attained, MOS’s have been formulated. A MOS is defined as a comparative requirement which must be satisfied by the results of testing. The overall measure of success for the primary and secondary objectives will be that the error bounds and accuracy in RT-PID and Theil inequality coefficients in D-ICES algorithms meets or exceeds its non-atmospheric turbulence performance in atmospheric turbulence. This success criterion focuses on the performance of these two algorithms and not on ICEPro’s utility to mitigate a potentially hazardous icing encounter or on the pilot’s performance during such encounter. It assumes that if the RT-PID and D-ICES algorithms perform equally in and out of atmospheric turbulence the ICEPro system will perform equally. The following MOS are defined for the primary objective.

RT-PID Algorithm 5. Message accuracy Message error will be determined by the error bounds and accuracy of its associated stability and control coefficient. For example, the red pitch degrade logic is a function of elevator control power 25 . 0 C C t ice no _ m m e e G G The elevator control power must meet or exceed its non-atmospheric turbulence error bound and accuracy in atmospheric turbulence. This is repeated for all ICEPro messages.

6. ISP accuracy ISP error will be determined by comparing the true ISP value with the estimated ISP value. ISP is a function of five stability and control coefficients: Longitudinal static stability, elevator control power, dihedral effect, aileron control power, and normal lift coefficient. The true ISP value is known since the five true stability and control parameters are known to the simulation. The ISP error must meet or exceed its non-atmospheric turbulence error bound and accuracy in atmospheric turbulence.

NASA/CR—2014-218320 110 D-ICES Algorithm 2. Control deflection accuracy Control deflection error is determined by comparing the estimated deflection Theil inequality coefficient with the baseline no turbulence case. The control deflection error must meet or exceed its non-atmospheric turbulence error in atmospheric turbulence.

E.3.5 Organizational Responsibilities The organizational responsibilities for each task are stated in Section E.4. Through each task the responsible party’s name is highlighted by parenthesis.

E.4 Simulator Flight Test Tasks E.4.1 Task 1 (UTSI) Determine whether the scatter in the turbulence estimates arise from, 1) colored noise in the regressor(s), and if so what is the source of the colored noise and in which regressor(s), 2) insufficient model structure, and if so how should the model structure be adjusted or, 3) some other means.

E.4.2 Primary Objective—Performance Testing in Turbulence Accomplished in Task 2 by baseline turbulence E.4.3 Primary Objective—Development of Algorithms for Atmospheric Turbulence E.4.3.1 Task 2 (UTSI and BAR) Modify, test, and validate the use of RT-PID and D-ICES algorithms with atmospheric turbulence as an input for various levels of atmospheric turbulence. Verify MOS during each phase of flight for varying atmospheric turbulence levels and icing conditions. UTSI and BAR will use the full 6-DOF nonlinear twin otter database.

Autopilot Task (Batch Testing) 1. Isolate the RT-PID problem from D-ICES and executive logic (UTSI) a. Flaps up straight and level flight for 30 sec at 110 KIAS and 2500 ft (1) Perfect Turbulence measurement—Optimal Inputs (a) Turbulence levels i. Baseline (no turbulence) ii. Light iii. Moderate iv. Severe (b) Ice 0 and 2 (2) Degraded Turbulence measurement—Optimal Inputs (a) Turbulence levels i. Baseline (no turbulence) ii. Light iii. Moderate iv. Severe (b) Ice 0 and 2 (c) Vary turbulence measurement error 0.3 to 1.5 kt in increments of 0.3 kt 2. Isolate the D-ICES problem from RT-PID and executive logic (BAR) a. Repeat steps outlined in 1 NASA/CR—2014-218320 111 Software Requirements (Task 2 plus) 1. D-six Light, Moderate, and Severe Turbulence Models (BAR) 2. D-six level flight autopilot with OBES hook outside of the autopilot loop (BAR) 3. D-six software switch providing isolation of D-ICES or RT-PID algorithms (BAR) 4. Example script file that calls D-six within MATLAB, trims linear model to 110 KIAS, isolates an algorithm, selects a turbulence model, selects and icing state, runs the model for a given time, and saves one example parameter (BAR) 5. Batch script file for RT-PID and D-ICES testing (UTSI and BAR respectively) 6. Update RT-PID and D-ICES algorithms as required (UTSI and BAR respectively) E.4.3.2 Task 3 (UTSI and BAR) Modify, test, and validate the use of RT-PID and D-ICES algorithms with atmospheric turbulence as an input. Select an acceptable atmospheric turbulence measurement error based on findings from Task 2 and commercially available atmospheric turbulence sensors. Verify MOS during each phase of flight for varying atmospheric turbulence levels and icing conditions. UTSI and BAR will use the full 6-DOF nonlinear twin otter database.

Autopilot Task (Batch Testing) Phases of flight—Autopilot with Optimal Inputs 1. Flaps up straight and level flight at 110 KIAS and 2500 ft—30 sec duration 2. Flaps up straight and level flight at 110 KIAS and 2500 ft followed by a constant altitude flap transition (0° to 20°) to 80 KIAS – 1 min duration 3. Heading capture at 80 KIAS, 2500 ft, and flaps 20° from 090° to 360° – 30 sec duration 4. Descent from 3000 ft at 75 KIAS and flaps 20° followed by a flap transition (30°) to 70 KIAS with localizer and glide slope tracking – 2 min duration Ice Condition 1. Ice 0 2. Ice 2 in zero time Turbulence Levels 1. Baseline (no turbulence) 2. Light 3. Moderate 4. Severe Software Requirements (Task 1 plus) 1. D-six autopilots for Phases 2 to 4 (BAR) 2. Update RT-PID and D-ICES algorithms as required (UTSI and BAR respectively) E.4.4 Secondary Objective—Preliminary Feasibility Study without OBES E.4.4.1 Task 4 (UTSI and RR) Research Objectives 1. Evaluate the effectiveness of manual pilot inputs alone for making high confidence vehicle state estimates when performing typical maneuvering flight tasks under varying levels of atmospheric turbulence.

a. Determine the portion or portions of the maneuvers, which provide frequency content at the natural frequency of the short period, dutch-roll, and roll modes and thereby support the achievement of high confidence parameter estimates from RT-PID: (1) Without use of the OBES.

NASA/CR—2014-218320 112 (2) With OBES.

(3) Optionally, acquire pilot workload assessment for (1) and (2) Assumptions 1. Accomplishment of this objective assumes that the goal of system performance as stated in the primary objectives has been met.

Test Design 1. This test will be designed and executed to acquire data that will enable the researcher to compare the differences between the effect of manual only pilot control inputs and pilot inputs that are augmented by the operation of the OBES on RT-PID vehicle state estimates. This comparison will be made for four atmospheric turbulence conditions, which were previously defined under Task 1. The required data will come from maneuver time histories of pilot control inputs with and without OBES “buzzing” and from RT-PID derived confidence intervals of key stability and control coefficients in each of the aircraft’s three rotational axes. The a priori truth versus the current estimate for each stability and control coefficient in the ISP formulation will be used to determine the accuracy in the estimate and will be reflected in the magnitude error bars for each coefficient of interest. This will form the basis for pilot alerting when the state estimate needs to be improved. A power spectral density analysis of control activity with and without OBES, for each part of the maneuver where a poor vehicle state estimate exists will be used to determine the amount of energy in the inputs at the desired frequency. Coherence plots will then be developed will facilitate the comparison between the non-OBES and OBES control input condition.

Method of Test 1. Task 4 will be divided into two subtasks.

a. The first subtask will be an initial evaluation of normal manual pilot inputs for achieving vehicle state estimates while performing basic flight maneuvers. These maneuvers will include steady level flight, climbs and descents, changes in airspeed, turns to headings, wing flap extensions/retractions, and power transitions. These flight maneuvers are described in the following section, and will be performed with and without OBES operation. This subtask will be performed in smooth atmospheric conditions, and later, under various levels of atmospheric turbulence. The goal of this testing is to acquire data that establishes the relationship between pilot inputs alone, and pilot inputs with OBES when performing basic aircraft maneuvers, and their effect on achieving the required level of confidence in vehicle state estimation. The results of this initial testing will be used to verify system functionality, ensure readiness for more extensive data collection as described in the second subtask below, and to verify data collection, processing, and analysis methods In addition, this testing will identify and address any anomalies found with pilot cueing and messaging such nuisance alerts and toggling.

b. The second subtask will be an extensive series data collection tests flown by a test pilot to evaluate manual pilot control input gain on vehicle state estimates under various conditions of atmospheric turbulence. This will involve three repeats of each maneuver after the pilot has practiced and reached a consistent level of performance. These tests will be performed both with and without OBES operation. Two means of varying pilot control activity and gain will be employed during this subtask. The first is by increasing desired maneuver performance requirements, which involves specifying high and low error tolerances during each set of repeat maneuvers. The large tolerance will result in lower gain and control activity, and the small tolerances should result in higher gain and control activity. A second means of affecting pilot gain and control activity will be to simply alert the pilot via TBD methods (voice or possibly messaging) that more aggressive inputs are required in any or all of the pitch, roll, and yaw axes until the alerting requirements no longer exist. It is also expected that some maneuvers may not NASA/CR—2014-218320 113 provide sufficient vehicle excitation in all the aircraft axes, e.g., level climbs and descents may not provide lateral directional excitation, and it may be necessary to combine the two methods to obtain a satisfactory state estimate. In this case, the pilot may be alerted to make inputs in the affected axes if a full state estimate is desired for that test point. As a final evaluation of manual pilot inputs, an operationally representative flight profile consisting of an instrument approach procedure as shown in Figure E.1 will be performed under each of the turbulence conditions defined in this plan. In addition to evaluating the effectiveness of manual pilot inputs on vehicle state estimation, the data derived from this subset of testing will also assess the effect of manual inputs and OBES operation on pilot workload.

c. Pilot tasks will be performed under simulated IMC conditions.

Data Requirements 1. Test data will be sampled at a minimum of 10 Hz. The sampled parameters include time histories of the following: a. Pressure altitude Hp b. Airspeed VIAS c. Angle of attack (AOA) Į d. Angle of sideslip (AOS) ȕ e. Heading angle Ȍ f. Pitch angle ș g. Bank angle ij h. Control and wing flap position į e į a į r į f i. Body rates and accelerations p , q , r , A , A , A , X Y Z j. Localizer and GS errors Localizer and glide slope Theil inequality coefficient k. State coefficient estimates C , C , C , C , C , C estimate, error, and truth m m l l N n D į e ȕ į a D į r Pilot Performance Task Standards 1. To ensure consistency, flight standards are provided, which will cause pilot gain to be either high or low. Low gain will result from the larger error tolerance values, and high gain will result from the lower tolerance values. The appropriate standard will be specified on the test card for the maneuver event.

a. Altitude control for level flight: ± 150 and ± 20 ft b. Heading control: ± 5 ° and ± 1 ° c. Climb/descent rate: full power, idle power, power as required per test card d. Airspeed: ± 5 kt and ± 1 kt e. Roll rates: ± 10 °/sec and ± 5 °/sec f. Localizer course intercept angle: 45 ° and 30 ° g. Localizer error: ± 1 dot and zero lateral deviation + 10 kt X wind h. Glide slope error: ± 1 dot and zero vertical deviation + 10 kt X wind Maneuver Descriptions 1. The descriptions of each maneuver below are generalized. Detailed test cards will provide specific instructions as to how each maneuver is to be accomplished. Each evaluation will be conducted in four levels of turbulence. Icing configuration will be ICE 0 for the initial and final evaluation and ICE 02 for the instrument approach procedure as shown in Figure E.1. The 10 kt cross wind in addition to the turbulence will be applied for only the instrument approach procedure. Repeat each procedure 3 times using low gain standards, and then repeat 3 times using high gain standards. Initialization conditions will be specified for each maneuver on the test card.

2. Basic flight maneuvers a. Steady level flight at constant heading.

NASA/CR—2014-218320 114 (1) Perform at 125 KIAS and 1.3 versus (90 KIAS) with PFLF (2) Stabilize on condition with PFLF and call “mark” when stable (3) Hold heading and altitude within specified parameters and take data for 3 min b. Level turns to headings (1) Perform at 125 KIAS and 1.3 versus (90 KIAS) (2) Stabilize on condition with PFLF and call “mark” when stable (3) Remain steady level for 15 sec. Using power as required maintaining constant airspeed and altitude while performing the following: Turn right to a heading of 090 and maintain steady level flight for 15 sec. Turn left to 045 and maintain steady flight for 15 sec. Turn left to 030 and maintain steady level flight for 1 min.

c. Constant airspeed and vertical speed climbs/descents on a steady heading (1) Perform at 125 KIAS and 1.3 versus (90 KIAS) (2) Stabilize on condition with PFLF and call “mark” when stable.

(3) Using power as required, maintain constant airspeed and climb at specified climb rate. Level off at 6000 ft and remain steady for 15 sec. Using power as required, begin a constant airspeed descent at the specified rate and level off at 5000 ft. Maintain steady level flight for 15 sec.

d. Constant airspeed climbing and descending turns to headings (1) Perform at 125 KIAS and at 1.3 versus (90 KIAS) (2) Remain steady level for 15 sec. Select maximum power and turn right to a heading of 090.

Maintain constant airspeed in climb and level off at 6000 ft. Stabilize on airspeed and heading of 090 for 15 sec. Select idle power and turn left to 360. Maintain constant airspeed in the descent and level off at 5000 ft. Stabilize on airspeed and heading for 15 sec.

(3) Repeat (ii) and use power as required to establish a 300 fpm climb and descent rate e. Wing flap transitions (1) Initialize on heading 360, PFLF at 85 KIAS, wing flaps up. When stable, call “mark” and hold steady condition for 15 sec. Select wing flaps DOWN and use power as required to maintain airspeed and altitude while holding heading constant. After wing flaps reach DOWN, stabilize and hold the steady condition for 15 sec.

(2) Stabilize for 1 min with wing flaps full down at 85 KIAS. When stable, call “mark” and select wing flaps to UP and use power as required to maintain airspeed while keeping altitude and heading constant as the flaps retract to the fully up condition. Stabilize and hold the steady condition for 15 sec.

f. Power transitions: (1) Initialize on a 360 heading, at 85 KIAS, wing flaps up, PFLF. When stable, call “mark” and rapidly apply full power while maintaining heading and level flight until reaching the maximum speed attainable. When stable at maximum speed, hold condition for 15 sec.

(2) Initialize on a 360 heading, at maximum level flight speed, wing flaps up, PFLF. When stable, call “mark” and rapidly reduce power to idle while maintaining heading and altitude.

When approaching 90 KIAS, add power as required to maintain constant heading and airspeed. When stable at 90 KIAS, hold condition for 15 sec.

g. Operational evaluation—Instrument Approach Procedure (1) Initialize with wing flaps up, 3000 ft, heading 360, and 125 KIAS, ICE02 (2) Conduct the approach procedure per Figure E.1 without OBES, and provide pilot alerting when state estimates need to be refreshed with additional control activity (3) Evaluate workload using NASA TLX (4) Repeat the procedure with OBES functioning (5) Evaluate workload using NASA TLX NASA/CR—2014-218320 115 Turbulence Levels as Defined by MIL-F-8785C 1. Smooth air, no turbulence 2. Light turbulence 3. Moderate turbulence 4. Severe turbulence Test Cards 1. Detailed test cards will be developed for each of the two sub tasks from the maneuver descriptions above. These test cards will be provided in a standard UTSI test template format and retained as part of the data package for each test session. Any deviations from initially planned test procedures will be noted on the test cards.

Software Requirements 1. D-six executive logic software switch enabling / disabling OBES functionality (BAR) 2. Alerting the pilot that control excitations are required (BAR) a. Update excite button functionality b. Apply stop light color scheme (1) Tie color scheme to RT-PID confidence bounds (a) Parameters in ISP calculation define axis color a. Differentiate scheme depending on which axis or axes require excitation Figure E.1.—Test and evaluation profile. Total duration of test is approximately 15 min. Quotes are clearances read to the EP, italics are required EP actions per the test protocol.

NASA/CR—2014-218320 116