Document
NASA/TM—2014–216640
Complexity and Pilot Workload Metrics for
the Evaluation of Adaptive Flight Controls
on a Full Scale Piloted Aircraft
Curt Hanson, Jacob Schaefer, John J. Burken, and David Larson Dryden Flight Research Center, Edwards, California Marcus Johnson Ames Research Center, Moffett Field, California Click here: Press F1 key (Windows) or Help key (Mac) for help
February 2014
This page is required and contains approved text that cannot be changed.
NASA STI Program ... in Profile
Since its founding, NASA has been dedicated CONFERENCE PUBLICATION.
to the advancement of aeronautics and space Collected papers from scientific and science. The NASA scientific and technical technical conferences, symposia, seminars, information (STI) program plays a key part in or other meetings sponsored or helping NASA maintain this important role. co-sponsored by NASA.
The NASA STI program operates under the SPECIAL PUBLICATION. Scientific, auspices of the Agency Chief Information Officer. technical, or historical information from It collects, organizes, provides for archiving, and NASA programs, projects, and missions, disseminates NASA’s STI. The NASA STI often concerned with subjects having program provides access to the NASA substantial public interest.
Aeronautics and Space Database and its public interface, the NASA Technical Reports Server, TECHNICAL TRANSLATION.
thus providing one of the largest collections of English-language translations of foreign aeronautical and space science STI in the world. scientific and technical material pertinent to Results are published in both non-NASA channels NASA’s mission.
and by NASA in the NASA STI Report Series, which includes the following report types: Specialized services also include organizing and publishing research results, distributing TECHNICAL PUBLICATION. Reports of specialized research announcements and completed research or a major significant feeds, providing information desk and personal phase of research that present the results of search support, and enabling data exchange NASA Programs and include extensive data services.
or theoretical analysis. Includes compila- tions of significant scientific and technical For more information about the NASA STI data and information deemed to be of program, see the following: continuing reference value. NASA counter- part of peer-reviewed formal professional Access the NASA STI program home page papers but has less stringent limitations on at http://www.sti.nasa.gov manuscript length and extent of graphic presentations. E-mail your question to help@sti.nasa.gov TECHNICAL MEMORANDUM. Fax your question to the NASA STI Scientific and technical findings that are Information Desk at 443-757-5803 preliminary or of specialized interest, e.g., quick release reports, working Phone the NASA STI Information Desk at papers, and bibliographies that contain 443-757-5802 minimal annotation. Does not contain extensive analysis. Write to: STI Information Desk CONTRACTOR REPORT. Scientific and NASA Center for AeroSpace Information technical findings by NASA-sponsored 7115 Standard Drive contractors and grantees. Hanover, MD 21076-1320
NASA/TM—2014–216640
Complexity and Pilot Workload Metrics for
the Evaluation of Adaptive Flight Controls
on a Full Scale Piloted Aircraft
Curt Hanson, Jacob Schaefer, John J. Burken, and David Larson Dryden Flight Research Center, Edwards, California Marcus Johnson Ames Research Center, Moffett Field, California Insert conference information, if applicable; otherwise delete Click here: Press F1 key (Windows) or Help key (Mac) for help National Aeronautics and Space Administration Dryden Flight Research Center Edwards, CA 93523-0273
February 2014
Enter acknowledgments here, if applicable.
Click here: Press F1 key (Windows) or Help key (Mac) for help Available from: NASA Center for AeroSpace Information 7115 Standard Drive Hanover, MD 21076-1320 443-757-5802 Click here: Press F1 key (Windows) or Help key (Mac) for help
Abstract
Flight research has shown the effectiveness of adaptive flight controls for improving aircraft safety and performance in the presence of uncertainties. The National Aeronautics and Space Administration’s (NASA)’s Integrated Resilient Aircraft Control (IRAC) project designed and conducted a series of flight experiments to study the impact of variations in adaptive controller design complexity on performance and handling qualities. A novel complexity metric was devised to compare the degrees of simplicity achieved in three variations of a model reference adaptive controller (MRAC) for NASA’s F-18 (McDonnell Douglas, now The Boeing Company, Chicago, Illinois) Full-Scale Advanced Systems Testbed (Gen-2A) aircraft. The complexity measures of these controllers are also compared to that of an earlier MRAC design for NASA’s Intelligent Flight Control System (IFCS) project and flown on a highly modified F-15 aircraft (McDonnell Douglas, now The Boeing Company, Chicago, Illinois). Pilot comments during the IRAC research flights pointed to the importance of workload on handling qualities ratings for failure and damage scenarios. Modifications to existing pilot aggressiveness and duty cycle metrics are presented and applied to the IRAC controllers. Finally, while adaptive controllers may alleviate the effects of failures or damage on an aircraft’s handling qualities, they also have the potential to introduce annoying changes to the flight dynamics or to the operation of aircraft systems. A nuisance rating scale is presented for the categorization of nuisance side-effects of adaptive controllers.
Nomenclature
Acronyms ACAT Autonomous Collision Avoidance Technology ESA European Space Agency FAST full-scale advanced systems testbed Gen-1 first-generation IFCS adaptive flight control system Gen-2 second-generation IFCS adaptive flight control system Gen-2A update to the second-generation IFCS adaptive flight control system IFCS intelligent flight control system IRAC integrated resilient aircraft control MRAC model reference adaptive control NASA National Aeronautics and Space Administration NDI nonlinear dynamic inversion onMRAC MRAC with normalization and optimal control modification onMRAC+ onMRAC with an additional adaptive parameter PIO pilot-in-the-loop oscillation RFI request for information sMRAC simple MRAC SOHO Solar Heliospheric Observatory Mathematical Symbols ᠨ matrix of input coefficients ゃ ᠩ error feedback compensator 䘲 ᡘ estimate of applied aerodynamic and inertial moments 。 ᠵ inertia matrix ᠶ measure of complexity ᠶ measure of pilot aggressiveness 。 ᡰ reference model input ᡄ reference model s Laplace operator ᡲ time ᡵ weighting terms ᡶ state vector ᡶ䙢 state derivative vector ᡶ 㕖 state error vector ᡷ output vector ‒ surface command vector ‒ pilot roll stick input 〨ぃ ‒ᆑ steady state component of pilot roll stick input 〨ぃ ‒ pilot pitch stick input 〲ぃ ‒ᆑ steady-state component of pilot pitch stick input 〲ぃ ᡦ quantity measure for a category of complexity elements … 㕈 scalar, time-varying disturbance uncertainty estimate い time index elapsed time between data points vector of body-axis rotational rates Subscripts 0 initial ᡓ adaptive augmentation ᡕ compensator augmentation ᡕᡥᡖ command ᡗᡰᡰ NDI reference model tracking error ᡘ final ᠵ/ᡁ input / output ᡣ tunable parameter ᠸᡆ lookup table ᡥ MRAC reference model ᠹᠰ mode discrete ᡀᠸ nonlinear ᡰᡗᡘ reference model ᔖ ᡶ integration ᡶ䙢 differentiation ⡹⡩ ᡶ inversion Superscripts ᡥᡓᡶ maximum available control input ᡥᡡᡦ minimum available control input
Introduction
This report represents a proposed set of metrics to quantify the relative complexity of multiple flight control system designs and for evaluating changes in pilot workload due to the behavior of an adaptive controller. Over the past 50 years, flight research has demonstrated that adaptive flight controls can be an effective technology for improving aircraft safety and performance in the presence of uncertain and changing environments (refs. 1–9). However, the nonlinear, time-varying nature of adaptive algorithms continues to challenge traditional methods for the verification and validation of safety-critical flight control systems.
In April of 2009, The National Aeronautics and Space Administration’s (NASA)’s Integrated Resilient Aircraft Control (IRAC) project disseminated a request for information (RFI) to the adaptive controls community seeking ideas for potential flight experiments (ref. 10). A workshop was held in Chicago in August of 2009 with representatives from industry, academia, and other government agencies to discuss the wide variety of RFI responses received by NASA. Three focus areas were identified through this process.
1. Simple, yet effective, adaptive control algorithms should be investigated to help address the issue of verification and validation of adaptive flight controls to a safety-critical level.
2. The appropriate level of pilot awareness and interaction with adaptive control systems should be studied.
3. Techniques should be matured for incorporating feedback information, both static and dynamic, from the structure of an aircraft into the flight control system to ensure structural integrity during adaptation.
The first two of these focus areas prompted the development of a low complexity, textbook-like direct model reference adaptive control (MRAC) scheme (ref. 11) for the NASA Full-Scale Advanced Systems Testbed (FAST) (refs. 8–9). FAST is a highly modified F-18 (McDonnell Douglas, now The Boeing Company, Chicago, Illinois) aircraft that contains a research flight control system capable of housing advanced flight controls experiments.
Full-scale piloted in-flight experimentation has been shown to be an effective method for uncovering implementation issues of adaptive systems (refs. 1, 4–9). A philosophy of “simpler is better” motivated the development of the low complexity MRAC formulation under the assumption that simplification leads to lower implementation and verification costs, and ultimately to a greater likelihood of acceptance by aircraft manufacturers, operators, and certification authorities (refs. 12–13). Reduced complexity can also lead to a safer design by reducing the potential for software implementation errors and unexpected interactions, both within the aircraft systems and between the controller and the human pilot (refs. 14–17).
Pilot interaction with adaptive systems can be either beneficial or have unwanted consequences. Beneficial interactions include decisions made by the pilot based upon information regarding the performance and status of the system. For example, the pilot may configure the controller specifically to adapt to a failed control surface, poor pitch damping or for severe cross-axis coupling. While altering the closed-loop dynamics of the aircraft to accommodate failures or damage, adaptive control also has the potential to adversely affect the handling qualities and pilot-in-the-loop oscillation (PIO) characteristics of the system. The potential for adverse interactions is compounded by simultaneous changes in piloting technique as the pilot also adapts to the altered aircraft’s flight characteristics. Additional adverse interactions between the pilot and adaptive controller include increased workload for the pilot and nuisance side effects caused by the adaptation.
The purpose of this report is to suggest potential metrics that will aid the designer in selecting and evaluating an appropriate adaptive control design for their application. Other important metrics not addressed in this report include measures of the system’s robustness and performance, which have been addressed previously by other authors (ref. 18).
Complexity Metric
Any flight control design theory must buy its way onto an aircraft by meeting performance and robustness specifications without being overly burdensome to design, implement, and qualify for flight. Control theories that are complicated to implement and difficult to prove the safety of will be bypassed in favor of those that may not perform as well, but can meet the project’s cost and schedule requirements. Such a trade study requires the ability to evaluate a priori the difficulty in bringing a particular control design to flight. By documenting the implementation complexity for future researchers, these metrics begin to construct a knowledge-base of experience for the controls community.
Traditionally, metrics used for evaluating control methods are often nebulous in nature and a result of “tribal knowledge” within the controls community. The necessity of full-state feedback, the number of generated controller states, and the difficulty of gain scheduling are some examples of potential drawbacks that are often cited when considering various modern control methods. Implementation of adaptive control is relatively new to the flight controls community, and engineers may not have the requisite experience to draw from when making a determination of an adaptive algorithm’s suitability for a given application.
Formal complexity metrics are generally designed for application to the software implementation of a system (ref. 19). Examples include the Halstead complexity metrics, which include concepts such as volume, difficulty, and effort derived from the number and variety of operators and operands in a software program (ref. 20). However, no attempt is made within these metrics to differentiate between the relative complexity of different operations. The cyclomatic complexity measures of McCabe are based upon the number of linearly independent paths through a software program (ref. 21). Neither of these approaches accounts for differences in complexity between different elements of control system design.
Complexity metrics can be used to characterize the impacts of theory selection on the costs of designing, implementing, and flight qualifying a control system. The metrics are designed to evaluate both fundamental elements of the control theory and any additional elements that, while required for implementation, are often omitted from theoretical descriptions. While motivated by the need to evaluate adaptive control theories, these metrics are applicable to any control theory.
Quantitative Complexity Metric Description Quantitative metrics provide an objective means for comparison between competing control methodologies prior to selection and can measure the impact of modifications to the complexity of the algorithm during implementation. A proposed quantitative metric for measuring the complexity of a control system is given in equation (1). The metric is similar in approach to Halstead’s complexity metrics (ref. 20) in that it derives a measure of complexity by counting the number of different control system elements used in the design. The proposed method differs from Halstead’s primarily in that it accounts for variations in complexity between different types of elements.
ᠶ = ᡵ ᡦ + ᡵ 㐵ᡦ + ᡦ + ᡦ ㄧㄗ + ᡦ 㐹 + ᡵ ᡦ + ᡵ ᡦ + ᡵ ᡦ (1) ⡩ 【/〖 ⡰ け䙢 〕〓 ⡱ 〸 ⡲ 〓〡 ⡳ 〔々 ᔖ け け The scalar J is a measure of complexity for the controller. Each component of the control design is categorized and tallied, and category totals represented by the series of variables ⡸ ⡸ ᡦ ᒈ ␠ . Weighting terms ᡵ ᜐ ᡵ ᒈ ጷ are used to adjust for inherent differences in the impact ⡩ ⡳ to complexity of the various component categories.
Relative magnitudes for the weighting terms are not proposed in this report. Their determination will require a rigorous and detailed tracking of resource utilization during multiple control law development programs. The complexity metric function in equation (1) is a linear approximation that does not contain element interdependencies and other non-linear relationships such as those of the Halstead metrics (ref. 20). For example, an increase in the number of tunable parameters may cause the complexity of the controller to rise at a faster-than-linear rate due to their inter-dependencies. Identification of any higher-order terms would be subject to the same empirical investigation as the weighting terms.
Input and Output Plane Dimensionality 㐵↖ 㐹 Ⅵ/ⅱ The dimensions of the input and output planes impact control system development in several ways. Linear robustness analysis is typically performed at each feedback loop and each controller output. Noise, latency and sampling rate characteristics for each input and actuator dynamics for each output, must be identified and modeled. During testing, the failure modes of each input and output must be understood and the effects evaluated. Some examples of the importance of understanding possible failure modes of feedback signals include the crashes of the NASA X-31 (Enhanced Fighter Maneuverability (EFM) demonstrator) (ref. 22) and Air France A330-200 Flight 447 (ref. 14), both cases in which Pitot tube icing was a contributor.
Integrators, Derivatives, Inverses and Nonlinear Elements 㐵↖ , ↖ , ↖ ㄧ❸ , ↖ 㐹 ∆䙢 ⅰⅨ ᔖ ∆ ∆ Integrators introduce complications to the control system primarily through the need to ensure their proper initialization on startup and during mode changes. Examples of incidents caused by improper handling of integrators include the premature retirement of NASA’s Demonstration for Autonomous Rendezvous Technology (DART) spacecraft (ref. 23), and the mission interruption of the joint NASA and European Space Agency (ESA) Solar Heliospheric Observatory (SOHO) (ref. 24). Derivatives of signals introduce noise into the system, which can ultimately lead to numerical instability and may require filtering. The number of scalar divides and matrix inversions add to the complexity of the system by requiring special handling to avoid numerical errors. Nonlinear elements such as integrator limits, saturation and rate limits, nonlinear mathematical operations, and dead-bands typically complicate linear robustness analysis and create additional test scenarios. These four elements share a common weighting coefficient because they are estimated to have roughly the same impact on the cost of control system development.
(↖ ) Tunable Parameters ↓ Tunable parameters are elements of the controller such as gains, thresholds, and filter coefficients that can be adjusted to affect controller performance. These parameters tend to complicate the design and verification of the controller considerably due to the potential for unforeseen interactions between them. The fewer tunable parameters a system has, the easier it is for the designer to understand the impacts of adjusting the design to achieve the desired result. Tunable parameters also have an impact on the “fly-fix-fly” process of flight testing as further adjustments must be made to correct for a lack of model fidelity. In 1998, American Airlines flight 1340 crashed just short of the runway due to excessive pitch oscillations caused by a gain in the autopilot of the Boeing 727 that was not properly designed for the approach speed used by the pilots (refs. 17 and 25).
Lookup Table Dimensionality (↖ ) Ⅸⅶ A lookup table encodes a function for use in the control system. Examples of look-up tables include gain schedules and coefficient tables. This metric captures their total dimensionality by summing together the number of independent input variables. Ensuring proper interpolation within the boundaries of the table and extrapolation or limiting outside the table complicates the testing effort. These types of tables often also include additional complicating features such as pre-lookup calculations and index memory for efficient operation.
Mode Discretes (↖ ) ⅩⅠ Switchable modes within a control system, such as flap and gear settings, and autopilot modes require considerable effort to test, which is especially true when it comes to evaluating the interactions of multiple modes and the effects of switching from one mode to another. One example of the unexpected complexity effects of mode transitions is the crash of TWA Flight 843, in which the combination of a state transition from ground to flight and a faulty angle-of-attack sensor contributed to a false stall warning immediately after takeoff (ref. 16).
Complexity Comparison between Multiple Adaptive Controllers The complexity metrics are applied to three variations of the IRAC experiment model reference adaptive controllers (refs. 8–9) designed for the FAST aircraft, and to the Gen-2A controller from the Intelligent Flight Control System (IFCS) experiment (refs. 6–7). Both experiments were conducted under the NASA Aviation Safety program. The IFCS flight research program was conducted on a highly modified F-15 aircraft (McDonnell Douglas, now The Boeing Company, Chicago, Illinois) and concluded approximately 2 years prior to the beginning of the IRAC flights on the FAST F-18 aircraft. Despite implementation on different avionics systems and different aircraft, the two adaptive controls experiments shared a similar set of top-level design requirements. Both experiments were designed to evaluate direct adaptive model-reference algorithms integrated with nonlinear dynamic inversion (NDI) baseline controllers as pilot-initiated emergency systems. The simulated failures used to evaluate the controllers were also similar, including a destabilized pitch axis scenario and a failed stabilator.
Controller Descriptions Figure 1 shows the top-level block diagram of the integrated MRAC and NDI controllers from the IRAC experiment. Three variations of the MRAC were implemented and tested to explore the relationships between adaptive controller complexity and performance, robustness, and handling qualities. The three controllers differed from each other by adding successive levels of complexity rather than implementing three entirely independent designs. The positive and negative effects of the additional complexity could then be studied. The simplest design, referred to as sMRAC, consisted of a single adaptive parameter in each of the pitch and roll axes, computed using a basic gradient-based update law. A second design, onMRAC, increased the complexity of the update law, while the third, onMRAC+, included one additional adaptive parameter in each axis. A detailed description of the IRAC controllers is given in reference 26.
Figure 1. IRAC NDI and MRAC controller architectures.
The objective of the IFCS research effort was to develop, through flight research, adaptive algorithms to enhance control during primary control surface failures and other aerodynamic changes resulting from damage. The first flight phase, referred to as Generation 1 or Gen-1 for short, focused on an indirect adaptive aerodynamic parameter identification method (ref. 27).
The second generation, Gen-2, used neural networks to provide direct adaptive command augmentation signals. In principle, this approach does not require information on the nature of the failure or the extent of the damage. During flight evaluation with simulated failed control surfaces, the Gen-2 controller exhibited poor handling qualities. The goal of Gen-2A, shown in figure 2, was to modify the Gen-2 control architecture to improve command tracking and handling qualities with control surface failures (ref. 7).
Figure 2. IFCS Gen-2A controller architecture.
Complexity Metric Application The complexity metric measures for the IRAC and IFCS controllers are listed in table 1.
Values are given separately for the pitch and roll axes of each controller. Elements of the system that existed solely to facilitate flight testing or accommodate research hardware were not included in the determination of complexity. None of the controllers that were evaluated contained derivatives or lookup tables, so those categories are eliminated from table 1 in the interest of brevity. Integrators and nonlinear elements which contain tunable parameters were counted separately from their tunable parameters, and calculations that were used more than once were only counted once wherever possible.
Table 1. Complexity measures.
sMRAC onMRAC onMRAC+ IFCS Gen-2A pitch 3 3 3 7 I/O dimensions ᡦ 【/〖 roll 3 3 3 7 pitch 5 5 6 1 integrators ᡦ ᔖ け roll 1 1 2 1 pitch 0 1 2 4 inverses ㄧㄗ ᡦ け roll 0 1 2 4 pitch 10 18 20 61 non-linear terms ᡦ 〕〓 roll 11 16 19 61 pitch 14 16 22 47 tunable gains ᡦ 〸 roll 12 14 20 47 mode discretes both 3 3 3 2 ᡦ 〔々 One of the goals of the IRAC adaptive controls experiment was to design three controllers with varying levels of complexity. From table 1 it is apparent that the difference in complexity between the IRAC controllers is primarily due to the number of non-linear elements and tunable parameters. These differences are mostly a result of increases in the complexity of the adaptive update laws.
By four of the six measures in table 1, all three of the IRAC controllers are less complex than the IFCS Gen-2A control law. The key difference in complexity between the IRAC controllers and the IFCS controller is the number of inputs, identified by the measure, ᡦ . The larger 【/〖 number of inputs used by the IFCS controller drives the substantial increases in the number of nonlinear elements, ᡦ , and in the number of tunable parameters, ᡦ .
〕〓 〸 The seemingly incongruous decrease in the number of integrators required by the IFCS Gen-2A pitch axis when compared to the IRAC controllers is due to the inclusion of the integral of pitch rate as a controlled variable in the IRAC architecture. As described in reference 26, the influence of the integral of pitch rate is essentially zeroed out within the controller, but related calculations were included in the IRAC controller’s complexity measures for completeness.
The complexity metric in equation (1) was applied twice to each of the four adaptive controllers in table 1, using two different sets of arbitrary weightings. The resulting complexity measures are plotted in figure 3 along with handling quality ratings (ref. 28) gathered during research flights with simulated failures. All five controllers were evaluated with a failed stabilator.
The NDI and IRAC controllers were also tested with unstable pitch dynamics, unstable roll dynamics, and significant roll-to-pitch coupling; these failures were not included in the IFCS project. Handling quality ratings for the NDI and IRAC controllers were taken from references 8 and 9, while the IFCS controller ratings are previously unpublished.
Figure 3. Complexity measure versus handling qualities rating for simulated failures.
The complexity of the NDI controller is defined as a baseline of zero since it does not contain an adaptive controller. The complexity metric was first applied using a weighting set of unity. The metric was then applied a second time, in which the tunable parameter weighting was set to ten to reflect the difficulty of adaptive control tuning, while the input-output weighting and mode discrete weightings were both set to five, under the assumption that they are greater contributors to complexity than the mathematical elements, but not as significant as tunable parameters.
By either measure of complexity, the three IRAC controllers are significantly less complex than the IFCS Gen-2A controller, while providing approximately the same level of handling qualities performance in the presence of failures. Although the onMRAC+ controller is the most complex of the three IRAC designs, it exhibits the lowest and most compact set of handling qualities ratings, indicating that the additional complexity may provide improved performance.
Pilot Workload Metrics
Pilot comments during the IRAC research flights indicated that workload played a significant role in the pilot’s handling qualities ratings. Traditionally, Cooper-Harper handling qualities ratings (ref. 28) and the PIO rating scales are used to assess pilot workload. The Cooper-Harper scale, in particular, requires the pilot to assess the perceived level of compensation required to accomplish the desired task. A more qualitative measure of pilot workload can be calculated by cross-plotting measures of the pilot’s aggressiveness and duty cycle against each other, as proposed by Shepherd (ref. 29).
Aggressiveness and Duty Cycle Metrics Pilot aggressiveness is a measure of dynamic control deflection, defined by equation (2) as the time-averaged summation of the unbiased pilot roll and pitch stick inputs during the maneuver. The roll and pitch control bias, ‒ᆑ (ᡲ) and ‒ᆑ (ᡲ) respectively, is found by forward 〨ぃ 〲ぃ and backward low-pass filtering of the original signal. Filtering the data in both directions eliminates any filter-induced phase shift. Each unbiased control measure is normalized by the total allowable deflection range of the appropriate control inceptor.
ぇ ㊁゙ ( ) ㎘ ‒ᆑ ( )㘧 ( ) ㎘ ‒ᆑ ( )㘧 100% 㘧‒ 㘧‒ 〨ぃ 〨ぃ 〲ぃ 〲ぃ (2) ᠶ = 㔳 䙸 + 䙹 c 。 〨け 〶ぁ 〨け 〶ぁ ᡲ ㎘ ᡲ ‒ ㎘ ‒ ‒ ㎘ ‒ 〳 ⡨ 〨ぃ 〨ぃ 〲ぃ 〲ぃ ⢀ぇ ㄖ The aggressiveness metric proposed in equation (2) differs from that of Shepherd (ref. 29) (ᡲ) (ᡲ) by introducing a subtraction of the bias terms ‒ᆑ and ‒ᆑ . This modification avoids the 〨ぃ 〲ぃ errant categorization of steady, non-zero pilot deflections as highly aggressive commands.
Equation (2) also combines pitch and roll control deflections into a single metric to support the analysis of multi-axis maneuvers.
Pilot duty cycle is a measure of how frequently the pilot is making a significant adjustment to his or her input. Duty cycle is calculated by counting the number of peaks per second in both the pitch and roll control stick inputs. A peak is arbitrarily defined as a change in the signal magnitude greater than 0.5 percent of the inceptor’s total displacement range and in the direction opposite of the signal’s change identified in the previous time step. Two test pilots, referred to as Pilot A and Pilot B, participated in the IRAC adaptive controls experiment. Figure 4 shows example results of the peak detection algorithm for the pitch stick inputs of Pilot A during a 2g air-to-air tracking maneuver with a simulated reduced pitch damping failure and the sMRAC controller.
Figure 4. Duty cycle peaks for Pilot A pitch inputs during 2g air-to-air tracking with a reduced pitch damping failure and sMRA.
This calculation of duty cycle diverges from the method proposed by Shepherd (ref. 29) in that it is a determination of the frequency with which the pilot reverses control direction, whereas Shepherd measures the percentage of duration of the maneuver during which the pilot’s compensation is non-steady. Application of the non-steady compensation technique to the IRAC flight data failed to provide sufficient distinction between the different controllers as the pilots, regardless of the control system, tended to almost continuously vary their control inputs.
Significant differences were seen, however, in the frequency of control motion reversals between controllers.
Figure 5 shows a plot of pilot aggressiveness versus duty cycle for two pilots during 2g air-to-air tracking maneuvers with a simulated reduced pitch damping failure. The metrics were computed for the IRAC non-adaptive NDI controller with no failures or damage, for the NDI controller with a simulated failure and for all three IRAC adaptive controllers with the same simulated failure. When no failure is present (Unfailed NDI), both Pilot A and Pilot B exhibit very similar workload metrics. Following the failure (NDI), with no adaptation, both pilots experience an increase in the aggressiveness and duty cycle of their control inputs, although the workload increase for Pilot B is significantly higher than that of Pilot A. The simplest adaptive controller (sMRAC) restores the duty cycle of Pilot A to approximately the same value as that of the unfailed scenario. The two controllers with higher level of complexity (onMRAC, onMRAC+) have almost no effect on duty cycle but reduce the aggressiveness of Pilot A, with the most complex controller (onMRAC+) displaying nearly the same workload metrics as the unfailed case.
Figure 5. Aggressiveness versus duty cycle metrics for Pilot A and Pilot B during 2g air-to-air tracking with a reduced pitch damping failure.
The workload of Pilot B is also reduced with adaptive control in the presence of the failure, but in this case both aggressiveness and duty cycle are affected. There is no significant difference between the three adaptive controllers on workload for Pilot B. It is interesting to note that even with adaptive control, the workload of Pilot B is much higher than in the unfailed scenario. One possible explanation is that despite the improvement in reference model tracking attributable to adaptive augmentation, the gain of Pilot B remains elevated from his initial response to the failure. Consequently, even after the adaptive controller restores good pitch damping, Pilot B continues to operate at a higher workload than is necessary. This situation is a case where information provided to the pilot on the status of the adaptive controller might prove beneficial.
Nuisance Rating Scale A nuisance rating scale was developed as part of the Autonomous Collision Avoidance Technology (ACAT) project. The nuisance scale, shown in figure 6, is designed to categorize the nuisance effects of an autonomous system using a format similar to that of the Cooper-Harper rating scale for handling qualities (ref. 28) and the PIO rating scale (ref. 30). An adaptive flight control system, while restoring good aircraft handling qualities in the event of a failure or damage, may exhibit undesirable side effects ranging from annoying to unsafe.
Application of the nuisance rating scale provides a consistent, systematic framework for the classification of undesirable side effects caused by adaptive flight control systems.
Figure 6. Nuisance rating scale for autonomous systems on piloted aircraft.
During the MRAC research flights, the nuisance scale was initially applied during tests of the adaptive controllers with no simulated failures present. The absence of simulated failures allowed for a clearer identification of any nuisance characteristics inherent in the controllers.
Pilot A was the only pilot to apply the nuisance scale to all four controllers with no failures during the 2g air-to-air tracking task. Pilot A gave the NDI controller a nuisance rating of 1, which served as a baseline for evaluation of the adaptive controllers. The onMRAC+ controller was rated as a 4 on the nuisance scale by Pilot A due to objectionable characteristics in the pitch axis. The following are excerpts from Pilot A’s comments on 2g air-to-air tracking with the onMRAC+ controller and no simulated failures: A lot more bobble in gross acquisition than I would have expected . . . had to back out of the loop a little bit when doing the gross acquisition, let everything kind of die out; it dies out relatively quickly, but little bit bigger oscillations than I would have thought . . . might have been a little sensitive in pitch for the gross acquisition and that may have been what caused some of the stuff.
As described in reference 8, an adverse interaction occurred during this maneuver between the pilot and the onMRAC+ controller involving the … 㕈 adaptive parameter. A minor PIO い resulted, for which a simple corrective adaptive parameter update law modification term has subsequently been proposed (ref. 8). The time-varying pitch sensitivity of this controller was also revealed in the Cooper-Harper handling qualities ratings and PIO rating scale evaluations by Pilot A, making the nuisance scale a consistent metric, but in this particular case potentially redundant. Although no qualities were discovered for any of the MRAC controllers through the use of the nuisance scale that weren’t also revealed by the other rating systems, this may be more a function of the nature of the controllers’ behavior than a weakness of the nuisance scale itself.
Summary
Realization of the objectives for the IRAC experiment necessitated the development of non-traditional evaluation metrics. A complexity metric was created for measuring progress toward a simple yet effective adaptive control method. Control system complexity metrics can be useful when selecting a design method, or when making modifications to an existing design, in order to better account for the lifecycle impacts of complicating controller structures or features. However, significant work remains in determining, for the complexity metric presented here, the appropriate term weightings and potential applicability of higher-order term combinations.
Application of the complexity metric to the three IRAC adaptive controllers confirmed the expected trend of increased complexity with incremental design enhancements. A significant reduction in complexity was demonstrated between all three IRAC designs relative to an earlier similar adaptive controller, the IFCS Gen-2A, validating the IRAC project objective of achieving simplified adaptive controls.
Pilot workload metrics were developed and refined to better understand the interactions between the pilot and the adaptive controller. Workload metrics can help the control system designer evaluate the impact of control system performance changes, and better understand pilot handling qualities evaluations. Pilot workload metrics for aggressiveness and duty cycle in the presence of a simulated aircraft failure were computed and cross-plotted for flight data from three IRAC adaptive controllers and a non-adaptive baseline controller, as well as for the baseline controller with no simulated failure. The metrics indicated that the adaptive controllers reduced the pilot workload under a failure scenario, but the degree of reduction was different between the two pilots.
A novel nuisance scale metric was applied to adaptive controllers with no simulated aircraft failures or damage. The metric successfully identified a low-level pilot-in-the-loop oscillation as a nuisance. The results warrant further application and evaluation of the nuisance scale as a pilot-interaction metric for adaptive controllers.
References
1. Whitaker, H. P., “Massachusetts Institute of Technology Presentation,” Proceedings of the Self Adaptive Flight Control Systems Symposium , Wright Air Development Center, January 13-14, 1959, pp. 58-78.
2. Schuck, H. O., “Honeywell’s History and Philosophy in the Adaptive Control Field,” Proceedings of the Self Adaptive Flight Control Systems Symposium , Wright Air Development Center, January 13-14, 1959, pp. 123-145.
3. National Aeronautics and Space Administration Flight Research Center Staff, “Experience with the X-15 Adaptive Flight Control System,” NASA TN D-6208, 1971.
4. Thompson, Milton O., and James R. Welsh, “Flight Test Experience with Adaptive Control Systems,” Advanced Control System Concepts, AGARD CP No. 58, AGARD Joint Symposium of the Guidance and Control Panel, and the Flight Mechanics Panel of AGARD , Oslo Norway, September 3-5, 1968, pp. 141-147, 1970.
5. Monaco, J., D. Ward, R. Barron, and R. Bird, “Implementation and Flight Test Assessment of an Adaptive, reconfigurable Flight Control System,” AIAA 97-3738, 1997.
6. Bosworth, John T., and Peggy S. Williams-Hayes, “Stabilator Failure Adaptation from Flight Tests of NF-15B Intelligent Flight Control System,” Journal of Aerospace Computing, Information, and Communication , Vol. 6, March, 2009, pp. 187-206.
7. Burken, John J., Curtis E. Hanson, James A. Lee, and John T. Kaneshige, “Flight Test Comparison of Different Adaptive Augmentations to Fault Tolerant Control Laws for a Modified F-15 Aircraft,” AIAA 2009-2056, 2009.
8. Hanson, Curt, Jacob Schaefer, John J. Burken, Marcus Johnson, and Nhan Nguyen, “Handling Qualities Evaluations of Low Complexity Model Reference Adaptive Controllers for Reduced Pitch and Roll Damping Scenarios,” AIAA 2011-6607, 2011.
9. Schaefer, Jacob, Curt Hanson, Marcus A. Johnson, and Nhan Nguyen, “Handling Qualities of Model Reference Adaptive Controllers with Varying Complexity for Pitch-Roll Coupled Failures,” AIAA 2011-6453, 2011.
10. Krishnakumar, Kalmanje, “A Request for Information in Flight Validation of Adaptive Control to Prevent Loss-of-Control Events,” Solicitation Number: NNH09ZEA002L, National Aeronautics and Space Administration Headquarters Office of Procurement, April 2009.
11. Ioannou, Petros A., and Jing Sun, Robust Adaptive Control , Prentice Hall Inc., Upper Saddle River, New Jersey, 1996.
12. Younossi, Obaid, David E. Stem, Mark A. Lorell, and Frances M. Lussier, “Lessons Learned from the F/A-22 and F/A-18E/F Development Programs,” RAND Corporation, 2005, pp. 39-44.
13. Decadal Survey of Civil Aeronautics: Foundation for the Future , Steering Committee for the Decadal Survey of Civil Aeronautics, National Research Council, 2006.
14. “Final Report on the Accident on 1st June 2009 to the Airbus A330-203 registered F-GZCP operated by Air France flight AF 447 Rio de Janeiro–Paris,” BEA, July 2012, pp. 199-201.
15. Harris, Jeffrey Jay, and G. Thomas Black, “F-22 Control Law Development and Flying Qualities,” AIAA 96-3379, 1996.
16. National Transportation Safety Board, “Aircraft Accident Report: Aborted Takeoff Shortly After Liftoff, Trans World Airlines Flight 843 Lockheed L-1011, N11002, John F. Kennedy International Airport, Jamaica, New York, July 30, 1992,” NTSB/AAR-93/04, March 31, 1993.
17. National Transportation Safety Board, “Aircraft Accident Brief DCA98MA023, American Airlines 1340,” NTSB/AAB-01/01, May 14, 2001.
18. Stepanyan, Vahram, Kalmanje Krishnakumar, Nhan Nguyen, and Luarens Van Eykeren, “Stability and Performance Metrics for Adaptive Flight Control,” AIAA 2009-5965, 2009.
19. Weyuker, Elaine J., “Evaluating Software Complexity Measures,” IEEE Transactions on Software Engineering, Vol. 14, No. 9, September 1988.
20. Halstead, Maurice H., Elements of Software Science . Elsevier North-Holland, Inc., New York, 1977.
21. McCabe, Thomas, J., “A Complexity Measure,” IEEE Transactions on Software Engineering, Vol. SE-2, No. 4, December 1976.
22. Merlin, Peter W., Gregg A. Bendrick, and Dwight A. Holland, “Breaking the Mishap Chain,” NASA SP-2011-594, 2011, pp. 3-21.
23. NESC Review of Demonstration of Autonomous Rendezvous Technology (DART) Mission Mishap Investigation Board Review (MIB) , RP-06-119, December 2006.
24. SOHO Mission Interruption Joint NASA/ESA Investigation Board Final Report , August 31, 1998. http://umbra.nascom.nasa.gov/soho/SOHO_final_report.html, accessed January 9, 2013.
25. National Transportation Safety Board, “Autopilots on Boeing 727 Aircraft and Other Aircraft,” Joint Flight Standards Information Bulletin for Air Transportation, General Aviation, and Airworthiness, NTSB A-00-43/45, June 13, 2001.
26. Hanson, Curt, Jacob Schaefer, Marcus Johnson, and Nhan Nguyen, “Design of Low Complexity Model Reference Adaptive Controllers,” NASA/TM-2012-215972, May 2012.
27. Hageman, Jacob J., Mark S. Smith, and Susan Stachowiak, “Integration of Online Parameter Identification and Neural Network for In-Flight Adaptive Control,” NASA/TM-2003-212028, October 2003.
28. Cooper, George E., and Robert P. Harper Jr., “The Use of Pilot Rating in the Evaluation of Aircraft Handling Qualities,” NASA TN D-5153, April 1969.
29. Shepherd, Michael J., Adam MacDonald, William R. Gray III, and Richard G. Cobb, “Limited Simulator Aircraft Handling Qualities Evaluation of an Adaptive Controller,” IEEE Aerospace Conference Paper 1292, March 7-14, 2009.
30. U.S. Department of Defense, Flying Qualities of Piloted Aircraft , MIL-HDBK-1797, December 19, 1997.