Document
Certification strategies using run-time safety assurance for
part 23 autopilot systems
Loyd R. Hook Matthew Clark David Sizoo University of Tulsa – ECE Dept. Air Force Research Laboratory FAA Aircraft Certification Service, 800 S. Tucker Dr., Rayzor 1130 2210 Eighth St. Small Airplane Directorate, Tulsa, OK 74104 Wright-Patterson AFB, OH 45433 901 Locust St., Kansas City, MO 64106 918-631-3272 937-713-7044 816-329-4158 Loyd-hook@utulsa.edu Matthew.clark.20@us.af.mil David.sizoo@faa.gov Mark A. Skoog James Brady NASA-Armstrong Flight Research Center FAA Aircraft Certification Service, P.O. Box 273 / M.S. 4830E Small Airplane Directorate Edwards, CA 93523 901 Locust St., Kansas City, MO 64106 661-276-5774 816-329-4132 Mark.a.skoog@nasa.gov James.brady@faa.gov Abstract — Part 23 aircraft operation, and in particular general 1. I NTRODUCTION aviation, is relatively unsafe when compared to other common forms of vehicle travel. Currently, there exists technologies that The Federal Aviation Administration (FAA) has been could increase safety statistics for these aircraft; however, the interested in alternate certification strategies for small aircraft high burden and cost of performing the requisite safety critical systems for several years. The Agency recognizes that new certification processes for these systems limits their technologies are available that could significantly increase proliferation. For this reason, many entities, including the safety. However, many of these technologies are not being Federal Aviation Administration, NASA, and the US Air Force, implemented or certified due to several barriers. Some of are considering new options for certification for technologies these barriers include the certification burden of outdated which will improve aircraft safety. Of particular interest, are regulations. The Small Aircraft Revitalization Act (SARA) low cost autopilot systems for general aviation aircraft, as these of 2013 provides a framework to consider new certification systems have the potential to positively and significantly affect safety statistics. This paper proposes new systems and options. Of primary importance is reducing the certification techniques, leveraging run-time verification, for the assurance burden for systems which will improve overall aircraft safety, of general aviation autopilot systems, which would be used to which is consistent with the core purpose of the certification supplement the current certification process and provide a process.
viable path for near-term low-cost implementation. In addition, discussions on preliminary experimentation and building the The most frequent causes of fatal mishaps that afflict small assurance case for a system, based on these principles, is aircraft are: loss of control (LOC), controlled flight into provided.
terrain (CFIT), and component failure involving the power plant [1]. Of particular interest is LOC, which accounts for T ABLE OF C ONTENTS over 40% of the total fatal mishaps. In many instances, LOC statistics — as well as CFIT statistics - which are due to spatial disorientation or pilot distraction - could be significantly improved with the addition of even very simple autopilot 3. R EVITALIZED CERTIFICATION FOR PART 23 systems, such as a simple wing leveler. In addition, there are other automatic aircraft systems which will be able to improve mishap rates in many other categories in the near 5. E FFECT OF AUTOPILOT INCLUSION AND future. For instance, an automatic LOC prevention and recovery system could have a dramatic impact on safety of 6. A UTOPILOT RUN - TIME CERTIFICATION small aircraft. This would also be true for a flight director or automatic ground collision avoidance system (Auto GCAS) or an automatic forced landing system (Auto FLS). Even so, 7. H IGH LEVEL ASSURANCE CASE FOR SIMPLE these systems will all require an integrated autopilot system to provide safety decision actuation in order to achieve the 8. R UN TIME ASSURANCE FOR GA AND S MALL UAV safety enhancements. These facts have led researchers and regulators to conclude that inclusion of an integrated autopilot into small aircraft would provide and/or facilitate a significant increase in safety for this type of airplane.
Autopilots can be found on some new small aircraft; however, due to current certification costs, the business case 978-1-4673-7676-1/16/$31.00 ©2016 IEEE Approved for Public Release Distribution Unlimited CASE NUMBER 88ABW-2015-4093 is not favorable for development of low cost autopilots for 2. A CRONYMS the small aircraft retrofit market and many new lower cost AFRL Air Force Research Laboratory aircraft. For example, a simple 2 axis, rate based autopilot ATC Air Traffic Control (that was state of the art 15 years ago) costs $20,000- $25,000 Auto FLS Automatic Forced Landing System to install on a simple Cessna C-182. This high cost means that sometimes the hull value of the aircraft is less than the Auto GCAS Automatic Ground Collision Avoidance installed autopilot system. Furthermore, modern attitude CFIT Controlled Flight into Terrain based autopilots are even more expensive and harder to COTS Commercial off the Shelf justify on older retrofit aircraft.
FAA Federal Aviation Administration FAR Federal Aviation Regulations In response to this reality, the FAA is partnering with NASA, GA General Aviation with the University of Tulsa and AFRL, to develop strategies GAJSC General Aviation Joint Steering to ease the certification burden for small aircraft autopilots in IFR Instrument Flight Rules order to improve the business case for their inclusion in both IMC Instrument Meteorological Conditions existing aircraft and lower cost new aircraft. One particular LOC Loss of Control strategy for accomplishing this is to transfer the authority and NASA National Aeronautics and Space Admin.
certification burden to a simpler and standardized system that RTA Run Time Assurance would monitor an autopilot during operation to assure that it ROA Region of Action could not direct unintended or unsafe actions. This autopilot ROR Region of Recovery assurance system would observe both the input plane (aircraft SARA Small Aircraft Revitalization Act state sensor inputs) and output plane (control commands) of the autopilot to determine if the aircraft is being directed into SOUP Software of Unknown Pedigree an unsafe or unrecoverable region of its state space. If this is the case, the assurance system would disable the autopilot and return full control to the pilot-in-command in a condition 3. R EVITALIZED CERTIFICATION FOR PART 23 which mitigates loss of control during this transition (See AIRCRAFT Figure 1). Techniques to provably assure safety in this Part 23 of the Federal Aviation Regulations (FAR) details manner are currently being established based on work in the airworthiness standards for the certification of airplanes that hybrid systems verification and run-time verification fields as fall within the normal, utility, acrobatic, and commuter well as being used during testing of experimental air and categories. These categories consist of nearly all general spacecraft control systems. Therefore, confidence in this aviation airplanes along with other small commuter aircraft.
method of alternate certification is high; however, there Therefore, when the United States Congress wanted to tackle remains a large amount of work that must be accomplished the “overly prescriptive and outdated certification process before certification authorities will have the data required to [2] ” for general aviation (GA) aircraft and systems, they make decisions based on this alternate method of passed H.R. 1848, The Small Aircraft Revitalization Act of certification.
2013 , or SARA. SARA provides the FAA the opportunity to reorganize the certification requirements for part 23 aircraft in order to streamline the approval of new technologies designed to improve safety. In addition, SARA stipulates to remove prescriptive based certification requirements in favor of performance based regulations, thereby opening up untraditional methods to airplane certification.
In this context, the FAA has been looking into alternate methods of certification which will continue to assure safety in small aircraft without the current, overly burdensome and expensive certification process. This has led to collaboration with partners in government, academia, and industry in order to determine the best ways to solve this difficult issue. In particular, the FAA has begun coordinating with NASA to develop alternate methods of certification for autopilot systems which could enable the largest increase to overall safety for GA aircraft.
Figure 1. Autopilot monitor and control switch 4. S MALL AIRCRAFT SAFETY STATISTICS strategy which may relieve certification burden for an autopilot At the heart of the issue that SARA and the FAA are trying to address is the relatively poor safety record of general 978-1-4673-7676-1/16/$31.00 ©2016 IEEE Approved for Public Release Distribution Unlimited CASE NUMBER 88ABW-2015-4093 aviation travel compared to other common forms of transportation. In the ten year period from 2001-2010, the average number of general aviation accidents was over 1600 per year, with over 300 of those causing at least one fatality [3]. This produced over 550 fatalities per year on average or over 1.5 fatalities per day. When adjusted for the number of general aviation flight hours flown during these years and assuming a conservative average velocity of 100 miles/hr.
and average occupancy of 2 persons per vehicle, the fatality rate per personal mile traveled was over 11.6 fatalities per 100 million personal miles traveled. When comparing this rate to other common forms of transportation, the data reveal that GA pilots and occupants are over 11 times more likely to Figure 3. Categorization of fatal GA accidents from be killed per mile traveled compared to travel in a car. This the GAJSC [1] number increases to over 1100 times more likely when compared to commercial air travel. Only when compared 5. E FFECT OF AUTOPILOT INCLUSION AND against travel in motorcycles, which is known to be one of AUTOMATIC SYSTEMS ON GA SAFETY the most dangerous forms of travel, does general aviation have an advantage in safety and this advantage is only by a Fortunately, automated systems are currently available which factor of around 2.5 [4, 5].
are able to have a major impact on fatality statistics of these three major causes. Of immediate interest, loss of control and CFIT accidents caused by poor situational awareness produced by environmental, geographical, or time-of-day factors, we believe, could be significantly reduced by a simple altitude-hold/heading-hold autopilot system. This is stated more specifically by the FAA General Aviation Joint Steering Committee’s (GAJSC) findings that “LOC accidents at night and in IMC would drop by 50 percent simply by installing autopilots in the more than 100,000 IFR capable GA airplanes [1] ”. However, increases in safety produced by autopilot inclusion are not limited to this class of accident.
Other automated safety systems are available which would provide a significant increase in the safety for other accident Figure 2. Comparing fatality rates in transportation categories as well. For instance, automatic ground collision categories per personal mile traveled. [3, 4, 5] avoidance systems (Auto GCAS), have been developed and are being deployed on United States Air Force F-16s [6].
Of these relatively large number of fatal accidents, over 60% These Auto GCAS systems may have the ability to reduce can be attributed to three specific causes: loss of control CFIT accidents by as much as 98% in military aircraft, and (LOC), controlled flight into terrain (CFIT), and component development of similar systems for GA aircraft is underway.
failure of the power plant [1]. Of these 3 major causes, loss In addition, systems to automatically avoid and/or recover of control heavily dominates, being the cause of over 40% of from loss of control situations are being developed which, the total fatal mishaps in general aviation. Therefore, when applied to the GA regime, would have a dramatic targeting solutions to these three major causes, with special impact in safety for all types of loss of control situations.
emphasis on loss of control, would provide the largest Even fatalities from power plant failure could be significantly contributions to increases in safety for GA aircraft. reduced with the inclusion of automatic forced landing systems (Auto FLS) or flight directors with energy management cues that provide the highest probability for a safe and successful emergency landing [7]. These Auto FLS systems are currently under development for commercial and general aviation category aircraft. Each of these automatic safety systems could dramatically influence the safety statistics of GA aircraft in the future, but they all rely on an integrated autopilot (or flight director) system to actuate their automated decisions. Therefore, not only would the inclusion of a low cost autopilot in a large number of GA aircraft immediately provide substantial increases in safety and decreases in the fatal accident rates, it would also allow for 978-1-4673-7676-1/16/$31.00 ©2016 IEEE Approved for Public Release Distribution Unlimited CASE NUMBER 88ABW-2015-4093 more advanced automatic systems to be integrated providing Run Time Assurance even further safety enhancement.
RTA can be defined as a structured argument supported by evidence, justifying that a system is acceptably safe and secure, not through reliance on offline tests or verification methods, but through reliance on real time monitoring, prediction, and failsafe recovery mechanisms. As illustrated in Figure 5, a run time assurance system consists of at least three components: the untrusted (or lesser certified) component, a run time monitor or flight executive, and one or more recovery systems. The untrusted component contains functional subcomponents, which may not be sufficiently reliable or sufficiently verified according to current development or certification standards. There may be multiple reasons for having such components in a system: under normal conditions, they can provide improved Figure 4. USAF F-16 with integrated Auto GCAS performance or operational efficiency for the system or [16] enhance the user experience. In the case of a general aviation aircraft, a low cost autopilot could be considered the untrusted component. The core idea that enables the use of 6. A UTOPILOT RUN - TIME CERTIFICATION such components in a system is the presence of a safe fallback STRATEGIES mechanism that 1) reliably detects potential problems (the Currently, there exist low cost, off the shelf, advanced monitor or flight executive) and 2) invokes a recovery autopilots designed for the experimental market with a mixed mechanism that can ensure safe operation of the system, track record of capability and performance. In addition to possibly with reduced capabilities and performance. It is these low cost solutions, several companies have invested assumed that the RTA monitor and recovery systems are resources in creating more robust, commercially available certified at the highest criticality level required for the total autopilot options. These autopilot technologies, under a well- system to operate. For example, consider an RTA protected defined set of operating conditions, perform safely and subsystem with a potential failure mode that has been effectively. However, most of these systems have what is determined to be highest risk, endangering human life or considered commercially off the shelf software (COTS) or significant cost. This risk level would translate to the highest software of unknown pedigree (SOUP), software that has criticality level (referred to as level A critical for civilization been developed without investing the resources to ensure aviation). For the RTA protected system, the corresponding compliance with current FAA certification guidance processes, design approaches, and verification methods (specifically, the airworthiness design criteria highlighted prescribed for level A critical software and hardware must within the SAE ARP4761, ARP 4754A, and the RTCA DO- apply to the Run Time Monitor, Switch, and Recovery 178C standards). To address this concern, one approach System.
would be to invest the significant time, resources, and The key advantage to a Run Time Assurance approach is that funding required to create an autopilot that is Flight Critically lower cost autopilot systems can be employed without costly rated “safe to fly”. However, the end product most likely certification, allowing only the behaviors that are protected would be cost prohibitive for private owners, rated for by certified monitors and recovery systems. At the surface, specific vehicle configurations, and limited in usability [8].
this may seem concerning, allowing a system to function Another option could be to assume that these autopilot without exhaustive testing / analysis. However, such an systems, from the certification perspective, are considered inference relies on the assumption that current software capable at a lower certification standard. This would assume systems are exhaustively tested and are without errors or that, without additional testing and inspection, the autopilot defects, which is actually not the case. Rather, software is run software may not be compliant with all FAA certification through a series of quality steps, checklists, and verification standards at the rated criticality level. For these types of practices that increase the implicit confidence of that code. It systems, several military and civilian aviation documents is our claim that the functional capability that has been tested, have identified a notion of a real time monitor and failsafe examined, and proven safe in a particular context, can be switching system, referred to as Run Time Assurance (RTA), argued as safe even if the underlying software has not been as a key component to enabling the certification of created using a design assurance process. Within this automated, increasingly autonomous, and highly complex paradigm, a design approach called Assume-Guarantee systems that are either cost prohibitive or impossible to Reasoning might provide the offline design considerations certify using the current standards or guidelines.
and formalisms necessary for articulating the allowable and certifiable behaviors of an advanced system by constraining behaviors to only what is safe or recoverable.
978-1-4673-7676-1/16/$31.00 ©2016 IEEE Approved for Public Release Distribution Unlimited CASE NUMBER 88ABW-2015-4093 In 2013, AFRL started a Phase III Small Business project the pilot. However, much care has to be taken to ensure with Barron Associates Incorporated (BAI) to develop a Run that either the pilot is capable of recovering the aircraft Time Assurance framework for untrusted flight critical at the point of autopilot failure or that alternate means of software within any control layer from mission planning to recovery are in place, such as a deployable parachute trajectory planning to inner loop control. Below are some system.
design considerations that were noted within the program that (4) Each recovery region must have defined zones or safety may be applicable to a Run Time Assurance based regions that ensure proper timing for switching and certification paradigm for a low cost GA Autopilot system: recovery. BAI has defined these zones based on aircraft (1) The controller need not be a total black box. The capability, ensuring that within the given time interval, complete certification case is better suited with at least the flight executive or RTA monitor has enough time to some evidence the controller is capable within a portion engage a recovery controller before the next time of the flight envelope under specific assumed operating interval.
conditions ( i.e. , assuming the GA autopilot is being fed reliable inertial and guidance inputs). Under these 7. H IGH LEVEL ASSURANCE CASE FOR SIMPLE defined assumptions, the autopilot must be designed RTA/ AUTOPILOT SYSTEMS with an RTA mechanism in mind or the autopilot code In what may be the most tractable near-term implementation must be instrumented to provide insight into the of an RTA system which could provide benefit for general reasoning behind the calculations being made at real aviation aircraft, a COTS type non-safety-critical autopilot time.
would be monitored by a configurable and certified RTA (2) The RTA framework can be implemented using multiple system. The “recovery” controller (from Figure 5) for this implementation, would be the human pilot in command recovery or failsafe mechanisms, which cover differing which is always allowed to control the aircraft as a result of areas of the operating envelope. Previous research the pilot training process. So, in essence, the human pilot limited the recovery controller to just one region of would be the certified backup to the uncertified autopilot.
attraction (ROA) or region of recovery (ROR). This This setup, which will be referred to for the remainder of this constraint made it difficult to justify a performance gain out of the advanced controller (or non-safety critical section as a “Non -Critical Autopilot - Run Time Assured - with Manual Pilot Recovery” System (NCA -RTA-MPR), is autopilot) since the performance was limited to one shown in Figure 6.
recovery system that was fully certified using conventional standards. A better approach would be to allow the untrusted code to operate under specific, tested conditions only if specific recovery mechanisms were in place to take over if the autopilot failed.
(3) For the GA aircraft, if the autopilot fails during operation, the predominant recovery controller may be Figure 5. Generic Run Time Assurance Architecture 978-1-4673-7676-1/16/$31.00 ©2016 IEEE Approved for Public Release Distribution Unlimited CASE NUMBER 88ABW-2015-4093 Argument 1. The pilot in command is responsible for the safety of the aircraft, including separation from other aircraft, ground avoidance, ATC compliance, weather avoidance, controllability, and general aviation “rules of the road.” Required evidence: x The pilot in command has been trained to be responsible for these safety factors and is “certified” to command the aircraft.
Argument 2. The autopilot system will only be able to be used under the authority of the pilot-in- command.
Sub-arguement a. The autopilot can only be engaged by the pilot in command.
Figure 6. Non-Critical Autopilot - Run Time Assured - with Manual Pilot Recovery Sub-arguement b. The autopilot can be disengaged (NCA-RTA-MPR) System at any time by the pilot in command.
Required evidence: We agree that the most important component of a Run Time Assurance based certification approach may be the x The RTA system must be assured through Assurance Case (or Safety Case) itself. Fundamentally, the appropriate safety-critical certification activities to overarching Run Time Assurance claim is that a subsystem, enable autopilot control only through the input of by itself, does not provide enough acceptable evidence to the pilot in command and allow disengagement at achieve the level of confidence required for the any time by the pilot in command.
predetermined level of risk, but that, in combination with a higher confidence monitoring and recovery system, the entire Argument 3. The autopilot will not be allowed to operate system provides sufficient evidence to achieve the level of in an unsafe or uncontrollable region of its flight confidence required for the predetermined risk. Our goal is to state space.
provide a NCA-RTA-MPR system that will NOT REDUCE the confidence in an existing GA aircraft and will lay the Sub-arguement a. If the aircraft is within the unsafe framework for future safety and recovery systems that rely on portion of its flight state space and under manual an autopilot. The goal of these future systems is to actually control of the pilot in command, the autopilot will INCREASE confidence in future GA, providing evidence to not be allowed to be enabled.
support the claim that aircraft will have a higher confidence of safety with the existence of these systems than without.
Sub-arguement b. If, while under autopilot control, the aircraft enters into the unsafe or uncontrollable However, this approach in many ways does not align with region (whether due to aircraft failure, existing design and verification processes as prescribed in environmental anomaly, or other emergency or documents such as the SAE DO-178C standard. It is assumed unknown reason), the autopilot will be disengaged.
that the standard processes will be followed, where feasible, to achieve a sufficient level of confidence in a NCA-RTA- Required evidence: MPR system. However, it is understood that the underlying assurance argument that governs such processes is implicit.
x The RTA system must be assured through Therefore, if any deviation to the existing standards is appropriate certification activities to be able to proposed, much care must be taken in constructing a new monitor aircraft state and disengage the autopilot if explicit argument and evidence to achieve the level of the state falls outside the safe region.
confidence desired. To further illustrate this point, the following examples would need to be constructed to x The aircraft must be assured to be safe and articulate the explicit high level arguments, sub-arguments, controllable within a pre-defined region of and required evidence which might support an NCA-RTA- operation. Any state space outside this safe region MPR assurance case. A complete and thorough assurance is considered unsafe for these purposes.
case is better suited for follow-on research and engineering efforts and is out of scope for this initial paper; however, the Argument 4. The autopilot will not be allowed to cause following is offered for example purposes. loss of control or entry into an unsafe or uncontrollable region of the aircraft operating space.
978-1-4673-7676-1/16/$31.00 ©2016 IEEE Approved for Public Release Distribution Unlimited CASE NUMBER 88ABW-2015-4093 Sub-arguement a. If the aircraft is tending toward out during post flight analysis that the Auto GCAS system the unsafe region of operation, the RTA system will actually initiated and flew the maneuver slightly before the disengage the autopilot in a timely manner to allow pilot [6]. In addition, after experimentation with delaying the for the pilot in command to accomplish an pilot alert of RTA switching, it was found that prompt and appropriate recovery action such that the aircraft aggressive indications should be added to rapidly alert the pilot he was being transferred control. Each of these findings never enters into the unsafe region of operation.
indicate the importance of the development effort that must Required evidence: be applied to the pilot vehicle interface for this type of system.
x The RTA system must be assured through appropriate certification activities to be able to Because of the small scale and limited safety risk of testing monitor aircraft state and disengage the autopilot in these small scale UAVs in highly controlled environments, time to allow for appropriate recovery actions to be making the safety case for their testing was rather performed by the pilot.
straightforward. However, testing of the NCA-RTA-MPR system on larger scale UAVs and manned general aviation x The pilot must be qualified and have the time needed aircraft is planned beginning in late 2015 continuing through to react and respond to the condition causing 2016. On these test platforms, the safety assurance case will disengagement of the autopilot in order to keep the be a critical factor in determination of flight safety and thus aircraft in the safe and controllable region of its the ability to perform the requisite testing. It is thus operation space.
envisioned that the results of this testing will be twofold, one being the design and implementation of the system itself, and More information and background on the construction and the other being the process required to convince experimental usage of assurance and safety cases can be found in Reinhart, airworthiness certification authorities at NASA and the FAA et al [9].
of the safety of the system and aircraft. The results of both should provide much needed direction to the 8. R UN TIME ASSURANCE FOR GA AND S MALL NASA/USAF/FAA group and the community at large.
UAV AUTOPILOTS : PRELIMINARY 9. R ECOMMENDATIONS AND F UTURE W ORK EXPERIMENTATION As has been discussed, the development of the Run Time Experimentation and implementation of the systems and Assurance methodologies, to date, has been largely concepts advocated in this paper (that being of the NCA- academic. However, this group is interested in not only the RTA-MPR system) is beginning to be applied at the NASA theoretical basis but also early experimentation and Armstrong Flight Research Center with support from the implementation to uncover problems early in the FAA and NASA. Initial testing on small unmanned aircraft development process. In pursuing this theme, work has has already provided limited but successful results and begun on identifying a candidate autopilot system considered proved the feasibility of testing both in simulation and on the for small GA aircraft to test the concepts presented in this small scale. For these initial tests, a small UAV autopilot was paper. With this system, we plan on utilizing conventional driven by intentionally unreliable position data. When the but novel assume-guarantee reasoning techniques to abstract data source predictably failed, the vehicle would be sent into allowable and non-allowable behaviors from the candidate an out of control situation. An RTA monitor was established autopilot. In addition, plans to identify and implement that looked at the change in this position data from frame to recovery mechanisms like pilot takeover, collision frame. When the monitor tripped pre-set values (limits), avoidance, etc., and instrument the candidate autopilot to which indicated it was likely that the position solution was monitor undesired behavior or behavior that induces an invalid, control was immediately switched from autopilot unsafe or unrecoverable condition are underway.
control to a backup controller (in this case, the human pilot).
After development, certification, and flight testing of the This limited example provided invaluable experience into the NCA-RTA-MPR system, other systems that are of interest to implementation of such a system. For example, the need for the community could be developed as well. For instance, the comprehensive instrumentation of the RTA monitor for flight FAA is interested in the concept of adaptive autopilots which testing was found to be critical to understanding the behavior can change their control properties based on changes to of the system. For instance, due to the nature of the position aircraft control or aerodynamics. These types of autopilots data source, the RTA monitor was tripped multiple times are designed to provide controllability even in the event of a during each flight test. Having RTA switch from autopilot to control surface failure and therefore may be highly desirable.
pilot was such a trivial and regular event from a pilot However, certification of these highly complex systems has perspective that accurate determination of who was not been successfully accomplished to date, making them controlling the aircraft was at times ambiguous. This great candidates for certification under an RTA approach. In unexpected result was also seen in the USAF Automatic addition, RTA acceptance could open up many further areas Ground Collision Avoidance System flight testing as the of research for autonomy in part 23 aircraft which could pilots thought they flew the recovery maneuver only to find 978-1-4673-7676-1/16/$31.00 ©2016 IEEE Approved for Public Release Distribution Unlimited CASE NUMBER 88ABW-2015-4093 increase safety and open up general aviation to a much larger Autopilot assurance – A structured argument, supported by percentage of the population. evidence that an autopilot system is acceptably safe and secure within the specific operational context in which it A PPENDIX was intended.
A. D EFINITION OF T ERMS Hybrid System Verification – The discipline and methods to verify Hybrid Systems, or systems that contain both Assurance Case – A structured argument, supported by discrete decisions and continuous dynamics.
documentation, which builds confidence of safety and security to an acceptable level within a particular context.
Assume-Guarantee Reasoning – A form of compositional The assurance case provides a means to structure the proof, performed by systematically defining and verifying reasoning that engineers implicitly use to gain confidence the pre-conditions (assumptions) and post-conditions that systems will work as expected. It also becomes a key (guarantees) that govern the interconnections between all element in the documentation of the system and provides a subcomponents within a system [13, 14, 15].
mapping to more detailed information. The concept of an assurance case has been derived from the safety case, a Region of Attraction (ROA) – The region or multi- construct that has been used successfully in Europe for over dimensional constraint space that guarantees a system, a decade to document safety for nuclear power plants, given the initial conditions start within the ROA, will transportation systems, automotive systems, and avionics always remain within the ROA.
systems. Much like a legal case presented in a courtroom, an assurance case requires arguments linking evidence with Region of Recovery (ROR) – The region or multi- claims of conformance to dependability-related dimensional constraint space that is defined by a Run Time requirements. [10] Several certification standards and Assurance recovery system, guaranteeing that once a guidelines in the defense, transportation (aviation, transition from advanced control to recovery control automotive, rail), and healthcare domains now recommend occurs, the system will safely traverse from the failed state and/or mandate the development of assurance cases for space to a less capable, certified controller’s region of software-intensive systems [11, 12] attraction (ROA).
Design Time – The period within a system lifecycle (RTA) Untrusted or Uncertified Component – The pertaining to all design, integration, verification and software component within an RTA system not certified at validation activities performed PRIOR to full rate the same criticality level that is required of the system as a production.
whole (i.e., a software system being used within a safety critical application that has not been tested to current safety Run Time – The period within a system lifecycle after critical standards).
deployment, fielding, or full rate production as opposed to referring to the system during the “design time” or design (RTA) Recovery System – Set of transition and baseline phase prior to full rate production.
components providing overall assurance that at any given time, the RTA protected system can recover from an Run Time Assurance (RTA) - A structured argument untrusted component failure.
supported by evidence, justifying that a system is acceptably safe and secure not through reliance on offline (RTA) Baseline Component – Software component(s) tests but through reliance on real time monitoring, certified to maintain RTA protected critical functions under prediction, and failsafe recovery.
specific and limited conditions using deterministic and reliable decision procedures.
Run Time methods – Methods and techniques to monitor, diagnose, and evaluate pre-defined (at design time) (RTA) Transition Component – Software component(s) constraints that always must hold. Other terms may refer to certified to transition the system from any condition at fault diagnosis, isolation, and recovery with the exception which the untrusted component failed to an operating that FDI & R methods predominantly refer to off-nominal condition suitable for the baseline controller to engage.
physical component failures and not necessarily to software based failures not previously identified at design time.
(RTA) Monitor & Switch – Certified run time executive that compares the untrusted component behavior with a set of Run Time Assurance Based Certification – The act of known, acceptable constraints based on the assume- providing acceptable arguments and evidence that leads to guarantee contracts of each subcomponent interaction and the certification of a system that contains a design time behavior. The monitor determines, based on the violation uncertified subsystem. The certification argument relies on of a constraint and the time required to recover, when to additional, complementary subsystem components switch from the untrusted to recovery components.
designed to monitor, interrupt, and recover from a failure from the uncertified subsystem component.
978-1-4673-7676-1/16/$31.00 ©2016 IEEE Approved for Public Release Distribution Unlimited CASE NUMBER 88ABW-2015-4093 (RTA) Instrumentation Considerations – The process by 1, 2004.
which an untrusted or uncertified component is designed [15] M. Huth and M. Ryan, Logic in Computer Science, with software instrumentation (i.e., software based triggers, Modelling and Reasoning about Systems, outputs, or assertions) such that run time monitoring against Cambridge: Cambridge University Press, 2004.
pre-defined constraints can be performed.
[16] Photo by Jim Ross, "NASA Dryden Past Projects: Automatic Collision Avoidance Technology / Fighter Worst Case Execution Time (WCET) – The maximum Risk Reduction Project," 2013. [Online]. Available: time a particular algorithm, compiled on the target http://www.nasa.gov/centers/dryden/research/ACAT platform, requires to perform all executions within a given _FRRP/.
period of time .
REFERENCES [1] General Aviation Joint Steering Committee (GAJSC) B IOGRAPHY Loss of Control Work Group, "Approach and Loyd R. Hook received a Ph.D.
Landing Report," 2012.
from the University of Oklahoma in [2] 113th US Congress, "H.R. 1848," 2013.
Electrical and Computer [3] National Transportation Safety Board, "Preliminary Engineering in 2012. He is Aviation Statistics, Data for years 2001-2010," 2013.
currently an assistant professor at the University of Tulsa in the [4] U.S. Department of Transportation, "Fatality Electrical and Computer Reporting System Data for years 2001-2010," 2012.
Engineering Department and [5] Insurance Institute of Highway Safety, "Traffic director of the TU Vehicle Safety Facts 2011, Data for years 2002-2011," 2011.
Autonomy and Intelligence Lab (TU [6] D. E. Swihart, A. Barfield, E. Griffin, R. Lehmann, S.
VAIL). Previously, he worked for NASA ’s Armstrong Flight Whitcomb, B. Flynn, M. Skoog and K. Processor, Research Center where he served as principle investigator "Automatic Ground Collision Avoidance System and flight test engineer primarily pursuing projects Design, Integration, & Flight Test," IEEE Aerospace investigating and implementing automatic and autonomous and Electronic Systems Magazine, vol. 26, no. 5, pp.
air vehicle systems. His current research focus is the 4-11, 2011.
development of safety assured autonomous vehicle systems.
[7] L. Hook and C. Tomlin, "Development of a “Where - Matthew A Clark is the Technical Area to- Land” Decision Function for an Expert Piloting Lead for the verification and validation Systems (EPS) in Man -rated Autonomous Air of autonomous control systems within Vehicles," NASA NARI, 2013.
the Autonomous Controls Branch, [8] J. Rushby, "New challenges in certification for AFRL/RQQA. Mr. Clark started his aircraft software," in Proceedings of the ninth ACM career in the Air Force Research Lab in international conference on Embedded software , 1998 supporting large scale aircraft 2011.
component thermal, acoustic, and static [9] D. Reinhart, J. Knight and J. Rowanhill, "Current combined environment structural Practices in Constructing and Evaluating Assurances testing. In 2000, 2010 respectively he Cases with Applications to Aviation," NASA received his Bachelor’s and Master’s Degree in Electrical Technical Report, 2015.
Engineering from Wright State University in Dayton, OH.
From 2000 to 2005, Mr. Clark worked as an industrial power [10] "Assurance Cases," [Online]. Available: and control engineer at Delphi Automotive, in Warren Ohio.
http://www.sei.cmu.edu/dependability/tools/assuran In 2005, Mr. Clark returned to AFRL as Technical Area Lead cecase/. [Accessed August 2015].
for the combined environment structural testing facility. In [11] 1st International Workshop on Assurance Cases for 2010, Mr. Clark served at the Air Force Material Command Software-Intensive Systems (ASSURE 2013).
headquarters providing support for the test and evaluation [12] R. Hawkins, I. Habli and T. Kelly, "The Principles of infrastructure, strategic planning, and operational cyber Software Safety Assurance," Boston, 2013.
security, receiving the Exemplary Civilian Service Award. In [13] K. L. McMillan, "Circular Compositional Reasoning 2011 he returned to the Air Force Research Laboratory to about Liveness," Cadence Berkeley Labs, , Berkeley, work on the verification and validation of autonomous CA, 1999. control systems and applications. His research interests include verifiable intelligent control systems and Run Time [14] G. Frehse, Z. Han and B. Krogh, "Assume-Guarantee Assurance of intelligent systems.
Reasoning for Hybrid I/O Autonomata by Over- Approximation of Continuous Interaction," Decision David Sizoo earned both M.S. and B.S. degrees in Aerospace and Control 2004, CDC 43rd IEEE Conference, vol.
978-1-4673-7676-1/16/$31.00 ©2016 IEEE Approved for Public Release Distribution Unlimited CASE NUMBER 88ABW-2015-4093 Engineering from M.I.T. He has been an Experimental Test Pilot and Human Factors Specialist with the FAA for 6 years.
Prior to joining the FAA, his 23 year career spanned work in both the military and civilian sectors. In the U.S. Air Force and at Gulfstream Aerospace he served as a Developmental Test Pilot on programs including the F-35 Joint Strike Fighter, F-16, and Gulfstream G-150. His passion is General Aviation flying and he is currently leading research to bring advanced technology and safety enhancements to the aviation community.
Mark A. Skoog works for NASA’s Armstrong Flight Research Center as the Principle Investigator for their Automatic Systems Project Office which leads the Center’s autonomy research. He also leads the Collision Avoidance Technical Stewardship Group for the Office of the Undersecretary of Defense for Personnel and Readiness. He graduated from California Polytechnic State University at San Luis Obispo. Over the past 34 years he has supported numerous NASA and the Air Force fighter and UAV research efforts as well as initial flight test of the B-2. Focus areas have included the integration of flight controls and avionics with high authority autopilots to automatically accomplish all phases of fighter combat missions. More recently, much of his career has been in the development of full vehicle autonomy and automatic collision avoidance systems, for both ground and air.
James Brady received a B.S. in Aerospace Engineering from Wichita State University, a M.S. in Engineering from Kansas University, a MBA from Avila University and a Ph.D. from Ohio State University. He spent more than 25 years with Lucent Technologies Bell Laboratories where he supervised product engineering and system groups for military and space projects. In 2006 he joined the FAA as the Avionics and Electrical Systems Specialist in the Small Airplane Directorate 978-1-4673-7676-1/16/$31.00 ©2016 IEEE Approved for Public Release Distribution Unlimited CASE NUMBER 88ABW-2015-4093