Document
NASA/ TP - 20240011784
Assessment of Some IASMS - relevant Data
Sources for Aviation Safety
Nikunj C. Oza, Ph.D.
Ames Research Center, Moffett Field, California Chad L. Stephens Langley Research Center, Hampton, Virginia Click here: Press F1 key (Windows) or Help key (Mac) for help
September 2024
This page is required and contains approved text that cannot be changed.
NASA STI Program ... in Profile Since its founding, NASA has been dedicated • CONFERENCE PUBLICATION.
to the advancement of aeronautics and space Collected papers from scientific and science. The NASA scientific and technical technical conferences, symposia, seminars, information (STI) program plays a key part in or other meetings sponsored or helping NASA maintain this important role. co - sponsored by NASA.
The NASA STI program operates under the • SPECIAL PUBLICATION. Scientific, auspices of the Agency Chief Information Officer. technical, or historical information from It collects, organizes, provides for archiving, and NASA programs, projects, and missions, disseminates NASA’s STI. The NASA STI often concerned with subjects having program provides access to the NTRS Registered substantial public interest.
and its public interface, the NASA Technical Report s Server, thus providing one of the largest • TECHNICAL TRANSLATION.
collections of aeronautical and space science STI English - language translations of foreign in the world. Results are publ ished in both non - scientific and technical material pertinent to NASA channels and by NASA in the NASA STI NASA’s mission.
Report Series, which includes the following report types: Specialized services also include organizing and publishing research results, distributing • TECHNICAL PUBLICATION. Reports of specialized research announcements and completed research or a major significant feeds, providing information desk and personal phase of research that present the results of search support, and enabling data exchange NASA Programs and include extensive data services.
or theoretical analysis. Includes compila - tions of significant scientific and technical For more information about the NASA STI data and information deemed to be of program, see the following: continuing reference value. NASA counter - part of peer - reviewed formal professional • Access the NASA STI program home page papers but has less stringent limitations on at http://www.sti.nasa.gov manuscript length and extent of graphic presentations. • E - mail your question to help@sti.nasa.gov • TECHNICAL MEMORANDUM. • Phone the NASA STI Information Desk at Scientific and technical findings that are 757 - 864 - 9658 preliminary or of specialized interest, e.g., quick release reports, working • Write to: papers, and bibliographies that contain NASA STI Information Desk minimal annotation. Does not contain Mail Stop 148 extensive analysis. NASA Langley Research Center Hampton, VA 23681 - 2199 • CONTRACTOR REPORT. Scientific and technical findings by NASA - sponsored contractors and grantees.
NASA/ TP - 20240011784
Assessment of Some IASMS - relevant Data
Sources for Aviation Safety
Nikunj C. Oza, Ph.D.
Ames Research Center, Moffett Field, California Chad L. Stephens Langley Research Center, Hampton, Virginia Click here: Press F1 key (Windows) or Help key (Mac) for help National Aeronautics and Space Administration Ames Research Center Moffett Field, CA 94035 - 1000
September 2024
Acknowledgments We thank Bryan Matthews, Daniel Weckler, Shannon Walker, Fasil Alemante, Rami Houssami , Ezequiel Juarez Garcia, and Nicholas Napoli for performing much of the work represented within this report. Acknowledgements to the NASA Aeronautics Research Mission Directorate (ARMD) System - Wide Safety (SWS) project and its management for supporting this work.
Click here: Press F1 key (Windows) or Help key (Mac) for help This report is available in electronic form at https://ntrs.nasa.gov/citations/20240011784 1. Introduction An In - time Aviation Safety Management System (IASMS) [1,2] is a set of services, functions, and capabilities (SFCs) necessary for monitoring known hazards and emergent risks, assessing safety data for anomalies, precursors, and trends, mitigating hazards t hat reach safety thresholds, and assuring efficacy of controls in mitigating hazards. An IASMS will continually monitor the NAS to collect data on the status of aircraft, air traffic management systems, weather, and airports. Within the NASA Aeronautics R esearch Mission Directorate (ARMD) System - Wide Safety (SWS) project’s technical challenge called In - time Aviation Safety Management Systems (IASMS) for Commercial Aviation Operations, which we often refer to as Technical Challenge 6 (TC - 6), we have perform ed an assessment of several aviation data sources we have found that are relevant to assessing the safety of the National Airspace System (NAS) in the context of an IASMS. This assessment includes understanding the nature of the data themselves and using s ome data analytics tools on these data to show how they can be used to identify potential safety issues. We also describe how the data and analytics are part of a system that can allow for other data and analytics to be performed and for the results to be visualized for use by appropriate operators to identify potential safety issues and develop mitigations. This report is a step toward the ultimate goal of TC - 6, which is to develop a prototype IASMS system that demonstrates the potential of an IASMS and in spire operators to build analogous systems to make the best possible use of the significant investments that they make in collecting, storing, and managingdata related to their operations.
2. FAA SWIM Data within NASA’s Digital Information Platform The NASA Digital Information Platform (DIP) [3] enables aviation data providers to provide data; service providers to provide data integration, analytics, and visualization services; and aviation data consumers to be able to access all of these provided it ems to enable safe and efficient operations. We are working to set up a demonstration data analytics service within DIP to show the benefit of an in - house developed anomaly detection algorithm that can be used for vulnerability discovery. This section desc ribes the data that we used and the results obtained so far.
Within the Federal Aviation Administration’s (FAA) System Wide Information Management (SWIM) data, we used two feeds to obtain the track and touchdown time data for our application.
These are SWIM Terminal Data Distribution System (STDDS) for the track dat a and SWIM Flight Data Publication Service (SFDPS) to determine the touchdown time for the monitored flights. The important variables that we used from the track data are latitude, longitude, altitude, heading, and ground speed. These allow us to calculate vertical speed, calculate distance remaining in the flight, and determine the runway on which the flight is landing. The SFDPS data allow us to monitor the arrival time of the flight. Once the flight has landed, the last 20 nautical miles of the flight ar e retained in our processed data. The timeseries is binned into 0.25 nautical mile bins. These multivariate binned timeseries data are used by the machine learning model generated by the in - house developed Convolutional Variational AutoEncoder (CVAE) algor ithm [4] to compute the anomaly score. The initial deployment of CVAE on DIP is focused on SFO arrivals using Northern California TRACON (NCT) data. We intend to extend our work to other airports within NCT before expanding to other TRACON regions. Figure 1 illustrates the original track data overlayed with the 20 nm binned data.
Figure 1. Last 20 NM binned flights (red) superimposed over the full unbinned flight paths (blue).
In - Time CVAE Program The In - Time CVAE program consists of 3 separate parts: training, track data capture, and metadata capture. These parts, together with the data on which they rely and their output, are depicted in Figure 2.
Figure 2: CVAE generates anomaly report from DIP SWIM Feed Training the CVAE model : We bootstrap the model training with Sherlock [5] data, which is essentially NASA’s archived version of various SWIM feeds with some extra processing and calculations. Since the Sherlock data and live SWIM feeds are identical, the offline batch training with Sherlock is appropriate for use until enough data is collected from the live SWIM feeds. The SWIM feed, however, has the advantage of being a live service — enabling in - time processing and anomaly detection, and evolution of the model as the NAS changes. W e intend to train the model entirely on SWIM data once enough is gathered.
Track Data: Our CVAE data processing program takes track data from the live streaming STDDS and SFDPS SWIM feeds, filters out unnecessary parameters, and processes the data when it lands. The SFDPS data and STDDS track data programs capture data independently, but ar e linked together within a shared database. The track program captures and stores the flight track data to a database which contains all flights currently considered to be in the airspace. Once a flight has been marked as landed, it extracts tha t flight from the database (removing it to save memory), bins the flight, and applies CVAE to the multivariate data. The resulting anomaly score is stored in another database. If a flight has not received new data for a set amount of time, the program will remove it from the database and add it in the data processing error logs (to see if it is a data quality error or an error with the capture program).
Meta d ata Capture: The SFDPS SWIM data feed program simply keeps track of both the estimated and “actual” landing times from this metadata stream as well as origin and destination airports. Once a flight’s landing data has been marked as “actual” (or the “estimated” time is after the current time), the landing time is written to a database which is constantly being checked by the track program. Once the track program detects a landing from this database, it also removes the entry from the landing database and initiates the p rocessing of the flight track data.
Figure 3 gives an example anomaly report generated by CVAE. It depicts the flight track, which illustrates a potential misalignment on landing. On the right side, the top plot shows the anomaly score relative to the distribution of anomaly scores over all flights on which it was trained --- the anomaly score is clearly greater than the threshold for marking the flight as an anomaly. The bottom plots show several variables relative to their distribution over the training flights as a function of the distance t o landing. Safety analysts can use these plots to better understand the nature of the anomaly and decide whether it represents a safety issue.
Application Programmers’ Interface (API) The API has been developed using FastAPI [6] and allows the user to request the anomaly scores for specific flights in a specified date (or date range) from the database that is continuously being updated with flights once they land. The API also implement s the use of “*” wildcards, which allow the user to request, for example, all flights from a specific airline (e.g. SWA* for all Southwest Airlines flights). These requests are done via a POST request using either Python or a curl command. Model versions a re tracked and the API also supports requesting the model parameters used in the calculation. These model parameters include the date range of the flights used to compute the model, how many flights the model was trained on, hyperparameter settings of the model as well as the mean and standard deviation of the anomaly scores used to compute the anomaly threshold, for which we used mean plus three times the standard deviation for this work.
Figure 3: Example of anomaly found on SWIM data --- potential misalignment Value to Stakeholders The CVAE DIP service will provide users the ability to track atypical flight trajectories on approach to airports in the NAS. Stakeholders can use this tool to monitor anomaly trends over time and identify hot spots where anomalies in aircraft trajectories show signs of increasing. This can help track whether operations are changing in - time and assist in determining if mitigation is needed to address an emerging safety concern. Operators ca n also compare their anomaly rates against the overall rates at various airports. In case an operator’s anomaly rate is higher, that operator can choose to examine anomalous flights in more detail and develop mitigations.
While DIP will have data sets that are open to all users, operators can choose to simultaneously use proprietary data to obtain greater insights into their own operations. Additionally, as more data sources and services are made available within DIP, we pl an to investigate their use as inputs into our and other available Machine Learning algorithms to make them even more useful for vulnerability discovery/anomaly detection and precursor identification.
We originally planned to demonstrate the CVAE DIP service at a DIP industry day in September 2024. However, the industry day has been delayed to March 2025.
3. FAA SWIM Data for Risk of Unstable Approach Our partners at Booz Allen Hamilton (BAH) used SWIM data to identify flights that are at risk for unstable vertical approach using three different models and demonstrated a way to visualize the results through their Advancing Risk Precursor Identification (ARPI) system. Unstable vertical approach is defined as a descent of greater than 1000 feet per minute. The goal of this ARPI demonstration was to characterize the risk of an unstable approach for a flight and give a prediction as early as possible within a visualization tool to aid dispatchers. The architecture of the system is shown in Figure 4.
Figure 4: ARPI architecture for SWIM demonstration for unstable vertical approach risk As with the CVAE DIP service, the ARPI prototype required some transformations to the SWIM data before passing to time series models for calculating the unstable approach risk. In particular, the data from STDDS was first run through a Java parser to conve rt the data from a JSON (hierarchical format) to CSV (flat) format. Also, the velocity was smoothed by averaging the last two time steps’ velocities because the resolution for velocity is rounded to the nearest 100 feet/second. Without smoothing, there wou ld be a large number of false alarm unstable approaches. The velocity was then converted to feet/minute. In SWIM, the altitude is provided in the form of feet above mean sea level (MSL) so the runway’s mean sea level (MSL) was subtracted to get the altitud e above ground level (AGL).
Three different well - known prediction models were used in this demonstration: AutoRegressive Integrated Moving Average (ARIMA), Survival Analysis (Kaplan - Meier estimator), and Long Short Term Memory (LSTM). LSTM performed best within this demonstration and yielded a visualization like what is shown in Figure 5. This visualization presents a summa ry of the risk of unstable vertical approach. It allows the user to select individual flights to look up more information about them.
Figure 5: ARPI Demonstration with SWIM data 4. Flight Operations Quality Assurance (FOQA) Data FOQA data are collected onboard commercial aircraft and are generally proprietary. FOQA data contain many parameters; however, we and our partners have typically examined a limited number of parameters that are relevant to takeoff and landing. In particula r, we have used the following data in various analyses: Location: Latitude, Longitude Configuration: Left and right aileron positions, left elevator position, trailing edge flap position, rudder position, average core speed average Attitude: Corrected angle of attack, glideslope deviation, pitch angle, roll angle Heading: Selected heading, localizer deviation, selected course, drift angle, true heading Atmosphere: Barometric correct altitude, total pressure Movement: Computed airspeed, vertical acceleration, wind speed, selected airspeed Figure 6: Examples of anomalies found in FOQA data Figure 6 gives examples of some anomalies that we have found in FOQA data that turned out to be safety concerns. The left figure shows a drop in airspeed during takeoff where the airspeed dropped to within 12 knots of stall speed. The middle plot shows an example of a human pilot and autopilot fighting over which of two autopilot modes should be in effect which ultimately led to the human pilot recycling the flight director. The right plot shows an example of an unstable approach with greater than 25 degree roll at less than 500 feet. Within the ARPI prototype described earlier, BAH also demonstrated the analysis of FOQA data using CVAE to identify anomalous flights. Although this was mainly done to demonstrate the generality of the ARPI platform.
Our partners at University of Florida (UF) used FOQA data to identify risk which impacts safety of flights (e.g., unstable approach) using a technique complementary to deep learning models used extensively in aviation anomaly detection studies. This altern ative approach is intended to address and enable end - user interpretability. Aiming to maintain the same level of interpretability as traditional threshold - exceedance methods, UF continued development of prediction models using ordinal patterns and their di stributions throughout the flight.
Specifically, a study presented at AIAA SciTech Forum 2024 [7] extends work into multiclass anomaly detection using sensor fusion based on Dempster - Shafer theory (DST), a second - order probability theory used to combine in formation from different sources of evidence. The approach uses DST to reduce the uncertainty in the class predictions of an ensemble of classifiers. These classifiers rely on the similarity between flight data and class templates to make a prediction of t he state of the aircraft. The approach aims to take advantage of simple models trained on interpretable features (ordinal patterns) to correctly predict an anomaly and identify the flight dynamics linked to the anomaly. Results of the study show an improve ment when using DST - based sensor fusion over a majority voting approach. The results of the UF team’s work provides insight into aircraft states linked to rare high - risk anomalies.
Since FOQA data are considered proprietary, these analyses are examples of what would be performed separately by each operator, perhaps in conjunction with analysis performed on SWIM data of the type described earlier in this report. Each flight’s risk ass essment can be performed based on an appropriate combination of proprietary data and SWIM or other data.
5. Other Data Sources and Analyses There are other past and ongoing aviation safety data analysis examples that may be relevant to the development of an IASMS. A past SWS project report [8] examined several “non - traditional” data sources may have value in assessing aviation safety. In parti cular, this report examined Aviation Safety Reporting System (ASRS) [ 9 ] reports, the Boeing Alertness Model (BAM), and crowdsourced ADS - B data. These data are available with varying latencies and; therefore, the value of these within an IASMS will also vary. SWIM and ADS - B data are currently being used to calculate other measures of risk in ongoing work funded by NASA and likely by others. Voice recordings, such as what are available from LiveATC.n et, are also being examined in ongoing NASA - funded work and may be useful in identifying and assessing safety incidents. Some of these datasets, such as SWIM, likely can be analyzed continually, while other datasets, such as voice tracks, may need to be an alyzed only conditionally, such as when SWIM data appear anomalous. What data are analyzed when will involve tradeoffs between the desire for identifying anomalies and precursors, the time within which these need to be identified, how soon the data can be made available, and the amount of computation required. As new data sources are made available, we within SWS will assess how they can be used to assess aviation safety.
References [1] National Academies of Sciences, Engineering, and Medicine, “In - time Aviation Safety Management: Challenges and Research for an Evolving Aviation System,” 2018.
https://doi.org/10.17226/24962.
[2] Ellis, K., Prinzel, L., Krois, P., Davies, M., Oza, N., Stephens, C., Mah, R., Koelling, J., and Infeld, S., “A Future In - time Aviation Safety Management System (IASMS) Perspective for Commercial Air Carriers,” AIAA Aviation, 2022.
[3] Digital Information Platform, https://nari.arc.nasa.gov/atmx - dip/ [4] Memarzadeh, M., Matthews, B., and Avrekh, I., “Unsupervised Anomaly Detection in Flight Data Using Convolutional Variational Auto - Encoder,” Aerospace, Vol. 7, 2020, 115 .
( https://www.mdpi.com/2226 - 4310/7/8/115 ).
[5] Sherlock Data Warehouse, https://aviationsystems.arc.nasa.gov/sherlock/ [6] FastAPI, https://fastapi.tiangolo.com [7] Garcia, E.J., Beres, S.L., Mulvihill, M.L., Stephens, C.L., & Napoli N.J. (2024) Multiclass Flight Anomaly Detection Using Sensor Fusion Based on Dempster - Shafer Theory. AIAA Scitech Forum 2024 Conference.
[8] Nikunj C. Oza, Chad Stephens, Fasil Alemante, Rami Houssami, Shannon Walker, Immanuel Barshi, Stephen Casner, Ilya Avrekh, Erin Flynn - Evans, Lucia Arsintescu, and Rachel Jansen, Assessing Several Non - Traditional Data Sources for Value in Aviation Safet y, NASA Technical Report.
[9] NASA Aviation Safety Reporting System (2023) Aviation Safety Reporting System Program Overview. https://asrs.arc.nasa.gov/docs/ASRS_ProgramBriefing.pdf Retrieved 2024 - 9 - 13.