Uncovering Resilient Behavior in the Aviation Safety Reporting System Using Large Language Models

Uncovering Resilient Behavior in the Aviation Safety Reporting System Using Large Language Models 2025 - 09 - 18 Bryan Matthews KBR, Inc.

Data Science Group NASA Ames Research Center

Outline

• Background and Concept of Resiliency • Aviation Safety Reporting System • Approach using LLMs • Findings • Discussion • Future Work

What is Resiliency?

• Types of resilient behavior defined by Erik Hollnagel et al*.

Anticipate Monitor React Learn * E. Hollnagel , R. L. Wears, and J. Braithwaite, “From safety - i to safety - ii: a white paper,” The resilient health care net: published simultaneously by the University of Southern Denmark, University of Florida, USA, and Macquarie University, Australia, 2015.

How Do We Currently Capture Resilient

Behavior?

• Human factors studies: • Surveys • Interviews • Eye tracking • Simulation scenarios • Flight deck observations: • Line Operational Safety Audit (LOSA) • Learning Improvement Team (LIT)

Can We Leverage Existing Data Sources In

New Ways?

ASRS Report 1. “SOCAL Approach Control cleared ASRS has over 2M voluntarily our flight for the ILS 24R via the CRCUS transition.

submitted reports.

We were following a B787 - 9. To help increase the • Capture safety events that space between our airplanes the Los Angeles Center operators were concerned Controller instructed us to slow to 250 KIAS while about.

on the ANJLL4 arrival which we complied with.

• Context includes before during Looking at our TCAS display, I estimated the 787 was and after the event.

approximately 5 miles ahead of us. SOCAL approach • Potentially contains corrective appropriately cautioned us for wake turbulence actions and preventative since we were following the heavy 787. Our flight was measures that pilots and normal until we reached CRCUS waypoint where we controllers exhibit to make the system more resilient. encountered the 787's wake”.

How Can We Leverage Large Language Models?

• Large Language Models (LLM) • LLMs are designed for token/word predictions given prior context and instructions • Retrieval Augmented Generation (RAG) • Limits the task’s scope to use specific provided text when crafting response • Open sourced Llama 3.1 8B Instruct Q6 8 Billion trained parameters • Multilingual Text • 128k context length (number of tokens) • Model uses 6 bit post quantization (~6.6GB fits within laptop memory)

Example schema and context

What Are We Looking For?

• Extracting evidence of resilient characteristics in the text • Use specific prompt components • Persona • Task • Context • Format

What Are the Trends?

• GA and Commercial follow similar reporting trends.

• GA has slightly higher monitoring rates.

What Does Resiliency Look Like?

Monitor : Anticipate: • Monitoring ATC communications • Weather forecasting and anticipation • Watching other aircraft’s position and movement • Fuel management and planning • Inspecting aircraft systems and instruments • Air traffic control and communication • Monitoring weather conditions • System malfunctions and failures • Watching for potential hazards and obstacles • Emergency procedures and protocols • Listening to radio communications for traffic • Altitude and terrain considerations information • Wake turbulence and airspeed deviations • Monitoring fuel levels and consumption • Communication with dispatch and • Scanning for traffic in the vicinity maintenance • Watching taxiways and runways for potential • Alternate routes and airports planning hazards • Pre - flight inspections and preparations • Listening to radio communications with other aircraft or ground control

What Does Resiliency Look Like?

Respond: Learn: • Responding to ATC instructions • Verify Information • Correcting mistakes or errors • Double - check Procedures • Declaring an emergency or taking evasive • Communicate Clearly action • Stay Vigilant and Focused • Communicating with ATC and other aircraft • Follow Standard Procedures • Taking evasive action to avoid collisions • Be Aware of Surroundings and • Following procedures for abnormal situations Environment • Responding to system malfunctions or failures • Monitor Systems and Instruments • Correcting navigation errors or • Take Responsibility for Actions and misunderstandings Decisions • Coordinating with crew members and air • Improve Situational Awareness traffic control • Review and Analyze Data • Reporting incidents or issues to authorities

How Do Commercial and General Aviation Compare?

How Do Commercial and General Aviation Compare?

How Do Commercial and General Aviation Compare?

How Do Commercial and General Aviation Compare?

What Does Resilient Behavior Look Like Across Different

Safety Events?

Airborne Conflict [ 21,778 ] Near Mid Air Collision (NMAC) [ 11,677 ] Altitude Deviation Overshoot [ 10,664 ] Runway Ground Excursion/Incursion [ 12,399 ] Weather Turbulence [ 16,529 ]

What Can We Learn From This?

• Communication: • Across all 4 categories of resilience.

• Both in GA and Commercial operations.

• Across different safety events.

• Sharing knowledge, intensions, asking for assistance.

• GA vs Commercial • Commercial operations had more focus on “following procedures and checklist” in both monitor and respond .

• GA operations relied upon “problem solving” in the unique respond category.

• Anticipate: threats and pressures to the original flight plan and/or procedures.

• Monitor: constantly survey the environment and other actors to gather information about how the situation is unfolding.

• Respond: take proper action and relay information to the appropriate people involved.

• Learn: from the event to better understand how to prepare for a similar situation in the future.

What Are The Next Steps?

• Comprehensive examination of all anomaly categories in ASRS.

• Complement existing SMS programs for ongoing assessment of resilient proficiencies.

• Analyze other safety reports: • National Transportation Safety Board accident investigation reports.

• LIT or LOSA observation narratives.

• Apply this technique to other safety reporting domains.

Acknowledgments

• Human Contributions to Safety Team • NASA’s Aviation Safety Reporting System • Work was funded by NASA’s System - Wide Safety Project under NASA’s Aeronautics Mission Directorate’s Airspace Operations and Safety Program.

Questions?