Document
Uncovering Resilient Behavior in the Aviation
Safety Reporting System Using Large Language
Models
st 1 Bryan Matthews KBR, Inc.
NASA Ames Research Center Mountain View, CA, USA bryan.l.matthews@nasa.gov Abstract —Resiliency is present in everyday life, both in system resilient factors that, when present in the system, add to the design and exhibited by the operators that function within critical layers of safety that millions of air travelers rely these systems. This includes the National Airspace System (NAS) upon. Additionally, quantifying resilient actions performed by where pilots and controllers make positive decisions and take humans can further highlight requirements that will be needed preventative or corrective actions every day even in unsafe for autonomous systems to function safely within a future situations. Pilot safety reports filed after an event are rich text narratives that detail the conditions around an event and can hybrid human/autonomous airspace. While pilots and air traffic provide additional context leading up to, during, and describe controllers are managing aircraft within the NAS they are how the situation was resolved. This yields useful insights into constantly facing a variety of pressures that require them to the resilient positive actions and corrective steps that may have adapt and pivot from their original flight plan to safely navigate transpired to prevented a safety incident from degrading further.
aircraft to their destination. These pressures range from envi- Analyzing large archives of these reports can be impractical for subject matter experts to properly extract evidence of resilient ronmental factors, to managing traffic separation, to handling behavior. However, Large Language Models have demonstrated issues with equipment and automation. To successfully address the potential to extract useful insights from extensive bodies of the impact of these pressures to the operations, the pilots and text. This work proposes to utilize the Llama3.1 Instruct model to controllers depend upon the resiliency of the system to ensure identify examples of resilient behavior within four categories on the safety of the NAS. Components of the system’s resiliency over 250,000 narratives from NASA’s Aviation Safety Reporting System (ASRS). The analysis will reveal how similar and different rely upon standard operating procedures, safety nets such resilient behaviors are present within various ASRS anomaly cat- as the traffic collision avoidance system (TCAS) and other egories such as airborne conflict, near mid air collision, altitude warning systems, and human decision making and responses deviation overshoot, runway excursion/incursions, and responses that are both strategic and tactical. Furthermore, human fac- to external factors such as weather turbulence. Additionally, the tors such as: effective communication on the flight deck, analysis will compare resilient behavior between general aviation and commercial operation events as well as temporal trends monitoring for readback errors during air traffic controller within the archive of reports. The analysis aims to uncover how communication, or adherence to standard operating procedures operators are practicing positive resilient behavior in situations are well known skills and qualities that are covered in training described within the corpus of the report archive. This method and are considered good airmanship. However, situations arise provides a new lens into these valuable safety reports that can be during the operations that require pilots and controllers to used to inform and improve safety monitoring systems from this human resiliency perspective. The benefit can lead to highlighting lean upon their years of experience and training to address a operator proficiencies within the community and identify any situation that, if not handled properly, could lead to something knowledge gaps to ultimately improve safety within the NAS.
more serious such as damage to the aircraft or even injuries Index Terms —Aviation Safety,Large Language Models or loss of life. Often pilots and controllers will have to adapt to a situation and implement a work around or utilize a coun- I. I NTRODUCTION termeasure [2] when automation or procedural failures occur Since the 2020 COVID-19 pandemic the National Airspace within daily line operations. This points to a need to learn from System (NAS) has quickly returned to and exceeded the historical events to better understand which resilient behaviors historically high traffic volume levels of the pre-pandemic are present in the existing NAS and how they are associated era. The U.S. Bureau of Transportation Statistics reported with different safety critical events. NASA’s Aviation Safety that U.S. Airline traffic for July of 2024 hit an all-time high Reporting System (ASRS) is an established archive where of 91.8M passengers and August of 2024 was up 4.5% for pilots and controllers voluntarily and anonymously report the same month of the previous year [1]. This increasing safety incidents that they have observed on recent flights. The demand along with other emerging operations that include reports are comprehensive narratives that describe the events both piloted and unmanned aircraft requires a need to identify and conditions leading up to the safety event. Since the reports are narratives and self-reflective in nature, there are insights 19 pandemic and with the expected integration of emerging into the pilot’s thinking at the time of the event and the Unmanned Aircraft System (UAS) operations, there is a need author will often recount lessons learned from their experience. to identify what aspects of the current system demonstrates Further examination of an event leading up to and in response resilience. These behaviors are important to ascertain so that to a threat, can yield an understanding of the positive actions they can be considered when defining new requirements in that averted a more serious adverse outcome. This approach future concepts of operations.
of viewing a situation is commonly referred to as Safety II [3] B. Related Work as opposed to Safety I where the focus is more on the factors that went wrong during an event. These Safety II aspects One program proposed and implemented by American of the operations can be analyzed to highlight successful Airlines is the Learning Improvement Team (LIT) [5]. This elements with the objective to gain a better understanding framework has the ability to capture pilot proficiencies that and ultimately reinforce these systems and behaviors in future fall within the Safety II paradigm. In this program, pilot operations. Historically Safety I has focused on the threats, peer observers take notes in the jump seat during a flight to errors, and adverse conditions that lead to a safety incident or identify the pressures the pilots face during line operations.
accident and aims to better understand how to prevent such The observer is trained to recognize pilot responses and code an event in the future. However, since adverse safety events their proficiencies after the flight is completed. Although the are relatively rare in the operations a Safety II perspective data is a rich source for capturing these resilient qualities, has the potential to draw from significantly more examples. some shortcomings, are that there is a limit to how many Positive actions taken by the operators exemplify resilient observable flights that can be collected every year and not qualities that when objectively identified can help augment all airlines have the resources to implement such a program.
current safety monitoring systems and ultimately reinforce However, the idea proposed by LIT of first capturing a narra- operator proficiencies. Although ASRS reports are primarily tive that describes activities within the context of operations focused on the adverse event that the author is reporting and then assessing for coded proficiencies can be extended to on, these rich text narratives also offer the opportunity to other types of narratives to yield other forms of resiliency.
extract evidence of resilient behaviors and characterize them In [6] a novel way to examine Aviation Safety Reporting within four constructs: anticipate, monitor, respond, and learn . System (ASRS) reports was proposed using sentiment analysis Given the large corpus of narratives that exist within ASRS, and clustering of positive pilot actions to uncover themes of this poses a daunting task for a subject matter experts to resilient behavior. However, the approach is unsupervised in manually sift through, extract these examples, and organize the sense that it does not specifically categorize the behavior into actionable recommendations to improve safety. However, into a predetermined resilient framework. Advancements in recent advances in Large Language Models (LLMs) have Large Language Models (LLM) have expanded the capabilities demonstrated the ability to sift through large bodies of text to review and analyze considerably large amounts of text in and extract out key insights that the prompt engineer has a highly scalable way where it would have previously been requested. Once the resilient behavior evidence is extracted infeasible for subject matter experts. Additionally the LLMs the models can also be used to summarized and help organize have the ability to be instructed with prompts and use context the factors. This knowledge, can in turn, be shared within the to help guide the output towards the desired objective, making aviation safety community and increase the effectiveness of this task potentially more feasible to scale.
existing safety monitoring systems. This paper will describe C. Aviation Safety Reporting System the concepts of human resilience and demonstrate how ASRS can be leveraged with the assistance of LLMs to uncover NASA’s ASRS [7] database contains a repository of nearly 2 aspects of both similar and different resilient qualities across million safety reports. The reports are voluntarily submitted by various operations and incident types. members of the aviation community including pilots, air traffic controllers, and flight attendants from both commercial airline A. Resiliency operations [8] and general aviation (GA) [9]. Pilots that file Bertoni [4] describes a resilient framework concept where Safety Action Program (ASAP) reports and controllers that file qualities of human resilience fall under four categories: antic- Air traffic Safety Action Program (ATSAP) with their Safety ipate, monitor, respond, and learn . Everyday actions taken by Management System (SMS) can choose to send a de-identified the operators demonstrate these resilient qualities to maintain version of the report to ASRS to preserve the anonymity of safety. This is also apparent when work imagined does not the people and organizations involved. The reports consist of always translate to work done such as in the case where a text narrative describing a safety event of concern including decision making is required to manage new developments the conditions leading up to the event, what happened during in the operations. Mapping these actions to their respective the event, actions taken by those involved, and the outcome. In resilient categories can better capture the characteristics of addition to the report narrative, demographics such as whether work done which can, in turn, uncover positive behavior that a report involved GA aircraft or a commercial airline. ASRS is required to continue to maintain a safe NAS. The NAS analysts also provide labels for the types of events as coded has continued to grow year over year since the COVID- anomaly categories that can be used to group the reports for analyzing and understanding common characteristics around Although it can be useful to compare the performance between specific types of events. The actions taken by the operators in different LLMs, the objective of this paper is to demonstrate the reports as well as the thought process and self reflective an approach to extract resilient behavior and illustrate how the nature of the narratives hold potential examples of resilient tool can support Safety II objectives.
Apart from being able to provide answers to queries by behavior.
drawing from an extremely large corpus of training data, these models have the ability to answer specific questions within the scope of a provided set of text. In general, this {"Anticipate": methodology is called Retrieval-Augmented Language Models ["We expected some wake turbulence (RALM) [11] and in specific implementations, referred to following a heavy.", , → as Retrieval Augmented Generation (RAG). This technique "Thunderstorms were reported in the can improve the relevant retrieval performance since the in- area so we planned to carry , → formation extracted by the LLM is limited to the scope of extra fuel."], , → the provided text and less likely susceptible to hallucinations "Monitor": where the model infers an inaccurate response due to the fact ["We were listening to ATC that it is focused on next word completion, drawing from the communications to understand , → enormous variety of text and domains that it was trained upon which runways to expect.", , → that may or may not be relevant to the prompted task. Using "We were watching the temperature the RALM technique to focus the text completion task, the and inspecting the wings for , → user can prompt the LLM to perform a specific task such potential icing."], , → as identify resilient behavior constrained to only information "Respond": from the provided report narrative. Additionally, the prompt ["We followed the TCAS RA and can be augmented with in-context [12] examples of what the leveled off to avoid a loss of , → user would like to retrieve from the provided report. In the separation.", , → domain of resilient behavior, the user may provide a different "ATC set us up high on a final categories of resiliency in the form of an example sentence approach so we had to bleed off , → and category. This serves two functions: 1) the in-context some altitude and speed."], , → examples help guide the LLM’s response towards the desired "Learn": information that the user is requesting and 2) it defines a ["Next time I will make sure to ask structured output such as JSON schema formatting to help for clarification when in , → with parsing and organizing the output. Fig. 1 shows the in- doubt.", , → context augmented text provided with the LLM prompt that "I will be sure to look over the was used to guide the model to retrieve examples for the airport taxiways in case there , → resilient categories in each of the reports.
is an unexpected change to , → Prompting is an important component in guiding the LLM them."] , → to extract the information that the user is attempting to retrieve } and may take some iterations to refine. Prompt priming is one way to give the LLM a subject matter expertise background so that the responses align with the domain the user is asking the Fig. 1: In-context JSON schema LLM to respond. The prompt should provide some description of the RAG text the LLM will be examining. Additionally D. Large Language Models explicit instructions are needed to ensure the LLM does not stray from the task or provide extraneous information. The With the recent advancements in LLMs such as: OpenAI’s prompt for the Llama3.1 Instruct model to analyze each ASRS ChatGPT, Meta’s Llama, and Google’s Gemini, the potential report was: for extracting and summarizing insights from large bodies of text have now become more feasible, opening the door to You are an aviation expert with a human factors novel ways of investigating report narratives on safety events at background. Review a report written by a member scale. An advantage that the open sourced Llama model has is of the aviation community. The report describes a that it can be run locally with on premises hardware allowing safety event that took place. The report may or for tighter data control. Although ASRS reports are publicly may not contain resilient behavior exhibited by the available, the proposed approach described in this paper can be pilot or controller in the report. If the report does applied to sensitive ASAP reports as well. In the case of ASAP not contain resilient behavior leave the list blank.
reports, airlines require that the data not be shared externally If there is resilient behavior, quote each example.
to preserve the confidentiality of the pilots who have filed the The text following the EXAMPLE SCHEMA: are reports. For the purposes of this work the Llama3.1 8B Instruct some examples of resilient behavior but the findings Model with post training 6 bit quantization was used [10]. should not be limited to only these specific exam- ples. Provide output in a valid JSON format using the EXAMPLE SCHEMA. Only quote instances of resilient behavior. Do not summarize or explain why the cited text is characterized as resilient behavior.
Do not perform other tasks.
This LLM task was run independently for each ASRS narrative spanning January 1988 through December 2024, comprising a total of 254 , 126 narratives. For each narrative the LLM produced a set of extracted evidence sentences from the narrative that were categorized into the 4 different resilient cat- egories. Narratives would yield multiple categories of evidence sentences or none at all. The total number for each category were as follows: Respond ( 146 , 517 ), Learn ( 101 , 816 ), Monitor Fig. 2: Rate of categories of resilient behavior over time and ( 48 , 359 ), and Anticipate ( 244 ).
volume of ASRS reporting II. R ESULTS A. Resilient Category Summaries • Communicating with ATC and other aircraft Once the LLM identified the evidence sentences from each • Taking evasive action to avoid collisions report that support the four resilient behavior categories the • Following procedures for abnormal situations LLM was prompted to summarize the top 10 themes for each • Responding to system malfunctions or failures of these categories. The sentences were supplied to the LLM • Correcting navigation errors or misunderstandings as a list across all the reports for each of the categories.
• Coordinating with crew members and air traffic control For some categories the total number of sentences exceeded • Reporting incidents or issues to authorities the LLM’s token limit and the summarization task had to be Learn: performed in smaller batches. After multiple top 10 lists were compiled across the smaller batches, the LLM was used again • Verify Information to combine these summary list. The top 10 overall themes for • Double-check Procedures each of the four resilient categories are as follows: • Communicate Clearly Anticipate: • Stay Vigilant and Focused • Follow Standard Procedures • Weather forecasting and anticipation • Be Aware of Surroundings and Environment • Fuel management and planning • Monitor Systems and Instruments • Air traffic control and communication • Take Responsibility for Actions and Decisions • System malfunctions and failures • Improve Situational Awareness • Emergency procedures and protocols • Review and Analyze Data • Altitude and terrain considerations • Wake turbulence and airspeed deviations B. Temporal Trends • Communication with dispatch and maintenance Even though the exact date of the ASRS reports are de- • Alternate routes and airports planning identified, the month is still preserved. This allows analysts • Pre-flight inspections and preparations to look at seasonal trends or the ability to track emerging Monitor: topics such as the rise of the term COVID-19 in 2020. In • Monitoring ATC communications addition to the month there are demographics on what type of • Watching other aircraft’s position and movement operation was involved in the observed safety situation. This • Inspecting aircraft systems and instruments includes GA aircraft as well as commercial airline operations.
• Monitoring weather conditions Fig 2 show the trend over time of the rate of the three • Watching for potential hazards and obstacles resilient categories monitor, respond, and learn for both GA • Listening to radio communications for traffic information and commercial operations. Note: anticipate is not displayed • Monitoring fuel levels and consumption due to the low overall count. A three month moving average • Scanning for traffic in the vicinity window was used to smooth the trends. The gray bars in the • Watching taxiways and runways for potential hazards background and corresponding right hand y-axis illustrate the • Listening to radio communications with other aircraft or volume of reports over time.
ground control C. Commercial vs General Aviation Respond: • Responding to ATC instructions To compare the summary themes between the commercial • Correcting mistakes or errors airline reports vs the GA reports the evidence sentences • Declaring an emergency or taking evasive action were summarized for each type of operation by the LLM (a) Anticipate (b) Monitor (c) Respond (d) Learn Fig. 3: Venn diagrams showing resilient behavior between commercial vs general aviation reports separately. When comparing the two sets of themes there was Airborne Conflict: good overlap, with a majority of the themes being shared • Anticipate: Traffic conflicts and altitude management.
by both commercial and GA operations, however, there were • Anticipate: Expectation bias and loss of radio con- some unique to each across the four resilient categories. Fig.
tact/communication breakdowns.
3 illustrates the shared and unique themes across the four • Monitor: Monitoring TCASII system to understand traffic resilient categories as Venn diagrams.
situation • Monitor: Inspecting surroundings for potential traffic or D. Anomaly Categories hazards The ASRS archive has coded anomaly categories for each • Monitor: Maintaining visual contact with other aircraft report. The top five most frequent anomaly categories were air- and monitoring altitude and position of other aircraft borne conflict ( 21 , 778 ), weather turbulence ( 16 , 529 ), runway • Respond: Maintaining visual separation with other air- ground excursion/incursion ( 12 , 399 ), Near Mid Air Collision craft (NMAC) ( 11 , 677 ), and altitude deviation overshoot ( 10 , 664 ).
• Respond: Issuing traffic alerts or warnings The LLM was prompted to summarize the top 10 themes • Respond: Reporting near misses or conflicts within the four different resilient categories for each anomaly • Learn: Equipment issues and automation misuse category. Using the top 10 themes, the LLM was prompted • Learn: Airspace design/management and controller-pilot to determine which themes were shared between each of the coordination anomaly categories. TABLE I lists the anomaly categories NMAC: and overlapping resilient themes across anticipate, monitor, respond and learn . In addition to identifying where themes • Anticipate: Monitoring communications, anticipating pi- overlapped between anomaly categories, the LLM also iden- lot behavior, and assessing situational awareness.
tified themes that were unique to each anomaly category. The • Anticipate: Includes items related to recognizing potential following lists the anomaly categories and the unique themes conflicts and adjusting flight plans.
identified by the LLM. • Monitor: Making position reports and announcements on CTAF/Unicom frequency • Monitor: Using TCAS or ADS-B to track traffic • Respond: Go Around/Missed Approach • Respond: Increase Rate of Descent/Decelerate ASRS Anomaly Categories • Learn: Using technology, such as TCAS, effectively Air- Weather Over- Run- borne NMAC Turbu- • Learn: Reporting incidents and near-misses to authorities shoot way Conflict lence Overshoot: Anticipate • Anticipate: Factors that can contribute to an aircraft Weather x x overshooting its intended target or destination.
Conditions Air Traffic • Anticipate: Includes items like autopilot reliance, power Control x x management, and climb rate management.
Instructions • Monitor: Inspecting Autopilot Performance and Settings Safety x Precautions • Monitor: Monitoring Flight Instruments and Systems for Situational Potential Issues, Monitoring Autopilot and Flight Director x x Awareness Systems Monitor • Respond: Recovering from Autopilot Failure or Malfunc- Monitoring ATC x x x x x communications tion Scanning for • Respond: Correcting for Altimeter Setting Error traffic visually • Learn: Altitude Management Errors or using radar/ x x x x x TCAS/Traffic • Learn: Pilot Experience and Familiarity with Air- Awareness craft/Systems System (TAS) Watching other Runway: aircraft’s position x x x x x • Anticipate: Situational awareness during takeoff and land- and altitude ing.
Listening to CTAF • Anticipate: Includes items related to runway and taxiway communications x x x x x situations, fuel management, and emergency situations.
or Unicom • Monitor: Watching taxiway and runway layout frequency Monitoring • Monitor: Using airport diagrams, charts, or maps to weather understand taxi routes and runways conditions x x x x x • Respond: Misunderstanding or Miscommunication with and wind direction/speed ATC / Failure to Follow ATC Instructions Respond • Respond: Loss of Control or Directional Issues / Failure Communicating x x x x x to Follow Standard Procedures with ATC • Learn: Overconfidence and Arrogance Taking evasive action or corrective x x x Weather Turbulence: action • Anticipate: Reviewing weather forecasts, assessing poten- Changing course x x or altitude tial hazards, and planning for contingencies.
Reporting • Monitor: Monitoring weather radar incidents x x • Monitor: Inspecting aircraft systems for potential issues or near misses to ATC • Monitor: Coordinating with ATC Responding to • Respond: Declaring an emergency TCAS RA or x x • Respond: Requesting assistance or clarification traffic alerts Learn • Respond: Declaring minimum fuel or emergency fuel Communication x x x x x • Learn: Weather conditions Human Error or x x x • Learn: Airspace and navigation issues Mistake Situational x x III. A NALYSIS A ND D ISCUSSION Awareness Fatigue and One consistent reoccurring theme across the different con- Workload x x x x ditions used to split up the reports was ”communication”. All Management Procedural four resilient categories in the overall summaries (in section Errors II-A) had one form or another of communicating with ATC.
x x and Adherence In the comparison between commercial and GA operations (in to Procedures Fig. 3) all of the categories had some type of communication TABLE I: Top 5 anomaly categories and overlapping resilient as a shared theme. And in the anomaly categories (in TABLE behavior.
I) communication was also highlighted across the five anomaly categories in monitor, respond , and learn . This aligns well with resiliency since communication allows pilots and controllers to share knowledge and intention with other operators, making Transportation Safety Board accident investigation reports, decisions and actions clear and more predictable for other LIT observation narratives, or Line Operations Safety Audit actors in the airspace. Additionally the temporal trends report observation narratives. Other domains include the Confidential rates shown in Fig. 2 across both commercial and GA in the Close Call Reporting System [13] which is an ASRS style resilient categories of monitor, respond and learn have similar safety reporting system for rail road operations or the Patient patterns. Safety Reporting System [14]. All of these diverse types Another observation of note is that in the anomaly category of narratives may capture a different perspective of resilient analysis in TABLE I, all the monitor category themes were behavior, and highlighting these qualities can serve to improve shared across all five of the anomaly categories. These specific the overall resiliency of their prospective domains.
themes point to a more general theme of enhancing situational VI. A CKNOWLEDGMENTS awareness around what is happening in the airspace.
To be added out upon acceptance.
However, in contrast to having shared themes, there were some unique behaviors found primarily in commercial opera- R EFERENCES tions that were not highlighted in the GA reports. For example [1] U. B. of Transportation, “August 2024 u.s. airline traffic data up in both resilient categories: monitor and respond the theme 4.5% from the same month last year,” Bureau of Transportation of ”following procedures and checklist” was unique in the Statistics website, Nov 2024, accessed 2025-04-24. [Online].
Available: https://www.bts.gov/newsroom/august-2024-us-airline-traffic- commercial reports. This is a understandable, since airlines data-45-same-month-last-year have developed procedures and checklists as a cornerstone to [2] M. Stewart and B. Matthews, “Resilient strategies in commercial avi- their pilot training and culture.
ation,” in 22nd International Symposium on Aviation Psychology , ser.
Proceedings, Rochester, NY, 2023.
IV. C ONCLUSION [3] E. Hollnagel, R. L. Wears, and J. Braithwaite, “From safety-i to safety-ii: a white paper,” The resilient health care net: published simultaneously This proposed approach demonstrated a ways to leverage by the University of Southern Denmark, University of Florida, USA, and Macquarie University, Australia , 2015.
LLMs to extract evidence of resilient behavior within pilot [4] V. B. Bertoni, T. A. Saurin, F. S. Fogliatto, A. Falegnami, and and controller safety reports and that this same tool can R. Patriarca, “Monitor, anticipate, respond, and learn: Developing successfully organize the evidence into themes of resilience.
and interpreting a multilayer social network of resilience abilities,” Safety Science , vol. 136, p. 105148, 2021. [Online]. Available: Previously this type of analysis would be impractical for https://www.sciencedirect.com/science/article/pii/S0925753520305440 subject matter experts due to the amount of time it would [5] American Airlines’ Department of Flight Safety (2020). Trailblazers into take and volume of reports that would need to be examined Safety-II: American Airlines’ Learning and Improvement Team, A White Paper Outlining AA’s Beginnings of a Safety-II Journey.
to extract these themes. Using this process can yield insights [6] B. Matthews, I. Barshi, and J. Feldman, “An approach to identifying into operator proficiencies in routine situations, as well as aspects of positive pilot behavior within the aviation safety reporting in safety critical events. In these situation, both pilots’ and system,” in 42nd AIAA/IEEE Digital Avionics Systems Conference Proceedings , 2023.
controllers’ training and experience is evident in how they: [7] NASA Aviation Safety Reporting System, “Asrs database online,” 1) anticipate threats and pressures to the original flight plan https://asrs.arc.nasa.gov/search/database.html, 2023.
and/or procedures, 2) constantly survey the environment and [8] F. A. Administration, “14 cfr part 121 operating requirements,” Federal Aviation Administration website, 04 2025, accessed 2025-04- other actors to gather information about how the situation will 24. [Online]. Available: https://www.ecfr.gov/current/title-14/chapter- unfold, 3) take proper action and relay information to the I/subchapter-G/part-121 appropriate people involved, and 4) learn from the event to [9] ——, “14 cfr part 135 operating requirements,” Federal Aviation Administration website, 04 2025, accessed 2025-04- better understand how to prepare for a similar situation in the 24. [Online]. Available: https://www.ecfr.gov/current/title-14/chapter- future. These qualities exemplify how the human component I/subchapter-G/part-135 in a complex system demonstrates the flexibility to ensure [10] “bartowski/meta-llama-3.1-8b-instruct-gguf,” accessed: 2024-08- 07. [Online]. Available: https://huggingface.co/bartowski/Meta- the primary mission of keeping the aircraft and all the souls Llama-3.1-8B-Instruct-GGUF/blob/main/Meta-Llama-3.1-8B-Instruct- aboard safe. SMS programs can benefit from identifying Q6 K L.gguf what proficiencies exist in their own operations so that they [11] K. Guu, K. Lee, Z. Tung, P. Pasupat, and M. Chang, “Retrieval augmented language model pre-training,” in Proceedings of the 37th can ensure that these qualities are maintained either through International Conference on Machine Learning , ser. Proceedings of pilot bulletins, training, or other means to disseminate these Machine Learning Research, H. D. III and A. Singh, Eds., vol.
characteristics throughout the community.
119. PMLR, 13–18 Jul 2020, pp. 3929–3938. [Online]. Available: https://proceedings.mlr.press/v119/guu20a.html [12] O. Ram, Y. Levine, I. Dalmedigos, D. Muhlgay, A. Shashua, V. F UTURE W ORK K. Leyton-Brown, and Y. Shoham, “In-context retrieval-augmented Potential areas to extend this work may include compre- language models,” Transactions of the Association for Computational Linguistics , vol. 11, pp. 1316–1331, 11 2023. [Online]. Available: hensively examination of all anomaly categories and other https://doi.org/10.1162/tacl a 00605 conditional demographics to identify unique and consistent [13] Confidential Close Call Reporting System, “C RS database online,” resilient behavior across the various event report types. Other https://c3rs.arc.nasa.gov/, 2025.
[14] Patient Safety Reporting System, “Psrs database online,” steps would be to work with stakeholders to discuss how https://psrs.arc.nasa.gov/programoverview/index.html, 2025.
this could be incorporated into existing SMS programs for ongoing assessment of these proficiencies. The tool could be used to examine other report types such as the National