SafeAeroBERT: Towards a Safety-Informed Aerospace-Specific Language Model

SafeAeroBERT: Towards a Safety -

Informed Aerospace - Specific

Presentation Title

Language Model

1, 2 Sequoia Andrade Hannah Walsh Author 1 2 HX5 LLC. ; NASA Ames Research Center Company/Organization th th AIAA Aviation, June 12 - 16 2023 Conference Name, Conference Dates Conference Location This material is a work of the U.S. Government and is not subject to copyright protection in the United States Published by the American Institute of Aeronautics and Astronautics, Inc., with permission.

Motivation

➢ Aviation generates many documents, such as compliance documents, mishap safety reports, and requirements ➢ The jargon in aviation documents limits the effectiveness of state - of - the - art machine learning model ➢ BERT models [1]: ➢ Have been successfully trained for various domain - specific use cases (e.g., biomedical [2], legal [2], finance [3]) ➢ Show superior performance on nearly all natural language processing (NLP) tasks Figure 1: Example ASRS [5] worksheet

Natural Language Processing

➢ In Aviation ➢ Classification for cause identification and severity categorization ➢ NER for NOTAM digitalization and FMEA extraction ➢ QA and Summarization ➢ The proposed SafeAeroBERT model could improve aviation NLP results by learning aviation jargon

BERT Model

➢ Bi - directional masked language model ➢ Provides contextual meaning of words within a sentence ➢ Next sentence prediction ➢ Provides contextual meaning of sentences in a document ➢ Pre - training: upfront training on millions of documents to learn word context ➢ Fine - Tuning: additional training on fewer documents for a specific task, such as classification

Method

Pre - Training: Data

➢ ASRS [5] ➢ 262,935 documents ➢ 5 possible narrative text sections ➢ NTSB ➢ 161,460 documents ➢ 6 possible narrative text sections ➢ Total of 2,283,435 narrative text sections

Pre - Training

➢ Training: ➢ Trained for > 2 epochs ➢ Batch size = 8 ➢ Gradient accumulation = 16 ➢ Total batch = 64 ➢ 13,000 steps ➢ ~ 2 months run time on 3 GPUs

Fine - Tuning

Contributing Factor Train Validation Test

➢ Classification

Aircraft Present 2083 260 260 Not Present 5917 740 740

➢ Compared SciBERT , base

Total 8000 1000 1000

BERT, and SafeAeroBERT

Human Factors Present 3610 451 451

➢ 5 epochs Not Present 4390 549 549

Total 8000 1000 1000

➢ Batch size of 32

Procedure Present 1172 146 146

➢ 3 GPUs

Not Present 6828 854 854 Total 8000 1000 1000

➢ ~1.2 hour to train each model

Weather Present 1035 129 129

for each classification task

Not Present 6965 871 871 Total 8000 1000 1000

Classification Results

Contributing Metric BERT SciBERT SafeAeroBERT

➢ SafeAeroBERT outperforms BERT

Factor Aircraft Accuracy 0.747 0.726 0.740

and SciBERT on procedure and

Precision 0.716 0.691 0.548

weather document classification

Recall 0.747 0.726 0.740 F - 1 0.719 0.699 0.629

➢ BERT outperforms SciBERT and

Human Accuracy 0.608 0.557 0.549

SafeAeroBERT on aircraft and human

Factors Precision 0.618 0.586 0.527 Recall 0.608 0.557 0.549

factors

F - 1 0.572 0.426 0.400 Procedure Accuracy 0.766 0.755 0.845

➢ SafeAeroBERT tends to have a higher

Precision 0.766 0.762 0.742

accuracy and recall, indicating that it

Recall 0.766 0.755 0.845 F - 1 0.766 0.758 0.784

has better success at identifying all of

Weather Accuracy 0.807 0.808 0.871

the positive cases, potentially at the

Precision 0.803 0.769 0.759 Recall 0.807 0.808 0.871

cost of a lower precision and higher

F - 1 0.805 0.788 0.811

false positive rate

Discussion

➢ SafeAeroBERT performs ➢ Entity Relation, Information better than both SciBERT retrieval, and Summarization still and BERT on some need to be tested downstream tasks ➢ Ideally, SafeAeroBERT would ➢ SafeAeroBERT can be used also include NER in pre - training as a backbone in a ➢ Future aviation specific BERT models must identify the set of knowledge management relevant entities and build a training toolkit set ➢ Knowledge Discovery ➢ Intended for safety applications ➢ Information Retrieval

Conclusion ➢ In this research, we pre - trained a custom safety - informed aviation - specific large language model on NTSB and ASRS reports ➢ The resulting SafeAeroBERT model shows promise for improving NLP applications in aviation when used in place of standard BERT or SciBERT ➢ Will be made publicly available in the Future via NASA GitHub

Contact Information

➢ Additional Resources: ➢ Emails: ➢ Robust Software Engineering: ➢ Sequoia.r.Andrade@nasa. gov https://ti.arc.nasa.gov/tech/rse/ ➢ H annah .s .Walsh @nasa.gov

References

[1] J. Devlin, M. - W. Chang, K. Lee and K. Toutanova, "BERT: Pre - training of Deep Bidirectional Transformers for Language Understanding," arXiv , 2018.

[2] J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So and J. Kang, " BioBERT : a pre - trained biomedical language representation model for biomedical text mining," Bioinformatics, 2019.

[3] I. Chalkidis , M. Fergadiotis , P. Malakasiotis , N. Aletras and I. Androutsopoulos , "LEGAL - BERT: The Muppets straight out of Law School," in Findings of the Association for Computational Linguistics: EMNLP 2020, 2020 [4] D. T. Araci , " FinBERT : Financial Sentiment Analysis with Pre - trained," arXiv , 2019.

[5] B. L. Hooey, "ASRS Program Briefing," NASA, 2021.