The Federal Aviation Administration and MITRE have unveiled a benchmark to facilitate the assessment of large language models, or LLMs, for aerospace tasks.
Table of Contents
ALUE Benchmark
MITRE said Wednesday the Aerospace Language Understanding Evaluation, or ALUE, benchmark is designed to streamline the inference and evaluation of LLMs using information specific to the aerospace domain.
ALUE supports open-source and domain-specific LLMs, custom datasets, user-defined prompts and various quantitative performance metrics. LLM evaluations are important in assessing a model’s performance and understanding its potential risks and limitations, including biases, hallucinations and privacy concerns.
“ALUE allows the FAA and the aerospace community to create a definitive library of diverse and specific aviation nomenclature and terms that will enable the agency to harness the power of AI for tools and tasks that will continuously improve safety and efficiency today and into the future,” said Kerry Buckley, vice president at MITRE and director of the Center for Advanced Aviation System Development.
MITRE noted that ALUE will help ensure artificial intelligence tools are fit to improve the safety of the National Airspace System.
Ongoing & Future Work Related to ALUE Benchmark
According to MITRE, ongoing work will continue to expand the ALUE benchmark’s scope to address more complex aerospace challenges, such as developing tasks for extracting complex data from charts.
For future work, the nonprofit organization said it expects the benchmark to integrate tasks that require LLMs to consult aircraft operational manuals and other external data sources to determine thrust and flap settings and other parameters under specific conditions.
Related Articles
Four U.S. senators have proposed a bipartisan bill that seeks to address price gouging by defense contractors and improve transparency in maintenance contracts. Sens. Elizabeth Warren, D-Mass., and Elissa Slotkin, D-Mich., introduced the Transparency in Contract Pricing Act of 2025 with Sens. Chuck Grassley, R-Iowa, and Joni Ernst, R-Iowa. In a statement published Wednesday, Warren said some contractors threaten U.S. military readiness by introducing unreasonably high prices into military contracts for spare parts and other basic materials. “Our common-sense bill puts these contractors on notice and requires them to justify major price increases — and they can bet we’ll double-check
Savannah River National Laboratory has named Dana Hewit as its deputy laboratory director for operations. Hewit, who previously led the Office of Integrated Performance Management at Oak Ridge National Laboratory, will officially join SRNL in mid-October, the South Carolina lab said Monday. In her new role, she will oversee the laboratory’s facilities and infrastructure and ensure that SRNL is operating safely, securely, efficiently and in compliance with the Department of Energy’s mission. “I am honored to join SRNL at such an exciting time for the laboratory,” stated the official “I look forward to working with the talented operations team and
Scientists and engineers at the Johns Hopkins Applied Physics Laboratory are developing a new approach that aims to speed up the process of designing, optimizing and testing metal components for the defense industrial base using artificial intelligence and robotics. APL’s TETRA Initiative APL said Wednesday the Transforming Evaluation and Testing via Robotics and Acceleration, or TETRA, effort is funded by the Pentagon’s Industrial Base Analysis and Sustainment program. “When developing materials for defense needs, it’s not just about the composition of the alloy or system — it’s also about how you shape, treat and refine it,” said Morgan Trexler, head