The National Institute of Standards and Technology is asking industry, government and research stakeholders to weigh in on a new draft framework aimed at improving how language models are evaluated through automated benchmarking.
NIST said Friday that its Center for AI Standards and Innovation, or CAISI, released an initial public draft of NIST AI 800-2, “Practices for Automated Benchmark Evaluations of Language Models,” and is accepting public comments through March 31.
The Potomac Officers Club’s 2026 Artificial Intelligence Summit on March 18 will bring together federal, defense and GovCon leaders to discuss how AI is being integrated into mission and enterprise environments. Through keynotes and panels, the event will highlight practical approaches to scaling AI, modernizing legacy systems, and building the data and infrastructure foundations needed for responsible adoption across government. Register now.
Table of Contents
Why Is NIST Issuing Guidance on Automated Benchmark Evaluations?
Automated benchmark evaluations are increasingly used to support AI procurement and deployment decisions, particularly when organizations face limited time or resources. However, NIST cautions that benchmarks are not suitable for every evaluation need. This reflects a growing concern that while these tests have become essential tools for assessing artificial intelligence performance, consistent standards for ensuring valid, reproducible and transparent results are still in their infancy.
The draft organizes guidance around three areas: defining evaluation objectives and select benchmarks, implementing and running evaluations, and analyzing and reporting results. It notes that automated benchmarks work best when tasks are structured, verifiable and stable over time, but are less effective for subjective, dynamic or human-in-the-loop evaluations.
What Does CAISI Recommend for Benchmark Design and Reporting?
One of the central recommendations is that evaluators should begin by clearly documenting what they are trying to measure and how results will be used.
CAISI emphasizes that evaluation objectives should specify both the intended use of the measurements and the underlying capability or construct being assessed. It also urges organizations to carefully select benchmarks, documenting what each benchmark actually measures and whether it directly aligns with the evaluation goal or serves only as a proxy.
Beyond benchmark selection, CAISI highlights the importance of evaluation protocol design — the operational procedures that shape results.
The draft identifies several emerging principles, including:
- Comparability across models
- External validity tied to real-world use
- Cost control, since a higher reasoning effort can inflate performance safeguards against evaluation “cheating,” such as models searching for answers online
CAISI notes that providing internet access during evaluations is a particularly consequential decision, since it can introduce contamination and undermine benchmark integrity.
The draft also calls for stronger norms around statistical analysis and reporting. It recommends that evaluators quantify uncertainty through confidence intervals or standard errors, rather than treating benchmark scores as absolute measures. CAISI further advises that organizations should make qualified claims and avoid overgeneralizing benchmark outcomes beyond their intended scope.
The draft reflects CAISI’s growing mission as the federal government’s primary industry-facing hub for testing frontier AI models. Recent CAISI initiatives include seeking AI experts to work on national security risk evaluations, AI red-teaming and secure deployment guidance as part of the Trump administration’s AI Action Plan.
NIST has also separately requested industry input on security risks and safeguards for agentic AI systems, highlighting threats such as backdoor attacks and data poisoning.
Related Articles
President Donald Trump on Tuesday signed a consolidated appropriations measure into law to end a partial government shutdown and fund the Department of War and other federal agencies through the end of September, according to a White House notice. The signing came shortly after the House approved the measure. Breaking Defense reported that the lower chamber voted 217-214 on Tuesday to pass the funding package, which includes a short-term funding measure for the Department of Homeland Security. The Senate had sent the package of five fiscal year appropriations bills to the House after approving the measure Friday by a 71-29
The Senate has confirmed Maj. Gen. Luke Cropsey for promotion to lieutenant general, elevating him to a three-star role as he becomes the military deputy in the Office of the Assistant Secretary of the Air Force for Acquisition, Technology and Logistics. The Potomac Officers Club’s 2026 Air and Space Summit will bring together leaders from the U.S. Air Force, U.S. Space Force and industry to discuss current priorities across the air and space domains. Taking place July 30, the event will feature keynotes, panel discussions and opportunities to connect with peers while exploring emerging challenges and technologies shaping the future
The Department of War has selected 25 companies to compete in the initial phase of an acquisition reform initiative aimed at rapidly fielding low-cost, small unmanned aerial systems designed to perform one-way attack missions. DOW said the selected vendors will participate in the Drone Dominance Program’s initial evaluation phase, known as the Gauntlet, which will kick off on Feb. 18 at Fort Benning in Georgia. The Pentagon’s Drone Dominance Program highlights how rapidly evolving acquisition efforts continue to shape the defense landscape. These broader trends in air and space operations will bring government and industry leaders together at the Potomac