The Department of War, in coordination with the Office of the Director of National Intelligence, is seeking industry proposals for an evaluation harness and government-defined benchmarks that would enable rigorous, reproducible and vendor-agnostic testing of artificial intelligence systems against criteria specified by the government.
Sign up for the Potomac Officers Club’s 2026 Artificial Intelligence Summit on March 18 to hear Cameron Stanley, chief digital and AI officer at the Department of War, and other federal, defense and industry leaders discuss the impact of AI, machine learning and automation.
Table of Contents
What Features Are Required in the Evaluation Harness?
According to the commercial solutions opening notice published by the Defense Innovation Unit, the War Department is pursuing an evaluation harness that connects to AI models, facilitates evaluation workflows and measures their performance against benchmarks. The harness should support human-in-the-loop, agentic and adversarial evaluations. It should simulate an integrated environment to continuously test and monitor an AI model performance in challenging settings. Furthermore, the harness should generate evaluation reports and manage benchmark execution.
What Standards Must the New Benchmarks Meet?
Vendors must provide methodologies for creating benchmarks across unclassified, secret and top secret workflows that are resistant to gaming, adaptable as requirements and AI models evolve, and supported by training materials. These benchmarks should identify capabilities for particular missions, break those capabilities into measurable tasks and create realistic evaluation scenarios. They should also define clear scoring criteria, establish fair performance baselines using open models and ensure benchmarks are valid, reliable and capable of distinguishing different levels of performance.
Why Is the Government Expanding AI Evaluation Capabilities?
The government is pursuing new evaluation systems to address the rapid advancement of AI technologies. The new infrastructure should be able to evaluate newly released AI models against mission-specific benchmarks. In addition, the system should assess human-machine collaboration to determine whether joint operations yield better mission outcomes than either humans or automated systems alone.
The effort, dubbed “Mystic Depot,” follows calls by Pentagon leadership to accelerate the adoption of AI across warfighting and administrative operations, DefenseScoop reported. Interested vendors can submit their responses to the CSO by March 24.
Related Articles
Daniel Morgan, the Department of Transportation’s inaugural chief data officer, has been appointed as DOT’s chief product and technology officer and has joined the Senior Executive Service. In a LinkedIn post, Morgan said the move marks a new chapter after more than a decade overseeing DOT’s enterprise data strategy. He also highlighted the collaborative nature of his work, noting that many of the recognitions earned during his tenure reflected team accomplishments across the department. “Data will always be a team sport, and I am proud of all we got done – inside DOT, across government and with the Chief Data
The 2026 Wash100 Popular Vote race is deep into the middle of the competition with supporters across the government contracting community continuing to rally behind their favorite leaders. The Wash100 Popular Vote is an annual contest where the GovCon community can show their support for the most influential government and industry leaders recognized by the prestigious Wash100 Award. Hung Cao, under secretary of the Navy, has widened his lead and now holds 2,206 votes, maintaining a commanding first-place position as the competition heads deeper into its latest week. But the race is far from over—and several leaders remain within striking distance
The U.S. Army is expanding its use of enterprise contracts to streamline procurement, promote competition and leverage enterprisewide buying power as part of efforts to modernize acquisition. The Army’s push toward enterprise contracts reflects ongoing changes in how the service approaches acquisition and modernization. Reserve your seat at the Potomac Officers Club’s 2026 Army Summit on June 18 to hear from government and industry leaders discussing priorities, technology developments and other trends shaping the military service. The service said Wednesday it has awarded 14 enterprise contracts in the past eight months, consolidating 118 separate agreements into unified vehicles and enabling