The National Institute of Standards and Technology has released a new report examining the difficulties organizations face when monitoring artificial intelligence systems after deployment, an increasingly important issue as AI tools are adopted across government and industry.
Published on Monday, the “NIST AI 800-4: Challenges to the Monitoring of Deployed AI Systems” was developed by the Center for AI Standards and Innovation following practitioner workshops and an extensive literature review aimed at mapping the emerging landscape of post-deployment AI monitoring, an area the agency says remains fragmented despite growing demand for real-world oversight of AI technologies.
The Potomac Officers Club’s 2026 Artificial Intelligence Summit on March 18 will bring together federal, defense and GovCon leaders to discuss the policies, standards and operational considerations shaping AI adoption across government. Register now to hear keynotes and practitioner-led panels on deploying AI responsibly and scaling emerging technologies in mission environments.
Why Monitoring AI Systems Matters
According to NIST, AI systems behave differently from traditional software because they can evolve, degrade or produce unexpected outputs over time. These characteristics make continuous monitoring essential for ensuring systems operate as intended once deployed.
The report notes that post-deployment monitoring can help organizations detect issues such as performance drift, security vulnerabilities and unintended impacts on users, all of which could undermine trust in AI technologies.
What Are the Key Monitoring Categories?
Researchers identified six major categories of AI monitoring that organizations should consider when evaluating deployed systems:
- Functionality: Tracks performance to ensure the system consistently executes its intended tasks.
- Operations: Monitors infrastructure health and service reliability.
- Human factors: Evaluates user interaction, output transparency and the practical utility of AI insights.
- Security: Identifies vulnerabilities to adversarial attacks or unauthorized misuse.
- Compliance: Ensures adherence to evolving regulations, industry standards and internal governance.
- Societal impact: Assesses the large-scale consequences of AI deployment.
What Challenges Did Researchers Identify?
Despite growing interest in monitoring AI systems, the report identifies multiple obstacles that complicate implementation.
Among the key challenges are limited research on human-AI feedback loops, fragmented system logging across distributed infrastructure and difficulty detecting performance degradation over time.
Researchers also point to broader systemic barriers, including the absence of widely accepted standards for monitoring tools, limited mechanisms for sharing incident information and a shortage of qualified AI specialists.
NIST said the findings highlight areas where further research and collaboration could strengthen oversight practices.
