Systemic drift and independent oversight: why complex systems may need continuous monitoring as they evolve

Complex systems may continue functioning while their decisions and outcomes begin to diverge from their original purpose. Identifying these changes can be challenging when multiple processes interact without a clear point of failure. The oversight challenge may lie in recognizing such departures early enough for organizations to consider appropriate responses before unintended consequences emerge.

The National Institute of Standards and Technology’s (NIST) Center for AI Standards and Innovation examines this challenge in its 2026 report. The report discusses difficulties associated with observing AI systems in real-world settings, suggesting that post-deployment monitoring may provide valuable information alongside controlled evaluations. This perspective offers a useful starting point for considering how oversight can continue after a system enters operation.

Andy Hyman, creator of Marginal Point of Systemic Drift (MPOSD), has developed an external monitoring system informed by decades of experience with complex organizational processes. His work explores how internal safeguards and pre-deployment testing might be complemented by independent visibility into system behavior during live operation. The aim is to help organizations examine whether observed outcomes continue to correspond with a system’s stated purpose and boundaries. This is mirrored in Bill Gates’ interview with The Verge.

Hyman’s interest in systemic drift dates back approximately four decades, when he observed how changes in insurance product design, corporate governance, and tax law could produce outcomes that differed from their original intentions. His experience in the insurance sector informed his interest in how incentives and interconnected decisions influence results over time. He shares, “My observations prompted a question that continues to guide my work: Could we build an external system to monitor a complex system against its defined purpose and identify when its behavior begins to depart from it?”

Systemic drift describes the possibility that a system’s behavior may diverge from its original design as decisions, feedback loops, incentives, and operational conditions interact. Individual changes may appear reasonable in isolation, while their combined effects produce outcomes that warrant reconsideration. Because these influences can develop across multiple processes, identifying the relationship between an original intention and an eventual result may require sustained observation.

This possibility gives independent monitoring a potential role in responsible system governance. “Internal teams possess important knowledge of their systems’ objectives, requirements, and operating conditions,” Hyman states. “An external monitoring layer could complement that knowledge by comparing observed behavior with predefined purposes, boundaries, and expected actions.” He notes that such oversight may help identify emerging patterns while leaving interpretation and intervention to the appropriate people.

Artificial intelligence provides a particularly relevant example. As AI capabilities expand, evaluating how systems behave across changing environments can become increasingly complex. The International AI Safety Report 2026 discusses limitations in predicting real-world performance, malicious use, cyberattacks, and biological and chemical risks. Its findings provide context for examining how continued observation could expose these problems before they become damaging.

AI-generated content and criminal activity: AI systems are being misused to generate content for scams, fraud, blackmail, and non-consensual intimate imagery. Although the occurrence of such harms is well-documented, systematic data on their prevalence and severity remains unknown.

Interest in external scrutiny also appears in discussions about AI evaluation. Proposals involving external evaluators reflect consideration of how systems might be examined beyond the teams responsible for their development. Hyman’s work extends this monitoring question across complex systems, with AI representing a particularly relevant application given its evolving capabilities and potential downstream effects.

MPOSD has been developed to address this need for external visibility. Designed to monitor systems while they operate, it compares observed behavior with defined purposes, destinations, boundaries, and specified actions. “The system is intended to assess multiple signals over time, distinguish ordinary variation from potentially meaningful drift, and identify conditions that may warrant human attention,” Hyman explains. A visual dashboard is planned to communicate emerging indicators as the monitored system operates.

Its intended scope also includes verification evidence, feedback delays, institutional pressures, governance arrangements, and factors that may influence monitoring independence. Considering multiple signals could help reviewers interpret developing conditions within a broader operational context. When evidence appears insufficient for a reliable assessment, MPOSD is designed to flag the situation for review, allowing uncertainty to inform subsequent decisions.

Hyman likens the concept to an electrocardiogram, which observes physiological signals externally to support an assessment of changing conditions. “MPOSD is intended to observe a system without taking over its operation,” Hyman says. “This separation allows independent monitoring to complement internal safeguards while preserving human responsibility for interpreting findings and determining appropriate responses.”

According to Hyman, testing has demonstrated external observation of a running artificial intelligence system, differentiation between ordinary variation and meaningful drift, monitoring of specified actions, and assessment against defined references using multiple signals. Synthetic test cases have also been used to examine whether observed behavior corresponds with specified actions. These results offer an initial basis for further evaluation. According to Hyman, the system is ready for assessment in real-time operating environments.

Overall, as complex systems continue to evolve, responsible governance may benefit from treating oversight as an ongoing responsibility that extends throughout operation. “Organizations must consider how continuous, independent monitoring complements existing safeguards, supports informed review, and helps maintain accountability as system behavior changes,” Hyman remarks. Participating in the evaluation of approaches such as MPOSD offers an opportunity to examine how these principles may be applied in practice.