Human in the Loop: Why human supervision is not a usability feature
Artificial intelligence thrives on data. In medical technology, however, it is not the quantity of data that determines the success of an AI system, but rather its quality, traceability and regulatory compliance. This is precisely where many AI projects stall, often not at the outset, but shortly before approval or during later operation.
Data in medtech is not a neutral raw material. It is generated in clinical contexts, is tied to devices, processes, and patient groups, and is subject to strict legal requirements. Every dataset is the result of deliberate decisions. Which patients were included? Under what conditions or in what environments was the data collected? What clinical standards were in effect at that time?
These questions are crucial because they directly shape the performance of an AI model. A system can function excellently during training and yet fail in real-world use if training and deployment conditions do not align. This phenomenon is described as “dataset shift,” one of the most common causes of performance degradation in clinical AI during field operations.
The EU AI Act explicitly addresses this risk through regulation. For high-risk AI, manufacturers are required to systematically document their training, validation, and test data. The origin, composition, processing, and known biases must be described in a transparent manner. This transforms data governance from a recommended practice into a mandatory requirement.
What matters here is not that datasets are perfect. What is crucial is that their limitations are made transparent. From a regulatory perspective, the problem is not data with limitations, but data whose limitations are not documented. The white paper therefore recommends structured documentation approaches that create precisely this transparency and make the quality of the data foundation traceable.
At the same time, the GDPR remains a central framework. Health data falls under the categories of data requiring special protection, and the white paper makes it clear that true anonymization is often nearly impossible to achieve in medical practice. Image data, time series, or clinical texts can still allow inferences about individual persons even after the removal of obvious identifiers. Data protection is thus not a formal step at the end of a project, but a design factor for AI systems.
Many AI projects fail precisely at this stage. Not because the model is technically immature, but because the legal framework for data use remains unclear or data protection impact assessments are conducted too late. The white paper clearly demonstrates that a solid legal foundation for data use is just as important as the technical quality of the data itself.
Data governance becomes particularly relevant beyond the approval phase. AI systems evolve during operation. New generations of devices, changes in patient populations, or updated clinical standards and workflows lead to shifts in data distributions. Without structured post-market monitoring, these changes remain undetected for a long time. Performance declines only become apparent when they are clinically relevant and by then it may already be too late.
That is why market surveillance, drift detection, and secure further development are explicitly understood as the foundation of data governance. This connection becomes particularly clear when it comes to continual learning. Systems intended for further development after approval must define their change limits in advance. Predetermined Change Control Plans enable exactly that, but only if training data, model versions, and performance metrics are properly versioned and documented.
Ultimately, it becomes obvious that data in medtech is not a byproduct of AI development. It is an integral part of product safety. By structuring data early on, ensuring its legal compliance, and continuously monitoring it, companies lay the foundation for AI systems that are not only technically sound but also remain safe and reliable in the long term.
The white paper “AI in Medical Devices: Secure Integration, Validated Approval” explains how data governance, GDPR, and AI Act requirements interact in practice.
