A medical AI label is not automatically a fact. It is the output of an evidence source, a protocol, qualified reviewers, and a disagreement process that must remain traceable.
By ModAstera
01 Sep 2026
Medical AI teams often describe their target labels as “ground truth.” The phrase sounds reassuring. It suggests that every image, waveform, specimen, or record contains one correct answer and that the dataset simply reveals it.
In practice, a label may come from a pathology result, culture, follow-up event, report, billing code, device measurement, expert reader, panel consensus, or an earlier model. Each source answers a different question. Each has its own timing, missingness, error modes, and relationship to the intended use.
That does not make supervised medical AI impossible. It means the reference standard is part of the system being designed. If the label process is vague, a model can optimize against an unstable target and still produce an impressive metric. If the process is explicit, the team can understand what the metric means, where disagreement comes from, and whether the evidence supports the claim.
A reference standard is the method used to determine the target condition or outcome against which a model is trained or evaluated. Sometimes it is close to a directly observed fact. Often it is a structured judgment under incomplete information.
Consider four examples:
These labels are not interchangeable. A report label may reflect clinical workflow and incomplete documentation. A panel label reflects its reader composition, instructions, available context, and consensus rules. An outcome label depends on the observation window and event definition. A mixed reference standard may introduce systematic differences between cases.
The first discipline is therefore linguistic: treat “ground truth” as shorthand, then document the actual reference standard precisely.
The label protocol should not begin with the annotation tool. It should begin with the decision the model is intended to support.
Define:
A unit mismatch can create misleading evidence. A model may operate on images while the outcome is defined at the patient level. Several images from one patient may share one label. If those images are treated as independent facts, both split integrity and uncertainty estimates can fail.
This is why reference-standard design and leakage-safe train, validation, and test splits belong in the same conversation.
The strongest available evidence depends on the task. Histopathology may be appropriate for one diagnostic question, longitudinal outcome for another, and expert visual assessment for a third. No source becomes a universal gold standard merely because it is expensive or clinically familiar.
For each label, record:
Proxy labels deserve special caution. Report keywords, diagnosis codes, procedure codes, and treatment decisions can provide scale, but they may encode documentation habits, billing rules, local practice, or access to care. They can be useful targets when the intended task matches what they represent. They should not be described as a different clinical fact.
A label guideline is an executable specification for human judgment. It should include inclusion and exclusion rules, boundary conventions, handling of poor-quality inputs, uncertain states, examples, counterexamples, and escalation criteria.
Version the protocol. Record which cases were labeled under each version. When a definition changes, do not silently overwrite history. Decide whether earlier labels remain valid, require re-review, or need a transformation that can be explained and reproduced.
Pilot the protocol on a deliberately difficult sample before launching a large annotation round. Include borderline cases, artifacts, uncommon presentations, and cases where relevant context is missing. The purpose of the pilot is not to maximize agreement. It is to expose ambiguous instructions while changes are still inexpensive.
Annotation tools can accelerate this process, but tooling is not the reference standard. Click assistance, pre-labels, and auto-annotation may reduce manual effort. They can also anchor reviewers toward a machine suggestion. If assistive features are used, record when and how, and evaluate whether the workflow changes reviewer behavior.
“Expert labeled” is not enough. Expertise is task-specific.
Record reviewer qualifications, relevant training, experience with the modality or condition, protocol training, calibration exercises, and conflicts of interest where relevant. Also record the information available during review:
More context can improve clinical judgment while making the label less independent from the data source being evaluated. Less context can improve blinding while creating an artificial task. The correct design depends on intended use. The important point is to choose deliberately and report it.
When two qualified reviewers disagree, the disagreement is information. It may reveal a vague guideline, poor input quality, a difficult phenotype, a continuous boundary forced into a binary category, or genuine limits in current knowledge.
A defensible process defines the next state before review begins. Options include:
These methods answer different questions. Majority vote summarizes a group. Adjudication produces a final operational label under a specified rule. Consensus may improve consistency but can erase the original distribution of opinions. None automatically reveals an unknowable biological truth.
Keep the raw reader labels even when the model uses an adjudicated target. Multi-rater data can support sensitivity analyses, disagreement maps, probabilistic targets, or separate evaluation of clear and ambiguous cases. Research such as Multi-rater Prism illustrates why variation across expert labels can be modeled rather than discarded.
Forcing every case into positive or negative may make a dataset easier to train on while making the evidence less honest.
Uncertainty can appear at several levels:
These states should not all collapse into one label. Define which cases are excluded, deferred, retained as uncertain, or resolved using more evidence. Report how those choices affect cohort size and class balance.
This principle matches a broader human-in-the-loop design: uncertainty should route work to an appropriate reviewer or process, not disappear behind an automated answer.
Development labels are often iterative. Teams refine instructions, correct obvious errors, and inspect model failures. That is useful during development, but the same feedback can contaminate an evaluation set.
For a locked evaluation cohort, define who may access cases, when labels can change, how corrections are authorized, and whether the model output is hidden from readers. Keep any post hoc correction in an audit trail and report analyses before and after the change where appropriate.
This does not mean evaluation labels can never be corrected. It means a correction is a controlled evidence event, not a quiet edit made after seeing an unfavorable result.
A headline model metric cannot be interpreted without understanding the label process behind it. Useful label-system measures may include:
Agreement is not validity, and one statistic is not appropriate for every task. A high agreement rate can reflect an easy cohort. A lower rate can reflect a difficult but clinically important boundary. Use measures that fit the label type and report the underlying counts.
The CLAIM 2024 update and STARD 2015 are useful reporting anchors because they push teams to make methods visible enough for critical appraisal. The goal is not checklist theater. It is to make clear what was measured and against what standard.
For every final label, a reviewer should be able to resolve:
This is where annotation workflow and data lifecycle infrastructure meet. MAEA is designed to keep samples, labels, datasets, and downstream experiment records connected. The platform can support that chain, but qualified teams still have to define the intended use, reference source, protocol, reviewer process, and acceptance criteria.
Label lineage also matters during external validation. A performance difference between sites can reflect population or device shift, but it can also reflect a changed reference method. Without lineage, those explanations are difficult to separate.
Before training or evaluating a medical AI model, ask:
If several answers depend on memory, spreadsheets, or overwritten files, the risk is not only operational. It changes what the model result means.
A defensible reference standard does not eliminate uncertainty. It makes the uncertainty, decisions, and evidence path inspectable. That is a stronger foundation for evaluating medical AI beyond accuracy, and for deciding whether a model is ready for the next stage of evidence.
Sizing a medical AI dataset is arithmetic. Keeping that dataset provable, versioned, and reusable across years of development is the part that fails, and it is what MAEA is built to hold together.
There is no single number. There is a data budget, a timeline, and an endpoint that finally makes the number computable. A full report on sizing evidence for regulated medical AI.
A practical guide to testing whether a locked medical-AI system transfers beyond its development data, including cohort design, thresholds, subgroup analysis, workflow evaluation, and failure review.
