How MAEA Solves the Data Lifecycle Problem

Working out how much data a medical AI model needs is, in the end, arithmetic. Pick the claim, pick the endpoint, decide how much statistical uncertainty is acceptable, and the required number of target events falls out of a formula. We walk through that calculation in detail in the companion article, how much data is needed to train a medical AI model.

The arithmetic is not what fails.

What fails is everything the arithmetic quietly assumes: that you can say where each sample came from, who labelled it and under which protocol version, which model version was trained on it, whether it ever touched the test set, and whether you can still answer all of those questions two years later when a modification needs fresh sequestered evidence.

That is the data lifecycle problem. It is the reason well-funded medical AI programs stall between a promising prototype and a defensible product, and it is the problem MAEA — ModAstera's Medical AI Engineering Agent — is built around.

The problem, stated precisely

A medical AI program has to keep eight things simultaneously true, from first data intake through post-market change:

1. Every sample has a known origin and a known permission. Which site, which system, which date range, under which consent or legal basis, with which rights for commercial use, transfer, and retention for future retraining.
2. Every label has a known author and a known protocol. Which annotator, which version of the annotation guideline, whether it was single-read, double-read, or adjudicated, and how disagreement was resolved.
3. Every dataset is a named, versioned object. Not a folder, not a shared drive, not "the export that came through in March".
4. Split boundaries hold at the correct unit of independence — permanently. Patient, case, specimen, encounter, site, or time period, whichever the claim requires, enforced every time the data is reused rather than re-derived by hand.
5. Coverage is measurable against the claim. Not just total N, but counts by site, device, protocol, demographic stratum, disease subtype, and failure mode.
6. Every result traces back to an exact state. A metric on a slide should resolve to a dataset version, a configuration, a code state, and a run.
7. The frozen candidate is portable. What was trained on one machine has to be reproducible somewhere else, including by someone who was not in the room.
8. Production behaviour is observable and comparable to the evidence baseline. Otherwise monitoring is a dashboard, not a control.

Any one of these is easy in isolation. The difficulty is that they must all hold at once, across years, across staff turnover, and across a toolchain that was never designed to preserve them.

Why teams lose it

The standard medical AI stack is assembled from parts that each do one job well and none of which carry context to the next: a PACS or LIS export, a spreadsheet of case identifiers, a labelling tool, a bucket of object storage, a set of notebooks, a training script, and eventually a serving stack.

Each handoff drops a fact. The export loses the query that produced it. The spreadsheet loses the consent basis. The labelling tool loses which guideline version was active. The notebook loses which dataset version it read. By the time a result is on a slide, no one can reconstruct the chain — and reconstructing it is exactly what a regulator, a notified body, a clinical partner, or a diligence process will ask for.

This is not a marginal overhead. Willemink and colleagues estimate that preprocessing and organising data for algorithm ingestion accounts for 80% or more of the effort in a medical imaging machine learning project. That is where most of a medical AI budget actually goes, and it is spent on work that produces no evidence unless the resulting structure is preserved.

The failure modes are familiar:

• Leakage discovered late. Related images, slides, or encounters from one patient end up on both sides of an evaluation boundary, and the discovery invalidates months of results. We cover the mechanics in data leakage in medical AI.
• A test set that quietly became a development set. It was consulted during tuning often enough that it no longer estimates anything.
• Annotation rounds that cannot be compared. The guideline changed between rounds and nobody recorded when.
• A number that cannot be reproduced. The model that produced it was overwritten, or the dataset that produced it was mutated in place.
• A coverage gap found at the end. The aggregate N was large; the claimed subgroup had 40 cases.
• A modification with nowhere to test it. The only untouched data left is the test set already used in the submission.

How MAEA is organised around the lifecycle

MAEA treats the lifecycle as the product, rather than treating model training as the product and the lifecycle as paperwork around it. Each stage below is a place where a fact is created and where MAEA's job is to make sure that fact survives to the next stage.

1. Intake: data becomes an object, not a folder

Samples enter MAEA as members of named datasets, with tags, slice extraction for volumetric and multi-frame inputs, and per-dataset access control. Storage configuration, allowed hosts, API keys, and organisation roles are part of the same system rather than a separate infrastructure concern.

The practical effect is that "which data is this" has an answer that does not depend on a filename convention or on whoever set up the export.

2. Labelling: the reference standard is defined where the data lives

MAEA's annotation workspace covers polygon and bounding-box annotation, video and multi-frame controls, DICOM windowing, orientation handling, default label sets, and keyboard-driven review — alongside assistive click-to-annotate and auto-annotation, and direct mask upload for labels produced elsewhere.

Two things matter more than the drawing tools. First, labels stay attached to the sample and the dataset rather than living in a parallel system. Second, assistive annotation is treated as an accelerator for the human reference standard, not as a substitute for it. The distinction is what keeps the resulting labels usable as evidence.

3. Composition: training datasets are declared, not assembled by hand

Training datasets in MAEA are built from filters, tags, and sample queries over a dataset that already exists as an object. The composition is a declaration that can be inspected, re-applied, and reasoned about, instead of a one-off copy of files into a directory.

This is where split integrity is won or lost. A declared composition can hold a boundary at the patient or site level every time it is reused. A hand-assembled folder holds it exactly once, until the next person rebuilds it.

4. Coverage: dataset analysis makes the sparse strata visible

Dataset analysis and sample-intelligence views let a team see the distribution of a dataset — target values, tags, sources, and sample-level detail — before training rather than after a disappointing subgroup result.

This is the operational answer to the point that matters most in dataset sizing: aggregate N hides sparse strata. A dataset of 50,000 cases with 40 examples of a claimed subgroup supports a claim about the population and nothing about the subgroup. That gap is only actionable if someone can see it while there is still time to collect.

5. Evidence: pipelines and experiments carry their own history

Training and evaluation run as pipelines with configurable metrics, fine-tuning, result visualisation, and side-by-side pipeline comparison. Above them, MAEA's experiment layer composes, compiles, runs, and repairs experiment configurations, and keeps an artifact timeline, versioned configurations, insight charts, and an inspectable configuration record for each run.

The point is not that the platform trains models. Many things train models. The point is that a metric on a chart resolves to the configuration and the run that produced it, without anyone having to remember.

6. Freeze and portability: snapshots

Pipeline snapshots can be exported and imported. A frozen candidate is therefore a transferable object — reviewable by a partner, reproducible on other infrastructure, and retrievable later when a change needs to be compared against the version that was actually validated.

Design freeze is a documentation event in most stacks. Here it is an artifact.

7. Deployment and monitoring: the evidence baseline stays connected

Deployments, deployment usage, API call volume, and latency metrics live in the same system as the datasets and experiments that produced the deployed model. Monitoring is a comparison against a known baseline rather than an isolated dashboard, which is the distinction that makes drift and workflow signals actionable. We expand on the monitoring side in AI model monitoring for regulated workflows.

8. Change control: modification evidence has somewhere to come from

Because compositions are declared, configurations are versioned, and snapshots are portable, a modification can be evaluated against sequestered data that is demonstrably independent — rather than against whatever data happens to remain. That is the practical precondition for a controlled change process, and it is the part teams most often discover they do not have at the moment they need it.

What this changes about the data budget

The sizing article argues that the real answer to "how much data do I need" is a data budget on a timeline: buy data in tranches, refit the learning curve after each one, reserve the locked test cohort before model and threshold selection, and account for coverage by stratum rather than by total.

Every one of those depends on lifecycle infrastructure:

• Buying in tranches requires that a dataset can grow as a versioned object and that a learning curve can be refit against a stable definition of "the same task".
• Reserving the test cohort early requires that the reservation is enforced by the system rather than remembered by a person.
• Coverage accounting requires that strata are queryable attributes, not something reconstructed from filenames.
• Reusing evidence across stages requires that a proof-of-concept result remains interpretable when the program is three stages further on.

Without that infrastructure, the budget is a spreadsheet that stops matching reality in month two. This is also why traceability is a data-planning topic rather than a documentation topic.

What MAEA does not do

Being clear about the boundary is part of being useful:

• It does not make a system compliant. Intended use, risk management, clinical evaluation, quality-system obligations, and submission strategy remain the manufacturer's responsibility. MAEA produces the traceable record that supports those activities; it does not replace them.
• It does not create data you do not have, or permission you do not hold. Sourcing, partnerships, and data rights are a separate problem.
• It does not choose the intended use, the endpoint, or the acceptance criteria. Those are clinical and regulatory decisions, and they determine the sample size before any tooling is involved.
• Auto-annotation is not a reference standard. It reduces the cost of producing labels that clinical experts still have to own.
• It does not remove the need for external validation. No lifecycle system can tell you how a model behaves at a site whose data it has never seen.

Where to start

If your team is somewhere between a prototype and a product, the useful first move is not a platform migration. It is an audit:

1. Take one result you would have to defend — a headline metric on a recent slide.
2. Try to resolve it to an exact dataset version, split definition, label protocol version, configuration, and code state.
3. Note every step where the chain breaks.
4. Ask what it would cost to reconstruct the missing links today, and what it would cost in two years.

That exercise usually settles the question of whether the data lifecycle is a real problem for your program or a theoretical one. Where it is real, MAEA is designed to close the specific breaks it finds — and where it is not, the audit is cheap.

If you are earlier than that, AI data readiness before model building is the better starting point, and ModAstera can review dataset structure, split logic, and evidence readiness before model comparisons become expensive to unwind.

References

• How much data is needed to train a medical AI model
• Willemink et al., Preparing Medical Imaging Data for Machine Learning, Radiology 2020
• IMDRF N88, Good Machine Learning Practice for Medical Device Development: Guiding Principles
• FDA, Marketing Submission Recommendations for a Predetermined Change Control Plan for AI-Enabled Device Software Functions
• MDCG 2025-6 / AIB 2025-1, interplay between the MDR, IVDR, and the AI Act
• Data leakage in medical AI: building trustworthy train, validation, and test splits
• AI model monitoring for regulated workflows
• AI traceability in regulated workflows