Radical honesty
We publish what works and what doesn't — negative results included. A medical decision-support tool earns trust by being auditable, not by hiding its limits.
The production MS head is the V7 SSL-foundation ensemble — held-out Dice 0.645, at/above the measured 7-expert inter-rater ceiling (0.632), with calibration ECE 0.209→0.032. The table below is the honest experiment history that led there.
| Model | Data | Val Dice | Status |
|---|---|---|---|
| SegResNet (FLAIR-only) | 60 real MS patients, patient-level split | 0.52 | in production |
| SegResNet (T1+T2+FLAIR, naive registration) | same, affine-header alignment | 0.45 | rejected |
| SegResNet (T1+T2+FLAIR, proper SimpleITK registration) | same, intensity-based registration | 0.47 | rejected — still < FLAIR-only |
Honest finding: even with correct registration, multi-sequence did not beat FLAIR-only here (the ground-truth mask is FLAIR-derived). We kept the simpler, stronger model rather than ship a worse one for novelty.
FLAIR → MNI152 affine registration (SimpleITK) + a Harvard-Oxford region atlas. Each lesion is classified by tight abutment: ≤3 mm to a ventricle = periventricular, ≤1 mm to cortex = juxtacortical, brain-stem/cerebellum = infratentorial.
An earlier atlas-free guess was tested and dropped as unreliable — anatomical labels that contradict reality are worse than none.
Confidence is recalibrated (ECE), then a conformal selector certifies a distribution-free bound: committed-error ≤ α at probability ≥ 1−δ (Clopper-Pearson). The agent abstains and escalates rather than commit beyond its certified threshold.
Deterministic, auditable orchestrator (not a generic agent loop) · FlammaGuard blocks final-diagnosis language and forces specialist review · FlammaCrossCheck catches image↔report contradictions (e.g. laterality flips) even at high confidence · self-consistency voting + multi-agent debate on hard cases · every output carries the disclaimer and is logged for clinician validation.
The exact vision model in production, its held-out Dice, training data and robustness — plus the full version lineage. Every assessment can state which agent version produced it.