Target Leakage, Not Models, Explains the 0.89 AUROC

Every few months a paper reports near-0.9 AUROC predicting heart attack from a national health survey, and every time the implicit promise is the same: a better model class got us here. A new audit of the 2022 Behavioral Risk Factor Surveillance System — 442,067 respondents, ten classifiers spanning linear models, gradient-boosted trees, neural nets, a glass-box explainable boosting machine, and a tabular foundation model — ran the experiment the field skipped: it varied the feature set instead of the model. Removing two post-diagnostic features cost every single model 0.049–0.051 AUROC and collapsed the entire ten-model field into a band 0.0045 wide. The reported headroom was never in the learner. It was in the leakage.

The Problem: Accuracy as a Property of the Spreadsheet

This is the medical-ML version of a story we've told before on Dispatch: evaluation practice, not capacity, is the binding constraint. The authors' framing is blunt — AUROC near 0.89 on survey-based cardiovascular screening is routinely reported, but nobody could say whether it reflects learning or target leakage. Post-diagnostic features (things recorded because care was already delivered) quietly encode the answer. The audit tiers features by leakage risk and asks what survives: discrimination, calibration, fairness at a fixed screening threshold, conformal coverage, explanation faithfulness, and inference cost — then freezes everything and re-tests on 430,755 respondents of the 2023 survey.

The Method: Five Tiers, One Frozen Threshold

Ten classifiers × five feature tiers of decreasing leakage risk, on BRFSS 2022. Each model was audited jointly, then applied — model and threshold frozen — to 2023 data. The design choice that matters is the frozen threshold: most papers retune the operating point on each dataset, which hides degradation. Here, nothing is retuned. Transport fidelity and calibration drift are measured, not assumed.

The Results: A 0.0045 Band and a 104× Speed Gap

Once the two leaky features are gone, model choice is statistically noise: a 0.0045-wide AUROC band across ten architectures. The glass-box explainable boosting machine (EBM) was non-inferior to every alternative within a pre-specified 0.005 margin — while scoring the cohort roughly 104× faster than the strongest tabular foundation model. The fairness numbers are the uncomfortable ones: one fixed threshold detected 75.4% of women's infarctions against 89.0% of men's. Editing the EBM's shape functions — something only possible because the model is interpretable — cut the gap to 0.010. Marginal conformal prediction gave 0.86 coverage to men but 0.82 to adults over 60; Mondrian (stratified) calibration repaired every stratum. Frozen models transported to 2023 within 0.002 AUROC.

---
config:
  theme: dark
---
xychart-beta
    title "AUROC by feature tier (illustrative of the collapse)"
    x-axis ["All features (leaky)", "Post-diagnostic removed", "High-risk tier"]
    y-axis "AUROC" 0.80 --> 0.92
    bar [0.89, 0.84, 0.835]
    line [0.89, 0.84, 0.835]
            

Read the bar chart's second and third columns together: after leakage removal the bars are flat across model classes too. The variance in the literature lives almost entirely in tier one.

Limitations

One survey, one outcome (prevalent myocardial infarction), self-reported data — BRFSS is questionnaire-based, so "post-diagnostic features" here are survey items, not claims/lab records; the leakage mechanism may differ in EHR data. One year-pair for transport testing. The fairness repair via shape-function editing was demonstrated, not deployed, and editing shape functions for fairness can trade off calibration elsewhere. And "tabular foundation models change nothing here" doesn't mean they're useless everywhere — this is a wide, shallow, low-signal tabular regime, close to their worst case.

Why Builders Should Care

If you're shipping a clinical-screening (or any tabular) model: audit features for leakage before shopping for architectures, because your model-selection experiments are probably measuring your spreadsheet, not your learner. Transparency cost nothing measurable here — the interpretable model matched the foundation model while making fairness repair and uncertainty conditioning directly auditable, 104× cheaper. The authors' closing line is the field's verdict: "Evaluation practice, not model capacity, is the binding constraint." That sentence applies well past cardiology.