Clean Time-Series Hold-Out: Gains Track Corpus, 28% MASE

Every leaderboard number you've seen for time-series foundation models — TimesFM, Chronos, Moirai — was earned on archives that predate the models and ship in the same public form pretraining corpora are scraped from. A new paper from BrightMind AI and UT Arlington does the one thing nobody does: evaluates on data that did not exist yet. Thirteen forecasters, seven groups, five domains, every observation published after the last model's release. The result doesn't kill the foundation-model story. It does something worse — it shows the story can't be repaired by recency alone.

The problem: a temporal hold-out fixes the wrong half of contamination

Contamination in time-series eval is usually framed as "the model saw the test window." The obvious fix is a test window that postdates training. This paper runs that fix and finds it insufficient: a model that read a decade of Wikipedia pageview series has learned what pageview series do, and January 2026 pageviews still do it. Moving the window forward removes memorisation of a window but leaves domain familiarity untouched. The authors' framing is the paper in one line — the practitioner's question isn't which model is better, it's whether your domain is one the model was raised on.

The method: thirteen forecasters, seven groups, zero API keys

Four classical statistical methods (SeasonalNaive, Theta, AutoETS, AutoARIMA), three trained per dataset (LightGBM, LSTM, N-BEATS), six pretrained (Chronos-Bolt S/B, Chronos-2, TimesFM, TimesFM-3, Moirai-2). Seven groups from five domains — Wikipedia pageviews at three granularities, hourly weather, hourly air quality, hourly electricity, daily exchange rates — with the entire test window after every model's release date, all rebuildable from released fetchers. MASE for point accuracy, weighted quantile loss and 80% coverage for probabilistic, Nemenyi intervals and corrected paired tests instead of decimal-place bravado.

The results: 5 of 7, and the exceptions are the story

The defensible count is 5 wins for pretrained models, one loss, one null. The loss is electricity: Theta ranks first, and grid load — the most regular series in the study — is exactly the domain the foundation-model literature claims most often, normally on ETT. That's the receipt nobody wanted: ETT is 2016–2018 data that sits in every pretraining corpus, so the field's favorite claim was tested on the one domain shape every model grew up on. The null is exchange rates, where the Friedman test doesn't reject and all thirteen methods — seasonal naive included — sit in one tied set. The authors count that as the benchmark working, not failing.

Then the negative result that earns the paper its keep. The natural hypothesis is that some intrinsic property of the series explains where pretrained models win. It fails: seasonal strength is negatively associated with the advantage (Wikipedia series measure 0.00–0.10 seasonal strength and show +16.9% to +28.4% gains; weather measures 0.77 and shows +12.5%). Spectral entropy fails the other way — exchange rates are as high-entropy as Wikipedia and show nothing.

What does track is the corpus. TimesFM's authors describe its pretraining corpus as dominated by Wikipedia pageviews — ~10¹¹ time points at exactly the granularities tested, differing only in time window. The largest gain (28% lower MASE, weekly pageviews) falls there. The clever part: because all six pretrained models forecast identical series, series difficulty cancels in a within-family comparison. The TimesFM family outranks the Chronos family by −0.53 ranks on Wikipedia versus −0.09 everywhere else (1,500 vs. 754 series, Mann–Whitney p < 10⁻⁵). The model with Wikipedia in its corpus is specifically better at Wikipedia, and only there.

mindmap
  root((Where pretrained wins land))
    Wikipedia pageviews
      +25.8% to +28.4% MASE
      TimesFM corpus is pageviews
      within-family gap −0.53 ranks
    Weather / air quality
      +12.5% / +12.6%
      no corpus story
    Electricity
      −10.6%: Theta wins
      ETT in every corpus
    Exchange rates
      null: nothing beats naive

There's a second result hiding in the probabilistic table that the abstract doesn't mention: calibration splits along family lines. Every pretrained model's nominal 80% interval under-covers (0.61–0.79 across domains), while classical methods over-cover (0.83–0.97). Point-accuracy leaderboards don't show this. A production forecaster whose uncertainty bands are systematically too narrow is a different risk profile than one that's merely less accurate — you optimize inventory against it.

Limitations: what the authors downplayed

The authors are unusually honest, but three things stay soft. First, the corpus-familiarity mechanism is a hypothesis the design "supports," not demonstrates — only TimesFM's corpus is disclosed well enough to test, so the effect is confirmed for exactly one family, on one domain, with a modest rank-biserial of −0.13. Second, the panels are small where it matters: 46 electricity series and 29 exchange-rate series widen intervals enough that the electricity "loss" is a rank-first-with-six-tied, and the authors themselves won't call the exchange-rate null proof of absence. Third, seven groups is seven data points for the familiarity gradient — no correlation analysis at that n can be significant, and the paper says so. The strongest defensible claim is directional, not quantitative.

What it means if you build things

Three practical moves. One: stop trusting cross-domain averages — they're dominated by the well-published domains most likely to be in a pretraining corpus. Ask instead whether your domain is one the model was raised on; a bursty, web-scale, continuously-published series is a very different bet than a physically bounded grid signal. Two: if you're evaluating forecasters, add a domain hold-out stated relative to each model's disclosed corpus — the paper argues, correctly, that this needs corpus disclosure the field currently doesn't provide, which is the actual ask. Three: check coverage before deploying, not just MASE. The pretrained models here were confidently wrong in a way the tables most people read can't show.