How to Evaluate Time Series Foundation Models for Classification

How to evaluate time series foundation models for classification

Choose a task scope and label budget before reading a ranking. These steps keep comparisons within the same published setting.

1. Choose the dataset scope. Use Overall, Univariate or Multivariate according to the input type you need. The Standard Overall ranking requires complete coverage of the 198-dataset core; other views show their own coverage.
2. Fix the label budget. Standard uses the official training split. Low-shot reports 5-shot and 10-shot separately, with that many training labels per class on the feasible subset for each budget. Do not compare their aggregate scores as if they used the same tasks.
3. Identify the prediction approach. In-context learning predicts from labeled support without fitting a target-specific classifier. Frozen-probe evaluation keeps the pretrained backbone fixed and fits the standardized lightweight probe. These approaches can be filtered within a budget setting.
4. Compare like with like. On the leaderboard, read Average Accuracy alongside Average Rank and Coverage. Open a method page to check the exact backbone, classifier configuration and source references behind a result.

For example, to compare models with five labels per class on univariate tasks, select Low-shot, 5-shot and Univariate on the leaderboard. Compare the displayed configurations and their coverage within that view.

Standard Evaluation

Use official train/test splits. Each dataset produces one accuracy, and each dataset receives equal weight in aggregate reporting.

Few-shot Time Series Classification (Low-shot)

The 5-shot and 10-shot settings use five or ten training labels per class, respectively. Use matched class-stratified context sets at these fixed label budgets. Report low-shot results separately from Standard scores. Low-shot uses a separately evaluated dataset subset, further filtered by the label budget, rather than the full Standard dataset universe.

Training-Free / In-Context Learning

Methods infer predictions from labeled context examples without target-specific parameter updates or per-dataset classifier fitting.

Frozen-Probe Evaluation

Keep the pretrained backbone frozen and fit only the standardized lightweight probe on the official training split.

Ranking Rules

• Each dataset receives equal weight.
• The primary metric is Average Accuracy.
• Ranking is descending by Average Accuracy.
• Only methods completing all required datasets enter the official Overall Leaderboard.
• Average Rank is computed per dataset against the full eligible configuration set for the selected budget and dataset scope. Accuracy differences of at most 1e-12 on the 0–1 scale are treated as ties and receive average ranks.
• ICL and Frozen-Probe views filter Standard Overall. Their Track Rank is local to the displayed group; Overall Avg. Rank retains the full Standard Overall comparison set.
• Low-shot results must not be mixed into the Standard Overall score.

Submission Requirements

Initial submissions must provide public code and checkpoint repositories, model provenance, the requested evaluation track, and a contact for review. License and environment details are requested when available; maintainers may require missing reproducibility material before publication.

Reproducibility

Before a submitted result can be published, its immutable code and checkpoint revisions are locked. The review then records the environment, command, seed, dataset version, dataset-level results, runtime, and peak memory required by the protocol.