Reproducibility
Evaluation Protocol
A protocol-first framework for comparable, auditable evaluation across time series classification model families.
Overview
The benchmark records dataset-level outputs first and computes aggregate statistics from verified records. Tracks remain distinct to prevent adaptation settings from being conflated.
Standard Evaluation
Use official train/test splits. Each dataset produces one accuracy, and each dataset receives equal weight in aggregate reporting.
Low-shot Evaluation
Use matched class-stratified context sets at fixed label budgets. Report low-shot results separately from Standard scores.
Training-Free / In-Context Learning
Methods infer predictions from labeled context examples without target-specific parameter updates or per-dataset classifier fitting.
Frozen-Probe Evaluation
Keep the pretrained backbone frozen and fit only the standardized lightweight probe on the official training split.
Ranking Rules
- Each dataset receives equal weight.
- The primary metric is Average Accuracy.
- Ranking is descending by Average Accuracy.
- Only methods completing all required datasets enter the official Overall Leaderboard.
- Average Rank is reported as a secondary statistic.
- Low-shot results must not be mixed into the Standard Overall score.
Evaluation pipeline
This visual describes the planned hosted workflow; it is not active in the current static demo.
Submission Requirements
Submissions must provide reproducible code, environment requirements, model and checkpoint provenance, a declared license, and a contact for evaluation issues.
Reproducibility
Every future official run will record the code revision, environment image, command, seed, dataset version, dataset-level results, runtime, and peak memory before administrator review.