Skip to content

Benchmark Datasets

Pre-configured DataLoader factories for identibench benchmark datasets.

create_dls_from_spec 

create_dls_from_spec(spec: BenchmarkSpec, **kwargs) -> DataLoaders

Create DataLoaders from an identibench benchmark specification.

Download/preparation is delegated to identibench (spec.ensure_datasets_exist, cached under the identibench data root). Files are resolved through the spec's role accessors (spec.train_files/valid_files/test_files), never by parsing the on-disk layout; the test split is the union of the spec's named test sets. Evaluation parameters are read off spec.task (e.g. window sizing for prediction benchmarks).

Requires a trainable spec (the workshop/robot/quad/ship benchmarks and the combined RIANN orientation benchmark). All-test evaluation specs (per-source orientation, DFJIMU) have no DataLoader factory here.

Parameters:

Name Type Description Default
spec BenchmarkSpec

benchmark specification from identibench

 required

Raises:

Type Description
ValueError

If the spec defines no train/valid split (spec.is_trainable is False).
Source code in tsfast/tsdata/benchmark.py 
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
def create_dls_from_spec(
    spec: idb.BenchmarkSpec,
    **kwargs,
) -> DataLoaders:
    """Create DataLoaders from an identibench benchmark specification.

    Download/preparation is delegated to identibench (`spec.ensure_datasets_exist`,
    cached under the identibench data root). Files are resolved through the spec's
    role accessors (`spec.train_files`/`valid_files`/`test_files`), never by
    parsing the on-disk layout; the test split is the union of the spec's named
    test sets. Evaluation parameters are read off `spec.task` (e.g. window sizing
    for prediction benchmarks).

    Requires a trainable spec (the workshop/robot/quad/ship benchmarks and the
    combined RIANN orientation benchmark). All-test evaluation specs
    (per-source orientation, DFJIMU) have no DataLoader factory here.

    Args:
        spec: benchmark specification from identibench

    Raises:
        ValueError: If the spec defines no train/valid split (`spec.is_trainable`
            is False).
    """
    if not spec.is_trainable:
        raise ValueError(
            f"{spec.name} is an all-test evaluation spec (no train/valid split); "
            "it cannot provide training DataLoaders."
        )
    spec.ensure_datasets_exist()

    dataset = {
        "train": [str(f) for f in spec.train_files()],
        "valid": [str(f) for f in spec.valid_files()],
        "test": [str(f) for f in spec.test_files()],
    }
    spec_kwargs = {
        "u": spec.u_cols,
        "y": spec.y_cols,
        "dataset": dataset,
    }

    # Prediction specs only contribute window sizing here; the prediction input
    # concatenation is left to the learner factories (prediction_concat transform).
    if isinstance(spec.task, idb.Prediction):
        spec_kwargs.update(
            {
                "win_sz": spec.task.horizon + spec.task.init_window,
                "valid_stp_sz": spec.task.step,
            }
        )

    if spec.name in BENCHMARK_DL_KWARGS:
        spec_kwargs.update(BENCHMARK_DL_KWARGS[spec.name])

    dl_kwargs = {**spec_kwargs, **kwargs}
    return create_dls(**dl_kwargs)