Example 13: Benchmarking with IdentiBench¶

IdentiBench provides standardized benchmarks for comparing system identification methods. This example shows how to run your TSFast models on IdentiBench benchmarks for fair, reproducible comparison with other methods.

Prerequisites¶

Setup¶

In [1]:

Copied!

import identibench as idb

from tsfast.tsdata.benchmark import create_dls_from_spec
from tsfast.inference import InferenceWrapper
from tsfast.training import RNNLearner, fun_rmse
import identibench as idb

from tsfast.tsdata.benchmark import create_dls_from_spec
from tsfast.inference import InferenceWrapper
from tsfast.training import RNNLearner, fun_rmse

What is IdentiBench?¶

IdentiBench is a benchmarking framework that provides standardized datasets, evaluation protocols, and metrics for system identification. Each benchmark defines:

A dataset with specified train/validation/test splits
Input and output column names (e.g., voltage in, displacement out)
Evaluation metrics (typically NRMSE -- normalized root mean square error)
A standard API that all methods must follow, ensuring fair comparison

The workshop_benchmarks dictionary contains the benchmarks used in the IdentiBench workshop -- a curated set covering different system types and difficulties.

The Build Model Function¶

IdentiBench requires a build_model function that takes a TrainingContext and returns a callable model for evaluation. The context provides:

context.spec -- the benchmark specification (column names, file resolution). Evaluation parameters live on the task: context.spec.task.init_window, context.spec.task.horizon, ...
context.hyperparameters -- your model's hyperparameters, passed through from the benchmark runner

The returned model must accept numpy arrays: model(u, y_init, attrs) for simulation benchmarks, where u is the full input signal and y_init is the initial output window.

In [2]:

Copied!





def build_model(context: idb.TrainingContext):
    """Build and train a TSFast model for an IdentiBench benchmark."""
    dls = create_dls_from_spec(context.spec)

    lrn = RNNLearner(
        dls,
        rnn_type=context.hyperparameters.get('model_type', 'lstm'),
        num_layers=context.hyperparameters.get('num_layers', 1),
        hidden_size=context.hyperparameters.get('hidden_size', 40),
        n_skip=context.spec.task.init_window,
        metrics=[fun_rmse],
    )

    lrn.fit_flat_cos(n_epoch=10, lr=3e-3)
    wrapper = InferenceWrapper(lrn)
    return lambda u, y_init, attrs: wrapper(u, y_init)
def build_model(context: idb.TrainingContext):
    """Build and train a TSFast model for an IdentiBench benchmark."""
    dls = create_dls_from_spec(context.spec)

    lrn = RNNLearner(
        dls,
        rnn_type=context.hyperparameters.get('model_type', 'lstm'),
        num_layers=context.hyperparameters.get('num_layers', 1),
        hidden_size=context.hyperparameters.get('hidden_size', 40),
        n_skip=context.spec.task.init_window,
        metrics=[fun_rmse],
    )

    lrn.fit_flat_cos(n_epoch=10, lr=3e-3)
    wrapper = InferenceWrapper(lrn)
    return lambda u, y_init, attrs: wrapper(u, y_init)

Key details:

create_dls_from_spec automatically extracts column names, window sizes, and prediction settings from the benchmark spec. It also applies benchmark-specific DataLoader defaults (e.g., batch size, step size) from TSFast's BENCHMARK_DL_KWARGS table.
n_skip=context.spec.task.init_window uses the benchmark-defined initialization window to skip the initial transient in the loss. This matches IdentiBench's evaluation protocol, which discards the first init_window timesteps.
InferenceWrapper wraps the trained learner into a numpy-in, numpy-out callable wrapper(u, y_init). IdentiBench's evaluation harness calls the model with three positional arguments -- model(u, y_init, attrs) -- so a small lambda adapts the wrapper to that contract. The wrapper forwards y_init to models that consume an initial output window (e.g. prediction models trained with prediction_concat) and ignores it for simulation models driven by u alone.

Configure and Run Benchmarks¶

We define a hyperparameter dictionary and pass it along with the benchmarks to idb.run_benchmarks. The runner:

Downloads each dataset (on first use)
Calls build_model with the spec and hyperparameters
Evaluates the returned model on the held-out test set
Collects metrics into a pandas DataFrame

In [3]:

Copied!





model_config = {
    'model_type': 'lstm',
    'num_layers': 1,
    'hidden_size': 40,
}

benchmarks = [idb.BenchmarkWH_Simulation, idb.BenchmarkSilverbox_Simulation, idb.BenchmarkCascadedTanks_Simulation]
results = idb.run_benchmarks(benchmarks, build_model, model_config)
model_config = {
    'model_type': 'lstm',
    'num_layers': 1,
    'hidden_size': 40,
}

benchmarks = [idb.BenchmarkWH_Simulation, idb.BenchmarkSilverbox_Simulation, idb.BenchmarkCascadedTanks_Simulation]
results = idb.run_benchmarks(benchmarks, build_model, model_config)

--- Starting benchmark run for 3 specifications, repeating each 1 times ---

-- Repetition 1/1 --

[1/3] Running: BenchmarkWH_Simulation (Rep 1)

Epoch 1/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 1/10:  93%|█████████▎| 279/300 [00:00<00:00, 557.18it/s]

Epoch 1/10: 100%|██████████| 300/300 [00:00<00:00, 557.18it/s, train=0.0365 | valid=0.0125 | fun_rmse=0.0159]

Epoch 1/10: 100%|██████████| 300/300 [00:00<00:00, 562.87it/s, train=0.0365 | valid=0.0125 | fun_rmse=0.0159]

Epoch 2/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 2/10: 100%|██████████| 300/300 [00:00<00:00, 713.67it/s, train=0.0110 | valid=0.0072 | fun_rmse=0.0095]

Epoch 2/10: 100%|██████████| 300/300 [00:00<00:00, 713.06it/s, train=0.0110 | valid=0.0072 | fun_rmse=0.0095]

Epoch 3/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 732.78it/s, train=0.0085 | valid=0.0091 | fun_rmse=0.0115]

Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 732.10it/s, train=0.0085 | valid=0.0091 | fun_rmse=0.0115]

Epoch 4/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 4/10: 100%|██████████| 300/300 [00:00<00:00, 774.87it/s, train=0.0071 | valid=0.0094 | fun_rmse=0.0112]

Epoch 4/10: 100%|██████████| 300/300 [00:00<00:00, 774.18it/s, train=0.0071 | valid=0.0094 | fun_rmse=0.0112]

Epoch 5/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 5/10: 100%|██████████| 300/300 [00:00<00:00, 764.40it/s, train=0.0071 | valid=0.0079 | fun_rmse=0.0094]

Epoch 5/10: 100%|██████████| 300/300 [00:00<00:00, 763.64it/s, train=0.0071 | valid=0.0079 | fun_rmse=0.0094]

Epoch 6/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 6/10: 100%|██████████| 300/300 [00:00<00:00, 753.34it/s, train=0.0062 | valid=0.0039 | fun_rmse=0.0052]

Epoch 6/10: 100%|██████████| 300/300 [00:00<00:00, 752.55it/s, train=0.0062 | valid=0.0039 | fun_rmse=0.0052]

Epoch 7/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 7/10: 100%|██████████| 300/300 [00:00<00:00, 737.71it/s, train=0.0059 | valid=0.0034 | fun_rmse=0.0047]

Epoch 7/10: 100%|██████████| 300/300 [00:00<00:00, 737.01it/s, train=0.0059 | valid=0.0034 | fun_rmse=0.0047]

Epoch 8/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 8/10: 100%|██████████| 300/300 [00:00<00:00, 751.62it/s, train=0.0056 | valid=0.0060 | fun_rmse=0.0077]

Epoch 8/10: 100%|██████████| 300/300 [00:00<00:00, 750.88it/s, train=0.0056 | valid=0.0060 | fun_rmse=0.0077]

Epoch 9/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 9/10: 100%|██████████| 300/300 [00:00<00:00, 774.96it/s, train=0.0038 | valid=0.0025 | fun_rmse=0.0036]

Epoch 9/10: 100%|██████████| 300/300 [00:00<00:00, 774.22it/s, train=0.0038 | valid=0.0025 | fun_rmse=0.0036]

Epoch 10/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 10/10: 100%|██████████| 300/300 [00:00<00:00, 754.85it/s, train=0.0023 | valid=0.0023 | fun_rmse=0.0033]

Epoch 10/10: 100%|██████████| 300/300 [00:00<00:00, 754.20it/s, train=0.0023 | valid=0.0023 | fun_rmse=0.0033]

  -> Success: BenchmarkWH_Simulation (Rep 1) completed.

[2/3] Running: BenchmarkSilverbox_Simulation (Rep 1)

Epoch 1/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 1/10: 100%|██████████| 300/300 [00:00<00:00, 763.09it/s, train=0.0195 | valid=0.0035 | fun_rmse=0.0050]

Epoch 1/10: 100%|██████████| 300/300 [00:00<00:00, 762.23it/s, train=0.0195 | valid=0.0035 | fun_rmse=0.0050]

Epoch 2/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 2/10: 100%|██████████| 300/300 [00:00<00:00, 770.23it/s, train=0.0037 | valid=0.0027 | fun_rmse=0.0041]

Epoch 2/10: 100%|██████████| 300/300 [00:00<00:00, 769.40it/s, train=0.0037 | valid=0.0027 | fun_rmse=0.0041]

Epoch 3/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 755.62it/s, train=0.0032 | valid=0.0023 | fun_rmse=0.0039]

Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 754.94it/s, train=0.0032 | valid=0.0023 | fun_rmse=0.0039]

Epoch 4/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 4/10: 100%|██████████| 300/300 [00:00<00:00, 744.06it/s, train=0.0029 | valid=0.0028 | fun_rmse=0.0042]

Epoch 4/10: 100%|██████████| 300/300 [00:00<00:00, 743.32it/s, train=0.0029 | valid=0.0028 | fun_rmse=0.0042]

Epoch 5/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 5/10: 100%|██████████| 300/300 [00:00<00:00, 787.35it/s, train=0.0030 | valid=0.0026 | fun_rmse=0.0040]

Epoch 5/10: 100%|██████████| 300/300 [00:00<00:00, 786.54it/s, train=0.0030 | valid=0.0026 | fun_rmse=0.0040]

Epoch 6/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 6/10: 100%|██████████| 300/300 [00:00<00:00, 745.65it/s, train=0.0031 | valid=0.0036 | fun_rmse=0.0048]

Epoch 6/10: 100%|██████████| 300/300 [00:00<00:00, 744.83it/s, train=0.0031 | valid=0.0036 | fun_rmse=0.0048]

Epoch 7/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 7/10: 100%|██████████| 300/300 [00:00<00:00, 745.64it/s, train=0.0028 | valid=0.0031 | fun_rmse=0.0042]

Epoch 7/10: 100%|██████████| 300/300 [00:00<00:00, 744.90it/s, train=0.0028 | valid=0.0031 | fun_rmse=0.0042]

Epoch 8/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 8/10: 100%|██████████| 300/300 [00:00<00:00, 752.68it/s, train=0.0027 | valid=0.0024 | fun_rmse=0.0039]

Epoch 8/10: 100%|██████████| 300/300 [00:00<00:00, 751.99it/s, train=0.0027 | valid=0.0024 | fun_rmse=0.0039]

Epoch 9/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 9/10: 100%|██████████| 300/300 [00:00<00:00, 771.13it/s, train=0.0023 | valid=0.0019 | fun_rmse=0.0035]

Epoch 9/10: 100%|██████████| 300/300 [00:00<00:00, 770.45it/s, train=0.0023 | valid=0.0019 | fun_rmse=0.0035]

Epoch 10/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 10/10: 100%|██████████| 300/300 [00:00<00:00, 748.51it/s, train=0.0018 | valid=0.0018 | fun_rmse=0.0035]

Epoch 10/10: 100%|██████████| 300/300 [00:00<00:00, 747.85it/s, train=0.0018 | valid=0.0018 | fun_rmse=0.0035]

  -> Success: BenchmarkSilverbox_Simulation (Rep 1) completed.

[3/3] Running: BenchmarkCascadedTanks_Simulation (Rep 1)

Epoch 1/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 1/10: 100%|██████████| 300/300 [00:00<00:00, 970.32it/s, train=1.9624 | valid=0.5362 | fun_rmse=0.8617]

Epoch 1/10: 100%|██████████| 300/300 [00:00<00:00, 969.22it/s, train=1.9624 | valid=0.5362 | fun_rmse=0.8617]

Epoch 2/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 2/10: 100%|██████████| 300/300 [00:00<00:00, 1087.53it/s, train=0.8016 | valid=0.6418 | fun_rmse=0.9207]

Epoch 2/10: 100%|██████████| 300/300 [00:00<00:00, 1086.06it/s, train=0.8016 | valid=0.6418 | fun_rmse=0.9207]

Epoch 3/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 1034.80it/s, train=0.2498 | valid=0.6752 | fun_rmse=0.9793]

Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 1033.56it/s, train=0.2498 | valid=0.6752 | fun_rmse=0.9793]

Epoch 4/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 4/10: 100%|██████████| 300/300 [00:00<00:00, 1023.22it/s, train=0.1914 | valid=0.6561 | fun_rmse=0.9279]

Epoch 4/10: 100%|██████████| 300/300 [00:00<00:00, 1021.98it/s, train=0.1914 | valid=0.6561 | fun_rmse=0.9279]

Epoch 5/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 5/10: 100%|██████████| 300/300 [00:00<00:00, 914.39it/s, train=0.1508 | valid=0.6470 | fun_rmse=0.9350]

Epoch 5/10: 100%|██████████| 300/300 [00:00<00:00, 913.19it/s, train=0.1508 | valid=0.6470 | fun_rmse=0.9350]

Epoch 6/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 6/10: 100%|██████████| 300/300 [00:00<00:00, 1032.34it/s, train=0.1300 | valid=0.6473 | fun_rmse=0.9933]

Epoch 6/10: 100%|██████████| 300/300 [00:00<00:00, 1031.03it/s, train=0.1300 | valid=0.6473 | fun_rmse=0.9933]

Epoch 7/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 7/10: 100%|██████████| 300/300 [00:00<00:00, 1001.44it/s, train=0.1227 | valid=0.6773 | fun_rmse=1.0309]

Epoch 7/10: 100%|██████████| 300/300 [00:00<00:00, 999.79it/s, train=0.1227 | valid=0.6773 | fun_rmse=1.0309]

Epoch 8/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 8/10: 100%|██████████| 300/300 [00:00<00:00, 1079.17it/s, train=0.1105 | valid=0.4993 | fun_rmse=0.7901]

Epoch 8/10: 100%|██████████| 300/300 [00:00<00:00, 1077.76it/s, train=0.1105 | valid=0.4993 | fun_rmse=0.7901]

Epoch 9/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 9/10: 100%|██████████| 300/300 [00:00<00:00, 1046.31it/s, train=0.0941 | valid=0.5981 | fun_rmse=0.9314]

Epoch 9/10: 100%|██████████| 300/300 [00:00<00:00, 1044.72it/s, train=0.0941 | valid=0.5981 | fun_rmse=0.9314]

Epoch 10/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 10/10: 100%|██████████| 300/300 [00:00<00:00, 1066.48it/s, train=0.0798 | valid=0.5782 | fun_rmse=0.8998]

Epoch 10/10: 100%|██████████| 300/300 [00:00<00:00, 1065.07it/s, train=0.0798 | valid=0.5782 | fun_rmse=0.8998]

  -> Success: BenchmarkCascadedTanks_Simulation (Rep 1) completed.

--- Benchmark run finished. 3/3 individual runs completed successfully. ---

Analyze Results¶

The results DataFrame shows the benchmark name, metric score, and training/test times for each benchmark.

In [4]:

Copied!

print(results)
print(results)

                      benchmark_name          datasets  \
0             BenchmarkWH_Simulation              [wh]   
1      BenchmarkSilverbox_Simulation       [silverbox]   
2  BenchmarkCascadedTanks_Simulation  [cascaded_tanks]   

                                     hyperparameters        seed  \
0  {'model_type': 'lstm', 'num_layers': 1, 'hidde...   186187610   
1  {'model_type': 'lstm', 'num_layers': 1, 'hidde...  1694128257   
2  {'model_type': 'lstm', 'num_layers': 1, 'hidde...  2728191559   

   training_time_seconds  test_time_seconds benchmark_type metric_name  \
0               4.683816           0.072302     Simulation     rmse_mV   
1               3.983859           0.068976     Simulation     rmse_mV   
2               2.950451           0.001991     Simulation        rmse   

   metric_score  test_sets.test.rmse_mV  test_sets.multisine.rmse_mV  \
0      3.367967                3.367967                          NaN   
1      0.668582                     NaN                     0.668582   
2      0.950440                     NaN                          NaN   

   test_sets.arrow_full.rmse_mV  test_sets.arrow_no_extrapolation.rmse_mV  \
0                           NaN                                       NaN   
1                      3.280731                                  0.585499   
2                           NaN                                       NaN   

   test_sets.test.rmse  
0                  NaN  
1                  NaN  
2              0.95044

Trying Different Configurations¶

One of IdentiBench's strengths is making it easy to compare different model architectures on the same benchmarks. Here we try a GRU with 2 layers instead of a single-layer LSTM.

In [5]:

Copied!





model_config_v2 = {
    'model_type': 'gru',
    'num_layers': 2,
    'hidden_size': 40,
}

results_v2 = idb.run_benchmarks(benchmarks, build_model, model_config_v2)
model_config_v2 = {
    'model_type': 'gru',
    'num_layers': 2,
    'hidden_size': 40,
}

results_v2 = idb.run_benchmarks(benchmarks, build_model, model_config_v2)

--- Starting benchmark run for 3 specifications, repeating each 1 times ---

-- Repetition 1/1 --

[1/3] Running: BenchmarkWH_Simulation (Rep 1)

Epoch 1/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 1/10: 100%|██████████| 300/300 [00:00<00:00, 609.51it/s, train=0.0281 | valid=0.0097 | fun_rmse=0.0131]

Epoch 1/10: 100%|██████████| 300/300 [00:00<00:00, 609.04it/s, train=0.0281 | valid=0.0097 | fun_rmse=0.0131]

Epoch 2/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 2/10: 100%|██████████| 300/300 [00:00<00:00, 651.99it/s, train=0.0097 | valid=0.0105 | fun_rmse=0.0132]

Epoch 2/10: 100%|██████████| 300/300 [00:00<00:00, 651.43it/s, train=0.0097 | valid=0.0105 | fun_rmse=0.0132]

Epoch 3/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 599.08it/s]

Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 599.08it/s, train=0.0086 | valid=0.0045 | fun_rmse=0.0061]

Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 594.74it/s, train=0.0086 | valid=0.0045 | fun_rmse=0.0061]

Epoch 4/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 4/10: 100%|██████████| 300/300 [00:00<00:00, 662.41it/s, train=0.0068 | valid=0.0066 | fun_rmse=0.0077]

Epoch 4/10: 100%|██████████| 300/300 [00:00<00:00, 661.84it/s, train=0.0068 | valid=0.0066 | fun_rmse=0.0077]

Epoch 5/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 5/10: 100%|██████████| 300/300 [00:00<00:00, 632.64it/s, train=0.0066 | valid=0.0051 | fun_rmse=0.0066]

Epoch 5/10: 100%|██████████| 300/300 [00:00<00:00, 632.07it/s, train=0.0066 | valid=0.0051 | fun_rmse=0.0066]

Epoch 6/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 6/10:  94%|█████████▍| 282/300 [00:00<00:00, 563.48it/s]

Epoch 6/10: 100%|██████████| 300/300 [00:00<00:00, 563.48it/s, train=0.0060 | valid=0.0056 | fun_rmse=0.0072]

Epoch 6/10: 100%|██████████| 300/300 [00:00<00:00, 557.19it/s, train=0.0060 | valid=0.0056 | fun_rmse=0.0072]

Epoch 7/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 7/10: 100%|██████████| 300/300 [00:00<00:00, 653.04it/s, train=0.0056 | valid=0.0047 | fun_rmse=0.0060]

Epoch 7/10: 100%|██████████| 300/300 [00:00<00:00, 652.45it/s, train=0.0056 | valid=0.0047 | fun_rmse=0.0060]

Epoch 8/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 8/10: 100%|██████████| 300/300 [00:00<00:00, 659.50it/s, train=0.0047 | valid=0.0075 | fun_rmse=0.0095]

Epoch 8/10: 100%|██████████| 300/300 [00:00<00:00, 658.94it/s, train=0.0047 | valid=0.0075 | fun_rmse=0.0095]

Epoch 9/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 9/10: 100%|██████████| 300/300 [00:00<00:00, 615.91it/s, train=0.0037 | valid=0.0018 | fun_rmse=0.0029]

Epoch 9/10: 100%|██████████| 300/300 [00:00<00:00, 615.40it/s, train=0.0037 | valid=0.0018 | fun_rmse=0.0029]

Epoch 10/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 10/10: 100%|██████████| 300/300 [00:00<00:00, 625.81it/s, train=0.0016 | valid=0.0016 | fun_rmse=0.0026]

Epoch 10/10: 100%|██████████| 300/300 [00:00<00:00, 625.15it/s, train=0.0016 | valid=0.0016 | fun_rmse=0.0026]

  -> Success: BenchmarkWH_Simulation (Rep 1) completed.

[2/3] Running: BenchmarkSilverbox_Simulation (Rep 1)

Epoch 1/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 1/10: 100%|██████████| 300/300 [00:00<00:00, 647.81it/s, train=0.0144 | valid=0.0034 | fun_rmse=0.0048]

Epoch 1/10: 100%|██████████| 300/300 [00:00<00:00, 647.33it/s, train=0.0144 | valid=0.0034 | fun_rmse=0.0048]

Epoch 2/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 2/10: 100%|██████████| 300/300 [00:00<00:00, 647.71it/s, train=0.0033 | valid=0.0028 | fun_rmse=0.0042]

Epoch 2/10: 100%|██████████| 300/300 [00:00<00:00, 647.23it/s, train=0.0033 | valid=0.0028 | fun_rmse=0.0042]

Epoch 3/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 633.95it/s, train=0.0031 | valid=0.0028 | fun_rmse=0.0041]

Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 633.48it/s, train=0.0031 | valid=0.0028 | fun_rmse=0.0041]

Epoch 4/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 4/10: 100%|██████████| 300/300 [00:00<00:00, 622.31it/s, train=0.0031 | valid=0.0026 | fun_rmse=0.0039]

Epoch 4/10: 100%|██████████| 300/300 [00:00<00:00, 621.83it/s, train=0.0031 | valid=0.0026 | fun_rmse=0.0039]

Epoch 5/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 5/10: 100%|██████████| 300/300 [00:00<00:00, 623.55it/s, train=0.0030 | valid=0.0033 | fun_rmse=0.0046]

Epoch 5/10: 100%|██████████| 300/300 [00:00<00:00, 622.99it/s, train=0.0030 | valid=0.0033 | fun_rmse=0.0046]

Epoch 6/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 6/10: 100%|██████████| 300/300 [00:00<00:00, 625.47it/s, train=0.0027 | valid=0.0022 | fun_rmse=0.0036]

Epoch 6/10: 100%|██████████| 300/300 [00:00<00:00, 624.84it/s, train=0.0027 | valid=0.0022 | fun_rmse=0.0036]

Epoch 7/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 7/10: 100%|██████████| 300/300 [00:00<00:00, 651.44it/s, train=0.0026 | valid=0.0024 | fun_rmse=0.0038]

Epoch 7/10: 100%|██████████| 300/300 [00:00<00:00, 650.89it/s, train=0.0026 | valid=0.0024 | fun_rmse=0.0038]

Epoch 8/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 8/10: 100%|██████████| 300/300 [00:00<00:00, 635.15it/s, train=0.0028 | valid=0.0026 | fun_rmse=0.0038]

Epoch 8/10: 100%|██████████| 300/300 [00:00<00:00, 634.61it/s, train=0.0028 | valid=0.0026 | fun_rmse=0.0038]

Epoch 9/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 9/10: 100%|██████████| 300/300 [00:00<00:00, 636.87it/s, train=0.0022 | valid=0.0018 | fun_rmse=0.0034]

Epoch 9/10: 100%|██████████| 300/300 [00:00<00:00, 636.29it/s, train=0.0022 | valid=0.0018 | fun_rmse=0.0034]

Epoch 10/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 10/10: 100%|██████████| 300/300 [00:00<00:00, 642.46it/s, train=0.0018 | valid=0.0018 | fun_rmse=0.0034]

Epoch 10/10: 100%|██████████| 300/300 [00:00<00:00, 641.90it/s, train=0.0018 | valid=0.0018 | fun_rmse=0.0034]

  -> Success: BenchmarkSilverbox_Simulation (Rep 1) completed.

[3/3] Running: BenchmarkCascadedTanks_Simulation (Rep 1)

Epoch 1/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 1/10: 100%|██████████| 300/300 [00:00<00:00, 845.86it/s, train=1.4020 | valid=0.9565 | fun_rmse=1.0593]

Epoch 1/10: 100%|██████████| 300/300 [00:00<00:00, 844.99it/s, train=1.4020 | valid=0.9565 | fun_rmse=1.0593]

Epoch 2/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 2/10: 100%|██████████| 300/300 [00:00<00:00, 841.23it/s, train=0.4048 | valid=0.4045 | fun_rmse=0.6045]

Epoch 2/10: 100%|██████████| 300/300 [00:00<00:00, 839.81it/s, train=0.4048 | valid=0.4045 | fun_rmse=0.6045]

Epoch 3/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 639.11it/s, train=0.1989 | valid=0.4276 | fun_rmse=0.6415]

Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 638.63it/s, train=0.1989 | valid=0.4276 | fun_rmse=0.6415]

Epoch 4/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 4/10: 100%|██████████| 300/300 [00:00<00:00, 872.60it/s, train=0.1507 | valid=0.4589 | fun_rmse=0.6292]

Epoch 4/10: 100%|██████████| 300/300 [00:00<00:00, 871.67it/s, train=0.1507 | valid=0.4589 | fun_rmse=0.6292]

Epoch 5/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 5/10: 100%|██████████| 300/300 [00:00<00:00, 739.03it/s, train=0.1202 | valid=0.3950 | fun_rmse=0.5936]

Epoch 5/10: 100%|██████████| 300/300 [00:00<00:00, 738.25it/s, train=0.1202 | valid=0.3950 | fun_rmse=0.5936]

Epoch 6/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 6/10: 100%|██████████| 300/300 [00:00<00:00, 775.42it/s, train=0.0973 | valid=0.3405 | fun_rmse=0.5152]

Epoch 6/10: 100%|██████████| 300/300 [00:00<00:00, 774.61it/s, train=0.0973 | valid=0.3405 | fun_rmse=0.5152]

Epoch 7/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 7/10: 100%|██████████| 300/300 [00:00<00:00, 710.67it/s, train=0.0920 | valid=0.3366 | fun_rmse=0.4939]

Epoch 7/10: 100%|██████████| 300/300 [00:00<00:00, 709.92it/s, train=0.0920 | valid=0.3366 | fun_rmse=0.4939]

Epoch 8/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 8/10: 100%|██████████| 300/300 [00:00<00:00, 878.51it/s, train=0.0787 | valid=0.3515 | fun_rmse=0.5054]

Epoch 8/10: 100%|██████████| 300/300 [00:00<00:00, 877.36it/s, train=0.0787 | valid=0.3515 | fun_rmse=0.5054]

Epoch 9/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 9/10: 100%|██████████| 300/300 [00:00<00:00, 644.65it/s, train=0.0666 | valid=0.3543 | fun_rmse=0.5223]

Epoch 9/10: 100%|██████████| 300/300 [00:00<00:00, 644.13it/s, train=0.0666 | valid=0.3543 | fun_rmse=0.5223]

Epoch 10/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 10/10: 100%|██████████| 300/300 [00:00<00:00, 784.05it/s, train=0.0569 | valid=0.3540 | fun_rmse=0.5115]

Epoch 10/10: 100%|██████████| 300/300 [00:00<00:00, 783.31it/s, train=0.0569 | valid=0.3540 | fun_rmse=0.5115]

  -> Success: BenchmarkCascadedTanks_Simulation (Rep 1) completed.

--- Benchmark run finished. 3/3 individual runs completed successfully. ---

In [6]:

Copied!

print(results_v2)
print(results_v2)

                      benchmark_name          datasets  \
0             BenchmarkWH_Simulation              [wh]   
1      BenchmarkSilverbox_Simulation       [silverbox]   
2  BenchmarkCascadedTanks_Simulation  [cascaded_tanks]   

                                     hyperparameters        seed  \
0  {'model_type': 'gru', 'num_layers': 2, 'hidden...   295740690   
1  {'model_type': 'gru', 'num_layers': 2, 'hidden...  1028283253   
2  {'model_type': 'gru', 'num_layers': 2, 'hidden...   227945766   

   training_time_seconds  test_time_seconds benchmark_type metric_name  \
0               4.829163           1.157594     Simulation     rmse_mV   
1               4.765215           1.324464     Simulation     rmse_mV   
2               3.947175           0.027554     Simulation        rmse   

   metric_score  test_sets.test.rmse_mV  test_sets.multisine.rmse_mV  \
0      2.304379                2.304379                          NaN   
1      0.696366                     NaN                     0.696366   
2      0.639207                     NaN                          NaN   

   test_sets.arrow_full.rmse_mV  test_sets.arrow_no_extrapolation.rmse_mV  \
0                           NaN                                       NaN   
1                      2.744697                                  0.663485   
2                           NaN                                       NaN   

   test_sets.test.rmse  
0                  NaN  
1                  NaN  
2             0.639207

Key Takeaways¶

IdentiBench provides standardized, reproducible benchmarks for fair comparison across system identification methods.
The build_model function follows a simple API: receive a training context, build and train a model, return a callable matching IdentiBench's model(u, y_init, attrs) contract -- here an InferenceWrapper behind a three-argument lambda.
create_dls_from_spec handles dataset-specific configuration automatically -- column names, window sizes, and prediction settings are all extracted from the benchmark spec.
Compare different architectures (LSTM vs. GRU, depth, width) on the same benchmarks with minimal code changes.
Results are directly comparable with other methods in the IdentiBench ecosystem.