Example 02: Simulation -- Training on Multiple Datasets¶
Simulation is the most common mode in system identification: the model predicts
output y(t) from input u(t) alone, with no access to past measured outputs. This
example trains simulation models on benchmark datasets and introduces
InferenceWrapper for numpy-based inference.
Prerequisites¶
This notebook builds on concepts from Examples 00 and 01. Make sure you are familiar with creating DataLoaders and training a basic model before proceeding.
Setup¶
from tsfast.tsdata.benchmark import create_dls_silverbox, create_dls_wh
from tsfast.inference import InferenceWrapper
from tsfast.training import RNNLearner, fun_rmse
What is Simulation?¶
In simulation mode, the model sees only the input signal u(t) and must predict the output y(t). The model has no access to measured outputs -- it must simulate the system's behavior purely from the input.
This is the simplest and most common mode for system identification. Think of it as a black-box model that takes a control signal and predicts what the system will do, without ever "peeking" at the real measurements during inference.
Load the Silverbox Dataset¶
The Silverbox is a standard benchmark in system identification. It is an electronic circuit that mimics a nonlinear mass-spring-damper system.
bs=16: batch size of 16 windows per training stepwin_sz=500: each training window is 500 timesteps longstp_sz=10: consecutive windows are offset by 10 timesteps (overlapping windows)
dls = create_dls_silverbox(bs=16, win_sz=500, stp_sz=10)
Train an LSTM with n_skip¶
RNNs start with a zero hidden state, so the first N predictions are unreliable
because the network hasn't "warmed up" yet. The n_skip parameter excludes the
first N timesteps from the loss computation, so the model isn't penalized for the
transient warmup period.
Key parameters:
rnn_type='lstm': use an LSTM cell (alternatives:'gru','rnn')n_skip=50: exclude the first 50 timesteps from the losshidden_size=40: 40 hidden units in the LSTM layermetrics=[fun_rmse]: track root mean squared error during training
lrn = RNNLearner(dls, rnn_type='lstm', n_skip=50, hidden_size=40, metrics=[fun_rmse])
lrn.show_batch(max_n=4)
lrn.fit_flat_cos(n_epoch=10, lr=3e-3)
Epoch 1/10: 0%| | 0/300 [00:00<?, ?it/s]
Epoch 1/10: 100%|██████████| 300/300 [00:00<00:00, 636.78it/s, train=0.0203 | valid=0.0051 | fun_rmse=0.0074]
Epoch 1/10: 100%|██████████| 300/300 [00:00<00:00, 636.28it/s, train=0.0203 | valid=0.0051 | fun_rmse=0.0074]
Epoch 2/10: 0%| | 0/300 [00:00<?, ?it/s]
Epoch 2/10: 100%|██████████| 300/300 [00:00<00:00, 747.81it/s, train=0.0033 | valid=0.0030 | fun_rmse=0.0041]
Epoch 2/10: 100%|██████████| 300/300 [00:00<00:00, 747.07it/s, train=0.0033 | valid=0.0030 | fun_rmse=0.0041]
Epoch 3/10: 0%| | 0/300 [00:00<?, ?it/s]
Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 843.70it/s, train=0.0027 | valid=0.0020 | fun_rmse=0.0031]
Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 842.72it/s, train=0.0027 | valid=0.0020 | fun_rmse=0.0031]
Epoch 4/10: 0%| | 0/300 [00:00<?, ?it/s]
Epoch 4/10: 100%|██████████| 300/300 [00:00<00:00, 904.18it/s, train=0.0024 | valid=0.0029 | fun_rmse=0.0039]
Epoch 4/10: 100%|██████████| 300/300 [00:00<00:00, 903.24it/s, train=0.0024 | valid=0.0029 | fun_rmse=0.0039]
Epoch 5/10: 0%| | 0/300 [00:00<?, ?it/s]
Epoch 5/10: 100%|██████████| 300/300 [00:00<00:00, 895.93it/s, train=0.0023 | valid=0.0034 | fun_rmse=0.0045]
Epoch 5/10: 100%|██████████| 300/300 [00:00<00:00, 894.87it/s, train=0.0023 | valid=0.0034 | fun_rmse=0.0045]
Epoch 6/10: 0%| | 0/300 [00:00<?, ?it/s]
Epoch 6/10: 100%|██████████| 300/300 [00:00<00:00, 850.60it/s, train=0.0024 | valid=0.0023 | fun_rmse=0.0034]
Epoch 6/10: 100%|██████████| 300/300 [00:00<00:00, 849.58it/s, train=0.0024 | valid=0.0023 | fun_rmse=0.0034]
Epoch 7/10: 0%| | 0/300 [00:00<?, ?it/s]
Epoch 7/10: 100%|██████████| 300/300 [00:00<00:00, 900.60it/s, train=0.0025 | valid=0.0016 | fun_rmse=0.0026]
Epoch 7/10: 100%|██████████| 300/300 [00:00<00:00, 899.61it/s, train=0.0025 | valid=0.0016 | fun_rmse=0.0026]
Epoch 8/10: 0%| | 0/300 [00:00<?, ?it/s]
Epoch 8/10: 100%|██████████| 300/300 [00:00<00:00, 937.34it/s, train=0.0022 | valid=0.0017 | fun_rmse=0.0027]
Epoch 8/10: 100%|██████████| 300/300 [00:00<00:00, 936.25it/s, train=0.0022 | valid=0.0017 | fun_rmse=0.0027]
Epoch 9/10: 0%| | 0/300 [00:00<?, ?it/s]
Epoch 9/10: 100%|██████████| 300/300 [00:00<00:00, 884.02it/s, train=0.0016 | valid=0.0009 | fun_rmse=0.0022]
Epoch 9/10: 100%|██████████| 300/300 [00:00<00:00, 882.93it/s, train=0.0016 | valid=0.0009 | fun_rmse=0.0022]
Epoch 10/10: 0%| | 0/300 [00:00<?, ?it/s]
Epoch 10/10: 100%|██████████| 300/300 [00:00<00:00, 891.94it/s, train=0.0009 | valid=0.0009 | fun_rmse=0.0021]
Epoch 10/10: 100%|██████████| 300/300 [00:00<00:00, 890.89it/s, train=0.0009 | valid=0.0009 | fun_rmse=0.0021]
Visualize Results¶
show_results overlays the model's predictions against the true output on
validation windows. The model has never seen these windows during training.
lrn.show_results(max_n=3)
Evaluating on the Validation Set¶
validate() runs the model on the validation set and returns a tuple of
(loss, {metric_name: value}). You can pass a different DataLoader via
dl= to evaluate on other splits (e.g., lrn.validate(dl=dls.test)).
One caveat for the test split: the test DataLoader yields each file as a
single full-length sequence at batch size 1, and evaluation concatenates
results across batches -- so this only works when all test files have the
same length.
val_loss, val_metrics = lrn.validate()
print(f"Validation loss: {val_loss}")
print(f"Validation metrics: {val_metrics}")
Validation loss: 0.0008781144279055297
Validation metrics: {'fun_rmse': 0.0021425785962492228}
Plotting Training History¶
The recorder stores training and validation loss (plus any metrics) for
each epoch. This is useful for diagnosing underfitting, overfitting, or
unstable training.
import matplotlib.pyplot as plt
epochs = range(1, len(lrn.recorder) + 1)
train_losses = [row[0] for row in lrn.recorder]
val_losses = [row[1] for row in lrn.recorder]
fig, ax = plt.subplots()
ax.plot(epochs, train_losses, label='Train loss')
ax.plot(epochs, val_losses, label='Valid loss')
ax.set_xlabel('Epoch')
ax.set_ylabel('Loss')
ax.legend()
ax.set_title('Training History')
plt.show()
Getting Predictions¶
get_preds returns a tuple of (predictions, targets) as tensors. This is
useful for custom analysis, plotting, or computing metrics that aren't built
into tsfast.
preds, targs = lrn.get_preds()
print(f"Predictions shape: {preds.shape}")
print(f"Targets shape: {targs.shape}")
Predictions shape: torch.Size([30, 500, 1]) Targets shape: torch.Size([30, 500, 1])
Training on a Different Dataset¶
The same workflow applies to any benchmark dataset. Here we train on the Wiener-Hammerstein benchmark, which models a different nonlinear dynamic system. The only change is the DataLoader factory function -- the model architecture and training loop are identical.
dls_wh = create_dls_wh()
lrn_wh = RNNLearner(dls_wh, rnn_type='lstm', n_skip=50, hidden_size=40, metrics=[fun_rmse])
lrn_wh.fit_flat_cos(n_epoch=10, lr=3e-3)
Epoch 1/10: 0%| | 0/300 [00:00<?, ?it/s]
Epoch 1/10: 100%|██████████| 300/300 [00:00<00:00, 675.80it/s, train=0.0445 | valid=0.0102 | fun_rmse=0.0136]
Epoch 1/10: 100%|██████████| 300/300 [00:00<00:00, 675.28it/s, train=0.0445 | valid=0.0102 | fun_rmse=0.0136]
Epoch 2/10: 0%| | 0/300 [00:00<?, ?it/s]
Epoch 2/10: 100%|██████████| 300/300 [00:00<00:00, 766.55it/s, train=0.0097 | valid=0.0080 | fun_rmse=0.0104]
Epoch 2/10: 100%|██████████| 300/300 [00:00<00:00, 765.88it/s, train=0.0097 | valid=0.0080 | fun_rmse=0.0104]
Epoch 3/10: 0%| | 0/300 [00:00<?, ?it/s]
Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 739.21it/s, train=0.0076 | valid=0.0063 | fun_rmse=0.0087]
Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 738.57it/s, train=0.0076 | valid=0.0063 | fun_rmse=0.0087]
Epoch 4/10: 0%| | 0/300 [00:00<?, ?it/s]
Epoch 4/10: 100%|██████████| 300/300 [00:00<00:00, 751.30it/s, train=0.0084 | valid=0.0048 | fun_rmse=0.0071]
Epoch 4/10: 100%|██████████| 300/300 [00:00<00:00, 750.56it/s, train=0.0084 | valid=0.0048 | fun_rmse=0.0071]
Epoch 5/10: 0%| | 0/300 [00:00<?, ?it/s]
Epoch 5/10: 100%|██████████| 300/300 [00:00<00:00, 748.33it/s, train=0.0055 | valid=0.0119 | fun_rmse=0.0152]
Epoch 5/10: 100%|██████████| 300/300 [00:00<00:00, 747.67it/s, train=0.0055 | valid=0.0119 | fun_rmse=0.0152]
Epoch 6/10: 0%| | 0/300 [00:00<?, ?it/s]
Epoch 6/10: 100%|██████████| 300/300 [00:00<00:00, 747.45it/s, train=0.0080 | valid=0.0080 | fun_rmse=0.0097]
Epoch 6/10: 100%|██████████| 300/300 [00:00<00:00, 746.77it/s, train=0.0080 | valid=0.0080 | fun_rmse=0.0097]
Epoch 7/10: 0%| | 0/300 [00:00<?, ?it/s]
Epoch 7/10: 100%|██████████| 300/300 [00:00<00:00, 786.25it/s, train=0.0062 | valid=0.0045 | fun_rmse=0.0059]
Epoch 7/10: 100%|██████████| 300/300 [00:00<00:00, 785.54it/s, train=0.0062 | valid=0.0045 | fun_rmse=0.0059]
Epoch 8/10: 0%| | 0/300 [00:00<?, ?it/s]
Epoch 8/10: 100%|██████████| 300/300 [00:00<00:00, 775.25it/s, train=0.0046 | valid=0.0038 | fun_rmse=0.0051]
Epoch 8/10: 100%|██████████| 300/300 [00:00<00:00, 774.47it/s, train=0.0046 | valid=0.0038 | fun_rmse=0.0051]
Epoch 9/10: 0%| | 0/300 [00:00<?, ?it/s]
Epoch 9/10: 100%|██████████| 300/300 [00:00<00:00, 733.98it/s, train=0.0036 | valid=0.0023 | fun_rmse=0.0037]
Epoch 9/10: 100%|██████████| 300/300 [00:00<00:00, 733.36it/s, train=0.0036 | valid=0.0023 | fun_rmse=0.0037]
Epoch 10/10: 0%| | 0/300 [00:00<?, ?it/s]
Epoch 10/10: 100%|██████████| 300/300 [00:00<00:00, 739.97it/s, train=0.0021 | valid=0.0022 | fun_rmse=0.0034]
Epoch 10/10: 100%|██████████| 300/300 [00:00<00:00, 739.17it/s, train=0.0021 | valid=0.0022 | fun_rmse=0.0034]
lrn_wh.show_results(max_n=3)
Using Your Model: InferenceWrapper¶
After training, you often want to run inference with numpy arrays -- for example, in a deployment pipeline or when integrating with scipy/control toolboxes.
InferenceWrapper handles the full pipeline automatically:
- Converts numpy input to a PyTorch tensor
- Applies the same input normalization used during training
- Runs the model forward pass
- Converts the output back to a numpy array
wrapper = InferenceWrapper(lrn)
xb, yb = next(iter(dls.valid))
np_input = xb.cpu().numpy()
y_pred = wrapper.inference(np_input)
print(f"Input shape: {np_input.shape}")
print(f"Output shape: {y_pred.shape}")
Input shape: (16, 500, 1) Output shape: (16, 500, 1)
Key Takeaways¶
- Simulation models predict output from input alone (no output feedback). The model must learn the full system dynamics from the excitation signal u(t).
n_skiphandles the RNN warmup transient by excluding early timesteps from the loss, so the model isn't penalized while its hidden state initializes.lrn.recorderstores per-epoch training and validation loss (plus metrics). Plot it to diagnose training progress.- Pass
dl=to evaluate on a specific split: e.g.dl=lrn.dls.testfor the test set (defaults to validation). The test loader yields full-length files at batch size 1, so this requires equal-length test files. InferenceWrapperprovides numpy-in / numpy-out inference with automatic normalization, making it easy to use trained models outside of the training loop.