Example 11: Autoregressive Models¶

Autoregressive (AR) models feed their own predictions back as input, enabling free-running multi-step-ahead simulation. During training, they use "teacher forcing" (feeding true outputs), then switch to their own predictions at inference time.

Prerequisites¶

This notebook builds on concepts from Examples 00-04. Make sure you are familiar with simulation (Example 02), prediction mode (Example 03), and model architectures (Example 04) before proceeding.

Setup¶

In [1]:

from tsfast.tsdata.benchmark import create_dls_silverbox
from tsfast.training import AR_RNNLearner, RNNLearner, AR_TCNLearner, fun_rmse

from tsfast.tsdata.benchmark import create_dls_silverbox from tsfast.training import AR_RNNLearner, RNNLearner, AR_TCNLearner, fun_rmse

What is Autoregressive Prediction?¶

In standard simulation (Example 02), the model maps input u(t) to output y(t) in a single forward pass. In autoregressive mode:

• Training (teacher forcing): the model receives [u(t), y_true(t-1)] as input. The prediction_concat transform (the default transform in AR_RNNLearner) concatenates the true target to the input.
• Inference (free-running): the model uses its own prediction [u(t), y_pred(t-1)]. The AR_Model wrapper implements this feedback loop. This tests whether the model is stable -- errors can accumulate and cause divergence.

AR models are more powerful for long-horizon prediction but require stronger regularization to stay stable.

Load the Dataset¶

In [2]:

dls = create_dls_silverbox(bs=16, win_sz=500, stp_sz=10)

dls = create_dls_silverbox(bs=16, win_sz=500, stp_sz=10)

Standard Simulation Baseline¶

Train a standard RNN for comparison. This model sees only the input u(t) and must predict y(t) without any output feedback. It serves as a baseline to highlight the difference autoregressive models make.

In [3]:

lrn_std = RNNLearner(dls, rnn_type='lstm', hidden_size=40, n_skip=50, metrics=[fun_rmse])
lrn_std.fit_flat_cos(n_epoch=10, lr=3e-3)
lrn_std.show_results(max_n=2)
print(f"Standard RNN: {lrn_std.validate()}")

lrn_std = RNNLearner(dls, rnn_type='lstm', hidden_size=40, n_skip=50, metrics=[fun_rmse]) lrn_std.fit_flat_cos(n_epoch=10, lr=3e-3) lrn_std.show_results(max_n=2) print(f"Standard RNN: {lrn_std.validate()}")

Epoch 1/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 1/10: 100%|██████████| 300/300 [00:00<00:00, 665.45it/s, train=0.0194 | valid=0.0075 | fun_rmse=0.0094]

Epoch 1/10: 100%|██████████| 300/300 [00:00<00:00, 664.92it/s, train=0.0194 | valid=0.0075 | fun_rmse=0.0094]

Epoch 2/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 2/10: 100%|██████████| 300/300 [00:00<00:00, 901.55it/s, train=0.0033 | valid=0.0018 | fun_rmse=0.0029]

Epoch 2/10: 100%|██████████| 300/300 [00:00<00:00, 900.55it/s, train=0.0033 | valid=0.0018 | fun_rmse=0.0029]

Epoch 3/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 902.86it/s, train=0.0026 | valid=0.0018 | fun_rmse=0.0027]

Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 901.91it/s, train=0.0026 | valid=0.0018 | fun_rmse=0.0027]

Epoch 4/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 4/10: 100%|██████████| 300/300 [00:00<00:00, 908.26it/s, train=0.0022 | valid=0.0015 | fun_rmse=0.0026]

Epoch 4/10: 100%|██████████| 300/300 [00:00<00:00, 907.23it/s, train=0.0022 | valid=0.0015 | fun_rmse=0.0026]

Epoch 5/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 5/10: 100%|██████████| 300/300 [00:00<00:00, 910.64it/s, train=0.0021 | valid=0.0016 | fun_rmse=0.0026]

Epoch 5/10: 100%|██████████| 300/300 [00:00<00:00, 909.68it/s, train=0.0021 | valid=0.0016 | fun_rmse=0.0026]

Epoch 6/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 6/10: 100%|██████████| 300/300 [00:00<00:00, 953.27it/s, train=0.0024 | valid=0.0020 | fun_rmse=0.0029]

Epoch 6/10: 100%|██████████| 300/300 [00:00<00:00, 952.11it/s, train=0.0024 | valid=0.0020 | fun_rmse=0.0029]

Epoch 7/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 7/10: 100%|██████████| 300/300 [00:00<00:00, 891.91it/s, train=0.0022 | valid=0.0027 | fun_rmse=0.0038]

Epoch 7/10: 100%|██████████| 300/300 [00:00<00:00, 891.02it/s, train=0.0022 | valid=0.0027 | fun_rmse=0.0038]

Epoch 8/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 8/10: 100%|██████████| 300/300 [00:00<00:00, 930.42it/s, train=0.0022 | valid=0.0022 | fun_rmse=0.0032]

Epoch 8/10: 100%|██████████| 300/300 [00:00<00:00, 929.19it/s, train=0.0022 | valid=0.0022 | fun_rmse=0.0032]

Epoch 9/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 9/10: 100%|██████████| 300/300 [00:00<00:00, 863.90it/s, train=0.0014 | valid=0.0010 | fun_rmse=0.0022]

Epoch 9/10: 100%|██████████| 300/300 [00:00<00:00, 863.02it/s, train=0.0014 | valid=0.0010 | fun_rmse=0.0022]

Epoch 10/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 10/10: 100%|██████████| 300/300 [00:00<00:00, 922.87it/s, train=0.0008 | valid=0.0008 | fun_rmse=0.0021]

Epoch 10/10: 100%|██████████| 300/300 [00:00<00:00, 921.74it/s, train=0.0008 | valid=0.0008 | fun_rmse=0.0021]

Standard RNN: (0.0008382503292523324, {'fun_rmse': 0.0020847057458013296})

Autoregressive RNN¶

AR_RNNLearner wraps the model with autoregressive behavior and adds ActivationRegularizer and TemporalActivationRegularizer automatically. alpha and beta control activation and temporal regularization respectively -- AR models need these for stability.

Key parameters:

• rnn_type='lstm': use LSTM cells for the recurrent layer.
• hidden_size=40: 40 hidden units in the LSTM.
• alpha=1.0: penalty weight for large activations (AR regularization).
• beta=1.0: penalty weight for abrupt activation changes between timesteps (TAR regularization).

In [4]:

lrn_ar = AR_RNNLearner(
    dls, rnn_type='lstm', hidden_size=40,
    alpha=1.0, beta=1.0, metrics=[fun_rmse]
)
lrn_ar.fit_flat_cos(n_epoch=10, lr=3e-3)
lrn_ar.show_results(max_n=2)
print(f"AR-RNN: {lrn_ar.validate()}")

lrn_ar = AR_RNNLearner( dls, rnn_type='lstm', hidden_size=40, alpha=1.0, beta=1.0, metrics=[fun_rmse] ) lrn_ar.fit_flat_cos(n_epoch=10, lr=3e-3) lrn_ar.show_results(max_n=2) print(f"AR-RNN: {lrn_ar.validate()}")

Epoch 1/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 1/10: 100%|██████████| 300/300 [00:00<00:00, 705.67it/s, train=0.0124 | valid=0.0003 | fun_rmse=0.0005]

Epoch 1/10: 100%|██████████| 300/300 [00:00<00:00, 705.09it/s, train=0.0124 | valid=0.0003 | fun_rmse=0.0005]

Epoch 2/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 2/10: 100%|██████████| 300/300 [00:00<00:00, 701.41it/s, train=0.0028 | valid=0.0002 | fun_rmse=0.0004]

Epoch 2/10: 100%|██████████| 300/300 [00:00<00:00, 700.81it/s, train=0.0028 | valid=0.0002 | fun_rmse=0.0004]

Epoch 3/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 793.72it/s, train=0.0020 | valid=0.0002 | fun_rmse=0.0004]

Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 793.00it/s, train=0.0020 | valid=0.0002 | fun_rmse=0.0004]

Epoch 4/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 4/10: 100%|██████████| 300/300 [00:00<00:00, 762.04it/s, train=0.0017 | valid=0.0003 | fun_rmse=0.0004]

Epoch 4/10: 100%|██████████| 300/300 [00:00<00:00, 761.27it/s, train=0.0017 | valid=0.0003 | fun_rmse=0.0004]

Epoch 5/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 5/10: 100%|██████████| 300/300 [00:00<00:00, 751.93it/s, train=0.0014 | valid=0.0002 | fun_rmse=0.0003]

Epoch 5/10: 100%|██████████| 300/300 [00:00<00:00, 750.89it/s, train=0.0014 | valid=0.0002 | fun_rmse=0.0003]

Epoch 6/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 6/10: 100%|██████████| 300/300 [00:00<00:00, 673.79it/s, train=0.0013 | valid=0.0001 | fun_rmse=0.0002]

Epoch 6/10: 100%|██████████| 300/300 [00:00<00:00, 673.26it/s, train=0.0013 | valid=0.0001 | fun_rmse=0.0002]

Epoch 7/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 7/10: 100%|██████████| 300/300 [00:00<00:00, 742.41it/s, train=0.0011 | valid=0.0004 | fun_rmse=0.0004]

Epoch 7/10: 100%|██████████| 300/300 [00:00<00:00, 741.75it/s, train=0.0011 | valid=0.0004 | fun_rmse=0.0004]

Epoch 8/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 8/10: 100%|██████████| 300/300 [00:00<00:00, 756.35it/s, train=0.0011 | valid=0.0003 | fun_rmse=0.0003]

Epoch 8/10: 100%|██████████| 300/300 [00:00<00:00, 755.56it/s, train=0.0011 | valid=0.0003 | fun_rmse=0.0003]

Epoch 9/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 9/10: 100%|██████████| 300/300 [00:00<00:00, 759.58it/s, train=0.0009 | valid=0.0001 | fun_rmse=0.0001]

Epoch 9/10: 100%|██████████| 300/300 [00:00<00:00, 758.94it/s, train=0.0009 | valid=0.0001 | fun_rmse=0.0001]

Epoch 10/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 10/10: 100%|██████████| 300/300 [00:00<00:00, 741.70it/s, train=0.0008 | valid=0.0000 | fun_rmse=0.0000]

Epoch 10/10: 100%|██████████| 300/300 [00:00<00:00, 741.04it/s, train=0.0008 | valid=0.0000 | fun_rmse=0.0000]

AR-RNN: (1.5940358935040422e-05, {'fun_rmse': 4.486130637815222e-05})

Autoregressive TCN¶

AR mode also works with temporal convolutional networks. AR_TCNLearner combines causal convolutions with autoregressive output feedback. The hl_depth parameter controls the number of TCN blocks (and therefore the receptive field, which is 2**hl_depth timesteps).

In [5]:

lrn_ar_tcn = AR_TCNLearner(dls, hl_depth=4, metrics=[fun_rmse])
lrn_ar_tcn.fit_flat_cos(n_epoch=10, lr=3e-3)
lrn_ar_tcn.show_results(max_n=2)
print(f"AR-TCN: {lrn_ar_tcn.validate()}")

lrn_ar_tcn = AR_TCNLearner(dls, hl_depth=4, metrics=[fun_rmse]) lrn_ar_tcn.fit_flat_cos(n_epoch=10, lr=3e-3) lrn_ar_tcn.show_results(max_n=2) print(f"AR-TCN: {lrn_ar_tcn.validate()}")

Epoch 1/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 1/10:  97%|█████████▋| 291/300 [00:00<00:00, 581.85it/s]

Epoch 1/10: 100%|██████████| 300/300 [00:00<00:00, 581.85it/s, train=0.1456 | valid=0.0219 | fun_rmse=0.0277]

Epoch 1/10: 100%|██████████| 300/300 [00:00<00:00, 578.40it/s, train=0.1456 | valid=0.0219 | fun_rmse=0.0277]

Epoch 2/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 2/10: 100%|██████████| 300/300 [00:00<00:00, 714.76it/s, train=0.0222 | valid=0.0019 | fun_rmse=0.0028]

Epoch 2/10: 100%|██████████| 300/300 [00:00<00:00, 714.04it/s, train=0.0222 | valid=0.0019 | fun_rmse=0.0028]

Epoch 3/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 665.18it/s, train=0.0109 | valid=0.0019 | fun_rmse=0.0024]

Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 664.67it/s, train=0.0109 | valid=0.0019 | fun_rmse=0.0024]

Epoch 4/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 4/10: 100%|██████████| 300/300 [00:00<00:00, 638.78it/s, train=0.0082 | valid=0.0014 | fun_rmse=0.0017]

Epoch 4/10: 100%|██████████| 300/300 [00:00<00:00, 638.24it/s, train=0.0082 | valid=0.0014 | fun_rmse=0.0017]

Epoch 5/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 5/10: 100%|██████████| 300/300 [00:00<00:00, 689.02it/s, train=0.0071 | valid=0.0020 | fun_rmse=0.0026]

Epoch 5/10: 100%|██████████| 300/300 [00:00<00:00, 688.42it/s, train=0.0071 | valid=0.0020 | fun_rmse=0.0026]

Epoch 6/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 6/10: 100%|██████████| 300/300 [00:00<00:00, 663.36it/s, train=0.0059 | valid=0.0008 | fun_rmse=0.0011]

Epoch 6/10: 100%|██████████| 300/300 [00:00<00:00, 662.79it/s, train=0.0059 | valid=0.0008 | fun_rmse=0.0011]

Epoch 7/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 7/10: 100%|██████████| 300/300 [00:00<00:00, 675.37it/s, train=0.0053 | valid=0.0012 | fun_rmse=0.0015]

Epoch 7/10: 100%|██████████| 300/300 [00:00<00:00, 674.76it/s, train=0.0053 | valid=0.0012 | fun_rmse=0.0015]

Epoch 8/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 8/10: 100%|██████████| 300/300 [00:00<00:00, 716.99it/s, train=0.0048 | valid=0.0005 | fun_rmse=0.0007]

Epoch 8/10: 100%|██████████| 300/300 [00:00<00:00, 716.36it/s, train=0.0048 | valid=0.0005 | fun_rmse=0.0007]

Epoch 9/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 9/10: 100%|██████████| 300/300 [00:00<00:00, 711.50it/s, train=0.0036 | valid=0.0004 | fun_rmse=0.0005]

Epoch 9/10: 100%|██████████| 300/300 [00:00<00:00, 710.94it/s, train=0.0036 | valid=0.0004 | fun_rmse=0.0005]

Epoch 10/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 10/10: 100%|██████████| 300/300 [00:00<00:00, 638.14it/s, train=0.0030 | valid=0.0003 | fun_rmse=0.0004]

Epoch 10/10: 100%|██████████| 300/300 [00:00<00:00, 637.67it/s, train=0.0030 | valid=0.0003 | fun_rmse=0.0004]

AR-TCN: (0.0002898429811466485, {'fun_rmse': 0.000445984594989568})

Stability and Regularization¶

AR models can diverge during free-running inference if prediction errors accumulate. Regularization helps:

• alpha penalizes large activations, keeping the model in a well-behaved region.
• beta penalizes abrupt changes in predictions, encouraging smoothness.
• Higher alpha and beta improve stability but may reduce accuracy on easy regions.

Train with stronger regularization to demonstrate the trade-off:

In [6]:

lrn_ar_strong = AR_RNNLearner(
    dls, rnn_type='lstm', hidden_size=40,
    alpha=3.0, beta=3.0, metrics=[fun_rmse]
)
lrn_ar_strong.fit_flat_cos(n_epoch=10, lr=3e-3)
lrn_ar_strong.show_results(max_n=2)

lrn_ar_strong = AR_RNNLearner( dls, rnn_type='lstm', hidden_size=40, alpha=3.0, beta=3.0, metrics=[fun_rmse] ) lrn_ar_strong.fit_flat_cos(n_epoch=10, lr=3e-3) lrn_ar_strong.show_results(max_n=2)

Epoch 1/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 1/10: 100%|██████████| 300/300 [00:00<00:00, 716.43it/s, train=0.0189 | valid=0.0003 | fun_rmse=0.0006]

Epoch 1/10: 100%|██████████| 300/300 [00:00<00:00, 715.60it/s, train=0.0189 | valid=0.0003 | fun_rmse=0.0006]

Epoch 2/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 2/10: 100%|██████████| 300/300 [00:00<00:00, 722.76it/s, train=0.0043 | valid=0.0002 | fun_rmse=0.0003]

Epoch 2/10: 100%|██████████| 300/300 [00:00<00:00, 722.08it/s, train=0.0043 | valid=0.0002 | fun_rmse=0.0003]

Epoch 3/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 641.65it/s, train=0.0029 | valid=0.0002 | fun_rmse=0.0003]

Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 640.64it/s, train=0.0029 | valid=0.0002 | fun_rmse=0.0003]

Epoch 4/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 4/10: 100%|██████████| 300/300 [00:00<00:00, 606.96it/s, train=0.0024 | valid=0.0002 | fun_rmse=0.0003]

Epoch 4/10: 100%|██████████| 300/300 [00:00<00:00, 606.45it/s, train=0.0024 | valid=0.0002 | fun_rmse=0.0003]

Epoch 5/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 5/10: 100%|██████████| 300/300 [00:00<00:00, 685.08it/s, train=0.0021 | valid=0.0004 | fun_rmse=0.0005]

Epoch 5/10: 100%|██████████| 300/300 [00:00<00:00, 684.51it/s, train=0.0021 | valid=0.0004 | fun_rmse=0.0005]

Epoch 6/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 6/10: 100%|██████████| 300/300 [00:00<00:00, 744.06it/s, train=0.0019 | valid=0.0004 | fun_rmse=0.0005]

Epoch 6/10: 100%|██████████| 300/300 [00:00<00:00, 743.41it/s, train=0.0019 | valid=0.0004 | fun_rmse=0.0005]

Epoch 7/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 7/10: 100%|██████████| 300/300 [00:00<00:00, 729.19it/s, train=0.0017 | valid=0.0003 | fun_rmse=0.0004]

Epoch 7/10: 100%|██████████| 300/300 [00:00<00:00, 728.47it/s, train=0.0017 | valid=0.0003 | fun_rmse=0.0004]

Epoch 8/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 8/10: 100%|██████████| 300/300 [00:00<00:00, 706.81it/s, train=0.0015 | valid=0.0003 | fun_rmse=0.0004]

Epoch 8/10: 100%|██████████| 300/300 [00:00<00:00, 706.17it/s, train=0.0015 | valid=0.0003 | fun_rmse=0.0004]

Epoch 9/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 9/10: 100%|██████████| 300/300 [00:00<00:00, 739.70it/s, train=0.0014 | valid=0.0001 | fun_rmse=0.0001]

Epoch 9/10: 100%|██████████| 300/300 [00:00<00:00, 739.03it/s, train=0.0014 | valid=0.0001 | fun_rmse=0.0001]

Epoch 10/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 10/10: 100%|██████████| 300/300 [00:00<00:00, 768.94it/s, train=0.0012 | valid=0.0000 | fun_rmse=0.0000]

Epoch 10/10: 100%|██████████| 300/300 [00:00<00:00, 768.15it/s, train=0.0012 | valid=0.0000 | fun_rmse=0.0000]

Key Takeaways¶

• AR models feed their own predictions back as input for multi-step-ahead simulation.
• Teacher forcing during training provides stable gradients; free-running at inference tests stability.
• AR_RNNLearner and AR_TCNLearner handle the autoregressive logic automatically.
• Regularization (alpha, beta) is essential to prevent error accumulation and divergence.
• Trade-off: stronger regularization leads to more stable but potentially less accurate predictions.