Example 07: Augmentations, Regularizers, and Training Options¶

TSFast provides composable building blocks that customize training without modifying model code:

• Augmentations modify training data on-the-fly (noise, bias, sequence length variation)
• Auxiliary losses add regularization terms to the main loss (activation smoothing, gradient penalties)
• Training options control optimizer behavior (gradient clipping)

This example demonstrates each category and shows how to combine them.

Prerequisites¶

• Example 00: Your First Model
• Example 01: Understanding the Data Pipeline
• Example 02: Simulation
• Benchmark RNN

Setup¶

In [1]:

from tsfast.tsdata.benchmark import create_dls_silverbox
from tsfast.models._core.scaling import unwrap_model
from tsfast.training import (
    RNNLearner,
    fun_rmse,
    ActivationRegularizer,
    TemporalActivationRegularizer,
    noise, bias, vary_seq_len, truncate_sequence,
    plot_grad_flow,
)

from tsfast.tsdata.benchmark import create_dls_silverbox from tsfast.models._core.scaling import unwrap_model from tsfast.training import ( RNNLearner, fun_rmse, ActivationRegularizer, TemporalActivationRegularizer, noise, bias, vary_seq_len, truncate_sequence, plot_grad_flow, )

Load the Dataset¶

In [2]:

dls = create_dls_silverbox(bs=16, win_sz=500, stp_sz=10)

dls = create_dls_silverbox(bs=16, win_sz=500, stp_sz=10)

Data Augmentations¶

Augmentations modify training data on-the-fly. They only apply during training (not validation or test), so your evaluation metrics stay comparable. Pass them as augmentations=[...] when creating the Learner.

noise¶

Adds Gaussian noise to input signals. std controls the noise magnitude.

In [3]:

lrn_noisy = RNNLearner(
    dls, rnn_type='lstm', metrics=[fun_rmse],
    augmentations=[noise(std=0.05)],
)

lrn_noisy = RNNLearner( dls, rnn_type='lstm', metrics=[fun_rmse], augmentations=[noise(std=0.05)], )

bias¶

Adds a constant offset per signal per sample. This simulates sensor drift or calibration errors, making the model more robust to such shifts.

In [4]:

lrn_bias = RNNLearner(
    dls, rnn_type='lstm', metrics=[fun_rmse],
    augmentations=[bias(std=0.05)],
)

lrn_bias = RNNLearner( dls, rnn_type='lstm', metrics=[fun_rmse], augmentations=[bias(std=0.05)], )

Training with Augmentation¶

Train three models -- one without augmentation, one with noise, one with bias -- to see the effect on validation performance.

In [5]:

lrn_base = RNNLearner(dls, rnn_type='lstm', metrics=[fun_rmse])
lrn_base.fit_flat_cos(n_epoch=5, lr=3e-3)

lrn_base = RNNLearner(dls, rnn_type='lstm', metrics=[fun_rmse]) lrn_base.fit_flat_cos(n_epoch=5, lr=3e-3)

Epoch 1/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 1/5:  50%|████▉     | 149/300 [00:00<00:00, 296.76it/s]

Epoch 1/5: 100%|██████████| 300/300 [00:00<00:00, 296.76it/s, train=0.0121 | valid=0.0054 | fun_rmse=0.0104]

Epoch 1/5: 100%|██████████| 300/300 [00:00<00:00, 339.79it/s, train=0.0121 | valid=0.0054 | fun_rmse=0.0104]

Epoch 2/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 2/5:  67%|██████▋   | 200/300 [00:00<00:00, 398.90it/s]

Epoch 2/5: 100%|██████████| 300/300 [00:00<00:00, 398.90it/s, train=0.0056 | valid=0.0083 | fun_rmse=0.0123]

Epoch 2/5: 100%|██████████| 300/300 [00:00<00:00, 395.98it/s, train=0.0056 | valid=0.0083 | fun_rmse=0.0123]

Epoch 3/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 3/5:  66%|██████▌   | 198/300 [00:00<00:00, 394.39it/s]

Epoch 3/5: 100%|██████████| 300/300 [00:00<00:00, 394.39it/s, train=0.0053 | valid=0.0055 | fun_rmse=0.0103]

Epoch 3/5: 100%|██████████| 300/300 [00:00<00:00, 394.84it/s, train=0.0053 | valid=0.0055 | fun_rmse=0.0103]

Epoch 4/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 4/5:  67%|██████▋   | 202/300 [00:00<00:00, 404.00it/s]

Epoch 4/5: 100%|██████████| 300/300 [00:00<00:00, 404.00it/s, train=0.0051 | valid=0.0041 | fun_rmse=0.0097]

Epoch 4/5: 100%|██████████| 300/300 [00:00<00:00, 402.39it/s, train=0.0051 | valid=0.0041 | fun_rmse=0.0097]

Epoch 5/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 5/5:  67%|██████▋   | 201/300 [00:00<00:00, 401.33it/s]

Epoch 5/5: 100%|██████████| 300/300 [00:00<00:00, 401.33it/s, train=0.0035 | valid=0.0029 | fun_rmse=0.0095]

Epoch 5/5: 100%|██████████| 300/300 [00:00<00:00, 401.15it/s, train=0.0035 | valid=0.0029 | fun_rmse=0.0095]

In [6]:

lrn_noisy.fit_flat_cos(n_epoch=5, lr=3e-3)

lrn_noisy.fit_flat_cos(n_epoch=5, lr=3e-3)

Epoch 1/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 1/5:  64%|██████▍   | 192/300 [00:00<00:00, 382.90it/s]

Epoch 1/5: 100%|██████████| 300/300 [00:00<00:00, 382.90it/s, train=0.0399 | valid=0.0312 | fun_rmse=0.0390]

Epoch 1/5: 100%|██████████| 300/300 [00:00<00:00, 384.19it/s, train=0.0399 | valid=0.0312 | fun_rmse=0.0390]

Epoch 2/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 2/5:  66%|██████▌   | 198/300 [00:00<00:00, 394.94it/s]

Epoch 2/5: 100%|██████████| 300/300 [00:00<00:00, 394.94it/s, train=0.0384 | valid=0.0351 | fun_rmse=0.0438]

Epoch 2/5: 100%|██████████| 300/300 [00:00<00:00, 391.41it/s, train=0.0384 | valid=0.0351 | fun_rmse=0.0438]

Epoch 3/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 3/5:  66%|██████▌   | 198/300 [00:00<00:00, 395.62it/s]

Epoch 3/5: 100%|██████████| 300/300 [00:00<00:00, 395.62it/s, train=0.0382 | valid=0.0332 | fun_rmse=0.0414]

Epoch 3/5: 100%|██████████| 300/300 [00:00<00:00, 394.11it/s, train=0.0382 | valid=0.0332 | fun_rmse=0.0414]

Epoch 4/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 4/5:  66%|██████▌   | 198/300 [00:00<00:00, 395.66it/s]

Epoch 4/5: 100%|██████████| 300/300 [00:00<00:00, 395.66it/s, train=0.0384 | valid=0.0337 | fun_rmse=0.0421]

Epoch 4/5: 100%|██████████| 300/300 [00:00<00:00, 394.81it/s, train=0.0384 | valid=0.0337 | fun_rmse=0.0421]

Epoch 5/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 5/5:  66%|██████▌   | 197/300 [00:00<00:00, 392.17it/s]

Epoch 5/5: 100%|██████████| 300/300 [00:00<00:00, 392.17it/s, train=0.0380 | valid=0.0336 | fun_rmse=0.0419]

Epoch 5/5: 100%|██████████| 300/300 [00:00<00:00, 391.61it/s, train=0.0380 | valid=0.0336 | fun_rmse=0.0419]

In [7]:

lrn_bias.fit_flat_cos(n_epoch=5, lr=3e-3)

lrn_bias.fit_flat_cos(n_epoch=5, lr=3e-3)

Epoch 1/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 1/5:  65%|██████▍   | 194/300 [00:00<00:00, 386.29it/s]

Epoch 1/5: 100%|██████████| 300/300 [00:00<00:00, 386.29it/s, train=0.0184 | valid=0.0081 | fun_rmse=0.0125]

Epoch 1/5: 100%|██████████| 300/300 [00:00<00:00, 385.35it/s, train=0.0184 | valid=0.0081 | fun_rmse=0.0125]

Epoch 2/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 2/5:  65%|██████▌   | 195/300 [00:00<00:00, 388.54it/s]

Epoch 2/5: 100%|██████████| 300/300 [00:00<00:00, 388.54it/s, train=0.0099 | valid=0.0095 | fun_rmse=0.0136]

Epoch 2/5: 100%|██████████| 300/300 [00:00<00:00, 386.80it/s, train=0.0099 | valid=0.0095 | fun_rmse=0.0136]

Epoch 3/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 3/5:  64%|██████▍   | 193/300 [00:00<00:00, 384.64it/s]

Epoch 3/5: 100%|██████████| 300/300 [00:00<00:00, 384.64it/s, train=0.0089 | valid=0.0075 | fun_rmse=0.0118]

Epoch 3/5: 100%|██████████| 300/300 [00:00<00:00, 386.03it/s, train=0.0089 | valid=0.0075 | fun_rmse=0.0118]

Epoch 4/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 4/5:  65%|██████▍   | 194/300 [00:00<00:00, 387.34it/s]

Epoch 4/5: 100%|██████████| 300/300 [00:00<00:00, 387.34it/s, train=0.0085 | valid=0.0075 | fun_rmse=0.0117]

Epoch 4/5: 100%|██████████| 300/300 [00:00<00:00, 387.74it/s, train=0.0085 | valid=0.0075 | fun_rmse=0.0117]

Epoch 5/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 5/5:  65%|██████▍   | 194/300 [00:00<00:00, 387.48it/s]

Epoch 5/5: 100%|██████████| 300/300 [00:00<00:00, 387.48it/s, train=0.0069 | valid=0.0047 | fun_rmse=0.0100]

Epoch 5/5: 100%|██████████| 300/300 [00:00<00:00, 389.65it/s, train=0.0069 | valid=0.0047 | fun_rmse=0.0100]

In [8]:

_, m_base = lrn_base.validate()
_, m_noisy = lrn_noisy.validate()
_, m_bias = lrn_bias.validate()
print(f"Without augmentation:    {m_base['fun_rmse']:.4f}")
print(f"With noise augmentation: {m_noisy['fun_rmse']:.4f}")
print(f"With bias augmentation:  {m_bias['fun_rmse']:.4f}")

_, m_base = lrn_base.validate() _, m_noisy = lrn_noisy.validate() _, m_bias = lrn_bias.validate() print(f"Without augmentation: {m_base['fun_rmse']:.4f}") print(f"With noise augmentation: {m_noisy['fun_rmse']:.4f}") print(f"With bias augmentation: {m_bias['fun_rmse']:.4f}")

Without augmentation:    0.0095
With noise augmentation: 0.0419
With bias augmentation:  0.0100

Activation Regularization¶

Two regularizers can be added to the loss:

• ActivationRegularizer: L2 penalty on RNN activations -- prevents activations from growing too large.
• TemporalActivationRegularizer: L2 penalty on temporal differences of activations -- encourages smooth predictions over time.

modules specifies which model components to regularize (typically the RNN layers). Pass them as aux_losses=[...] when creating the Learner.

In [9]:

lrn_reg = RNNLearner(dls, rnn_type='lstm', metrics=[fun_rmse])
lrn_reg.aux_losses.append(
    ActivationRegularizer(modules=[unwrap_model(lrn_reg.model).rnn], alpha=2.0)
)
lrn_reg.aux_losses.append(
    TemporalActivationRegularizer(modules=[unwrap_model(lrn_reg.model).rnn], beta=1.0)
)
lrn_reg.fit_flat_cos(n_epoch=5, lr=3e-3)
lrn_reg.show_results(max_n=2)

lrn_reg = RNNLearner(dls, rnn_type='lstm', metrics=[fun_rmse]) lrn_reg.aux_losses.append( ActivationRegularizer(modules=[unwrap_model(lrn_reg.model).rnn], alpha=2.0) ) lrn_reg.aux_losses.append( TemporalActivationRegularizer(modules=[unwrap_model(lrn_reg.model).rnn], beta=1.0) ) lrn_reg.fit_flat_cos(n_epoch=5, lr=3e-3) lrn_reg.show_results(max_n=2)

Epoch 1/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 1/5:  62%|██████▏   | 187/300 [00:00<00:00, 373.56it/s]

Epoch 1/5: 100%|██████████| 300/300 [00:00<00:00, 373.56it/s, train=0.0131 | valid=0.0085 | fun_rmse=0.0129]

Epoch 1/5: 100%|██████████| 300/300 [00:00<00:00, 375.98it/s, train=0.0131 | valid=0.0085 | fun_rmse=0.0129]

Epoch 2/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 2/5:  65%|██████▍   | 194/300 [00:00<00:00, 387.00it/s]

Epoch 2/5: 100%|██████████| 300/300 [00:00<00:00, 387.00it/s, train=0.0063 | valid=0.0056 | fun_rmse=0.0106]

Epoch 2/5: 100%|██████████| 300/300 [00:00<00:00, 386.25it/s, train=0.0063 | valid=0.0056 | fun_rmse=0.0106]

Epoch 3/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 3/5:  64%|██████▎   | 191/300 [00:00<00:00, 381.82it/s]

Epoch 3/5: 100%|██████████| 300/300 [00:00<00:00, 381.82it/s, train=0.0061 | valid=0.0078 | fun_rmse=0.0122]

Epoch 3/5: 100%|██████████| 300/300 [00:00<00:00, 383.57it/s, train=0.0061 | valid=0.0078 | fun_rmse=0.0122]

Epoch 4/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 4/5:  65%|██████▌   | 195/300 [00:00<00:00, 389.28it/s]

Epoch 4/5: 100%|██████████| 300/300 [00:00<00:00, 389.28it/s, train=0.0056 | valid=0.0080 | fun_rmse=0.0125]

Epoch 4/5: 100%|██████████| 300/300 [00:00<00:00, 387.17it/s, train=0.0056 | valid=0.0080 | fun_rmse=0.0125]

Epoch 5/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 5/5:  64%|██████▎   | 191/300 [00:00<00:00, 381.05it/s]

Epoch 5/5: 100%|██████████| 300/300 [00:00<00:00, 381.05it/s, train=0.0041 | valid=0.0030 | fun_rmse=0.0096]

Epoch 5/5: 100%|██████████| 300/300 [00:00<00:00, 382.03it/s, train=0.0041 | valid=0.0030 | fun_rmse=0.0096]

Gradient Clipping¶

Clips the gradient norm during backpropagation. This prevents exploding gradients, which are common with RNNs on long sequences. Pass grad_clip= when creating the Learner.

In [10]:

lrn_clip = RNNLearner(dls, rnn_type='lstm', metrics=[fun_rmse], grad_clip=10)
lrn_clip.fit_flat_cos(n_epoch=5, lr=3e-3)

lrn_clip = RNNLearner(dls, rnn_type='lstm', metrics=[fun_rmse], grad_clip=10) lrn_clip.fit_flat_cos(n_epoch=5, lr=3e-3)

Epoch 1/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 1/5:  60%|██████    | 180/300 [00:00<00:00, 359.00it/s]

Epoch 1/5: 100%|██████████| 300/300 [00:00<00:00, 359.00it/s, train=0.0122 | valid=0.0058 | fun_rmse=0.0106]

Epoch 1/5: 100%|██████████| 300/300 [00:00<00:00, 373.00it/s, train=0.0122 | valid=0.0058 | fun_rmse=0.0106]

Epoch 2/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 2/5:  67%|██████▋   | 201/300 [00:00<00:00, 401.70it/s]

Epoch 2/5: 100%|██████████| 300/300 [00:00<00:00, 401.70it/s, train=0.0054 | valid=0.0046 | fun_rmse=0.0100]

Epoch 2/5: 100%|██████████| 300/300 [00:00<00:00, 396.83it/s, train=0.0054 | valid=0.0046 | fun_rmse=0.0100]

Epoch 3/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 3/5:  68%|██████▊   | 203/300 [00:00<00:00, 405.11it/s]

Epoch 3/5: 100%|██████████| 300/300 [00:00<00:00, 405.11it/s, train=0.0053 | valid=0.0093 | fun_rmse=0.0133]

Epoch 3/5: 100%|██████████| 300/300 [00:00<00:00, 402.35it/s, train=0.0053 | valid=0.0093 | fun_rmse=0.0133]

Epoch 4/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 4/5:  67%|██████▋   | 202/300 [00:00<00:00, 402.25it/s]

Epoch 4/5: 100%|██████████| 300/300 [00:00<00:00, 402.25it/s, train=0.0049 | valid=0.0042 | fun_rmse=0.0098]

Epoch 4/5: 100%|██████████| 300/300 [00:00<00:00, 399.97it/s, train=0.0049 | valid=0.0042 | fun_rmse=0.0098]

Epoch 5/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 5/5:  67%|██████▋   | 200/300 [00:00<00:00, 398.86it/s]

Epoch 5/5: 100%|██████████| 300/300 [00:00<00:00, 398.86it/s, train=0.0036 | valid=0.0029 | fun_rmse=0.0095]

Epoch 5/5: 100%|██████████| 300/300 [00:00<00:00, 397.64it/s, train=0.0036 | valid=0.0029 | fun_rmse=0.0095]

Diagnosing Gradient Issues¶

When training is unstable or the model isn't learning, visualizing gradients helps identify the problem. plot_grad_flow shows the gradient magnitude at each layer -- vanishing gradients appear as near-zero bars, exploding gradients as very tall bars.

Gradients are cleared after every optimizer step, so we run one fresh forward/backward pass (without stepping the optimizer) to populate them before plotting. The model must be in training mode: fit leaves it in eval mode, and cuDNN only supports RNN backward in training mode.

In [11]:

lrn_clip.model.train()
xb, yb = lrn_clip.prepare_batch(next(iter(lrn_clip.dls.train)))
pred = lrn_clip.model(xb)
if isinstance(pred, tuple):
    pred = pred[0]
lrn_clip.compute_loss(pred, yb, xb).backward()
plot_grad_flow(lrn_clip.model.named_parameters())

lrn_clip.model.train() xb, yb = lrn_clip.prepare_batch(next(iter(lrn_clip.dls.train))) pred = lrn_clip.model(xb) if isinstance(pred, tuple): pred = pred[0] lrn_clip.compute_loss(pred, yb, xb).backward() plot_grad_flow(lrn_clip.model.named_parameters())

vary_seq_len¶

Randomly truncates sequences to different lengths each batch. This acts as data augmentation by preventing the model from overfitting to a fixed window size. min_len sets the minimum allowed length.

In [12]:

lrn_vary = RNNLearner(
    dls, rnn_type='lstm', metrics=[fun_rmse],
    augmentations=[vary_seq_len(min_len=100)],
)
lrn_vary.fit_flat_cos(n_epoch=5, lr=3e-3)

lrn_vary = RNNLearner( dls, rnn_type='lstm', metrics=[fun_rmse], augmentations=[vary_seq_len(min_len=100)], ) lrn_vary.fit_flat_cos(n_epoch=5, lr=3e-3)

Epoch 1/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 1/5:  92%|█████████▏| 276/300 [00:00<00:00, 550.84it/s]

Epoch 1/5: 100%|██████████| 300/300 [00:00<00:00, 550.84it/s, train=0.0147 | valid=0.0050 | fun_rmse=0.0102]

Epoch 1/5: 100%|██████████| 300/300 [00:00<00:00, 547.20it/s, train=0.0147 | valid=0.0050 | fun_rmse=0.0102]

Epoch 2/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 2/5:  94%|█████████▎| 281/300 [00:00<00:00, 560.22it/s]

Epoch 2/5: 100%|██████████| 300/300 [00:00<00:00, 560.22it/s, train=0.0076 | valid=0.0059 | fun_rmse=0.0106]

Epoch 2/5: 100%|██████████| 300/300 [00:00<00:00, 557.34it/s, train=0.0076 | valid=0.0059 | fun_rmse=0.0106]

Epoch 3/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 3/5:  96%|█████████▋| 289/300 [00:00<00:00, 577.89it/s]

Epoch 3/5: 100%|██████████| 300/300 [00:00<00:00, 577.89it/s, train=0.0077 | valid=0.0059 | fun_rmse=0.0106]

Epoch 3/5: 100%|██████████| 300/300 [00:00<00:00, 574.56it/s, train=0.0077 | valid=0.0059 | fun_rmse=0.0106]

Epoch 4/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 4/5:  96%|█████████▌| 288/300 [00:00<00:00, 575.67it/s]

Epoch 4/5: 100%|██████████| 300/300 [00:00<00:00, 575.67it/s, train=0.0074 | valid=0.0052 | fun_rmse=0.0103]

Epoch 4/5: 100%|██████████| 300/300 [00:00<00:00, 574.28it/s, train=0.0074 | valid=0.0052 | fun_rmse=0.0103]

Epoch 5/5:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 5/5:  96%|█████████▌| 287/300 [00:00<00:00, 573.79it/s]

Epoch 5/5: 100%|██████████| 300/300 [00:00<00:00, 573.79it/s, train=0.0061 | valid=0.0030 | fun_rmse=0.0095]

Epoch 5/5: 100%|██████████| 300/300 [00:00<00:00, 570.32it/s, train=0.0061 | valid=0.0030 | fun_rmse=0.0095]

truncate_sequence¶

Progressively increases sequence length during training. Starts with short sequences (easier for the model) and gradually increases to full length. This is a form of curriculum learning that helps the model learn short-term dynamics first before tackling longer dependencies.

In [13]:

lrn_trunc = RNNLearner(
    dls, rnn_type='lstm', metrics=[fun_rmse],
    augmentations=[truncate_sequence(truncate_length=100)],
)
lrn_trunc.fit_flat_cos(n_epoch=10, lr=3e-3)

lrn_trunc = RNNLearner( dls, rnn_type='lstm', metrics=[fun_rmse], augmentations=[truncate_sequence(truncate_length=100)], ) lrn_trunc.fit_flat_cos(n_epoch=10, lr=3e-3)

Epoch 1/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 1/10: 100%|██████████| 300/300 [00:00<00:00, 904.34it/s, train=0.0182 | valid=0.0091 | fun_rmse=0.0135]

Epoch 1/10: 100%|██████████| 300/300 [00:00<00:00, 903.16it/s, train=0.0182 | valid=0.0091 | fun_rmse=0.0135]

Epoch 2/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 2/10: 100%|██████████| 300/300 [00:00<00:00, 926.75it/s, train=0.0136 | valid=0.0050 | fun_rmse=0.0104]

Epoch 2/10: 100%|██████████| 300/300 [00:00<00:00, 925.57it/s, train=0.0136 | valid=0.0050 | fun_rmse=0.0104]

Epoch 3/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 827.54it/s, train=0.0108 | valid=0.0053 | fun_rmse=0.0103]

Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 826.71it/s, train=0.0108 | valid=0.0053 | fun_rmse=0.0103]

Epoch 4/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 4/10: 100%|██████████| 300/300 [00:00<00:00, 647.95it/s, train=0.0073 | valid=0.0073 | fun_rmse=0.0117]

Epoch 4/10: 100%|██████████| 300/300 [00:00<00:00, 647.43it/s, train=0.0073 | valid=0.0073 | fun_rmse=0.0117]

Epoch 5/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 5/10:  88%|████████▊ | 263/300 [00:00<00:00, 525.47it/s]

Epoch 5/10: 100%|██████████| 300/300 [00:00<00:00, 525.47it/s, train=0.0062 | valid=0.0049 | fun_rmse=0.0101]

Epoch 5/10: 100%|██████████| 300/300 [00:00<00:00, 517.84it/s, train=0.0062 | valid=0.0049 | fun_rmse=0.0101]

Epoch 6/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 6/10:  74%|███████▍  | 223/300 [00:00<00:00, 445.12it/s]

Epoch 6/10: 100%|██████████| 300/300 [00:00<00:00, 445.12it/s, train=0.0052 | valid=0.0051 | fun_rmse=0.0101]

Epoch 6/10: 100%|██████████| 300/300 [00:00<00:00, 434.65it/s, train=0.0052 | valid=0.0051 | fun_rmse=0.0101]

Epoch 7/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 7/10:  68%|██████▊   | 203/300 [00:00<00:00, 405.04it/s]

Epoch 7/10: 100%|██████████| 300/300 [00:00<00:00, 405.04it/s, train=0.0050 | valid=0.0082 | fun_rmse=0.0124]

Epoch 7/10: 100%|██████████| 300/300 [00:00<00:00, 405.75it/s, train=0.0050 | valid=0.0082 | fun_rmse=0.0124]

Epoch 8/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 8/10:  68%|██████▊   | 205/300 [00:00<00:00, 408.59it/s]

Epoch 8/10: 100%|██████████| 300/300 [00:00<00:00, 408.59it/s, train=0.0047 | valid=0.0035 | fun_rmse=0.0095]

Epoch 8/10: 100%|██████████| 300/300 [00:00<00:00, 405.99it/s, train=0.0047 | valid=0.0035 | fun_rmse=0.0095]

Epoch 9/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 9/10:  68%|██████▊   | 205/300 [00:00<00:00, 408.23it/s]

Epoch 9/10: 100%|██████████| 300/300 [00:00<00:00, 408.23it/s, train=0.0038 | valid=0.0034 | fun_rmse=0.0096]

Epoch 9/10: 100%|██████████| 300/300 [00:00<00:00, 405.78it/s, train=0.0038 | valid=0.0034 | fun_rmse=0.0096]

Epoch 10/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 10/10:  67%|██████▋   | 202/300 [00:00<00:00, 402.37it/s]

Epoch 10/10: 100%|██████████| 300/300 [00:00<00:00, 402.37it/s, train=0.0028 | valid=0.0029 | fun_rmse=0.0096]

Epoch 10/10: 100%|██████████| 300/300 [00:00<00:00, 396.29it/s, train=0.0028 | valid=0.0029 | fun_rmse=0.0096]

Combining Options¶

Augmentations, auxiliary losses, and gradient clipping can be combined freely. Pass them all at Learner creation time.

In [14]:

lrn_combined = RNNLearner(
    dls, rnn_type='lstm', metrics=[fun_rmse],
    grad_clip=10,
)
lrn_combined.aux_losses.append(
    ActivationRegularizer(modules=[unwrap_model(lrn_combined.model).rnn], alpha=2.0)
)
lrn_combined.aux_losses.append(
    TemporalActivationRegularizer(modules=[unwrap_model(lrn_combined.model).rnn], beta=1.0)
)
lrn_combined.fit_flat_cos(n_epoch=10, lr=3e-3)
lrn_combined.show_results(max_n=2)

lrn_combined = RNNLearner( dls, rnn_type='lstm', metrics=[fun_rmse], grad_clip=10, ) lrn_combined.aux_losses.append( ActivationRegularizer(modules=[unwrap_model(lrn_combined.model).rnn], alpha=2.0) ) lrn_combined.aux_losses.append( TemporalActivationRegularizer(modules=[unwrap_model(lrn_combined.model).rnn], beta=1.0) ) lrn_combined.fit_flat_cos(n_epoch=10, lr=3e-3) lrn_combined.show_results(max_n=2)

Epoch 1/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 1/10:  63%|██████▎   | 188/300 [00:00<00:00, 374.09it/s]

Epoch 1/10: 100%|██████████| 300/300 [00:00<00:00, 374.09it/s, train=0.0112 | valid=0.0063 | fun_rmse=0.0106]

Epoch 1/10: 100%|██████████| 300/300 [00:00<00:00, 373.88it/s, train=0.0112 | valid=0.0063 | fun_rmse=0.0106]

Epoch 2/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 2/10:  63%|██████▎   | 188/300 [00:00<00:00, 374.10it/s]

Epoch 2/10: 100%|██████████| 300/300 [00:00<00:00, 374.10it/s, train=0.0065 | valid=0.0063 | fun_rmse=0.0109]

Epoch 2/10: 100%|██████████| 300/300 [00:00<00:00, 374.57it/s, train=0.0065 | valid=0.0063 | fun_rmse=0.0109]

Epoch 3/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 3/10:  63%|██████▎   | 188/300 [00:00<00:00, 375.63it/s]

Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 375.63it/s, train=0.0061 | valid=0.0046 | fun_rmse=0.0100]

Epoch 3/10: 100%|██████████| 300/300 [00:00<00:00, 378.54it/s, train=0.0061 | valid=0.0046 | fun_rmse=0.0100]

Epoch 4/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 4/10:  63%|██████▎   | 189/300 [00:00<00:00, 376.41it/s]

Epoch 4/10: 100%|██████████| 300/300 [00:00<00:00, 376.41it/s, train=0.0058 | valid=0.0062 | fun_rmse=0.0109]

Epoch 4/10: 100%|██████████| 300/300 [00:00<00:00, 373.67it/s, train=0.0058 | valid=0.0062 | fun_rmse=0.0109]

Epoch 5/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 5/10:  64%|██████▎   | 191/300 [00:00<00:00, 380.76it/s]

Epoch 5/10: 100%|██████████| 300/300 [00:00<00:00, 380.76it/s, train=0.0059 | valid=0.0074 | fun_rmse=0.0118]

Epoch 5/10: 100%|██████████| 300/300 [00:00<00:00, 376.93it/s, train=0.0059 | valid=0.0074 | fun_rmse=0.0118]

Epoch 6/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 6/10:  62%|██████▏   | 185/300 [00:00<00:00, 368.84it/s]

Epoch 6/10: 100%|██████████| 300/300 [00:00<00:00, 368.84it/s, train=0.0055 | valid=0.0034 | fun_rmse=0.0096]

Epoch 6/10: 100%|██████████| 300/300 [00:00<00:00, 370.56it/s, train=0.0055 | valid=0.0034 | fun_rmse=0.0096]

Epoch 7/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 7/10:  64%|██████▎   | 191/300 [00:00<00:00, 380.39it/s]

Epoch 7/10: 100%|██████████| 300/300 [00:00<00:00, 380.39it/s, train=0.0055 | valid=0.0053 | fun_rmse=0.0102]

Epoch 7/10: 100%|██████████| 300/300 [00:00<00:00, 377.91it/s, train=0.0055 | valid=0.0053 | fun_rmse=0.0102]

Epoch 8/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 8/10:  63%|██████▎   | 189/300 [00:00<00:00, 377.08it/s]

Epoch 8/10: 100%|██████████| 300/300 [00:00<00:00, 377.08it/s, train=0.0053 | valid=0.0037 | fun_rmse=0.0097]

Epoch 8/10: 100%|██████████| 300/300 [00:00<00:00, 375.65it/s, train=0.0053 | valid=0.0037 | fun_rmse=0.0097]

Epoch 9/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 9/10:  64%|██████▎   | 191/300 [00:00<00:00, 380.18it/s]

Epoch 9/10: 100%|██████████| 300/300 [00:00<00:00, 380.18it/s, train=0.0044 | valid=0.0037 | fun_rmse=0.0096]

Epoch 9/10: 100%|██████████| 300/300 [00:00<00:00, 380.29it/s, train=0.0044 | valid=0.0037 | fun_rmse=0.0096]

Epoch 10/10:   0%|          | 0/300 [00:00<?, ?it/s]

Epoch 10/10:  62%|██████▏   | 187/300 [00:00<00:00, 372.60it/s]

Epoch 10/10: 100%|██████████| 300/300 [00:00<00:00, 372.60it/s, train=0.0032 | valid=0.0029 | fun_rmse=0.0095]

Epoch 10/10: 100%|██████████| 300/300 [00:00<00:00, 372.54it/s, train=0.0032 | valid=0.0029 | fun_rmse=0.0095]

Key Takeaways¶

• noise and bias augment training data for better generalization. Pass them as augmentations=[...] on the Learner.
• ActivationRegularizer and TemporalActivationRegularizer smooth predictions with activation and temporal penalties. Pass them as aux_losses=[...] or via lrn.aux_losses.append(...).
• grad_clip prevents exploding gradients on long sequences.
• plot_grad_flow visualizes gradient magnitudes per layer -- use it to diagnose vanishing or exploding gradients.
• vary_seq_len acts as augmentation by varying sequence length each batch.
• truncate_sequence implements curriculum learning with progressive sequence length.
• All options compose -- combine augmentations, auxiliary losses, and gradient clipping for best results.