Continual and Lifelong Learning

"They retrained me on the new quarter's data and I learned it beautifully. Then someone asked me about last quarter and I looked at them as though they had invented the months. I had not lost the old weights to malice. I had simply, gradient by gradient, written the new regime directly over the place the old one used to live."

A Neural Network Terrified of Forgetting Last Quarter

In Section 20.3 we treated online learning as a stream of single updates: the model sees one example, updates, and moves on, tracking a slowly drifting target. That framing assumed the drift was gradual and that forgetting the distant past was acceptable, even desirable. This section confronts the opposite regime. Sometimes the past matters and must be retained; sometimes the change is not a gentle drift but an abrupt switch to a new task or regime that the model must learn without discarding what it already knew. A volatility model that learned the calm regime of 2017 should not become useless the moment it is updated on the turbulent regime of 2020; it should hold both. The branch of machine learning that studies how to keep learning new things while retaining old ones is continual (or lifelong) learning, and it is the natural endpoint of a chapter on learning under change. We use the unified notation of Appendix A throughout: $\boldsymbol{\theta}$ for the model parameters, $\mathcal{L}$ for a loss, $\mathcal{D}_k$ for the data of task or regime $k$.

Why does a temporal book need a section on continual learning specifically? Because temporal data is the setting where the assumption that breaks continual learning, the assumption that training data is independent and identically distributed, breaks the hardest. A time series arrives in order, its distribution shifts as regimes change (the concept drift of Section 20.2), and a deployed temporal model is almost never trained once and frozen: it is retrained as new data accrues. Every one of those retrainings is a continual-learning step in disguise, and every one of them risks erasing the model's hard-won knowledge of earlier regimes. The methods in this section are what let a temporal model accumulate competence over its operational lifetime instead of perpetually relearning and forgetting. The four competencies installed here are: to explain catastrophic forgetting and the stability-plasticity dilemma; to place any continual-learning method in one of three families and state its mechanism; to map the formal incremental scenarios onto temporal regimes, series, and sensors; and to measure forgetting and transfer with the standard metrics. These carry directly into the adaptive systems of Section 20.5.

1. Catastrophic Forgetting and the Stability-Plasticity Dilemma Beginner

Catastrophic forgetting is the empirical fact that a neural network trained sequentially on task A and then task B typically loses most of its performance on A, often dropping to near-chance, even though the very same network had ample capacity to perform both. The cause is not lack of capacity but the mechanism of gradient descent on shared weights. When you fine-tune on task B, every gradient step moves $\boldsymbol{\theta}$ to lower the task-B loss, with no term in the objective that protects the task-A loss. The weights that task A had carefully arranged are simply reused as raw material for task B, and the configuration that solved A is overwritten by the configuration that solves B. Nothing in plain backpropagation remembers that those weights were already spoken for.

This is the concrete face of a deeper tension known as the stability-plasticity dilemma. A learning system must be plastic enough to acquire new knowledge from new data, and stable enough to retain old knowledge against the interference that new learning causes. The two pull in opposite directions: a perfectly stable model (frozen weights) never forgets but also never learns anything new, while a perfectly plastic model (unconstrained fine-tuning) learns each new thing instantly and forgets the previous one just as fast. Every continual-learning method is, at bottom, a particular answer to where on this spectrum to sit and how to spend a finite budget of plasticity without spending the stability that holds the past in place.

For temporal models the stakes are concrete and constant. The world a temporal model lives in is non-stationary by definition, so the model faces a steady arrival of new regimes (a market shifting from low to high volatility), new series (a retailer adding a product line), and new sensors (a factory installing a different make of vibration probe). Each is a new "task" the model must absorb. If absorbing it erases the model's competence on the previous regimes, the model is perpetually one retraining away from uselessness, and the operational history it accumulated is thrown away every time the data shifts. Continual learning is what converts that history from a liability into an asset.

2. The Three Families: Regularization, Replay, and Parameter Isolation Intermediate

Essentially every continual-learning method belongs to one of three families, distinguished by where they add the missing protection for old knowledge: into the loss (regularization), into the data (replay), or into the architecture (parameter isolation). Understanding the three mechanisms lets you place any method, including ones invented after this book, and reason about its costs.

Regularization: anchor the important weights

Regularization methods add a penalty to the loss that discourages changing the parameters that mattered most for previous tasks. The canonical method is Elastic Weight Consolidation (EWC), introduced by Kirkpatrick and colleagues in 2017. The idea rests on a Bayesian reading: after training on task A, the posterior over parameters $p(\boldsymbol{\theta} \mid \mathcal{D}_A)$ encodes what A needs; when we then learn task B we should stay in the region of high posterior probability for A. Approximating that posterior as a Gaussian centered at the task-A optimum $\boldsymbol{\theta}_A^\star$ with precision given by the Fisher information $F$, the log-posterior becomes a quadratic penalty. The EWC objective when learning task B is

where $\mathcal{L}_B$ is the loss on the new task, $\boldsymbol{\theta}_A^\star$ are the parameters after learning A, $\lambda$ sets the strength of the anchor, and $F_i$ is the $i$-th diagonal entry of the Fisher information matrix, estimated as the expected squared gradient of the log-likelihood at $\boldsymbol{\theta}_A^\star$:

The Fisher term $F_i$ is the key. It measures how sensitive the task-A loss is to each parameter: a large $F_i$ means parameter $i$ was important for A, so the penalty $F_i(\theta_i - \theta_{A,i}^\star)^2$ pins it hard near its old value; a small $F_i$ means A barely used parameter $i$, so it is left free for B to repurpose. EWC thus does not freeze the whole network (which would block all new learning); it selectively stiffens the directions A cared about while leaving the rest plastic, a direct, per-parameter resolution of the stability-plasticity dilemma. Two influential successors keep the same quadratic-penalty shape but estimate the per-parameter importance differently: Synaptic Intelligence (SI, Zenke and colleagues 2017) accumulates importance online along the training trajectory rather than from a separate Fisher pass, and Memory Aware Synapses (MAS, Aljundi and colleagues 2018) measures importance as the sensitivity of the learned function's output to each weight, an unsupervised quantity that needs no labels.

Replay: interleave the old data

Replay (rehearsal) methods sidestep the loss penalty and instead restore the missing old data. The simplest form, experience replay, keeps a small memory buffer of examples from previous tasks and mixes them into every minibatch while training on the new task, so the gradient again contains a term pulling toward old performance. The optimizer is once more reminded that the past exists, this time through data rather than a penalty. When storing raw old data is impossible (privacy, volume, or a non-stationary stream you cannot rewind), generative replay trains a generative model on each task and, when learning the next task, samples synthetic old examples from it to interleave, exactly the temporal generative models of Chapter 17 (VAEs, diffusion models, GANs over sequences) pressed into service as a compressed, replayable memory of past regimes. Replay is consistently among the strongest families empirically, at the cost of memory for the buffer or compute for the generator.

Parameter isolation: give the new task new capacity

Parameter-isolation (dynamic-architecture) methods avoid interference structurally by dedicating different parameters to different tasks. Progressive Neural Networks (Rusu and colleagues 2016) freeze the columns trained on old tasks and add a fresh column for each new task, with lateral connections that let the new column read the old features without being able to overwrite them, so old knowledge is literally untouchable. Lighter-weight variants keep a single frozen backbone and insert small trainable adapter modules per task, the same adapter idea used to specialize the foundation models of Chapter 15. Isolation gives essentially zero forgetting (the old parameters never change) but grows the model with each task and, in its pure form, needs to know which task a test input belongs to.

3. Continual-Learning Scenarios Mapped to Temporal Settings Intermediate

The continual-learning literature distinguishes three standard scenarios, formalized by van de Ven and Tolias, that differ in what changes between tasks and what the model is told at test time. The distinction matters because methods that look strong in the easy scenario can collapse in the hard one, and because each scenario has a direct temporal analogue.

• Task-incremental: tasks arrive in sequence and the task identity is given at test time (the model is told "this input is from task 3"). This is the easiest scenario, because a task label can route the input to task-specific parameters or output heads. Temporal analogue: a forecaster that is explicitly told which regime it is in (a known calm-versus-turbulent flag), so it may keep a head per regime.
• Domain-incremental: the task structure (the label space, the prediction target) stays the same but the input distribution shifts, and the task identity is not given. The model must perform the same job across changing domains without knowing which domain it faces. Temporal analogue: the same forecasting target measured by a new sensor with a different noise profile, or the same series entering a new regime the model is not told about, exactly the unsignalled concept drift of Section 20.2.
• Class-incremental: new classes (or new series, or new fault types) appear over time and the model must discriminate among all classes seen so far without being told which task an input belongs to. This is the hardest scenario, because the model must both learn the new classes and keep the decision boundaries against all old ones. Temporal analogue: an anomaly or activity classifier that must recognize a new fault type introduced after deployment while still detecting every previously known one.

The mapping is the practical payoff of this subsection: when a temporal system faces change, identify which scenario it is, because that fixes how hard the problem is and which methods apply. A new regime you are told about is task-incremental and nearly free; a new sensor you are not told about is domain-incremental and needs a method robust to unsignalled shift; a new fault class you must fold into a single ever-growing classifier is class-incremental and is where naive methods, and even some published ones, fail outright. The anomaly-and-change-point machinery of Chapter 8 is often what tells you a new task has begun in the first place, supplying the boundary that task-incremental methods assume is given.

4. Evaluation: Average Accuracy, Forgetting, and Transfer Intermediate

A single end-of-training accuracy number cannot tell you whether a model is doing continual learning or merely fine-tuning successfully on the last task, so the field uses a small set of metrics computed from the full accuracy matrix $R$, where $R_{i,j}$ is the model's test accuracy on task $j$ after it has finished training on task $i$. Three quantities summarize this matrix, all defined over $T$ tasks.

Average accuracy is the mean performance over all tasks after the final task has been learned, the headline number for how much total competence survives:

Forgetting (equivalently the negative of backward transfer) measures how much performance on each earlier task degraded from its peak by the end of training, averaging the drop over the earlier tasks:

import torch, torch.nn as nn, torch.nn.functional as F

torch.manual_seed(0)
device = "cpu"

def make_regime(rotation, n=2000):
    """Two-class temporal feature task; 'rotation' defines the regime's decision rule."""
    X = torch.randn(n, 2)
    c, s = torch.cos(torch.tensor(rotation)), torch.sin(torch.tensor(rotation))
    R = torch.tensor([[c, -s], [s, c]])                # rotate the class boundary
    Xr = X @ R.T
    y = (Xr[:, 0] + 0.5 * Xr[:, 1] > 0).long()         # regime-specific label rule
    return X.to(device), y.to(device)

# Regime A and regime B differ by a 60-degree rotation of the decision boundary.
XA, yA = make_regime(0.0)
XB, yB = make_regime(3.14159 / 3)

def make_model():
    return nn.Sequential(nn.Linear(2, 64), nn.ReLU(), nn.Linear(64, 64),
                         nn.ReLU(), nn.Linear(64, 2)).to(device)

def train(model, X, y, steps=400, lr=0.05, penalty=None, replay=None):
    opt = torch.optim.SGD(model.parameters(), lr=lr)
    for _ in range(steps):
        opt.zero_grad()
        loss = F.cross_entropy(model(X), y)
        if replay is not None:                          # interleave old examples
            Xr, yr = replay
            loss = loss + F.cross_entropy(model(Xr), yr)
        if penalty is not None:                         # add the EWC anchor
            loss = loss + penalty(model)
        loss.backward(); opt.step()
    return model

@torch.no_grad()
def acc(model, X, y):
    return (model(X).argmax(1) == y).float().mean().item()

# Baseline: learn A, then fine-tune on B with no protection.
m = make_model()
train(m, XA, yA)
accA_after_A = acc(m, XA, yA)
train(m, XB, yB)                                        # plain fine-tuning, forgets A
print(f"plain  | A after A: {accA_after_A:.3f} | "
      f"A after B: {acc(m, XA, yA):.3f} | B after B: {acc(m, XB, yB):.3f}")

plain  | A after A: 0.969 | A after B: 0.640 | B after B: 0.971

def fisher_diag(model, X, y):
    """Diagonal Fisher: expected squared gradient of the log-likelihood at theta_A*."""
    fisher = [torch.zeros_like(p) for p in model.parameters()]
    for i in range(X.shape[0]):
        model.zero_grad()
        logp = F.log_softmax(model(X[i:i+1]), dim=1)[0, y[i]]
        logp.backward()
        for f, p in zip(fisher, model.parameters()):
            f += p.grad.detach() ** 2                   # accumulate squared gradients
    return [f / X.shape[0] for f in fisher]

# Retrain A cleanly, snapshot the optimum theta_A* and its Fisher importances.
mA = make_model(); train(mA, XA, yA)
theta_star = [p.detach().clone() for p in mA.parameters()]
F_diag = fisher_diag(mA, XA, yA)

def ewc_penalty(model, lam=4000.0):
    """Fisher-weighted quadratic anchor toward theta_A*; large F_i pins important weights."""
    pen = 0.0
    for p, p0, f in zip(model.parameters(), theta_star, F_diag):
        pen = pen + (f * (p - p0) ** 2).sum()
    return 0.5 * lam * pen

# Fix 1: EWC. Start from theta_A* and learn B with the anchor active.
m_ewc = make_model()
with torch.no_grad():
    for p, p0 in zip(m_ewc.parameters(), theta_star): p.copy_(p0)
train(m_ewc, XB, yB, penalty=ewc_penalty)
print(f"EWC    | A after B: {acc(m_ewc, XA, yA):.3f} | B after B: {acc(m_ewc, XB, yB):.3f}")

# Fix 2: replay. Keep a small buffer of regime-A examples and interleave them.
idx = torch.randperm(XA.shape[0])[:200]                 # 200-example memory buffer
m_rep = make_model()
with torch.no_grad():
    for p, p0 in zip(m_rep.parameters(), theta_star): p.copy_(p0)
train(m_rep, XB, yB, replay=(XA[idx], yA[idx]))
print(f"replay | A after B: {acc(m_rep, XA, yA):.3f} | B after B: {acc(m_rep, XB, yB):.3f}")

EWC    | A after B: 0.918 | B after B: 0.944
replay | A after B: 0.951 | B after B: 0.957

from avalanche.training import EWC
from avalanche.models import SimpleMLP

# Avalanche owns the Fisher estimate, the theta* snapshot, and the penalty injection.
# Build the model once and bind the optimizer to that same model's parameters.
model = SimpleMLP(num_classes=2, input_size=2)
strategy = EWC(model,
               optimizer=torch.optim.SGD(model.parameters(), lr=0.05),
               criterion=nn.CrossEntropyLoss(),
               ewc_lambda=4000.0,                       # the same lambda as our anchor
               train_mb_size=128, train_epochs=4)

# benchmark is built in Code 20.4.4 below (the regime stream and metric pipeline).
for experience in benchmark.train_stream:               # one experience per regime
    strategy.train(experience)                          # Fisher + penalty handled inside
    strategy.eval(benchmark.test_stream)                # logs ACC and Forgetting for us

from avalanche.benchmarks.generators import tensors_benchmark
from avalanche.training import Replay, EWC, SynapticIntelligence

# One stream of regimes, one metric pipeline, swap the strategy by name.
benchmark = tensors_benchmark(
    train_tensors=[(XA, yA), (XB, yB)],                 # regime A then regime B
    test_tensors=[(XA, yA), (XB, yB)],
    task_labels=[0, 1])
# strategy = Replay(...)  or  EWC(...)  or  SynapticIntelligence(...)  (same API)