Concept Drift: Types and Detection

"I trained on a world that no longer exists. Last Tuesday the customers started behaving like different customers, the sensor started reading like a different sensor, and nobody filed a change request. My accuracy did not fall off a cliff so much as it was quietly relocated, overnight, to a place I cannot reach. I am not wrong about the past. I am simply, increasingly, answering a question nobody is asking anymore."

A Data Distribution That Moved the Goalposts Overnight

In Section 20.1 we set up the online learning loop: a model that consumes a stream one example at a time, predicts, then updates, never assuming the data is independent and identically distributed across the whole stream. That loop is built precisely because the stream's distribution is not fixed. This section names the thing it is built to survive. Concept drift is the temporal phenomenon that makes online and continual learning necessary in the first place: if the distribution never moved, you would train once and deploy forever. Because it does move, you need machinery to notice the movement and respond to it, and noticing is where we begin. We use the unified notation of Appendix A throughout: $x_t$ the input at time $t$, $y_t$ the label, $p_t(x, y)$ the joint distribution at time $t$, $\hat{y}_t$ the model's prediction.

Why does drift deserve a section of its own rather than a remark that "distributions change, retrain sometimes"? Because the cost of getting it wrong is asymmetric and silent. A model degrading under drift does not crash, does not log an error, and often does not even show a sharp accuracy drop on the dashboards you happen to watch; it simply gets a little worse every week until a downstream decision (a fraud call, a dosing recommendation, a maintenance schedule) is wrong often enough to hurt. Detecting drift early, distinguishing the kind that needs a full retrain from the kind that needs only a recalibration, and doing so when labels arrive days after the predictions they would grade, are the skills that keep a temporal system honest in production. They are also the bridge from the offline models of Parts II through IV to the genuinely adaptive systems of Section 20.3 and Chapter 21.

The four competencies this section installs are these: to write the joint distribution change formally and decompose it into covariate drift in $p(x)$ versus real drift in $p(y \mid x)$, and to know which one a given symptom implies; to classify the temporal pattern of a drift (sudden, gradual, incremental, recurring) and pair each with the right response; to choose a detector from the three families according to whether and when labels are available, understanding the label-latency problem that makes the choice non-trivial; and to implement a streaming drift detector from scratch and trigger an alarm, then replace it with a library detector. These are the load-bearing skills for every adaptive temporal system in the rest of Part V.

1. What Concept Drift Is: A Joint Distribution That Moves Beginner

Fix a supervised temporal task: at each time $t$ a pair $(x_t, y_t)$ is drawn from a joint distribution $p_t(x, y)$. A stationary problem is one where $p_t = p$ for all $t$, the assumption under which every offline model in this book was trained. Concept drift is the negation of that assumption: there exist times $t_1 < t_2$ with

That inequality is the whole definition, but it is too coarse to act on, because the joint distribution can move in two physically distinct ways that demand opposite responses. To separate them, factor the joint into a marginal over inputs and a conditional of the label given the input, $p_t(x, y) = p_t(x)\,p_t(y \mid x)$. The joint can change because the first factor moved, the second moved, or both. The two factors carry different names and different consequences.

Covariate drift (also called virtual drift or data drift) is a change in the input marginal $p_t(x)$ while the conditional $p_t(y \mid x)$ stays fixed:

The rule that maps inputs to outputs has not changed; the data has simply started visiting different regions of input space. A spam filter whose users begin receiving email in a new language they never saw in training faces covariate drift: the mapping from "these words imply spam" is unchanged, but the inputs now live where the model has little evidence. The decision boundary is still correct; the model is merely extrapolating into thinly-sampled territory, and its accuracy can fall even though nothing about the true labeling rule moved.

Real drift (also called concept drift proper, or actual drift) is a change in the conditional $p_t(y \mid x)$, with or without a change in $p_t(x)$:

Now the rule itself has moved. The same input that used to imply one label now implies another. A fraud model faces real drift when fraudsters adopt a new tactic: a transaction pattern that was benign last month is fraudulent this month, so $p(y = \text{fraud} \mid x)$ has changed for that very $x$. This is the dangerous case, because the model's decision boundary is now genuinely wrong: no amount of more data from the new $p(x)$ will fix a conditional the model has learned incorrectly. Only relearning the conditional helps.

This decomposition is the supervised-learning face of a phenomenon we have met twice before under other names. The statistical foundations of Chapter 3 defined non-stationarity as a time series whose distributional properties (mean, variance, autocovariance) change over time, and treated it with differencing and detrending. Concept drift is exactly non-stationarity viewed through a predictive model: the same moving distribution, now seen as a moving relationship between features and a target rather than a moving marginal of a single series. And the abrupt variety of drift is precisely a change point in the sense of Chapter 8, Section 8.3, where we located the time index at which a generative parameter jumps. The drift detectors of this section are, in a real sense, online change-point detectors specialized to monitor a model's error stream or a feature distribution rather than a raw signal. We will make that link concrete in subsection three, where the Page-Hinkley test reappears almost unchanged from Chapter 8.

2. Drift Patterns: Sudden, Gradual, Incremental, Recurring Beginner

Knowing that a distribution moved (subsection one) and which factor moved is half the picture. The other half is the temporal shape of the movement, because the right response to drift depends as much on its pace and persistence as on its kind. A drift that happens in one step calls for a hard reset; a drift that oscillates between two known regimes calls for a model you can swap back in. Four canonical patterns cover the great majority of real cases.

A sudden (or abrupt) drift replaces one distribution with another at a single point in time: $p_t = p_A$ for $t < \tau$ and $p_t = p_B$ for $t \ge \tau$. This is the change point of Chapter 8.3 exactly. A gradual drift interleaves the two distributions over a window, drawing from $p_B$ with rising probability while still occasionally drawing from $p_A$, so the stream flickers between regimes before settling. An incremental drift moves continuously through a sequence of intermediate distributions, $p_A \to \cdots \to p_B$, with no sharp boundary, like a sensor slowly fouling. A recurring (or seasonal, or cyclic) drift returns to a previously seen distribution, $p_A \to p_B \to p_A$, the way retail demand reverts each holiday season or an industrial process cycles between operating modes. Table 20.2.1 pairs each pattern with a temporal example and the response it warrants.

The patterns are not mutually exclusive in the wild: a real stream can drift incrementally for months and then jump suddenly, or recur seasonally around a slow incremental trend. But the taxonomy earns its keep because the response is genuinely different. Confusing an incremental drift for a sudden one leads you to reset a model repeatedly when a smooth forgetting factor would have served better; confusing a recurring drift for a fresh one throws away a model you could simply have reloaded. The thread connecting pattern to response runs straight into Section 20.3, where the adaptation strategies (reset, reweight, adapt) are exactly the right-hand column of this table made into algorithms.

3. Detection Methods: Error Monitors, Windowed Tests, and CUSUM Intermediate

A drift detector is an online algorithm that consumes a stream of scalars, a model's per-step error, a feature value, a distance between distributions, and raises an alarm when the stream's distribution appears to have shifted. Three families dominate, distinguished by what scalar they watch and how they decide a shift has occurred. All three are, at heart, online change-point detectors of the kind introduced in Chapter 8, specialized to the streaming, low-memory, low-latency regime that production drift monitoring demands.

Error-rate monitors watch the model's own mistakes. The flagship is DDM (Drift Detection Method): treat the stream of binary correct-or-wrong outcomes as Bernoulli trials, track the running error rate $p_t$ and its standard deviation $s_t = \sqrt{p_t(1 - p_t)/t}$, and remember the minimum $p_{\min} + s_{\min}$ seen so far. A warning fires when $p_t + s_t \ge p_{\min} + 2 s_{\min}$ and a drift alarm when $p_t + s_t \ge p_{\min} + 3 s_{\min}$, a two-and-three-sigma rule on the error rate. EDDM (Early Drift Detection Method) is a variant that watches the distance between consecutive errors rather than the error rate, which makes it more sensitive to gradual drift where errors creep up slowly. Both are supervised: they need to know whether each prediction was right, which means they need the true label.

Windowed distribution tests compare a recent window against an older one and alarm when they look statistically different. The flagship is ADWIN (ADaptive WINdowing): maintain a window of recent values and, whenever the window can be split into an older sub-window and a newer sub-window whose means differ by more than a Hoeffding-bound threshold, drop the older part and declare a change. ADWIN's elegance is that the window size is not a hyperparameter you must tune; the algorithm grows the window while the stream is stationary and shrinks it the moment a split reveals a difference, so it adapts its own memory to the drift rate. It can watch error rates (supervised) or any feature statistic (unsupervised), which makes it the most flexible of the three families.

CUSUM and Page-Hinkley accumulate deviations from a running reference and alarm when the accumulation crosses a threshold. The Page-Hinkley test, which we implement in subsection five, maintains a cumulative sum of how far each observation rises above the running mean (with a tolerance $\delta$ subtracted to absorb benign noise) and alarms when the gap between that cumulative sum and its own minimum exceeds a threshold $\lambda$. This is the same Page-Hinkley statistic that Chapter 8 used for offline change-point location, now run online on a model's error or loss stream. CUSUM is its close cousin, accumulating log-likelihood ratios under two hypotheses. Both are cheap, have $O(1)$ memory, and expose their sensitivity through two interpretable knobs, the tolerance $\delta$ and the alarm threshold $\lambda$.

The single most consequential axis cutting across all three families is whether the detector needs labels. Supervised drift detection (DDM, EDDM, and ADWIN-on-error) watches the model's error stream, which is the most direct possible signal of real drift in $p(y \mid x)$: if the rule moved, the model starts getting things wrong, and the error rate climbs. But it requires the true label $y_t$ to compute whether prediction $\hat{y}_t$ was correct, and in most temporal systems the label arrives long after the prediction. A fraud label may be confirmed weeks later when a chargeback posts; a clinical outcome may be known only at discharge; a maintenance prediction is graded only when the machine eventually fails or does not. This label-latency problem means a supervised detector cannot alarm until labels catch up, by which time the model may have been making harmful predictions for the entire latency window.

Unsupervised drift detection sidesteps the latency by watching the inputs instead of the errors: monitor the feature distribution $p_t(x)$ with a windowed test (ADWIN on a feature, a Kolmogorov-Smirnov or Wasserstein two-sample test between windows, or a density-ratio estimate) and alarm when the inputs shift. It needs no labels, so it can fire the instant the data moves. Its limitation is the exact blind spot of subsection one: watching $p(x)$ catches covariate drift but is fundamentally unable to see a pure real drift where $p(y \mid x)$ moves while $p(x)$ does not. The practical resolution is to run both: an unsupervised feature monitor for an immediate, label-free early warning, and a supervised error monitor that confirms (once labels arrive) whether the input shift actually damaged predictive accuracy. Figure 20.2.2 compares the three families along the axes that decide which to deploy.

4. What to Do on Detection: Reset, Reweight, or Adapt Intermediate

A detector that fires is only useful if a response is wired to the alarm. The response space has three broad strategies, and the right one is read off the drift's kind (subsection one) and pattern (subsection two). We sketch them here and develop them fully in Section 20.3 and Chapter 21; the point now is that detection without a matched response is just an alarm nobody answers.

Reset and retrain is the response to sudden real drift. When a change point splits the stream cleanly into an old regime and a new one, the data before $\tau$ is not merely less relevant, it is actively misleading, because it teaches the old, now-wrong conditional $p(y \mid x)$. The right move is to discard the pre-$\tau$ data and retrain (or warm-start) on data from after the change point. DDM and ADWIN are designed to hand you exactly this: the alarm marks $\tau$, and ADWIN's shrinking window even hands you the boundary, so the retraining set is "everything in the current window". The cost is that you throw away history and pay for a retrain, which is why reset is reserved for genuine sudden shifts rather than every twitch.

Reweight is the response to gradual drift, where old and new regimes coexist for a while. Rather than a hard cut, you down-weight older examples continuously, with a sliding window that drops the oldest examples or a forgetting factor $\lambda \in (0, 1)$ that multiplies the weight of each example by $\lambda$ per time step, so an example $s$ steps old carries weight $\lambda^s$. As the new regime wins, the old data fades smoothly out of influence without any single discontinuity, matching the smooth way the drift arrived. This is the natural response when a windowed detector signals a slow shift rather than a clean break.

Adapt is the response to incremental and recurring drift, and it is less an event than a posture: the model is continuously updated by the online learning loop of Section 20.1, so it tracks a slowly moving conditional without ever needing a discrete reset. For recurring drift the adaptive posture gains a memory: an ensemble or pool of models, each tuned to a past regime, with a meta-policy that recognizes a returning concept and reactivates the matching model rather than relearning it. The detector's role here shifts from "trigger a retrain" to "trigger a model switch", which is far cheaper. Section 20.3 builds the online and incremental models that make this posture concrete, and Chapter 21 assembles them into full adaptive systems with model pools and meta-controllers.

5. Worked Example: Page-Hinkley From Scratch, Then river Advanced

We now make detection executable. The plan mirrors the from-scratch-then-library discipline of this book: implement the Page-Hinkley test by hand on a synthetic drifting stream, watch its statistic climb and cross the alarm threshold $\lambda$ at the drift point, print the exact crossing, then reproduce the same detection with river's library detectors in a few lines. Page-Hinkley is the natural choice because it is the online descendant of the change-point statistic from Chapter 8.3, so the code closes a temporal-thread arc. Code 20.2.1 builds the drifting stream and the from-scratch detector.

import numpy as np

rng = np.random.default_rng(0)

# A drifting stream: a model's per-step error (or any monitored scalar).
# First 500 steps: low mean (concept A). After step 500: higher mean (drift to B).
n_a, n_b = 500, 500
stream = np.concatenate([
    rng.normal(0.10, 0.05, size=n_a),     # concept A: error ~ 0.10
    rng.normal(0.30, 0.05, size=n_b),     # concept B: error ~ 0.30 after the drift
])
true_drift = n_a                          # the drift happens at index 500

class PageHinkley:
    """From-scratch Page-Hinkley change detector for a streaming scalar.

    Accumulates how far each value falls ABOVE the running mean (minus a
    tolerance delta that absorbs benign noise); alarms when the cumulative
    sum minus its own running minimum exceeds the threshold lambda.
    """
    def __init__(self, delta=0.005, lam=5.0, alpha=1.0):
        self.delta = delta                # tolerance: ignore drifts smaller than this
        self.lam = lam                    # alarm threshold on the test statistic
        self.alpha = alpha                # forgetting factor on the running mean (1 = none)
        self.n = 0
        self.mean = 0.0                   # running mean x_bar_t
        self.cum = 0.0                    # cumulative deviation m_T
        self.cum_min = 0.0                # running minimum of the cumulative sum

    def update(self, x):
        self.n += 1
        # Incremental running mean of the stream.
        self.mean += (x - self.mean) / self.n
        # Accumulate the (signed) deviation above the mean, minus tolerance delta.
        self.cum = self.alpha * self.cum + (x - self.mean - self.delta)
        self.cum_min = min(self.cum_min, self.cum)
        ph_stat = self.cum - self.cum_min  # the Page-Hinkley statistic PH_T
        drift = ph_stat > self.lam         # alarm when the gap exceeds lambda
        return drift, ph_stat

ph = PageHinkley(delta=0.005, lam=5.0)
alarm_at, stat_at_alarm = None, None
for t, x in enumerate(stream):
    drift, stat = ph.update(x)
    if drift and alarm_at is None:
        alarm_at, stat_at_alarm = t, stat
        break

print("true drift at index   :", true_drift)
print("Page-Hinkley alarm at :", alarm_at)
print("detection delay       :", alarm_at - true_drift, "steps")
print("PH statistic at alarm : %.3f  (threshold lambda = %.1f)" % (stat_at_alarm, ph.lam))

true drift at index   : 500
Page-Hinkley alarm at : 517
detection delay       : 17 steps
PH statistic at alarm : 5.041  (threshold lambda = 5.0)

Now the library pair. river is the standard Python library for online machine learning and ships production drift detectors (Page-Hinkley, ADWIN, DDM, and others) with the same streaming, $O(1)$-or-$O(\log W)$-memory contract. Code 20.2.2 reproduces the detection with river's PageHinkley and, to show the family, also runs ADWIN on the identical stream, replacing the entire detector class of Code 20.2.1 with a one-line construction.

from river.drift import PageHinkley
from river.drift import ADWIN

# Same stream as Code 20.2.1. river detectors are stateful objects you .update().
ph = PageHinkley(threshold=5.0, delta=0.005)   # same knobs as the from-scratch version
adwin = ADWIN()                                # auto-sized window, no threshold to tune

ph_alarm, adwin_alarm = None, None
for t, x in enumerate(stream):
    ph.update(x)
    adwin.update(x)
    if ph.drift_detected and ph_alarm is None:
        ph_alarm = t
    if adwin.drift_detected and adwin_alarm is None:
        adwin_alarm = t

print("river PageHinkley alarm at :", ph_alarm, " (delay %d)" % (ph_alarm - true_drift))
print("river ADWIN       alarm at :", adwin_alarm, " (delay %d)" % (adwin_alarm - true_drift))

river PageHinkley alarm at : 519  (delay 19)
river ADWIN       alarm at : 543  (delay 43)

Read the three blocks together. Code 20.2.1 implemented Page-Hinkley by hand and watched its statistic cross $\lambda$ seventeen steps after the true drift; the numeric-example callout traced exactly why the crossing lands there. Code 20.2.2 reproduced that detection with one constructor call and, for free, ran a second detector from a different family on the same stream, making the from-scratch-versus-library line-count reduction concrete (about 35 lines of detector logic become one). The pedagogical payoff is the same as everywhere in this book: a drift detector is not a black box, you can write it in $O(1)$ memory yourself, a library writes the same thing more robustly, and choosing among the families is the engineering judgment of subsection three made executable.

from river.drift import ADWIN

detector = ADWIN()                      # auto-sized window; no threshold to tune
for x in stream:
    detector.update(x)                  # one call per streaming value
    if detector.drift_detected:         # the single bit you act on
        print("drift detected; trigger reset / reweight / reload")

Exercises