Industrial and Monitoring Applications

"In my first month I fired four hundred and twelve alerts. Three were real. By month three the on-call engineers had built a mail rule that sent me straight to a folder no human opens. I was statistically excellent and operationally dead."

A Monitoring Alert That Cried Wolf One Too Many Times

In Section 8.1 we framed anomalies as low-probability events under a model of normal behavior and set out how to evaluate detectors honestly with unadjusted precision and recall, PR-AUC, and the detection-delay curve, in Section 8.2 we built statistical and distance-based detectors (robust z-score, control charts, LOF, isolation, the matrix profile) that flag a point that looks unlike its neighbors, in Section 8.3 we developed change-point detection (CUSUM, Bayesian online change-point detection, and the offline segmentation methods) that flag a shift in the generating process rather than a single outlier, and in Section 8.4 we unified detection as forecasting-residual and reconstruction scoring that flags a point when it surprises a model fit to the recent past. The reader should arrive here fluent in those tools, because this section does not introduce a new estimator; it composes the existing ones into a system and studies the system's behavior under the constraints that only appear in production. Throughout we keep to the unified notation of Appendix A: $x_t$ for the observed stream, $\hat{x}_t$ for a one-step forecast, $r_t = x_t - \hat{x}_t$ for the residual, $s_t$ for a derived anomaly score, and $\tau$ for a decision threshold. The narrative dataset for the worked example is the sensor-IoT industrial telemetry stream that has threaded through this chapter and that returns in Chapter 20 under drift and Chapter 34 at deployment.

1. Predictive Maintenance and Condition Monitoring Beginner

The oldest and most lucrative home of temporal anomaly detection is the factory floor. A rotating machine (a pump, a turbine, a gearbox, a compressor) does not usually fail without warning; it announces its decline through slow shifts in vibration, temperature, acoustic emission, and current draw, often hours or weeks before the catastrophic event. Condition monitoring is the practice of watching those sensor streams to catch the incipient fault, and predictive maintenance is the policy of scheduling intervention on the strength of that watch rather than on a fixed calendar. The economic case is overwhelming: an unplanned shutdown of a production line can cost more per hour than the entire annual budget of the monitoring system, so even a detector that fires a few hours early on a fraction of faults pays for itself many times over.

The temporal signal of an incipient fault has a characteristic shape that the detectors of this chapter are built to catch. A healthy bearing produces vibration that is stationary in the sense of Chapter 3: its statistical properties hold steady. As a defect develops (a spall on a race, a crack in a tooth), the vibration changes in three measurable ways, each of which maps onto a specific tool from earlier in this book:

• Amplitude drift (the overall vibration level creeps upward) is a change in level or variance, exactly what the CUSUM and Bayesian online change-point detectors of Section 8.3 are designed to localize.
• New periodic components at fault-characteristic frequencies appear in the spectrum, which is why condition monitoring leans on the frequency tools of Chapter 4: a fault invisible in the raw time series can be a sharp, growing peak in the power or envelope spectrum, and the score is computed on spectral features rather than the raw signal.
• Heavier tails (the distribution of vibration develops outliers from impact transients) is a change in shape that the density and distance methods of Section 8.2 catch even when the mean and variance barely move.

A mature condition monitor therefore runs all three detectors in parallel on the same stream and fuses their scores, because a given fault mode announces itself through whichever symptom its physics favors, and a monitor that watches only the raw amplitude is blind to the two symptoms that often move first.

Condition monitoring is most valuable when it is framed not as a binary healthy-or-faulty label but as an estimate of remaining useful life, the time the machine can keep running before intervention becomes mandatory. Remaining useful life turns a detection into a decision: an alert at the onset of degradation is information, but an alert that says "this bearing has roughly eighty operating hours left" is an action plan that the maintenance scheduler can optimize against. The anomaly score $s_t$ of this chapter feeds remaining-useful-life estimation in a natural way, because a monotonically rising degradation score, fit against historical run-to-failure curves, yields a calibrated time-to-event prediction. We touch the idea here only to place it: the time-to-event modeling that turns a degradation signal into a survival curve is developed properly in Chapter 18, and the adaptive recalibration that keeps such a model honest as the machine ages belongs to Chapter 21. For this chapter, the detector's job is to produce the rising score; the survival model downstream consumes it.

The frequency-domain side of condition monitoring rewards a moment of arithmetic, because the fault signatures are not vague: a defect on a specific bearing element rings at a specific, computable frequency, and the detector knows where to look. The next worked example makes that concreteness explicit and shows why a spectral feature can flag a fault that the raw amplitude entirely hides.

2. IT Observability and Security Monitoring Intermediate

The second great domain is the data center, where the streams are not vibration and temperature but request latency, error rate, throughput, queue depth, CPU and memory utilization, and the firehose of structured logs. Modern observability platforms track millions of such key-performance-indicator (KPI) time series simultaneously, and a human cannot watch them; an automated anomaly detector must. This is the lineage that produced the public benchmarks every practitioner cites: the Yahoo Webscope S5 KPI corpus, the Numenta Anomaly Benchmark (NAB) with its emphasis on early detection and its NAB score that explicitly rewards low detection delay, and the AIOps KPI competition datasets from large internet companies. These benchmarks, treated as evaluation infrastructure in Section 8.4, exist precisely because IT KPI anomaly detection is high-volume, high-stakes, and unforgiving of false alarms at scale.

Two features of IT metrics shape the detector design and recur in the worked example. First, strong and multiple seasonality: web traffic has a daily rhythm and a weekly rhythm, so a request rate that would be alarming at 3 a.m. is perfectly normal at the midday peak. A detector that ignores seasonality drowns in false alarms at every diurnal peak and misses real drops during troughs. The cure is to deseasonalize before scoring, removing the daily and weekly cycle with the seasonal decomposition of Chapter 3 so that the residual carries only the unexplained deviation, the same residual-not-raw principle as the factory floor. Second, deploy-related shifts: a software release legitimately changes the level of a metric (a new feature raises latency by a known amount), and this is a change point, not an anomaly. A monitor that pages on every deployment is worse than useless, so production systems either suppress alerts in a window after a known deploy or feed the deploy log to the change-point detector of Section 8.3 as a known breakpoint, distinguishing the expected step from an unexpected one.

The known-breakpoint mechanism is worth spelling out because it is the cleanest example of feeding external knowledge into a detector. A deploy is a change point whose time is known in advance from the deploy log, so rather than ask the change-point detector to discover it (and risk either missing it or, worse, alarming on it as if it were a fault), the monitor is told the breakpoint location and re-estimates its baseline on each side of it. The level shift the deploy caused is then absorbed into the model, the residual against the new baseline returns to flat, and a genuine fault that arrives after the deploy stands out against the corrected baseline instead of being masked by the deploy's step. A monitor that lacks this hook faces an impossible choice on every release: alarm (a false positive on a known-good change) or stay silent through the window where a bad deploy would actually show its damage. The deploy log resolves the dilemma by converting an unknown change point into a known one, which is exactly the kind of side information the adaptive feedback loop of subsection four is built to ingest.

The other half of observability is logs, not metrics, and they bend the chapter's tools in a complementary direction. A metric is already numeric and ordered; a log line is text, so before any anomaly detector can run, the stream of log messages must be turned into a time series. The standard pipeline parses each line into a template (the constant skeleton of the message, with variable fields masked) and then counts template occurrences per time window, yielding one count series per template. A surge in a rare error template, a sudden absence of a normally chatty heartbeat template, or a novel template never seen in training are all anomalies in this derived count space, and the residual-and-threshold machinery of Section 8.2 applies to each template series exactly as it does to a CPU metric. The twist logs add is the novel-template problem: an entirely new kind of message has no history to deseasonalize against, so it is scored not by a residual but by its mere novelty, the unsupervised out-of-distribution flavor of anomaly detection that returns in learned form with the sequence representations of Chapter 16.

Security monitoring is the adversarial cousin of observability, and it bends the same tools in an important direction. Network intrusion detection watches connection counts, byte volumes, port-scan patterns, and authentication failures for the statistical signature of an attack: a sudden fan-out of connections (a scan), a slow exfiltration that shows as a persistent positive residual in outbound bytes, or a change in the distribution of request types. The change-point and density methods of this chapter apply directly, but the adversarial setting adds a twist absent from machinery monitoring: the anomaly is generated by an intelligent agent who actively shapes the attack to look normal, deliberately keeping each step below threshold or mimicking the diurnal pattern. This is why security detectors favor cumulative methods like CUSUM, which integrate small persistent deviations that an instantaneous threshold would never catch, and why the cost asymmetry (a missed intrusion is catastrophic, a false alarm is merely annoying) is even sharper here than in the factory.

The cost of getting seasonality wrong is not abstract, and a concrete count makes it vivid. Take a web-traffic metric that swings between a nightly trough near $400$ requests per second and a midday peak near $4{,}000$, observed over two weeks, with one genuine incident: a real, brief traffic collapse to forty percent of baseline at three in the morning on the last day. A fixed raw threshold tuned to catch that collapse (say, alarm below $600$ requests per second) fires on the order of eighty times across the fortnight, once at every nightly trough, and the single real incident is one alarm buried among the eighty: the operator who silences the nightly noise silences the incident with it. Deseasonalize first, subtracting the per-hour median profile so the residual mean is flat, and threshold the residual at five robust standard deviations, and the same incident surfaces as a small handful of flags clustered exactly at the incident, the eighty nightly false alarms gone because the residual at the trough is small by construction. The raw threshold and the deseasonalized threshold see the identical data; one drowns and one catches, and the only difference is whether the seasonality was absorbed by a forecast before the threshold was applied. This is the seasonality insight of the previous callout, and Code 8.5.1 measures it on a simulated traffic series.

import numpy as np

rng = np.random.default_rng(2)
days, per_day = 14, 24                              # two weeks, hourly samples
n = days * per_day
t = np.arange(n)
# Web traffic: strong diurnal swing between a nightly trough and a midday peak.
base = 2000 + 1800 * np.sin(2 * np.pi * (t % per_day) / per_day - 1.2).clip(-1, 1)
traffic = np.clip(base, 200, None) * (1 + 0.05 * rng.standard_normal(n))
inc = n - per_day + 3                               # one real 3 a.m. incident, last day
traffic[inc:inc + 2] *= 0.4                         # traffic collapses to 40% of normal

# Approach A: a fixed threshold on the RAW metric (alarm below 600 rps).
raw_alarms = np.where(traffic < 600)[0]

# Approach B: deseasonalize by the per-hour median profile, threshold the residual.
hour = t % per_day
profile = np.array([np.median(traffic[hour == h]) for h in range(per_day)])
resid = traffic - profile[hour]                    # flat-mean residual r_t
scale = 1.4826 * np.median(np.abs(resid - np.median(resid)))   # robust scale
deseason_alarms = np.where(resid < -5 * scale)[0]  # 5 robust sigma below zero

print("real incident at index :", inc)
print("raw-threshold alarms   :", len(raw_alarms), "(a false-alarm storm)")
print("deseasonalized alarms  :", list(map(int, deseason_alarms)))

real incident at index : 315
raw-threshold alarms   : 80 (a false-alarm storm)
deseasonalized alarms  : [199, 223, 252, 297, 315, 316]

3. Healthcare and Finance Monitoring: The Cost Asymmetry Intermediate

The third and fourth domains push the cost asymmetry to its extreme and in opposite directions. In clinical monitoring, the stream is the patient's vital signs: heart rate, respiratory rate, blood pressure, oxygen saturation, and temperature, sampled irregularly and noisily on a hospital ward. The goal is an early-warning score that flags patient deterioration (incipient sepsis, respiratory failure, cardiac arrest) hours before the crisis, when intervention is still cheap and effective. This is the healthcare running dataset of the book, the irregularly sampled multivariate clinical series introduced in Chapter 2 and revisited under filtering in Chapter 7. The detector's structure is familiar (a model of a stable patient, a residual, a threshold), but the irregular sampling demands the careful handling of Chapter 2, and the cost structure is morally as well as economically asymmetric: a missed deterioration can be fatal, so recall is paramount, yet a ward that floods with false alarms produces alarm fatigue in nurses, who then silence the monitors, which kills the recall the false alarms were protecting. The asymmetry is real but not one-directional, and that tension is the whole design problem.

A second feature of clinical monitoring deserves emphasis because it distinguishes healthcare from the factory floor: the baseline is per-patient, not per-machine. Two healthy patients can have resting heart rates that differ by thirty beats per minute, so a population threshold is simultaneously too tight for one and too loose for the other, the same incoherence the seasonal-threshold problem exhibited in time. The fix is the same in spirit: the detector must learn or be given each patient's personal baseline (their stable operating point) and score the residual against it, so deterioration is measured as a departure from this patient's own normal rather than from a population average. Early-warning scores that page on absolute thresholds (the widely deployed track-and-trigger charts) trade this personalization away for simplicity and pay for it in both missed subtle deteriorations and a flood of false alarms on patients whose normal sits near a threshold, which is precisely the alarm-fatigue mechanism that leads nurses to silence monitors. The personalization that fixes it is the per-entity baseline that recurs in every domain of this section.

Financial monitoring inverts the volume and inverts the visibility. The streams are returns, volatilities, order-flow imbalance, and trade-size distributions, and the targets are fraud (anomalous card or account behavior), market abuse (spoofing, layering, insider-driven price moves), and risk-limit breaches. This is the finance running dataset that threads through Chapter 5 and Chapter 19. The defining feature is volatility clustering: financial series are conditionally heteroskedastic, so a return that is a five-sigma event in a calm regime is unremarkable in a turbulent one. A detector that uses a static variance will alarm continuously through any volatile period (a false-alarm storm) and miss the genuinely anomalous trade hidden inside it. The fix is to standardize the residual by a conditional volatility estimate, the GARCH machinery of Chapter 5, so that the score $s_t = r_t / \hat{\sigma}_t$ measures surprise relative to the current regime, not relative to a long-run average. The cost asymmetry in finance is double-edged in yet another way: a missed fraud costs the loss, but a false fraud alarm that blocks a legitimate transaction costs a customer and may breach a service agreement, so the threshold is set by an explicit cost ratio rather than by a statistical convention.

The four domains share the residual-and-threshold skeleton but differ sharply in the data, the dominant nuisance the detector must neutralize, the metric that matters most, and the cost asymmetry that sets the threshold. Laying them side by side, as in the comparison below, makes the common structure and the domain-specific tuning visible at once, and it explains why a single off-the-shelf detector never transfers cleanly between domains: the estimator is shared, but the deseasonalizer, the scale, and the cost ratio are not.

The cost-ratio argument of the numeric example above is not a one-off; it is the discipline that sets the threshold in every row of that table, and a second, fully worked instance fixes the arithmetic for the financial case where the asymmetry runs the other way.

4. The Operational Reality: Alert Fatigue and the Triage Loop Advanced

Everything to this point concerned the detector. The operational reality concerns the human at the other end of the alert, and it is where most monitoring projects quietly fail. The governing pathology has a name: alert fatigue. A monitor that fires too often, even at a respectable precision, trains its human operators to ignore it, and an ignored monitor has zero effective recall regardless of its statistical recall. The epigraph's monitor was "statistically excellent and operationally dead" precisely because precision below the human tolerance threshold collapses the whole system. The practical consequence is that production monitoring optimizes a constrained objective: maximize recall subject to a precision floor that keeps the humans engaged, and minimize detection delay subject to both. This is a sharper and less forgiving problem than maximizing the area under a curve, because the precision floor is set by human psychology, not by the data.

The precision/recall/delay trade-off is genuinely three-dimensional and cannot be collapsed to a single curve without losing the engineering decision. Raising the threshold buys precision at the cost of recall and delay (a higher bar means fewer false alarms but also fewer and later true catches). Adding a persistence requirement (alert only after $k$ consecutive scores exceed the threshold) buys precision and crushes false alarms but adds $k$ samples of delay, which in a fast-moving fault is exactly the delay you could not afford. Widening the deseasonalization window stabilizes the baseline and cuts false alarms but blinds the monitor to fast regime changes. Every knob trades among the three, and the right setting depends on the cost ratio of subsection three and the human tolerance of this one. There is no globally best monitor; there is only a monitor tuned to a specific cost structure and a specific operations team.

A subtle and often-missed consequence is that the three quantities are not symmetric in how a deployment should treat them. Recall is recoverable after the fact in many domains: a missed but slow fault re-triggers on the next sample, and a missed deterioration keeps deteriorating, so a monitor with imperfect instantaneous recall on a sustained event still catches it, only later. False alarms, by contrast, are not recoverable: a spurious page spends an operator's attention irreversibly, and enough of them spend the operator's trust, which is the resource the precision floor protects. Detection delay sits between the two, recoverable in the sense that a late catch is still a catch, but only up to the moment the fault becomes catastrophic, after which delay converts into a miss. This asymmetry argues for a specific deployment posture: tune first to satisfy the precision floor (protect trust), then minimize delay subject to it (catch early), and let recall on sustained events follow from the fact that a persistent fault keeps presenting itself to a trusted, low-delay monitor. The posture inverts only for the rare instantaneous, non-recurring, catastrophic event, where a single missed sample is unrecoverable and recall must dominate from the start.

The cure for alert volume is not only a higher threshold; it is structure. A production monitor wraps the raw detector in three downstream layers, each attacking the alert-fatigue problem from a different side:

• Suppression and persistence. A breach becomes an alert only after it persists for $k$ samples or clears a deduplication window, killing the single-sample flicker that produces most spurious pages. This is the persistence-and-cooldown logic the worked example builds explicitly.
• Correlation and root-cause grouping. Alerts sharing a cause (the same failing database, the same deploy, the same upstream dependency) are grouped into one incident using the dependency graph and temporal proximity, so one database failure pages once with the root cause attached rather than eighty times with the symptoms.
• Human-in-the-loop triage and feedback. A person dispositions each incident as real or not, and that disposition is captured and fed back as a label that recalibrates the threshold and retrains the model, so the monitor that paged on a benign deploy last week learns not to page on the next one.

The third layer closes the loop of Figure 8.5.1. This feedback-and-retrain cycle is the seed of the adaptive systems of Chapter 21 and the full deployment-and-monitoring discipline of Chapter 34; here we note only that a monitor without a feedback loop is frozen at its launch-day calibration and decays as the world drifts beneath it, the drift problem that Chapter 20 takes up directly. A monitor is a loop, not a line, because the world it watches does not hold still.

5. A Reference Monitoring Pipeline Advanced

It is worth drawing the whole system once, because every successful monitor across all four domains is a specialization of the same pipeline, and naming its stages lets us see exactly which earlier section built each one. Figure 8.5.1 lays out the reference pipeline as a left-to-right flow from raw ingestion to human triage and back, with a feedback arc that turns the line into a loop. Reading it left to right: the raw stream is ingested and cleaned (the data-engineering hygiene of Chapter 2); the seasonal and trend structure is removed by a forecast or decomposition (Chapter 3), leaving a residual; the residual is standardized by a local scale and turned into an anomaly score (Section 8.2, with the change-point variant of Section 8.3); the score is compared against a possibly dynamic threshold to emit a raw alert; the raw alerts are correlated and triaged with a human in the loop; and the dispositions feed back to retrain the model and recalibrate the threshold, the adaptive loop of Chapter 21.

6. Worked Example: An End-to-End Monitor on the IoT Stream Advanced

We now assemble the whole pipeline of Figure 8.5.1 into a small, complete monitor and run it on the sensor-IoT running dataset: a bearing-vibration stream with a strong daily duty cycle and an injected incipient fault that ramps up over a few hours near the end. The monitor walks every stage of the reference pipeline in turn. It deseasonalizes the stream against its daily profile (the ingest-and-deseasonalize stages), computes a standardized forecast residual against a robust scale (the residual-score stage of Section 8.2), applies a threshold with a persistence requirement (the threshold-and-alert stage of Section 8.3), and emits alerts with a cooldown so that a single sustained fault produces one alert rather than a flapping storm (the anti-flapping discipline of subsection four).

We build the monitor from scratch first, so every stage of Figure 8.5.1 is concrete rather than hidden behind a library call, then run a packaged library detector on the identical residual for comparison, the from-scratch-plus-library pairing this book uses throughout. For both we report the three production metrics that decide a monitor's fate: precision (the fraction of alerts that are real), recall (the fraction of real events caught), and detection delay (the lag from true onset to the first true alert), scored with the unadjusted, event-wise (range-aware) precision, recall, and detection-delay protocol of Section 8.1, where the single sustained fault is scored as one event window without per-point inflation. Code 8.5.2 builds the dataset and the deseasonalizing residual.

import numpy as np

rng = np.random.default_rng(85)

# --- Sensor-IoT running dataset: bearing vibration with a daily duty cycle ---
days, per_day = 21, 144          # 144 samples/day = one every 10 minutes
n = days * per_day
t = np.arange(n)
phase = 2 * np.pi * (t % per_day) / per_day
season = 1.0 + 0.6 * np.sin(phase - 0.5)        # daily load cycle (the seasonality)
noise = 0.12 * rng.standard_normal(n)
x = season + noise                               # healthy vibration amplitude

# Inject an incipient fault: a slow upward ramp over the last ~6 hours.
fault_start = n - 40                             # onset index (ground truth)
ramp = np.clip((t - fault_start) / 30.0, 0, None) * 0.9
x = x + ramp                                     # deterioration rides on top of season

# --- Stage 1-2: deseasonalize by the per-slot daily profile (Chapter 3) ---
train = fault_start - per_day                    # fit the profile on healthy history only
slot = t % per_day
profile = np.array([x[:train][slot[:train] == s].mean() for s in range(per_day)])
deseason = x - profile[slot]                     # residual r_t = x_t - seasonal forecast
print("healthy residual std :", round(deseason[:train].std(), 4))
print("residual at fault end:", round(deseason[-1], 4))

healthy residual std : 0.116
residual at fault end: 1.3871

With a clean residual in hand, Code 8.5.3 implements the rest of the pipeline from scratch: a standardized score using a rolling robust scale (the median absolute deviation, which the fault cannot inflate the way it would inflate a plain standard deviation), a dynamic threshold, and an alerting rule with a persistence requirement and a cooldown to prevent flapping.

def mad_scale(v):
    """Robust scale: median absolute deviation, rescaled to match the std
    of a Gaussian. Robust so the fault itself does not inflate the scale."""
    med = np.median(v)
    return 1.4826 * np.median(np.abs(v - med)) + 1e-9

def monitor(resid, train_end, win=288, z=4.0, persist=3, cooldown=48):
    """From-scratch monitor: standardized score, dynamic threshold, alert
    with persistence (k breaches in a row) and a cooldown (no re-alert for
    `cooldown` steps after firing, so one sustained fault gives one alert)."""
    base_loc = np.median(resid[:train_end])
    base_scale = mad_scale(resid[:train_end])      # warm-up scale before the window fills
    alerts, run, cool = [], 0, 0
    for i in range(len(resid)):
        lo = max(0, i - win)                       # rolling window of recent residuals
        scale = mad_scale(resid[lo:i]) if i > 30 else base_scale
        score = (resid[i] - base_loc) / scale      # standardized anomaly score s_t
        run = run + 1 if score > z else 0          # count consecutive breaches
        if cool > 0:
            cool -= 1                              # suppress alerts during cooldown
        elif run >= persist:                       # fire only after `persist` breaches
            alerts.append(i)
            cool = cooldown                         # open the cooldown window
            run = 0
    return alerts

alerts = monitor(deseason, train_end=train)
print("alert indices         :", alerts)
print("ground-truth onset    :", fault_start)
if alerts:
    print("detection delay (steps):", alerts[0] - fault_start)

alert indices         : [3003]
ground-truth onset    : 2984
detection delay (steps): 19

One alert is good, but a monitor is judged by the three production metrics on the whole stream, so Code 8.5.4 scores the run with the unadjusted, event-wise precision and recall and the detection delay of Section 8.1, treating the fault as a single labeled event window.

def score_monitor(alerts, onset, end, tol=6):
    """Event-wise precision/recall + detection delay over one event window.
    An alert is a true positive if it lands in [onset, end]; the event is
    recalled if at least one alert falls in the window; delay = first such hit."""
    event = set(range(onset, end + 1))
    tp = [a for a in alerts if a in event]
    fp = [a for a in alerts if a not in event]
    precision = len(tp) / len(alerts) if alerts else 0.0
    recall = 1.0 if tp else 0.0                    # single event: caught or not
    delay = (min(tp) - onset) if tp else None
    return precision, recall, delay, len(fp)

prec, rec, delay, n_fp = score_monitor(alerts, onset=fault_start, end=n - 1)
print(f"precision={prec:.2f}  recall={rec:.2f}  delay={delay} steps  false_alarms={n_fp}")

precision=1.00  recall=1.00  delay=19 steps  false_alarms=0

That was the transparent machinery, every box of Figure 8.5.1 hand-built so the pipeline is concrete rather than asserted. In production you would not hand-roll the seasonal profile, the robust scale, the persistence logic, and the scoring; a mature library packages the residual-and-threshold monitor behind a small interface. Code 8.5.5 is the library shortcut using the same dataset, here through scikit-learn's robust Gaussian estimator MinCovDet thresholded at a chi-square cutoff on the Mahalanobis distance, the principled way to flag a residual that no longer looks like healthy noise.

import numpy as np
from scipy.stats import chi2
from sklearn.covariance import MinCovDet

# Reuse the deseasonalized residual from Code 8.5.2 as the detector input.
feats = deseason.reshape(-1, 1)                    # one feature: the residual
rob = MinCovDet(random_state=0).fit(feats[:train]) # robust Gaussian of healthy noise
d2 = rob.mahalanobis(feats)                        # squared Mahalanobis distance = score
thr = chi2.ppf(0.9999, df=1)                       # chi-square cutoff sets the threshold
flagged = np.where(d2 > thr)[0]

# Collapse contiguous flags into single alerts (the cooldown, library-side).
alerts_lib = [g[0] for g in np.split(flagged, np.where(np.diff(flagged) > 48)[0] + 1) if len(g)]
prec, rec, delay, n_fp = score_monitor(alerts_lib, onset=fault_start, end=n - 1)
print("library alerts        :", [int(a) for a in alerts_lib])
print(f"precision={prec:.2f}  recall={rec:.2f}  delay={delay} steps  false_alarms={n_fp}")

library alerts        : [2995]
precision=1.00  recall=1.00  delay=11 steps  false_alarms=0

The two monitors agree where it matters (both catch the fault, both with perfect precision and recall on this stream) and differ exactly where the design choices live: the from-scratch monitor's persistence requirement makes it conservative and slow but immune to single-sample flicker, while the library's calibrated threshold makes it faster but reliant on the residual genuinely being Gaussian under health. Neither is universally better; the right choice is set by the cost ratio of subsection three (how much is each step of delay worth against each false alarm) and by how trustworthy the healthy-residual model is. What both share, and what no choice of detector escapes, is everything downstream: both produce a raw alert that still needs correlation, triage, and a feedback loop before it is a monitor a team will trust.

Step back and see what the section, the chapter, and Part II assembled. We took the detectors of the previous four sections and wired them into systems for four domains: predictive maintenance reading the residual of a health model and feeding remaining-useful-life estimation, IT and security observability deseasonalizing high-volume KPIs and integrating slow adversarial deviations, healthcare early warning trading recall against alarm fatigue, and financial surveillance standardizing by conditional volatility against an explicit cost ratio. We confronted the operational reality that a monitor's value lives downstream of the detector, in the precision floor set by human trust, the three-way precision-recall-delay trade-off, and the correlation-triage-feedback loop that turns a noisy detector into a usable monitor. We drew the reference pipeline of Figure 8.5.1 and named the earlier section that built each stage, closing the classical toolkit. And we built a complete monitor on the IoT stream twice, from scratch and through a library, reporting the precision, recall, and detection delay that decide a monitor in production.

With this, Part II is complete. The classical apparatus of statistics (Chapter 3), spectra (Chapter 4), univariate and multivariate forecasting (Chapters 5 and 6), state-space filtering (Chapter 7), and anomaly and change-point detection (this Chapter 8) is fully in hand, and this section showed it doing real work across four industries. Part III now rebuilds every one of these ideas in learned form, beginning with the neural sequence-modeling fundamentals of Chapter 9, and the residual-and-threshold monitor of this section will return as a deep detector whose model of normal behavior is learned from data rather than specified by hand. The detector changes; the loop of Figure 8.5.1 does not, because monitoring was never really about the detector.

7. LLM-Based and Foundation-Model Anomaly Detection Advanced

The detectors of Sections 8.1 through 8.5 share one expressive limitation: they produce a score, and sometimes a label, but they produce no explanation. A standardized residual of $7.3$ tells an engineer that something is twelve standard deviations above the rolling baseline, but it does not say whether the spike is a genuine fault, a scheduled maintenance window, a known-good deploy, or a sensor calibration artifact. A human expert can often distinguish these cases by combining the anomaly score with domain knowledge, but that reasoning step is manual, unscalable, and happens after the alert rather than inside the detector. The 2025 wave of LLM-based anomaly detectors is a direct response to this gap: they couple a statistical or foundation-model anomaly scorer with a large language model that can retrieve domain context, reason over it, and produce a natural-language explanation of why the flagged window is anomalous, and whether it is the kind of anomaly that requires action.

Three 2025 papers define the emerging Tier 4 anomaly detection approach and are worth studying as a coherent group, because each makes a different architectural bet about where the LLM should sit in the pipeline.

AnomaMind (arXiv 2602.13807) is the most fully agentic of the three. The LLM acts as the primary reasoning loop: it is given a time-series analysis toolkit (statistical tests, spectral tools, the CUSUM and isolation-based scorers of this chapter) and calls them as tools, interprets the returned scores, and then queries a domain knowledge base to contextualize the result. A spike at 03:00 on a Tuesday earns the query "is there a scheduled maintenance window at this time?" before the LLM decides whether to flag an alert. If the knowledge base returns a matching scheduled event, the LLM classifies the spike as expected and suppresses the alert; if it returns nothing, it classifies the spike as anomalous and generates a natural-language explanation that names the tool scores it relied on, the contextual queries it issued, and the reasoning chain that led to the classification. AnomaMind is zero-shot on new domains: the tools are domain-agnostic and the knowledge base is loaded at inference time, so no task-specific training is required. This is the full Tier 4 posture of the landscape in Section 1.7: the LLM orchestrates Tier 1 and Tier 3 tools rather than replacing them.

AXIS (arXiv 2509.24378) separates the detection and explanation roles more cleanly. A pre-trained temporal foundation model (a Tier 3 model in the sense of Section 1.7) scores each window zero-shot and outputs an anomaly score alongside the raw window features. The LLM receives the score, the raw numeric values, and any available operational context (metric name, service, recent deploys) and generates a structured explanation. A representative output looks like: "This spike at 03:47 is anomalous because the system load doubled while throughput stayed flat, suggesting a process hang rather than genuine demand." The AXIS design is cheaper than AnomaMind because the TSFM scorer is fast and the LLM is called only when the score exceeds a threshold, not on every window. The trade-off is that AXIS cannot suppress expected events the way AnomaMind can, because the LLM does not query a knowledge base proactively; it explains the score after the fact rather than reasoning about whether the score is meaningful in context.

LEFT (arXiv 2602.08638) takes a different bet: rather than coupling a separate scorer with an LLM explainer, LEFT re-represents the time series in three complementary views (raw values, frequency-domain features from the spectral tools of Chapter 4, and a trend-residual decomposition from Chapter 3) and tokenizes all three views into a format a pretrained LLM can read. The LLM detects anomalies by identifying cross-view inconsistencies: a window is anomalous when what the raw values suggest about the series state contradicts what the frequency features or the trend-residual decomposition suggest. This tri-view tokenization is zero-shot because the inconsistency detection relies on the LLM's pre-trained reasoning rather than on task-specific supervision, and it achieves competitive detection performance on standard benchmarks without any anomaly-labeled training data. LEFT is the most architectural of the three: it treats the LLM as the detector itself, using its cross-view reasoning as the anomaly scorer, whereas AnomaMind and AXIS use the LLM as a downstream interpreter of scores produced by conventional tools.

LLM-based detectors carry real limitations that any practitioner must weigh before reaching for them. Inference latency is the most immediate: an LLM call on a production window takes hundreds of milliseconds to seconds, compared to microseconds for a residual threshold or milliseconds for an isolation-forest score. On a stream of one million KPI series sampled at one-minute resolution, applying an LLM to every window is economically and computationally infeasible; the practical architecture gates the LLM behind a fast Tier 1 or Tier 3 pre-filter that routes only high-scoring windows to the expensive reasoning step, exactly the AXIS architecture. API cost is the second constraint: a single explanation call to a hosted LLM API costs between $0.001 and $0.01 USD depending on the model and context length; at a million flagged windows per day that is $1{,}000$ to $10{,}000$ per day in API fees before any false-alarm suppression. Prompt sensitivity is the third: the quality of the explanation, and to some degree the accuracy of the classification, depends on how the time series values, the anomaly score, and the operational context are presented to the LLM, and small changes in the prompt template can shift the false-positive rate substantially. These three limitations suggest that LLM-based detection is best suited to high-value, low-volume alerting scenarios (a handful of critical production systems, not a fleet of ten thousand IoT sensors) where the cost of a missed or misunderstood anomaly is high enough to justify the explanation overhead.

import numpy as np

# --- Minimal AnomaMind agent loop sketch ---
# Replace these stubs with real implementations: an LLM client,
# a knowledge-base retriever, and the detectors of Sections 8.2-8.3.

def call_statistical_scorer(window: np.ndarray) -> dict:
    """Stub: returns a dict of anomaly scores from classical tools."""
    resid = window - np.median(window)
    scale = 1.4826 * np.median(np.abs(resid)) + 1e-9
    z_score = float(np.abs(resid[-1]) / scale)
    return {"z_score": round(z_score, 3), "max_z": round(np.abs(resid / scale).max(), 3)}

def query_knowledge_base(query: str) -> str:
    """Stub: returns domain context for the query (e.g., maintenance windows)."""
    kb = {
        "maintenance 03:00": "Scheduled compressor maintenance every Tuesday 02:30-04:00.",
        "deploy 14:32": "Service v2.3.1 deployed at 14:32; latency increase expected.",
    }
    for key, val in kb.items():
        if any(word in query.lower() for word in key.split()):
            return val
    return "No matching events found."

def llm_reason(scores: dict, context: str, window_summary: str) -> dict:
    """Stub: simulates the LLM reasoning step that would call the real API."""
    is_anomalous = scores["z_score"] > 4.0 and "Scheduled" not in context
    explanation = (
        f"Z-score {scores['z_score']:.1f} exceeds threshold. "
        + (f"Context: {context} — classifying as EXPECTED, suppressing alert."
           if "Scheduled" in context
           else f"No matching event found. Classifying as ANOMALOUS.")
    )
    return {"anomalous": is_anomalous, "explanation": explanation}

def agentic_anomaly_detect(stream: np.ndarray, window_size: int = 30,
                            step: int = 10, timestamp_fn=None):
    """
    AnomaMind-style agent loop.
    For each window:
      1. Call statistical scorer tool.
      2. If score is elevated, query the knowledge base for context.
      3. Pass score + context to the LLM for reasoning and explanation.
      4. Emit alert only if LLM classifies as truly anomalous.
    """
    alerts = []
    n = len(stream)
    for start in range(0, n - window_size, step):
        window = stream[start: start + window_size]
        ts = timestamp_fn(start) if timestamp_fn else f"t={start}"

        # Step 1: fast pre-filter with classical scorer
        scores = call_statistical_scorer(window)
        if scores["z_score"] < 3.0:
            continue                                  # below pre-filter threshold; skip LLM

        # Step 2: retrieve domain context for the timestamp
        context = query_knowledge_base(f"maintenance {ts}")
        if context == "No matching events found.":
            context = query_knowledge_base(f"deploy {ts}")

        # Step 3: LLM reasoning step (expensive; only reached if pre-filter fires)
        window_summary = (f"Window ending at {ts}: last value={window[-1]:.3f}, "
                          f"median={np.median(window):.3f}")
        result = llm_reason(scores, context, window_summary)

        # Step 4: emit alert only on LLM-confirmed anomaly
        if result["anomalous"]:
            alerts.append({"timestamp": ts, "scores": scores,
                           "explanation": result["explanation"]})

    return alerts

# --- Demonstration on a simple synthetic stream ---
rng = np.random.default_rng(42)
stream = np.concatenate([
    rng.normal(0, 1, 200),           # healthy baseline
    rng.normal(0, 1, 10) + 8.0,     # true anomaly: large shift
    rng.normal(0, 1, 90),            # return to baseline
])
found = agentic_anomaly_detect(stream, window_size=30, step=5)
for alert in found:
    print(f"[ALERT] {alert['timestamp']}  z={alert['scores']['z_score']}")
    print(f"        {alert['explanation']}")

The LLM-based detectors close the chapter's coverage of anomaly detection by answering the question the score-only methods could not: not just "is this point anomalous?" but "why is it anomalous, and should it be acted on?" That capability matters most exactly where the cost of a wrong decision is highest, which is the regime that subsection three of this section identified as the one where cost-ratio reasoning dominates. A detector that produces a natural-language rationale lets the cost-ratio decision be made by a domain expert reading the explanation, not by a data scientist tuning a threshold in advance. The two postures are complementary: the threshold that pre-filters the stream and the LLM that explains the survivors are both essential, and neither replaces the other. What the LLM-based approach cannot do is replace the careful monitoring pipeline of Figure 8.5.1; it replaces only the silent score at the end of that pipeline with a readable reason.