"They handed me one sequence of numbers and asked me to speak for an entire universe of sequences that were never observed. I obliged. I called my average a population mean, my wandering correlogram a law of nature, and the single road I happened to walk the only road there ever was. Believe me at your peril: I am a sample mean wearing the costume of the truth."
A Sample Mean Pretending to Be the Truth
Chapter Overview
Part II is where this book learns to forecast, and it opens not with a model but with a vocabulary. Every classical method that follows, the ARIMA family of Chapter 5, the spectral analysis of Chapter 4, the state-space filters of Chapter 7, rests on a small set of probabilistic ideas that the practitioner must hold precisely before fitting anything. A time series is not just a column of numbers indexed by time; it is one realization of a stochastic process, a single draw from an ensemble of histories that could have happened but did not. Almost everything difficult about forecasting follows from that one fact, and almost everything this chapter does is build the language to reason about it honestly.
The chapter assembles that language in five movements. It begins with the random process itself and the uncomfortable truth that we must infer the behavior of an entire ensemble from the one path we were given, a feat that is only possible when the process is ergodic enough to let time averages stand in for ensemble averages. It then formalizes stationarity, the assumption that the statistical character of the series does not drift with time, and the unit-root pathology that breaks it, along with the differencing that repairs it. With a stationary series in hand, the autocorrelation and partial autocorrelation functions become a diagnostic language: a correlogram is a fingerprint that tells you which model class generated the data. The chapter then separates the deterministic structure most series wear on their surface, trend and seasonality, from the stochastic structure underneath, and closes with decomposition, the practice of splitting a series into trend, season, and remainder so that each piece can be modeled or removed on its own terms.
None of this is busywork for the classical era alone. The statistics introduced here reappear, barely disguised, inside the learned models of Part III. Stationarity is exactly the assumption a transformer's reversible instance normalization tries to manufacture before attention ever sees the data; autocorrelation is the structure the Autoformer block explicitly computes; the random-walk baseline is the line that deep forecasters are still, embarrassingly often, failing to beat. The temporal thread that runs through this entire book begins here: each classical idea returns in neural form, and a reader who owns the classical idea recognizes the neural one as an old friend in new clothes rather than an unfamiliar trick. Learn the vocabulary once and it pays dividends across thirty more chapters.
The chapter is also disciplined about practice. Following this book's running-dataset convention, Part II foregrounds the finance returns-and-volatility series, the series whose near-random-walk behavior makes it the perfect adversary for every stationarity test and every forecasting claim, while the hourly sensor telemetry supplies the multiple-seasonality case that decomposition was built for. By the end of the chapter you will be able to take an unfamiliar series and, before writing a single line of modeling code, decide whether it is stationary, what order of differencing it needs, what ARMA structure its correlogram suggests, and whether what remains after decomposition is genuinely the white noise that every honest model hopes to be left holding.
Prerequisites
This chapter assumes you have read Part I. From Chapter 1: Introduction to Temporal Intelligence it carries forward the idea that a sequence carries dependence a table does not, and the limits-of-predictability framing that explains why no model can beat the irreducible noise floor this chapter learns to estimate. From Chapter 2: Temporal Data Engineering it inherits a clean, regularly sampled, leakage-free series, because every statistic defined here is only meaningful once timestamps are ordered and the train and test boundary is honest. On the mathematical side it assumes comfort with basic probability: expectation, variance, covariance, the normal distribution, and the idea of an estimator and its sampling variability. Readers who want a refresher will find one in the appendices listed in the Table of Contents (Appendix A for the unified notation, Appendix B for the probability and statistics this chapter leans on most heavily). No prior time-series theory is required; every concept is built from first principles and connected to a production library through the book's "Right Tool" callouts.
If you keep one idea from this chapter, keep stationarity: it is the assumption that makes the past a usable guide to the future. When the statistical character of a series does not drift with time, the patterns you measured yesterday are still the patterns that govern tomorrow, and estimation from a single realization becomes possible at all. Almost the whole of classical time series is two moves: first make a series stationary, by differencing away a unit root or by stripping out trend and seasonality, and then exploit the stationary remainder with a model whose guarantees only hold there. When a forecast looks too good or a test statistic looks too clean, the first question is rarely about the model. It is whether the series was ever stationary in the first place.
Chapter Roadmap
- 3.1 Random Processes and Ergodicity Why a single observed series is just one realization of a stochastic process, how the mean and autocovariance functions describe the whole ensemble, and the ergodicity that lets one path's time averages estimate quantities defined over all paths.
- 3.2 Stationarity and Unit Roots Strict versus weak stationarity, why it is the assumption every model quietly depends on, the random walk and its unit root, testing with ADF and KPSS, and differencing a series to its order of integration $I(d)$.
- 3.3 Autocorrelation and Partial Autocorrelation The ACF and PACF as the fingerprint of a process, reading their decay and cutoff to guess an ARMA order, and the Ljung-Box correlogram diagnostic that tests whether residuals are genuinely white.
- 3.4 Trend and Seasonality Separating the deterministic surface of a series from its stochastic core: deterministic versus stochastic trend, seasonal dummies and Fourier harmonics, and removing both before modeling the remainder.
- 3.5 Time-Series Decomposition Splitting a series into trend, season, and remainder with classical decomposition, STL (seasonal-trend decomposition using Loess), and the robust and multiple-seasonality variants needed when the data fights back.
Once you have worked through the five sections, the Hands-On Lab below chains them into a single pre-modeling workup. Each lab step maps to a section, so the lab doubles as a review: by the time the harness reports its verdict you have applied the stationarity tests of Section 3.2, the correlogram reading of Section 3.3, the trend-and-season separation of Section 3.4, and the decomposition of Section 3.5 with your own hands, on one real series.
Hands-On Lab: Diagnose a Series Before You Model It
Objective
Run a complete pre-modeling workup on one series from the running datasets and produce a written diagnosis before fitting a single forecasting model. Given a daily finance series (or the hourly sensor series for the seasonal variant), you will test stationarity with the agreeing-and-disagreeing pair of ADF and KPSS, difference the series to its integration order $I(d)$, read its ACF and PACF to propose a candidate ARMA order, decompose the series with STL, and check the remainder for whiteness with a Ljung-Box test. The deliverable is a short report that states, with evidence, whether the series is stationary, what order of differencing it needs, what model class its correlogram points to, and whether anything systematic survives decomposition. This is the workup a careful analyst performs every time a new series lands, and it is precisely the discipline that keeps the rest of Part II honest.
What You'll Practice
- Running and reconciling the ADF and KPSS tests, whose opposite null hypotheses make their agreement a strong stationarity signal and their disagreement a diagnosis in itself, the core skill of Section 3.2.
- Differencing a series to its order of integration and confirming the difference is stationary rather than over-differenced, following Section 3.2.
- Reading the ACF and PACF as a fingerprint to propose an ARMA order, the decay-versus-cutoff logic of Section 3.3.
- Decomposing a series into trend, season, and remainder with STL, the robust workhorse of Section 3.5.
- Testing the remainder for whiteness with Ljung-Box, the correlogram diagnostic of Section 3.3 that decides whether your decomposition left signal behind.
Setup
You need pandas for the series, statsmodels for every test and the STL decomposition, and matplotlib if you want to view the correlograms. A small synthetic generator ships in the lab so it runs out of the box, and the bibliography links real series (Yahoo Finance equity data for the finance variant, the Monash archive for evenly clocked series) for when you want to diagnose genuine messiness.
pip install pandas statsmodels matplotlibSteps
Step 1: Test stationarity with ADF and KPSS together
Run both tests on the raw series. The augmented Dickey-Fuller test takes a unit root as its null, so a small p-value argues for stationarity; the KPSS test takes stationarity as its null, so a small p-value argues against it. Run them as a pair, exactly as Section 3.2 recommends, because their crossed nulls turn agreement into confidence and disagreement into a precise diagnosis (trend-stationary versus difference-stationary).
import numpy as np
import pandas as pd
from statsmodels.tsa.stattools import adfuller, kpss
def stationarity_report(x):
adf_p = adfuller(x, autolag="AIC")[1] # H0: unit root (non-stationary)
kpss_p = kpss(x, regression="c", nlags="auto")[1] # H0: stationary
return {"adf_p": adf_p, "kpss_p": kpss_p} # read the pair, not either alone
Step 2: Difference to the order of integration $I(d)$
If the pair in Step 1 says the level series is non-stationary, difference it once and re-test, increasing $d$ until both tests agree the differenced series is stationary. Stop at the smallest such $d$: over-differencing injects spurious negative autocorrelation that will mislead Step 3. This is the search for the order of integration from Section 3.2, where most financial price series settle at $d=1$ (their returns are stationary) and many sensor series at $d=0$ once seasonality is handled.
def integrate_order(x, max_d=2):
d, series = 0, pd.Series(x).dropna()
while d < max_d:
r = stationarity_report(series)
if r["adf_p"] < 0.05 and r["kpss_p"] > 0.05: # both agree: stationary
break
series = series.diff().dropna() # difference once more
d += 1
return d, series # smallest d that works
Step 3: Read the ACF and PACF to propose an ARMA order
On the stationary differenced series, compute the autocorrelation and partial autocorrelation functions and read them as a fingerprint, following Section 3.3. A PACF that cuts off sharply after lag $p$ with a slowly decaying ACF suggests an AR($p$); an ACF that cuts off after lag $q$ with a decaying PACF suggests an MA($q$); both tailing off suggests a mixed ARMA. Record the candidate order; you are not fitting yet, only narrowing the search the way a careful analyst does before reaching for an automatic order selector.
from statsmodels.tsa.stattools import acf, pacf
def correlogram(series, nlags=24):
a = acf(series, nlags=nlags) # MA(q): cuts off after lag q
p = pacf(series, nlags=nlags) # AR(p): cuts off after lag p
bound = 1.96 / np.sqrt(len(series)) # 95% significance band
return a, p, bound
Step 4: Decompose with STL
Decompose the (level, not differenced) series into trend, seasonal, and remainder components with STL, the seasonal-trend decomposition using Loess from Section 3.5. STL is the right tool here because it estimates a smoothly evolving seasonal pattern and is robust to the outliers that derail classical moving-average decomposition. Set the seasonal period to match the data (a weekly cycle of seven for daily finance, a daily cycle of twenty-four for hourly sensor telemetry).
from statsmodels.tsa.seasonal import STL
def decompose(level_series, period):
result = STL(level_series, period=period, robust=True).fit()
return result.trend, result.seasonal, result.resid # the three additive parts
Step 5: Test the remainder for whiteness with Ljung-Box
Apply the Ljung-Box test to the STL remainder, the correlogram diagnostic of Section 3.3. Its null is that the remainder is white noise, with no autocorrelation left at any tested lag. A large p-value means decomposition has extracted everything systematic and what remains is the irreducible noise floor; a small p-value means structure survived, and the candidate ARMA order from Step 3 is exactly what you should fit to the remainder next. This is the verdict the whole workup builds toward.
from statsmodels.stats.diagnostic import acorr_ljungbox
def whiteness(resid, lags=10):
lb = acorr_ljungbox(pd.Series(resid).dropna(), lags=[lags]) # H0: white noise
p = float(lb["lb_pvalue"].iloc[0])
return p, ("white: nothing left to model" if p > 0.05
else "structure remains: fit the ARMA order from Step 3")
Expected Output
Running the workup on the daily finance series prints a non-stationary verdict on the level (ADF fails to reject, KPSS rejects), a clean stationary verdict after one difference ($d=1$, the returns), a correlogram with little significant structure beyond a short lag or two, and a Ljung-Box test on the STL remainder that often fails to reject whiteness: the unwelcome but honest finding that a near-efficient market leaves little for a linear model to exploit. Running it on the hourly sensor series tells a different story: a strong daily seasonal component STL pulls out cleanly, a remainder that Ljung-Box flags as still autocorrelated, and an ACF and PACF that point to a low-order ARMA worth fitting next. The instructive contrast is the point: the same five-step workup correctly refuses to promise signal in the finance series while clearly identifying it in the sensor series, which is exactly the judgment this chapter exists to teach.
The lab is written longhand so each diagnostic is visible. In practice you would not run the stationarity search and order guess by hand. pmdarima's auto_arima runs the ADF or KPSS differencing search, selects the ARMA order by information criterion, and fits the model in a single call, doing Steps 1 through 3 and the fit at once. Nixtla's statsforecast ships an AutoARIMA and an MSTL decomposition that scale the same pipeline across thousands of series. statsmodels itself bundles every primitive the lab calls. The roughly forty lines here collapse to a handful of library calls, with the library handling the differencing loop, the order search, and the seasonal periods. Run the workup by hand once so you can read the diagnosis; lean on the library afterward so you never skip a step.
Stretch Goals
- Replace STL with MSTL on a series carrying two seasonal cycles (daily and weekly), confirm both components are extracted cleanly, and re-run the Ljung-Box test on the smaller remainder, the multiple-seasonality case of Section 3.5.
- Deliberately over-difference the finance series to $d=2$ and watch the ACF develop the spurious negative spike at lag one that Section 3.2 warns marks an over-differenced series, then confirm $d=1$ was correct.
- Fit the candidate ARMA order from Step 3 to the sensor remainder and re-run Ljung-Box on its residuals, closing the loop that Section 3.3 opens: a good model leaves a white residual, and the same test that diagnosed the data now grades the fit.
What's Next?
This chapter built the statistical vocabulary, stationarity, autocorrelation, and decomposition, that the rest of Part II treats as given. The next chapter changes the lens entirely. Chapter 4: Frequency-Domain and Spectral Analysis takes the same stationary series and views it not as a sequence of values across time but as a sum of oscillations across frequency. The autocorrelation function this chapter made central has a Fourier twin, the spectral density, and the seasonal components decomposition pulled out by hand appear there as sharp peaks at their fundamental frequencies. The frequency view will sharpen everything learned here and return, much later, inside the frequency-domain neural forecasters of Part III, another strand of the temporal thread that this chapter started weaving.
Bibliography & Further Reading
Foundational Texts
Box, G. E. P., Jenkins, G. M., Reinsel, G. C., Ljung, G. M. "Time Series Analysis: Forecasting and Control," 5th edition. Wiley, 2015. wiley.com
The book that defined the identify-estimate-diagnose workflow this chapter's lab follows; the source of the ACF and PACF identification logic of Section 3.3 and the Ljung-Box whiteness test it ends on.
Hamilton, J. D. "Time Series Analysis." Princeton University Press, 1994. press.princeton.edu
The rigorous graduate reference for stationarity, unit roots, and the asymptotic theory behind the ADF test, the formal backbone of Sections 3.1 and 3.2.
Brockwell, P. J., Davis, R. A. "Time Series: Theory and Methods," 2nd edition. Springer, 1991. link.springer.com
The mathematically careful treatment of stochastic processes, autocovariance, and ergodicity that grounds Section 3.1's account of inferring an ensemble from one realization.
Key Books
Shumway, R. H., Stoffer, D. S. "Time Series Analysis and Its Applications: With R Examples," 4th edition. Springer, 2017. stat.pitt.edu/stoffer/tsa4
An applied classic with a freely available companion site; its worked correlogram and decomposition examples mirror the diagnostic workup the lab performs.
Hyndman, R. J., Athanasopoulos, G. "Forecasting: Principles and Practice," 3rd edition. OTexts, 2021. otexts.com/fpp3
The free modern standard; its chapters on stationarity and differencing and on STL decomposition track Sections 3.2 and 3.5 almost verbatim and supply the seasonal-naive baseline this part measures against.
Articles
Dickey, D. A., Fuller, W. A. "Distribution of the Estimators for Autoregressive Time Series with a Unit Root." Journal of the American Statistical Association 74(366), 1979. jstor.org
The original unit-root test; the ADF test the lab runs in Step 1 is the augmented form of this paper's statistic, with a unit root as its null hypothesis.
Kwiatkowski, D., Phillips, P. C. B., Schmidt, P., Shin, Y. "Testing the Null Hypothesis of Stationarity against the Alternative of a Unit Root." Journal of Econometrics 54(1-3), 1992. sciencedirect.com
The KPSS test, with stationarity as its null; pairing it against ADF, exactly as Section 3.2 and the lab's Step 1 do, turns two weak tests into one strong diagnosis.
Cleveland, R. B., Cleveland, W. S., McRae, J. E., Terpenning, I. "STL: A Seasonal-Trend Decomposition Procedure Based on Loess." Journal of Official Statistics 6(1), 1990. wessa.net
The paper that introduced STL; the robust, smoothly evolving decomposition the lab applies in Step 4 and the workhorse of Section 3.5.
Bandara, K., Hyndman, R. J., Bergmeir, C. "MSTL: A Seasonal-Trend Decomposition Algorithm for Time Series with Multiple Seasonal Patterns." arXiv:2107.13462, 2021. arxiv.org/abs/2107.13462
Extends STL to series with several seasonal cycles at once; the method behind Section 3.5's multiple-seasonality variant and the lab's first stretch goal.
Tools & Libraries
statsmodels time-series analysis (tsa) documentation. statsmodels.org
The reference for the ADF, KPSS, ACF, PACF, STL, and Ljung-Box routines the lab calls in every step; statsmodels is the library this chapter is built on.
statsmodels STL decomposition API reference. statsmodels.org
The exact STL class used in the lab's Step 4, with the robust and period options that make it the right tool for the outlier-heavy finance and seasonal sensor series.
pmdarima: ARIMA estimators for Python, with auto_arima. alkaline-ml.com/pmdarima
Automates the differencing search and ARMA order selection the lab does by hand in Steps 1 through 3, the "Right Tool" that collapses the manual workup to a single call.
Nixtla statsforecast: lightning-fast statistical forecasting, with AutoARIMA and MSTL. nixtlaverse.nixtla.io/statsforecast
Scales the entire diagnostic-and-fit pipeline, including MSTL decomposition, across thousands of series, the production form of this chapter's workflow.
Datasets & Benchmarks
Godahewa, R., Bergmeir, C., Webb, G. I., Hyndman, R. J., Montero-Manso, P. "Monash Time Series Forecasting Archive." NeurIPS Datasets and Benchmarks, 2021. forecastingdata.org
A large curated collection of forecasting series across domains; the source of the evenly clocked seasonal series the lab diagnoses alongside the finance variant.
Makridakis, S., Spiliotis, E., Assimakopoulos, V. "The M4 Competition: 100,000 Time Series and 61 Forecasting Methods." International Journal of Forecasting 36(1), 2020. sciencedirect.com
The benchmark whose hundred thousand series established that simple statistical methods, properly diagnosed, remain hard to beat, the empirical case for taking this chapter's workup seriously.