"They told me I was just one number in a long row of timestamps, a humble coefficient minding my own narrow band. Then the Fourier transform arrived and revealed the truth: I alone carry the heartbeat of every Monday, the surge of every summer, the hum of the machine that never sleeps. The time domain saw a flat line. I am the wave underneath it, and I have been here all along."
A Frequency Bin With Delusions of Grandeur
Chapter Overview
Chapter 3 taught you to read a series along its natural axis, value against time, and the autocorrelation function was the lens that made structure visible. This chapter rotates the picture ninety degrees. The same stationary series can be described not as a list of values across time but as a recipe of oscillations across frequency: how much of the signal lives at a daily rhythm, how much at a weekly one, how much is the slow swell of a trend and how much the fast jitter of noise. Nothing about the data changes. What changes is the coordinate system, and a problem that looks tangled in time often falls apart into simple, separable pieces in frequency. This is the second lens on the same data, and learning to switch between the two views deliberately is one of the most durable skills in the whole field.
The chapter develops that lens in five movements. It opens with the Fourier transform itself and the periodogram, the raw estimate of how power is distributed across frequency, where the dominant peaks reveal exactly the seasonal cycles that decomposition extracted by hand in the previous chapter. It then confronts an uncomfortable fact: the raw periodogram never settles down no matter how much data you collect, so it builds the smoothed and parametric estimators (Welch averaging, lag windows, autoregressive spectra) that turn a noisy estimate into a consistent one. With a reliable spectrum in hand it turns to filtering, where the convolution theorem reveals that a moving average is nothing more exotic than a low-pass filter, and where isolating a single cycle becomes a matter of keeping the frequencies you want and discarding the rest. The Fourier view has one blind spot, that it assumes the frequency content never changes, so the fourth movement introduces wavelets and the spectrogram to track frequencies that drift, appear, and vanish over time.
The fifth movement is the chapter's punchline, and it is why a textbook on temporal AI spends a chapter on signal processing at all. These spectral ideas did not stay in the twentieth century. They returned, barely disguised, inside the most modern architectures in this book. FNet replaces the entire attention mechanism with a Fourier transform and loses almost nothing. FEDformer and its relatives forecast by keeping only a handful of dominant frequency components, exactly the periodogram peaks of Section 4.1. The structured state-space models of Chapter 13 compute their long convolutions through the FFT because doing it directly would be too slow. The temporal thread that runs through this book, each classical idea returning in learned form, is unusually literal here: the operations a deep network performs on a sequence are, often enough, the very spectral operations this chapter derives from scratch.
The chapter keeps its feet on the ground throughout. Following the running-dataset convention, it foregrounds a sensor-and-IoT style energy series, the kind of multi-seasonal telemetry whose overlapping daily and weekly rhythms are nearly impossible to disentangle by eye but separate cleanly the moment you look at them in frequency. By the end you will be able to take an unfamiliar series, compute and trust a spectral estimate, read off its dominant periods, build a filter that isolates a single cycle, decide with a spectrogram whether that cycle is stable or drifting, and recognize the same machinery the next time it surfaces inside a neural forecaster.
Prerequisites
This chapter builds directly on Chapter 3: Statistical Foundations, and you should be comfortable with stationarity, the autocorrelation function, and the trend-and-seasonality decomposition it developed, because the spectral density introduced here is the Fourier counterpart of that autocorrelation, and the seasonal peaks you will read off the periodogram are the same cycles decomposition pulled out by hand. On the mathematical side the chapter assumes fluency with complex numbers (the Fourier transform lives in the complex plane, and Euler's formula $e^{i\theta} = \cos\theta + i\sin\theta$ is used without ceremony) and with basic calculus (integrals over frequency, the idea of a continuous transform that the discrete version approximates). None of this is heavy, but it is assumed. Readers who want a refresher on complex exponentials, the unified notation, or the probability that underlies the spectral density will find the relevant appendices listed in the Table of Contents (Appendix A collects the notation and the complex-number and Fourier identities used here). No prior signal-processing background is required; every transform is built from first principles and connected to a production library through the book's "Right Tool" callouts.
If you keep one idea from this chapter, keep this: the time domain and the frequency domain are two views of one signal, and choosing the right one can turn a hard problem into an easy multiplication. A convolution in time, the slow sliding-and-summing that defines every moving average and every filter, becomes a simple pointwise product in frequency. A seasonal pattern that smears across thousands of timestamps collapses into a single sharp peak. Most of the power of spectral methods comes from this one duality: when an operation is awkward in one domain, transform to the other, where it is often trivial, then transform back. The art is knowing which view to be in, and this chapter is a training course in that judgment.
Chapter Roadmap
- 4.1 Fourier Analysis and the Periodogram The frequency-domain view of a series, the discrete Fourier transform and its frequency bins, the Nyquist limit and aliasing, and the periodogram as a raw power-spectrum estimate (with the spectral leakage and windowing that keep it honest) used to recover planted cycles from a noisy multi-sinusoid signal.
- 4.2 Spectral Density Estimation Why the raw periodogram is unbiased but inconsistent, the Wiener-Khinchin link between the spectral density and the autocovariance, and the Bartlett, Welch, Daniell, and autoregressive (Yule-Walker, Burg) estimators that turn a noisy estimate into a trustworthy one.
- 4.3 Filtering in the Frequency Domain The convolution theorem and why a moving average is a low-pass filter, the frequency response of low-, high-, band-pass, and band-stop families, windowed-sinc and Butterworth design, zero-phase filtering, and the Hodrick-Prescott and Baxter-King filters that extract trend and cycle.
- 4.4 Wavelets and Time-Frequency Analysis Why the Fourier transform loses all time information, the short-time Fourier transform and its spectrogram, the time-frequency uncertainty trade-off, and the scaled-and-shifted wavelets that give multi-resolution analysis, denoising, and transient detection when frequencies drift.
- 4.5 Connections to Modern Architectures How the spectral ideas of this chapter return inside deep learning: spectral bias, Fourier token mixing (FNet), frequency-enhanced forecasters (Autoformer, FEDformer, FreTS), and the FFT view of state-space models, with a runnable frequency-mixer layer.
Once you have worked through the five sections, the Hands-On Lab below chains them into a single spectral investigation. Each lab step maps to a section, so the lab doubles as a review: by the time the harness reports the cycles it found you will have computed the periodogram of Section 4.1, the Welch spectrum of Section 4.2, the band-pass filter of Section 4.3, and the time-frequency analysis of Section 4.4 with your own hands, closing on the FFT-based smoother that previews the neural connection of Section 4.5.
Hands-On Lab: Find the Hidden Cycles
Objective
Take a real-ish multi-seasonal energy series, the kind of building or household power-consumption telemetry that carries a daily usage rhythm and a weaker weekly one buried under noise, and recover its hidden cycles using nothing but the spectral toolkit of this chapter. You will compute the periodogram to spot candidate periods, smooth it into a consistent Welch spectrum to confirm which peaks are real, build a band-pass filter to isolate the daily cycle as a clean reconstructed wave, use a spectrogram or continuous wavelet transform to decide whether that daily cycle holds steady or drifts across the record, and finally apply a tiny FFT-based smoother that previews how a neural frequency mixer operates. The deliverable is a short report naming the dominant periods you found, the cycle you isolated, your verdict on whether it is stationary in frequency, and the effect of the spectral smoother. This is exactly the workup an analyst runs when an unlabeled sensor stream lands and the question is simply: what rhythms live in here?
What You'll Practice
- Computing a periodogram and converting its peak frequencies into human-readable periods, the core skill of Section 4.1, including watching out for the spectral leakage that smears a sharp peak.
- Replacing the noisy raw periodogram with a Welch averaged spectrum so the real peaks survive and the noise floor flattens, the consistency fix of Section 4.2.
- Designing and applying a zero-phase band-pass filter to isolate a single cycle and reconstruct it as a clean wave, following Section 4.3.
- Reading a spectrogram or continuous wavelet transform to decide whether a cycle is stable or drifting in time, the time-frequency skill of Section 4.4.
- Applying an FFT-based low-pass smoother by zeroing high-frequency coefficients, the hand-built ancestor of the frequency-mixer layer of Section 4.5.
Setup
You need numpy and scipy for every transform and filter, and matplotlib if you want to view the spectra and spectrogram; PyWavelets is optional for the wavelet stretch. A small synthetic generator ships in the lab so it runs out of the box, planting a daily cycle, a weaker weekly one, and noise, but the bibliography links the real UCI individual-household electric-power-consumption series and the Monash energy series for when you want genuine messiness. Sampling is hourly throughout, so a period of twenty-four samples is one day and one hundred sixty-eight is one week.
pip install numpy scipy matplotlib pywaveletsSteps
Step 1: Compute the periodogram and read off candidate periods
Compute the raw periodogram of the hourly series and convert each strong frequency peak into a period in hours, following Section 4.1. With hourly sampling you expect a dominant peak near a frequency of one cycle per twenty-four samples (the daily rhythm) and a fainter one near one cycle per one hundred sixty-eight samples (the weekly rhythm). The raw periodogram will look jagged, that is the inconsistency Step 2 repairs, but its largest peaks are already informative. Detrend first so a slow drift does not pile spurious power at the lowest frequencies.
import numpy as np
from scipy import signal
def candidate_periods(x, fs=1.0, top=5):
f, pxx = signal.periodogram(x, fs=fs, detrend="linear") # raw spectrum, drift removed
f, pxx = f[1:], pxx[1:] # drop the zero-frequency bin
peaks = np.argsort(pxx)[-top:][::-1] # the strongest frequencies
return [(1.0 / f[i]) for i in peaks] # period in samples (= hours)
Step 2: Confirm the peaks with a Welch spectrum
The raw periodogram of Step 1 never gets smoother as you add data, so confirm which peaks are real by computing a Welch spectrum, the segment-averaging estimator of Section 4.2. Welch splits the series into overlapping windowed segments, computes a periodogram of each, and averages them, trading a little frequency resolution for a dramatic drop in variance. The genuine daily and weekly peaks survive the averaging because they sit at the same frequency in every segment; the noise, which scatters randomly, averages down toward a flat floor.
def welch_spectrum(x, fs=1.0, seg=512):
f, pxx = signal.welch(x, fs=fs, nperseg=seg, detrend="linear") # averaged, consistent
return f, pxx # peaks now stand clear
Step 3: Isolate the daily cycle with a band-pass filter
Build a zero-phase band-pass filter around the daily frequency and apply it to the series to reconstruct the daily cycle in isolation, following Section 4.3. A Butterworth band-pass keeps a narrow band of frequencies on either side of one cycle per twenty-four hours and rejects the rest, and applying it with filtfilt rather than lfilter runs the filter forwards and backwards so the output has zero phase delay and the recovered wave stays aligned with the original. The result is the daily rhythm, lifted clean of trend, noise, and the weekly cycle.
def isolate_cycle(x, fs=1.0, period=24.0, half_width=0.15):
f0 = 1.0 / period # center on the daily frequency
low, high = f0 * (1 - half_width), f0 * (1 + half_width) # narrow band around it
b, a = signal.butter(4, [low, high], btype="band", fs=fs)
return signal.filtfilt(b, a, x) # zero-phase: no time shift
Step 4: Check stability with a spectrogram
A single spectrum assumes the daily cycle is equally strong throughout the record, but real telemetry often has a rhythm that strengthens on workdays and fades on weekends or holidays. Compute a spectrogram, the short-time Fourier transform of Section 4.4, to see how the power at the daily frequency evolves across time. A horizontal band of constant brightness at one cycle per twenty-four hours means a stable cycle; a band that brightens and dims, or drifts to a slightly different frequency, means the cycle is non-stationary and a single Fourier spectrum was hiding that fact. The continuous wavelet transform from PyWavelets is the stretch-goal alternative when you want sharper time localization of brief transients.
def time_frequency(x, fs=1.0, seg=256):
f, t, sxx = signal.spectrogram(x, fs=fs, nperseg=seg, noverlap=seg // 2)
return f, t, sxx # power at each frequency over time
Step 5: Smooth with a tiny FFT-based low-pass
Finish by denoising the series with a smoother built entirely in the frequency domain, the hand-made ancestor of the neural frequency mixers of Section 4.5. Transform the series with the FFT, zero out every coefficient above a cutoff frequency (discarding the fast noise while keeping the slow structure), and transform back. This is a low-pass filter expressed as pure multiplication in frequency, the convolution theorem of Section 4.3 made concrete, and it is structurally the same move FNet and FEDformer make when they keep only low-frequency components and drop the rest.
def fft_smoother(x, keep_frac=0.05):
X = np.fft.rfft(x) # to the frequency domain
cutoff = int(len(X) * keep_frac) # keep only the lowest frequencies
X[cutoff:] = 0.0 # zero the high-frequency noise
return np.fft.irfft(X, n=len(x)) # back to a smoothed time series
Expected Output
Running the workup on the energy series prints a dominant period near twenty-four hours and a secondary one near one hundred sixty-eight hours from Step 1, both confirmed as clean peaks standing above a flat noise floor once Welch averaging is applied in Step 2. Step 3 returns a smooth daily sinusoid, aligned with the original peaks because the filter is zero-phase, that visibly tracks the day-night usage swing. Step 4 typically reveals that the daily band is bright and steady on workdays and dims across weekends, the honest finding that the cycle is present but not perfectly stationary, which is precisely why a spectrogram earns its place over a single spectrum. Step 5 returns a smoothed series that keeps the daily and weekly swells while shedding the high-frequency jitter, and comparing it to the band-pass reconstruction makes the point of the chapter concrete: the same series, viewed through frequency, gives up secrets that no amount of staring at the time plot would reveal.
The lab is written longhand so each spectral operation is visible, but every step is already a one-liner in a mature library. scipy.signal.welch gives a consistent spectrum, scipy.signal.butter with filtfilt gives a zero-phase band-pass, scipy.signal.spectrogram gives the time-frequency view, and pywt.cwt gives the continuous wavelet transform for sharper transient localization. The roughly forty lines here collapse to a handful of calls, with the library handling windowing, segment overlap, filter stability, and the FFT bookkeeping. Run the workup by hand once so you understand what each estimate is doing and why the raw periodogram could not be trusted; lean on scipy and PyWavelets afterward so you never reimplement a windowed transform under deadline.
Stretch Goals
- Swap the spectrogram of Step 4 for a continuous wavelet transform with PyWavelets and compare how the two localize a brief transient (a sudden spike in consumption), the multi-resolution advantage of Section 4.4.
- Down-sample the hourly series to one sample every three hours and watch the weekly peak shift or fold as the daily cycle approaches the Nyquist limit, making the aliasing warning of Section 4.1 concrete.
- Wrap the Step 5 smoother as a small PyTorch module operating on
torch.fft.rfftoutput, learn the keep-mask instead of fixing it, and you have built a one-layer version of the frequency mixer that Section 4.5 traces through FNet and FEDformer.
What's Next?
This chapter gave you a second lens, the frequency domain, and the toolkit to estimate, filter, and track a spectrum, then showed those same operations returning inside modern neural forecasters. The next chapter returns to the time domain to build the workhorses of classical forecasting. Chapter 5: Univariate Forecasting Models takes the stationary series you now know how to diagnose (Chapter 3) and analyze in frequency (this chapter) and finally fits models that predict its future: the AR, MA, and ARIMA family, exponential smoothing, and the seasonal extensions that turn the cycles you found here into forecasts. The spectral peaks of this chapter and the ARIMA parameters of the next are two descriptions of the same autocorrelation structure, and the ARIMA model will itself return, much later, as a form of linear attention, another strand of the temporal thread.
Bibliography & Further Reading
Foundational Texts
Oppenheim, A. V., Schafer, R. W. "Discrete-Time Signal Processing," 3rd edition. Pearson, 2009. pearson.com
The standard graduate reference for the discrete Fourier transform, filter design, and the convolution theorem; the formal backbone for the periodogram of Section 4.1 and the filtering of Section 4.3.
Percival, D. B., Walden, A. T. "Spectral Analysis for Physical Applications: Multitaper and Conventional Univariate Techniques." Cambridge University Press, 1993. cambridge.org
The careful treatment of spectral density estimation, bias, variance, and tapering that grounds Section 4.2's account of why the raw periodogram is inconsistent and how averaging repairs it.
Bloomfield, P. "Fourier Analysis of Time Series: An Introduction," 2nd edition. Wiley, 2000. wiley.com
An accessible bridge from time-series statistics to the frequency domain; its periodogram and smoothing chapters track Sections 4.1 and 4.2 for the statistically minded reader.
Key Books
Percival, D. B., Walden, A. T. "Wavelet Methods for Time Series Analysis." Cambridge University Press, 2000. cambridge.org
The definitive statistical treatment of the discrete and continuous wavelet transforms; the reference behind Section 4.4's multi-resolution analysis, denoising, and time-frequency localization.
Mallat, S. "A Wavelet Tour of Signal Processing: The Sparse Way," 3rd edition. Academic Press, 2008. elsevier.com
The canonical text on wavelets and time-frequency analysis; its treatment of the short-time Fourier transform and the uncertainty trade-off underpins Section 4.4.
Papers
Cooley, J. W., Tukey, J. W. "An Algorithm for the Machine Calculation of Complex Fourier Series." Mathematics of Computation 19(90), 1965. jstor.org
The fast Fourier transform; the algorithm that makes every periodogram, filter, and spectrogram in this chapter (and the FFT inside the state-space models of Chapter 13) computationally feasible.
Welch, P. D. "The Use of Fast Fourier Transform for the Estimation of Power Spectra: A Method Based on Time Averaging Over Short, Modified Periodograms." IEEE Transactions on Audio and Electroacoustics 15(2), 1967. ieeexplore.ieee.org
The segment-averaging spectral estimator the lab uses in Step 2; the method that turns the inconsistent periodogram of Section 4.2 into a usable, low-variance spectrum.
Lee-Thorp, J., Ainslie, J., Eckstein, I., Ontanon, S. "FNet: Mixing Tokens with Fourier Transforms." arXiv:2105.03824, 2021. arxiv.org/abs/2105.03824
Replaces self-attention with an unparameterized Fourier transform and loses almost nothing; the most literal return of this chapter's spectral ideas in deep learning, traced in Section 4.5.
Zhou, T., Ma, Z., Wen, Q., Wang, X., Sun, L., Jin, R. "FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting." arXiv:2201.12740, ICML 2022. arxiv.org/abs/2201.12740
Forecasts by keeping only a few dominant frequency components, exactly the periodogram peaks of Section 4.1; the headline example of Section 4.5's claim that spectral selection is a modern forecasting primitive.
Tools & Libraries
scipy.signal: signal-processing routines for Python (periodogram, welch, butter, filtfilt, spectrogram). docs.scipy.org
The library every lab step is built on; supplies the periodogram and Welch estimators, the Butterworth filter and zero-phase filtfilt, and the spectrogram used across Sections 4.1 through 4.4.
numpy.fft: the discrete Fourier transform in NumPy (rfft, irfft, fftfreq). numpy.org
The raw FFT the chapter derives from first principles and the lab's Step 5 smoother calls directly; the same transform that torch.fft mirrors inside the neural mixers of Section 4.5.
PyWavelets: wavelet transforms in Python (discrete and continuous). pywavelets.readthedocs.io
The library for the continuous and discrete wavelet transforms of Section 4.4 and the lab's wavelet stretch goal; supplies cwt for sharp time-localization of transients.
Datasets & Benchmarks
Hebrail, G., Berard, A. "Individual Household Electric Power Consumption." UCI Machine Learning Repository, 2012. archive.ics.uci.edu
A multi-year minute-level household power series with strong daily and weekly rhythms; the real-world energy telemetry the lab's hidden-cycle workup is designed for.
Godahewa, R., Bergmeir, C., Webb, G. I., Hyndman, R. J., Montero-Manso, P. "Monash Time Series Forecasting Archive." NeurIPS Datasets and Benchmarks, 2021. forecastingdata.org
A large curated collection that includes evenly clocked electricity and energy series; the source of the multi-seasonal data on which the spectral workup of this chapter is most instructive.