"For most of this book I lived on a line. The past trailed off to my left, the future opened to my right, and every value I ever predicted had exactly one neighbor before it and one after, a tidy little procession of moments. Then someone handed me a city. Suddenly a value was not just after another value, it was also next to it, across the river from it, three blocks downwind of it, and the jam on the bridge at noon was busy becoming the jam on the avenue at half past. I had spent my whole education learning that the future depends on the past, and now I had to learn the harder lesson, that the future here also depends on what is happening over there. A sequence I could follow. A map that moves I had to learn to think in, where every point keeps two kinds of company at once, its own history and its neighbors' present, and forecasting means listening to both."
A Sequence Model That Discovered It Had Neighbors
Chapter Overview
Every model in the book so far has treated a time series as a line: a value, the value before it, the value before that, a single chain of cause running backward into the past. This chapter adds the dimension the line was always missing. Real temporal data rarely lives in isolation; it lives somewhere. A traffic sensor reads not just its own recent history but the congestion arriving from the road upstream. A taxi's next location is shaped by where it has been and by the layout of the city around it. A pixel in a video frame means little alone and almost everything once you know what moved into it from the frame before. The moment you let a temporal signal sit in space, the question shifts from what comes next here to what comes next here given what is happening there, and that coupling of the temporal axis with a spatial structure (a graph, a grid, a coordinate plane) is the subject of spatio-temporal intelligence.
Five threads carry the chapter, each a different place where space and time intertwine. Mobility modeling treats human and vehicle movement as trajectories over a map, asking where someone will go next and confronting the surprising finding that human movement is far more predictable than it feels. Traffic forecasting puts sensors on a road network, represents the network as a graph, and predicts how congestion propagates along it, the canonical proving ground for spatio-temporal graph models. Video understanding treats a clip as a stack of frames, a spatial grid that evolves in time, and asks what is happening across both axes at once. Activity recognition reads streams from wearable and ambient sensors, or skeleton joints over time, to name what a person is doing. And underneath all of them sits the unifying abstraction: the spatio-temporal graph, where nodes carry time series, edges carry spatial relationships, and both the signal on the nodes and sometimes the graph itself evolve.
The recurring move is to combine a spatial operator with a temporal one. The sequence models of Part III still do the work along the time axis: a recurrent network, a temporal convolution, or an attention mechanism reads each location's history. What is new is a spatial operator working along the other axis: a graph convolution that mixes a node with its neighbors, a 2D or 3D convolution that mixes a pixel with the patch around it, an attention that lets distant locations talk. Architectures in this chapter are largely recipes for interleaving these two operators, diffusion convolution wrapped in a recurrence, spatial graph convolution stacked with temporal convolution, factorized space-and-time attention, so that a representation captures both how a place evolves and how places influence one another.
This is also where several of the book's running threads converge on something concrete. The temporal graphs introduced in Chapter 18 become the load-bearing data structure; the windowing and leakage discipline of Chapter 2 governs how spatio-temporal samples are cut so that the future never leaks across a node's neighbors; and the attention and transformer machinery of Part III reappears, now made to attend across space as well as time. The Sensor/IoT dataset that has threaded the book reaches its richest form here, no longer a single telemetry stream but a network of sensors whose readings forecast one another. By the end of the chapter a temporal model is no longer a creature of the line. It is a citizen of a moving map.
Prerequisites
This chapter assumes the sequence models of Part III, and especially Chapter 12: Attention and Transformers, because every architecture here pairs a spatial operator with one of the temporal operators built in that part, a recurrence, a temporal convolution, or an attention mechanism reading each location's history. It builds directly on the temporal-graph material of Chapter 18.3: temporal graphs, in Chapter 18: Event and Sequence Modeling, since the spatio-temporal graph (nodes carrying time series, edges carrying spatial relations, both evolving) is the central abstraction of this chapter and the generalization of the dynamic graphs introduced there. And it leans on the windowing and leakage discipline of Chapter 2: Temporal Data Engineering, because cutting spatio-temporal samples correctly is subtler than cutting a single series: the future must not leak across a node's spatial neighbors any more than across its own past. Readers wanting to refresh attention mechanisms, graph neural network basics, or temporal windowing before diving in will find the refreshers through the Table of Contents.
If you keep one idea from this chapter, keep this: a spatio-temporal model predicts the future at a location by listening to two things at once, that location's own history and its spatial neighbors' present, and almost every architecture here is a recipe for interleaving a temporal operator with a spatial one. Mobility models forecast movement over a map and exploit how predictable human travel actually is; traffic forecasters put sensors on a graph and predict congestion as it diffuses along the network; video and activity models treat a clip or a sensor stream as a spatial structure evolving in time; and the spatio-temporal graph unifies them all as nodes that carry time series, edges that carry spatial relationships, and a structure that can itself change.
Chapter Roadmap
- 32.1 Mobility Modeling Movement as trajectories over a map: next-location and destination prediction, the limits of predictability in human mobility, sequence models over visited places, and the tooling that turns raw GPS traces into trips, stays, and flows.
- 32.2 Traffic Forecasting Predicting congestion on a sensor network: representing a road network as a graph, diffusion convolutional recurrent networks (DCRNN), spatio-temporal graph convolution (STGCN) and Graph WaveNet, and the METR-LA and PEMS benchmarks that anchor the field.
- 32.3 Video Understanding A clip as a spatial grid evolving in time: 3D convolutions and the inflated I3D backbone, the SlowFast two-pathway design, video transformers such as TimeSformer with factorized space-time attention, and self-supervised video pretraining with VideoMAE.
- 32.4 Activity Recognition Naming what a person is doing from streams: wearable and ambient sensor windows, skeleton-based recognition with spatial-temporal graph convolution (ST-GCN), the windowing and segmentation discipline that makes it work, and the link back to the Sensor/IoT running thread.
- 32.5 Dynamic and Spatio-Temporal Graph Learning The unifying view: nodes carrying time series and edges carrying spatial relations, learned versus fixed adjacency, graphs whose structure itself evolves over time, and the PyTorch Geometric Temporal library that packages the whole family.
Once you have worked through the five sections, the Hands-On Lab below builds three spatio-temporal models in one sitting, a mobility predictor, a traffic forecaster on a sensor graph, and an activity classifier, so the lab doubles as a review. By the time you finish you will have predicted a next location with the mobility models of Section 32.1, forecast congestion on a graph with the methods of Section 32.2, classified an activity from sensor windows following Section 32.4, and seen all three expressed in the unifying spatio-temporal graph language of Section 32.5, with the video architectures of Section 32.3 as the fourth corner of the same idea.
Hands-On Lab: Predict Across Space and Time
Objective
Build three spatio-temporal predictors that share one idea expressed three ways: that the future at a location depends on both its own history and its neighbors. You will first build a next-location mobility predictor over a sequence of visited places, the trajectory-modeling task of Section 32.1, and measure how much a place's recent history alone can tell you before any spatial structure is added. You will then build a spatio-temporal traffic forecaster on a sensor graph, representing the road network as an adjacency matrix and interleaving a graph convolution with a temporal operator following Section 32.2, so that each sensor's forecast draws on the readings diffusing in from its neighbors. Finally you will build a sensor-based activity classifier over windowed multivariate streams, the recognition task of Section 32.4, and read all three models through the single lens of the spatio-temporal graph of Section 32.5. The thread is that a line is a special case of a graph: once you let a time series have neighbors, mobility, traffic, and activity become three views of one problem.
What You'll Practice
- Building a next-location predictor over a trajectory of visited places, the mobility-modeling task of Section 32.1.
- Representing a road network as a graph and forecasting congestion by interleaving a spatial graph operator with a temporal one, following Section 32.2.
- Classifying activity from windowed multivariate sensor streams, the recognition discipline of Section 32.4.
- Cutting spatio-temporal windows so the future leaks neither across a node's past nor across its spatial neighbors, the leakage discipline of Chapter 2 carried into space.
- Reading mobility, traffic, and activity as three instances of one spatio-temporal graph, the unifying view of Section 32.5.
Setup
You need a Python environment with numpy and pandas for the series, torch for the models, and torch-geometric-temporal for the traffic forecaster of Section 32.2 and the spatio-temporal graph view of Section 32.5; scikit-mobility is convenient for turning raw GPS traces into the trips and stays the mobility model of Section 32.1 consumes. Use the METR-LA or PEMS-BAY traffic benchmark for the forecaster, a public mobility trace or the Sensor/IoT running dataset cast as movements for the next-location model, and a windowed wearable-sensor dataset for the activity classifier. The discipline that governs the whole lab is the one from Chapter 2 extended into space: when you cut a training window, nothing from the future may leak in, not through a node's own later timesteps and not through a neighbor's. The helper below cuts a sliding window from a node-by-time array so that each sample sees a fixed past horizon and predicts a fixed future horizon, the sampling backbone shared by all three models.
import numpy as np
def make_st_windows(signal, in_len, out_len):
"""Cut sliding spatio-temporal windows from a (nodes, time) array.
Returns X of shape (samples, nodes, in_len) and Y of shape (samples, nodes, out_len),
with no future timestep ever appearing in an input window."""
n_nodes, n_time = signal.shape
X, Y = [], []
for t in range(n_time - in_len - out_len + 1):
X.append(signal[:, t:t + in_len]) # past horizon, all nodes
Y.append(signal[:, t + in_len:t + in_len + out_len]) # future horizon, all nodes
return np.stack(X), np.stack(Y) # leakage-free supervised pairs
Steps
Step 1: Build a next-location mobility predictor
Turn a set of GPS traces into sequences of visited places, then train a sequence model to predict the next place from the recent trajectory, the mobility task of Section 32.1. Use scikit-mobility to detect stays and trips, then feed the place sequence to a recurrent or attention model. Compare your top-1 accuracy against the predictability ceiling implied by the entropy of each user's history, the surprising result that human movement leaves far less room for a model than intuition suggests.
Step 2: Forecast traffic on a sensor graph
Load the METR-LA or PEMS-BAY sensor network, build its adjacency matrix from road distances, and train a spatio-temporal model that interleaves a graph convolution with a temporal operator, a DCRNN, an STGCN, or Graph WaveNet, following Section 32.2. Use the windowing helper above so each sample respects the leakage discipline, and contrast the graph model against a per-sensor baseline that ignores neighbors, reading the gap as the value of the spatial axis.
Step 3: Classify activity from sensor windows
Window a multivariate wearable or skeleton-sensor stream and train a classifier to name the activity in each window, the recognition task of Section 32.4. Start with a temporal model over flattened channels, then, if you use skeleton joints, add a spatial graph over the joints in the manner of ST-GCN and measure the lift, the same interleaving of space and time you used for traffic, now in service of a label rather than a forecast.
Step 4: Unify the three as spatio-temporal graphs
Re-express all three models in the single language of Section 32.5: the mobility model as a graph over places, the traffic model as a graph over sensors, the activity model as a graph over joints or channels. Where one model used a fixed adjacency and another a learned one, note the choice and try swapping it, observing how a learned graph can recover structure the fixed one missed.
Step 5: Write the spatio-temporal account
Put the three results together into one account of what the spatial axis bought you: how much accuracy the neighbors added in traffic, whether the joint graph helped activity, and where mobility was already near its predictability ceiling without any graph at all. Conclude with the one lesson the chapter exists to teach, that the line was always a graph with the edges left out.
Expected Output
The lab produces three working spatio-temporal models and one unifying view. The mobility predictor of Step 1 reaches a top-1 next-location accuracy that you can place against the entropy-based predictability ceiling, making concrete how little room human movement leaves. The traffic forecaster of Step 2 beats a neighbor-blind baseline by a clear margin, the spatial axis turned into a measured error reduction on the METR-LA or PEMS benchmark. The activity classifier of Step 3 names windows correctly, with a visible lift once a joint graph is added if you take the skeleton path. Step 4 shows all three as instances of one spatio-temporal graph, and Step 5 reads off where space helped most. The reader finishes able to build a mobility model, a graph traffic forecaster, and a sensor activity classifier, and, more importantly, able to see the single structure beneath them, a set of nodes that each carry a time series and listen to their neighbors, which is the whole idea of the chapter compressed into one data type.
The from-scratch window cutter of the setup is worth writing once so the leakage discipline stays visible, but you should rarely hand-build these architectures in production. PyTorch Geometric Temporal packages DCRNN, STGCN, Graph WaveNet, and a dozen other spatio-temporal graph models behind a uniform layer API and ships the METR-LA, PEMS-BAY, and Chickenpox datasets with loaders, collapsing the traffic forecaster of Section 32.2 and the unified graph view of Section 32.5 into a few dozen lines. For mobility, scikit-mobility turns raw GPS into trips, stays, flows, and the entropy-based predictability measures of Section 32.1 without your writing the trajectory parsing by hand. For video and skeleton activity, the reference I3D, SlowFast, TimeSformer, VideoMAE, and ST-GCN implementations come with pretrained weights, so the architectures of Section 32.3 and Section 32.4 are a fine-tune rather than a rebuild. The discipline the chapter teaches still governs the libraries: a learned adjacency is only as trustworthy as the leakage-free windows it was trained on, and a spatio-temporal benchmark flatters any model that lets a neighbor's future slip into the past. The library supplies the layer; it does not cut your windows, and the windows are where spatio-temporal claims live or die.
Stretch Goals
- Replace the fixed road-distance adjacency of the traffic forecaster in Section 32.2 with a learned adjacency in the Graph WaveNet style and compare the forecasts, observing how a data-driven graph can find influence the road map alone misses.
- Swap the recurrent backbone of the mobility model of Section 32.1 for an attention model and measure whether long-range trajectory context helps, weighing the predictability ceiling against the cost of the larger model.
- Take the skeleton activity classifier of Section 32.4 and ablate the spatial joint graph, then the temporal convolution, in turn, isolating how much each axis of the ST-GCN contributes and confirming that neither space nor time alone suffices.
What's Next?
This chapter completes Part VII: Building Intelligent Temporal Systems. Across its three chapters the book moved from systems that reason about time and cause (Chapter 30), to agents that perceive, decide, and act over time (Chapter 31), to models that ground time in space (Chapter 32), and with that the work of building intelligent temporal systems is done. What remains is making them trustworthy and putting them into the world. Part VIII opens with Chapter 33: Interpretability, Robustness, and Responsible Temporal AI, which asks whether the spatio-temporal forecasters and agents you have built can be explained, whether they hold up under distribution shift and adversarial pressure, and whether they treat the people whose mobility, vitals, and behavior they model fairly and privately. The spatial axis added here sharpens those questions: a model that forecasts where you will go and what you are doing is exactly the kind of model whose interpretability and privacy matter most. The full path through Part VIII and the rest of the book is laid out in the Table of Contents.
Bibliography & Further Reading
Mobility Modeling
Song, C., Qu, Z., Blumm, N., Barabasi, A.-L. "Limits of Predictability in Human Mobility." Science, 327(5968), 1018-1021, 2010. doi.org/10.1126/science.1177170
The landmark study showing that human movement carries a high theoretical predictability ceiling regardless of how far people travel, the result that frames every next-location model in Section 32.1.
Pappalardo, L., Simini, F., Barlacchi, G., Pellungrini, R. "scikit-mobility: A Python Library for the Analysis, Generation, and Risk Assessment of Mobility Data." Software library. scikit-mobility.github.io
The reference Python toolkit that turns raw GPS traces into trips, stays, flows, and predictability measures, the library that handles the trajectory parsing behind the mobility lab of Section 32.1.
Traffic Forecasting on Sensor Graphs
Li, Y., Yu, R., Shahabi, C., Liu, Y. "Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting (DCRNN)." ICLR, 2018. arXiv:1707.01926. arxiv.org/abs/1707.01926
The paper that models traffic as diffusion on a road graph and wraps a diffusion convolution inside a recurrence, introducing the METR-LA and PEMS-BAY benchmarks and anchoring Section 32.2.
Yu, B., Yin, H., Zhu, Z. "Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting (STGCN)." IJCAI, 2018. arXiv:1709.04875. arxiv.org/abs/1709.04875
The paper that stacks graph convolution with temporal convolution in a fully convolutional spatio-temporal block, the clean interleaving of space and time operators central to Section 32.2.
Wu, Z., Pan, S., Long, G., Jiang, J., Zhang, C. "Graph WaveNet for Deep Spatial-Temporal Graph Modeling." IJCAI, 2019. arXiv:1906.00121. arxiv.org/abs/1906.00121
The paper that learns a self-adaptive adjacency matrix alongside dilated temporal convolutions, showing a data-driven graph can capture influence the road map misses, the learned-adjacency idea of Sections 32.2 and 32.5.
Video Understanding
Carreira, J., Zisserman, A. "Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset (I3D)." CVPR, 2017. arXiv:1705.07750. arxiv.org/abs/1705.07750
The paper that inflates 2D image convolutions into 3D space-time kernels and bootstraps them from ImageNet weights, the I3D backbone and Kinetics benchmark behind Section 32.3.
Feichtenhofer, C., Fan, H., Malik, J., He, K. "SlowFast Networks for Video Recognition." ICCV, 2019. arXiv:1812.03982. arxiv.org/abs/1812.03982
The paper introducing a two-pathway design that reads slow spatial detail and fast motion on separate streams, the dual-rate architecture discussed in Section 32.3.
Bertasius, G., Wang, H., Torresani, L. "Is Space-Time Attention All You Need for Video Understanding? (TimeSformer)." ICML, 2021. arXiv:2102.05095. arxiv.org/abs/2102.05095
The paper that brings the transformer to video with factorized divided space-time attention, the convolution-free video architecture of Section 32.3.
Tong, Z., Song, Y., Wang, J., Wang, L. "VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training." NeurIPS, 2022. arXiv:2203.12602. arxiv.org/abs/2203.12602
The paper that masks most of a video and reconstructs it for label-free pretraining, the self-supervised video representation learning previewed in Section 32.3.
Activity Recognition
Yan, S., Xiong, Y., Lin, D. "Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition (ST-GCN)." AAAI, 2018. arXiv:1801.07455. arxiv.org/abs/1801.07455
The paper that builds a graph over body joints and convolves over it in both space and time, the skeleton-based activity model and joint-graph idea at the heart of Section 32.4.
Spatio-Temporal Graph Learning and Datasets
Rozemberczki, B., et al. "PyTorch Geometric Temporal: Spatiotemporal Signal Processing with Neural Machine Learning Models." CIKM, 2021. arXiv:2104.07788. arxiv.org/abs/2104.07788
The paper and library packaging DCRNN, STGCN, Graph WaveNet, and the field's benchmarks behind a uniform API, the off-the-shelf engine for the traffic forecaster and the unified spatio-temporal graph view of Sections 32.2 and 32.5.
"METR-LA and PEMS-BAY Traffic Datasets." Los Angeles loop-detector and Bay Area sensor networks, released with DCRNN. github.com/liyaguang/DCRNN
The two sensor-network benchmarks that have become the standard proving ground for spatio-temporal traffic forecasting, the datasets used throughout Section 32.2 and the chapter lab.