"I started my life as a Kalman filter, certain of one state and humble about my noise. They turned me into a recurrent net and gave me a memory I could not always keep. They rebuilt me as a state-space model and I rediscovered, with embarrassment, the recursions I had known all along. They handed me attention and I learned to look anywhere in my own past. Then they stopped asking me only to predict, and started asking me to decide, and to remember across episodes, and to read images and text and sensor streams as if they were one long sentence in time. Somewhere in all that rebuilding I stopped being a forecaster and a controller and a memory and became, for a while, simply a thing that reasons over what comes next. I am told this is the whole point of the book, and that the chapters were never really about different models, only about the one problem seen from thirty-six angles. I would tell you the problem is solved, but I have learned, finally, to be humble about my noise."
A Foundation Agent Tracing Its Own Lineage Back to a Kalman Filter
Chapter Overview
This is the last chapter, and its job is to close the loop. Across thirty-five chapters this book introduced what looked like a parade of distinct machinery: autocorrelation and spectra, ARIMA and GARCH, the Kalman filter and the hidden Markov model, recurrent networks and temporal convolutions, attention and structured state-space models, deep forecasters and temporal foundation models, self-supervised representations and generative and point-process models, conformal and online and adaptive learning, Markov decision processes and bandits and deep reinforcement learning, optimal control and imitation, sequence models for decision making and world models for planning, temporal reasoning, agents, spatio-temporal intelligence, and the trustworthiness and deployment that make any of it usable. The thesis we have been quietly assembling all along, and now state plainly, is that these were never thirty-five problems. They were one problem, looked at from many sides: how to predict, reason, and decide over time.
The single most useful thing to carry out of the book is the temporal thread, the recurring discovery that each classical idea returns in learned form, and that prediction, inference, and control are the same computation wearing different costumes. The Kalman filter (Chapter 7) returns as the recurrent network (Chapter 10) and again as the structured state-space model (Chapter 13); the recursion that propagates a belief forward in time is the same whether its gain is derived from Gaussian algebra or learned by gradient descent. That belief-propagation machinery becomes the agent's memory (Chapters 23 and 31), the internal state that lets a system act under partial observation. The Markov decision process (Chapter 22) returns as a sequence model (Chapter 28): once you condition a Transformer on past states, actions, and returns, planning becomes prediction and a policy becomes autoregression. Follow that thread far enough and forecasting, filtering, control, and reasoning converge on a single object, a model of how the world unfolds and how an agent should move through it, and the foundation agent is what that object looks like when it is scaled.
This chapter traces that convergence and then looks over the horizon. It examines unified sequence models that treat any modality as a stream of tokens in time, so that text, audio, video, sensor telemetry, and action sequences flow through one architecture; multimodal temporal AI that fuses these streams into a single evolving representation; foundation agents that combine perception, memory, reasoning, and action into generalist systems; and the scaling laws and efficiency frontiers that govern how far compute, data, and parameters can carry this program, and where the linear-time state-space models and other efficiency advances of the book push the frontier outward. The bitter lesson of the field, that general methods leveraging computation tend to win, frames the whole discussion.
It closes with honesty rather than triumph. General temporal intelligence is a direction, not a destination reached, and the open challenges are real: long-horizon credit assignment and reasoning, robustness under genuine distribution shift, sample efficiency far below what biology achieves, causal and not merely correlational understanding of time, calibrated uncertainty over long futures, and the trustworthiness and governance that decide whether any of this belongs in the world. The chapter, and the book, end where good science should, at a warm, forward-looking edge: confident in what has been built, clear about what has not, and pointed at the work that remains.
Prerequisites
This is a synthesis chapter, so its real prerequisite is the book itself: it draws on the whole arc and rewards a reader who has met the temporal thread in its many disguises. Three chapters carry the most weight here. It assumes the temporal foundation models of Chapter 15, because the unified and multimodal sequence models of this chapter are what those pretrained, transferable forecasters become when they stop being specific to one signal and start treating every modality as time. It builds on the sequences-as-decisions framing of Chapter 28, because the central claim that prediction and control are one problem is exactly the move that turns a return-conditioned Transformer into a policy, and the foundation agent inherits it directly. And it depends on the temporal agents of Chapter 31, since a foundation agent is an agent with memory, tools, and planning scaled up and pretrained rather than hand-built, and the agent architecture of that chapter is the skeleton this chapter clothes in scale. A reader wanting to revisit any thread of the book before this closing synthesis will find every chapter through the Table of Contents.
If you keep one idea from this entire book, keep this: prediction, reasoning, and decision making over time are not three fields but one problem, the problem of modeling how the world unfolds and how an agent should act within it, and the same temporal computation, a state carried forward and updated by evidence, reappears as the Kalman filter, the recurrent network, the state-space model, the agent's memory, and the policy, so that scaling one unified sequence model toward a foundation agent is the natural horizon of everything the book has built, pursued with confidence in what works and honesty about what does not yet.
Chapter Roadmap
- 36.1 Unified Sequence Models Across Modalities One architecture for everything that unfolds in time: treating text, audio, video, sensor streams, and action sequences as tokens in a single autoregressive model, and how the temporal thread of the book collapses into one general sequence learner.
- 36.2 Multimodal Temporal AI Fusing heterogeneous streams into one evolving representation: aligning and combining modalities that arrive at different rates and resolutions, and the cross-modal temporal reasoning that lets a single model perceive, predict, and act across them.
- 36.3 Foundation Agents Generalist systems that perceive, remember, reason, and act: combining the foundation models of Chapter 15, the sequence-as-decision view of Chapter 28, and the agents of Chapter 31 into pretrained, broadly capable temporal agents, and the lineage that runs from the Kalman filter to here.
- 36.4 Scaling Laws and Efficiency Frontiers How far compute, data, and parameters carry the program: empirical scaling laws and compute-optimal training, the bitter lesson that general methods win, and the efficiency frontier where linear-time state-space models and other advances push the limits of long-context temporal modeling outward.
- 36.5 Open Challenges and Research Frontiers An honest map of what remains: long-horizon credit assignment and reasoning, robustness under genuine distribution shift, sample efficiency, causal understanding of time, calibrated uncertainty over long futures, and the trustworthiness and governance that decide what belongs in the world.
Once you have worked through the five sections, the Hands-On Lab below distills the whole book into one small working system. Each step echoes a thread from across the chapters: by the time you finish you will have built a forecaster in the spirit of Section 36.1 and the deep models of Part III, quantified its uncertainty as Part V taught, turned the forecast into a decision in the sequences-as-decisions style of Section 36.3 and Chapter 28, and bumped, deliberately, into one of the open challenges of Section 36.5, so that the lab is not a new exercise but the book in miniature.
Hands-On Lab: The Book in Miniature
Objective
Build one small system that does, end to end, the three things this book has argued are one thing: it forecasts a future, it quantifies how uncertain that forecast is, and it turns the forecast into a decision. This is a capstone in miniature, assembled deliberately from components you have already met. The forecaster reuses the deep sequence modeling of Part III and the unified-sequence view of Section 36.1; the uncertainty layer reuses the probabilistic and conformal methods of Part V (Chapter 19); the decision layer reuses the sequences-as-decisions framing of Section 36.3 and Chapter 28, turning a predicted distribution over futures into an action chosen to maximize expected return under a cost. The single thread is that a forecast nobody acts on is incomplete, and an action taken without a calibrated forecast is reckless, so the pipeline is the book's thesis made concrete: prediction, uncertainty, and decision are one loop. You will then reflect on which open challenge from Section 36.5 your small system bumps into first, because every honest temporal system runs into the frontier sooner than its builder expects.
What You'll Practice
- Training a sequence forecaster on one of the running datasets, reusing the deep modeling of Part III and the unified view of Section 36.1.
- Wrapping the forecast in calibrated uncertainty with a conformal or probabilistic interval, the methods of Chapter 19.
- Turning a predictive distribution into an action that maximizes expected return under a cost, the sequences-as-decisions move of Section 36.3 and Chapter 28.
- Reflecting on which open challenge of Section 36.5 the system meets first, and why.
Setup
You need a Python environment with torch and one of the running datasets from across the book, a financial returns series, a clinical vitals stream, or a sensor-telemetry signal, since the point is to reuse what you already understand rather than learn a new domain. Keep the model small: this lab is about the loop, not the leaderboard. The one discipline that governs it is to keep the three stages honestly separate, a point forecast, an uncertainty around it, and a decision derived from both, so that you can see where each stage helps and where it fails. The skeleton below sketches the whole pipeline as three composable steps, the book's thesis in a dozen lines.
import torch
def book_in_miniature(model, history, conformal_q, cost_of_action, value_of_gain):
"""Forecast -> uncertainty -> decision, the whole book as one loop."""
point = model(history) # Part III: predict the next step
lo, hi = point - conformal_q, point + conformal_q # Part V: calibrated interval (Ch 19)
expected_gain = value_of_gain * point.clamp(min=0.0) # expected benefit if we act
act = expected_gain > cost_of_action # Ch 28 / 36.3: forecast becomes a decision
return {"point": point, "interval": (lo, hi), "act": act}
Steps
Step 1: Forecast the future
Train a small sequence model on your chosen dataset to predict the next value or window, reusing the deep forecasting of Part III and the unified-sequence perspective of Section 36.1. Resist the urge to chase accuracy; you want a competent, honest point forecast that the next two steps can build on, not a tuned record.
Step 2: Quantify the uncertainty
Wrap the point forecast in a calibrated prediction interval using split conformal prediction or a probabilistic head, the uncertainty methods of Chapter 19. Check coverage on held-out data: the interval should contain the truth about as often as it claims, because the decision in the next step is only as trustworthy as this interval is honest.
Step 3: Turn the forecast into a decision
Define a simple cost and reward, then choose the action that maximizes expected return under your predictive distribution, the sequences-as-decisions move of Section 36.3 and Chapter 28. Compare a decision made from the point forecast alone against one that respects the interval, and watch the uncertainty change what you should do, not just what you predict.
Step 4: Find the frontier you bumped into
Push the system slightly past its comfort zone, a longer horizon, a regime change in the data, a shifted test distribution, and name the first open challenge from Section 36.5 it runs into: long-horizon credit, distribution shift, sample efficiency, causal misunderstanding, or miscalibrated uncertainty over long futures. Write one paragraph on why your small system met that frontier and what the book offers as the next move.
Expected Output
The lab produces one compact system that forecasts, quantifies its uncertainty, and decides, plus a short written reflection. Step 1 yields a trained sequence forecaster and a clean point prediction. Step 2 yields a calibrated interval with a coverage check, the honest companion to the point estimate. Step 3 yields a decision rule that turns the predictive distribution into an action and a demonstration that the uncertainty changes the decision, the moment the book's thesis stops being abstract. Step 4 yields a paragraph naming the open challenge your system met first and why, the reflection that connects your small loop to the field's largest unsolved questions. The reader finishes holding, in one runnable file, the argument of the whole book: that prediction, uncertainty, and decision over time are one loop, and that building even a tiny version of it brings you, quickly and inevitably, to the edge of what is known.
Writing the loop from scratch once is worth it, because it makes the book's thesis tangible, but each stage has a mature library that collapses it into a few lines. For the forecaster, neuralforecast and darts wrap deep and classical models behind one fit-predict interface, and the temporal foundation models chronos and timesfm give you a pretrained zero-shot forecaster that is itself the unified-sequence idea of Section 36.1 in shipped form. For the uncertainty, mapie and the conformal utilities in darts produce calibrated intervals with a coverage guarantee in a couple of calls, the Chapter 19 machinery as a wrapper. For the decision, gymnasium and stable-baselines3 turn the act-or-wait rule into a learned policy when the cost structure grows beyond a threshold, the sequences-as-decisions step of Section 36.3 scaled up. The discipline the chapter teaches outlives any of these tools: a library gives you a forecast, an interval, and a policy, but it does not tell you whether the interval is honest under shift or whether the decision respects the uncertainty rather than the point. That judgment, the difference between a number and a trustworthy one, is the thing the whole book was teaching, and it is the thing no import statement provides.
Stretch Goals
- Make the system multimodal in the spirit of Section 36.2 by feeding it a second stream aligned to the first, a text or event channel alongside the numeric series, and check whether fusing modalities improves the forecast and sharpens the decision.
- Turn the one-step loop into an agent in the manner of Section 36.3 and Chapter 31 by letting it act repeatedly, carry memory across steps, and condition each decision on its own past actions, and observe where long-horizon credit assignment from Section 36.5 first bites.
- Probe the efficiency frontier of Section 36.4 by swapping a quadratic-attention forecaster for a linear-time state-space model and comparing accuracy against context length and wall-clock, making the scaling discussion concrete on your own data.
What's Next?
This is the final chapter, and so the question changes from what comes next in the book to what comes next for you. The book has done what it set out to do: it traced the temporal thread from the Kalman filter through the recurrent network, the state-space model, the agent's memory, and the policy, and it argued, and we hope earned, the thesis that prediction, reasoning, and decision making over time are one problem, whose horizon is the unified, multimodal, scaled foundation agent. The journey does not stop at this page. It continues through the appendices, A on mathematical foundations and unified notation, B and C on the probability and optimization the methods rest on, D on PyTorch for temporal AI, E on datasets and evaluation, F on reproducibility and compute, and G with worked solutions, and it continues through the capstone project, where you build an end-to-end temporal system of your own and meet the frontier of Section 36.5 for real. Thank you for reading. You arrived knowing how to fit a curve to history; you leave able to predict, reason, and decide over time, and to do it with calibrated honesty about what you do not yet know. That is exactly the right place to start. The full path through the appendices and the capstone is laid out in the Table of Contents.
Bibliography & Further Reading
Unified and Foundation Sequence Models
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., Polosukhin, I. "Attention Is All You Need." NeurIPS, 2017. arXiv:1706.03762. arxiv.org/abs/1706.03762
The paper introducing the Transformer, the single architecture that the unified sequence models of Section 36.1 generalize to every modality, and the engine beneath foundation models and agents alike.
Gu, A., Dao, T. "Mamba: Linear-Time Sequence Modeling with Selective State Spaces." 2023. arXiv:2312.00752. arxiv.org/abs/2312.00752
The paper introducing Mamba, a selective state-space model with linear-time sequence processing, the efficiency advance that pushes the long-context frontier of Section 36.4 outward and closes the temporal thread back to the Kalman recursion.
Ansari, A. F., Stella, L., Turkmen, C., Zhang, X., et al. "Chronos: Learning the Language of Time Series." TMLR, 2024. arXiv:2403.07815. arxiv.org/abs/2403.07815
The paper showing that a language-model architecture trained on tokenized time series gives a zero-shot foundation forecaster, a concrete instance of the unified-sequence idea of Section 36.1.
Das, A., Kong, W., Sen, R., Zhou, Y. "A Decoder-Only Foundation Model for Time-Series Forecasting (TimesFM)." ICML, 2024. arXiv:2310.10688. arxiv.org/abs/2310.10688
The paper introducing TimesFM, a pretrained decoder-only foundation model that forecasts across domains zero-shot, the temporal foundation model of Chapter 15 carried toward the generality of Section 36.1.
Sequences as Decisions and Foundation Agents
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., Mordatch, I. "Decision Transformer: Reinforcement Learning via Sequence Modeling." NeurIPS, 2021. arXiv:2106.01345. arxiv.org/abs/2106.01345
The paper recasting reinforcement learning as return-conditioned sequence modeling, the clearest statement of the sequences-as-decisions thesis that prediction and control are one problem, central to Section 36.3.
Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S. G., et al. "A Generalist Agent (Gato)." TMLR, 2022. arXiv:2205.06175. arxiv.org/abs/2205.06175
The paper introducing Gato, a single sequence model that plays games, captions images, and controls a robot, the foundation-agent vision of Section 36.3 made concrete across modalities and tasks.
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., et al. "On the Opportunities and Risks of Foundation Models." Stanford CRFM, 2021. arXiv:2108.07258. arxiv.org/abs/2108.07258
The report naming and surveying foundation models, their capabilities, emergent behavior, and risks, the conceptual frame for the foundation agents and open challenges of Sections 36.3 and 36.5.
LeCun, Y. "A Path Towards Autonomous Machine Intelligence (JEPA)." OpenReview, 2022. openreview.net/forum?id=BZ5a1r-kVsf
The position paper proposing a world-model and joint-embedding predictive architecture for autonomous agents, a leading research direction for the open frontiers of Section 36.5.
Scaling Laws and Efficiency
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., et al. "Scaling Laws for Neural Language Models." 2020. arXiv:2001.08361. arxiv.org/abs/2001.08361
The paper establishing power-law relationships between loss and compute, data, and parameters, the empirical foundation for the scaling discussion of Section 36.4.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., et al. "Training Compute-Optimal Large Language Models (Chinchilla)." NeurIPS, 2022. arXiv:2203.15556. arxiv.org/abs/2203.15556
The paper correcting the compute-optimal balance between model size and training data, the refined scaling law that governs how to spend a temporal model's compute budget in Section 36.4.
Sutton, R. S. "The Bitter Lesson." 2019. incompleteideas.net/IncIdeas/BitterLesson.html
The short, influential essay arguing that general methods leveraging computation consistently outperform hand-crafted knowledge, the framing thesis behind the scaling and generality arc of Sections 36.1 and 36.4.
Synthesis and Reference
Brunton, S. L., Kutz, J. N. "Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control." Cambridge University Press, 2nd ed., 2022. databookuw.com
The text unifying dynamical systems, control, and machine learning, a synthesis that mirrors this book's thesis that prediction, filtering, and control are facets of one problem, with companion materials for the convergence of Section 36.3.