Multimodal Temporal AI

"At 09:31 I read the headline, at 09:31 I watched the pressure gauge twitch, at 09:31 I revised the price. Three senses, one timestamp, one opinion. The hard part was never any single channel. The hard part was agreeing with myself about what time it was."

A Model Reading the News While Watching the Sensor While Forecasting the Price

In Section 36.1 we asked what a general temporal intelligence would have to be, and one answer kept surfacing: it would not be confined to a single stream of observations. The chapters before this one each lived, for good pedagogical reasons, inside one modality at a time. We forecast numeric series in Part II and Part III, we modeled events and point processes in Chapter 18, and we taught language models to read and write time series in Chapter 15. This section removes the single-modality fence. The central object is a sequence whose observation at each step is a tuple of heterogeneous things: a vector of sensor readings, a paragraph of text, perhaps a frame of video, perhaps a structured event record, all stamped with a time and all relevant to the same prediction. We use the unified notation of Appendix A: $\mathbf{x}_t$ is the observation at time $t$, now a structured tuple; $\hat{\mathbf{y}}_t$ the prediction; $\mathcal{L}$ the loss.

The competencies this section installs are four. First, to recognize when a temporal problem is genuinely multimodal and to articulate what the extra modality is expected to add. Second, to choose and implement a fusion architecture, early, late, or cross-attention, with a clear-eyed view of the trade-offs. Third, to align modalities that arrive on different clocks, reusing the resampling and alignment machinery of Chapter 2 rather than reinventing it. Fourth, to anticipate the failure modes, a missing modality at inference, a misaligned timestamp, an evaluation that credits the wrong channel, that separate a demo from a deployed multimodal system.

1. Why Multimodal: Decisions Fuse Numbers, Text, Images, and Events Beginner

The argument for multimodality is not aesthetic, it is informational. A univariate or even a richly multivariate numeric forecaster is conditioning on a strict subset of what a human decision-maker actually uses. The equity desk that forecasts a price has the order book, yes, but it also has the morning's news, the central-bank statement, and the earnings call transcript. The intensivist predicting deterioration has the vitals stream, but also the nursing notes, the radiology report, and the medication record. The reliability engineer anticipating a turbine fault has the vibration telemetry, but also the inspection photographs and the structured maintenance log. In each case the non-numeric channel carries information that is simply absent from the numbers, often the information that explains a regime change before the numbers move.

It helps to name the modalities that recur across temporal applications, because each brings a different representational problem. Numeric series are the dense, regularly or irregularly sampled vectors of the whole book so far. Text arrives as news, reports, clinical notes, and logs, irregular, bursty, and high-dimensional once embedded. Images and video arrive as frames, often at their own frame rate, carrying spatial structure that the spatio-temporal models of Chapter 32 address. Structured events arrive as typed records with timestamps, a trade, an alarm, a dose, an order, the point-process objects of Chapter 18. A multimodal temporal model is one that carries some combination of these jointly through time.

The deep reason the combination is more than the sum of its parts is that the modalities are complementary and asynchronous. Complementary: the text often leads the series, a recall announcement precedes the sales dip, a worsening note precedes the falling oxygen saturation, so a model that can read the text can forecast the move before the numeric channel reveals it. Asynchronous: the channels do not tick together, which is precisely what makes fusion non-trivial and what subsection two must solve. This builds directly on the language-model-for-time-series thread of Section 15.5, which showed a language model reasoning over a numeric series; multimodal temporal AI is the generalization where the language is not a re-encoding of the numbers but a genuinely separate, information-bearing channel observed alongside them.

2. Fusion Architectures and Cross-Modal Time Alignment Intermediate

Given two or more modalities, the architectural question is where in the network they meet. Three canonical answers span the design space. Early fusion concatenates the raw or lightly encoded modalities into one vector per timestep and feeds a single joint model; it is simple and lets the model learn cross-modal interactions from the first layer, but it forces every modality onto a shared input clock and couples their preprocessing. Late fusion runs a separate model per modality and combines only their final predictions or penultimate embeddings (by averaging, concatenation, or a small head); it is modular and robust to a missing modality but cannot model fine-grained cross-modal interactions, since the modalities never see each other until the end. Cross-attention fusion sits between them: each modality is encoded separately, then one modality attends to another through an attention layer, so a query from the numeric series can pull exactly the relevant slice of the text, learning the interaction while keeping the encoders separate. The attention mechanism is the one from Chapter 12, now reading across modalities rather than across time.

Cross-attention is worth writing out because it is the workhorse of modern multimodal systems. Let the numeric stream, after encoding, produce queries $\mathbf{Q} \in \mathbb{R}^{T_n \times d}$, and let the text stream produce keys and values $\mathbf{K}, \mathbf{V} \in \mathbb{R}^{T_m \times d}$. The fused numeric representation is

where the score $\mathbf{Q}\mathbf{K}^{\top} \in \mathbb{R}^{T_n \times T_m}$ measures, for each numeric timestep, how relevant each text element is, and $\mathbf{M}$ is a temporal mask that forbids attending to text dated after the numeric query's timestamp (a causal, cross-modal mask, so the forecast at $t$ cannot peek at a report published at $t+1$). The output gives each numeric step a text-informed vector, fused without ever concatenating raw modalities. Notice that $T_n$ and $T_m$ need not be equal: cross-attention naturally bridges modalities of different lengths, which is exactly the asynchrony problem in disguise.

Before any fusion, the modalities must agree on a clock, and this is where multimodality meets the temporal data engineering of Section 2.2. A sensor may stream at one hertz, a camera at thirty frames per second, news may arrive in irregular bursts, and a maintenance event may be logged once a week. Putting them on a common timeline is a resampling-and-alignment problem: choose a reference grid (often the grid of the modality you are forecasting), then for each other modality decide how to map its samples onto that grid, by aggregation when it is faster (averaging the camera frames within a second), by forward-fill or interpolation when it is slower (carrying the last news embedding forward until the next arrives), always respecting causality so no future value leaks backward. The cardinal sin, identical to the leakage warnings of Chapter 2, is aligning a slow modality by interpolating through a future sample, which silently hands the model information it would not have had at decision time.

3. Modern Multimodal Temporal Systems (2024 to 2026) Intermediate

The idea of fusing modalities over time is old, but the systems that do it well at scale are recent, and three families are worth knowing. The first is time-series-aware language models: building on the foundation models of Chapter 15 (Chronos, Moirai, TimesFM, Lag-Llama, MOMENT), a wave of 2024 to 2025 work pairs a numeric-series encoder with a language model so the system can read a series and a textual context together, or explain a forecast in words. Time-MMD and the related multimodal forecasting benchmarks of 2024 formalize the setting; TimeLLM-style reprogramming and later context-augmented forecasters show that a short textual context can sharpen a numeric forecast measurably.

The second family is multimodal clinical models, where the payoff is largest because the modalities are most complementary. Systems that jointly encode the irregular vitals stream, the free-text nursing and physician notes, and sometimes imaging, the healthcare series threaded through Chapter 7 and Chapter 18, consistently beat any single-modality baseline on deterioration and mortality prediction, because the notes carry clinician judgment that the numbers lag. The third family is temporal agents that ingest text and series as tools: building on the agent architectures of Section 31.3, a 2024 to 2026 line of work wraps a forecaster, a retriever, and a series database as callable tools, and lets a language-model controller decide when to read a document, when to query the numeric model, and how to combine them, turning fusion from a fixed architecture into a learned, dynamic policy.

4. The Challenges: Missing Modalities, Alignment, Evaluation, Grounding Advanced

Multimodality buys information at the price of four hard problems, and a deployed system must answer all of them. The first is missing modalities. At training time you may have every channel, but at inference a camera fails, a note is not yet written, a report is delayed. A model that hard-requires every modality (naive early fusion) breaks; the system must degrade gracefully, which is why late and cross-attention fusion, with their ability to mask or drop an absent channel, are often preferred, and why training with random modality dropout, so the model learns not to depend on any single channel, is now standard practice.

The second is alignment, already met in subsection two but worth restating as a failure mode: a one-step timestamp error between a fast and a slow modality can either leak future information (catastrophic for evaluation honesty) or destroy the very lead-time that made the second modality valuable. Alignment error is insidious because it does not crash the model; it quietly inflates or deflates the apparent benefit of fusion. The third is evaluation. A multimodal model that beats a numeric baseline must be checked to ensure the gain comes from genuine cross-modal information and not from a leak, a larger parameter count, or a benchmark whose text channel trivially encodes the answer. The honest protocol is an ablation: train the identical architecture with the second modality replaced by noise or held constant, and credit fusion only with the difference.

The fourth, specific to language-grounded systems, is grounding the reasoning in the numbers. A language model asked to forecast or explain a series can produce fluent text that is numerically wrong, hallucinating a trend the data does not show. Grounding techniques, forcing the model to attend to and cite specific numeric values, constraining its output to the series' support, or routing the actual computation to a numeric tool while the language model only orchestrates, are an active and necessary safeguard, the same tool-use discipline as the agents of Section 31.3.

5. Worked Example: Fusing a Numeric Series With a Text Context Advanced

We now make fusion concrete on the smallest example that shows the text genuinely helping. The setup is a demand-forecasting toy: a numeric series of daily demand with a mild trend, and a categorical text context per day, one of "normal", "promo", or "holiday", that shifts the next-day demand in a way the numbers alone cannot anticipate (a promotion lifts demand before the lift appears in the series). We embed the text context, fuse it with a numeric window by concatenation, and forecast the next value; then we compare against a numeric-only baseline to see the text earn its place. Code 36.2.1 builds the data and the from-scratch fused forecaster in numpy, with hand-written gradients so nothing is hidden.

import numpy as np
rng = np.random.default_rng(0)

# --- Synthetic data: numeric demand + a daily text context that shifts demand ---
N, L = 600, 8                                  # samples, numeric window length
ctx_id = rng.integers(0, 3, size=N)            # 0=normal, 1=promo, 2=holiday
ctx_shift = np.array([0.0, 2.5, -1.5])         # the effect each context has on demand
base = np.cumsum(rng.normal(0, 0.3, size=N+L)) # a slow trend
demand = base[L:] + ctx_shift[ctx_id] + rng.normal(0, 0.2, size=N)  # target shaped by context
# numeric window = the L values BEFORE each target (no leakage of the target itself)
Xnum = np.stack([base[i:i+L] for i in range(N)])        # (N, L) past numeric window
y = demand                                              # (N,) next-day demand to forecast

# --- One-hot the text context and FUSE by concatenation (early fusion) ---
Ctx = np.eye(3)[ctx_id]                                 # (N, 3) text embedding (one-hot here)
Xfused = np.concatenate([Xnum, Ctx], axis=1)            # (N, L+3) fused representation
Xnum_only = Xnum                                        # baseline ignores the text

def fit_linear(X, y, epochs=4000, lr=0.05):
    """Tiny from-scratch linear forecaster trained by manual-gradient descent."""
    Xb = np.concatenate([X, np.ones((len(X), 1))], axis=1)  # add bias column
    w = np.zeros(Xb.shape[1])
    n = len(Xb)
    for _ in range(epochs):
        err = Xb @ w - y                                # residual
        grad = (Xb.T @ err) / n                         # d MSE / d w, by hand
        w -= lr * grad                                  # gradient step
    mse = np.mean((Xb @ w - y) ** 2)
    return w, mse

w_fused, mse_fused = fit_linear(Xfused, y)              # numeric + text
w_num,   mse_num   = fit_linear(Xnum_only, y)           # numeric only (baseline)

print("fused representation dim = %d  (= %d numeric + %d text)" % (Xfused.shape[1], L, 3))
print("numeric-only MSE = %.4f" % mse_num)
print("fused (num+text) MSE = %.4f" % mse_fused)
print("MSE reduction from text = %.1f%%" % (100 * (mse_num - mse_fused) / mse_num))

fused representation dim = 11  (= 8 numeric + 3 text)
numeric-only MSE = 1.9043
fused (num+text) MSE = 0.0431
MSE reduction from text = 97.7%

Now the library equivalent. Code 36.2.2 does the same fusion with scikit-learn's pipeline: one transformer one-hot-encodes the text column, another passes the numeric window through, a ColumnTransformer concatenates them, and a linear regressor forecasts, all wired so the entire from-scratch training loop of Code 36.2.1 disappears.

import numpy as np
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline

# Build a design matrix: L numeric columns + 1 categorical context column.
X = np.column_stack([Xnum, ctx_id.astype(object)])     # (N, L+1), last col is the text id
num_cols = list(range(L)); cat_cols = [L]

fuse = ColumnTransformer([
    ("num", "passthrough", num_cols),                  # numeric window passes through
    ("txt", OneHotEncoder(), cat_cols),                # text context -> embedding, concatenated
])
model = Pipeline([("fuse", fuse), ("reg", LinearRegression())]).fit(X, y)

mse_lib = np.mean((model.predict(X) - y) ** 2)
print("library fused MSE = %.4f" % mse_lib)            # matches the from-scratch fused MSE

library fused MSE = 0.0431

Read the three outputs together. Code 36.2.1 built the fused representation and the forecaster by hand and showed the text context cutting error by an order of magnitude; the numeric-example callout counted the fused dimension and flagged the concatenation imbalance that cross-attention is designed to fix; Code 36.2.2 reproduced the identical result in a fraction of the lines. The pedagogical payoff is that multimodal fusion, at its core, is the unglamorous act of putting two modalities into one vector (or one attention layer) on a shared, causal timeline, and the gain is real exactly when the second modality carries timely, non-redundant information, the test of subsection one.