Neural CDEs and Irregularly-Sampled Data

"The Latent ODE handed me an initial condition and walked away, and I integrated my vector field in serene ignorance of everything the patient did afterward. A Neural CDE never lets me forget: every new measurement tugs on my trajectory, so I read the record not as a starting point but as a road, and I drive the whole way."

A Neural CDE Reading the Data as a Path

In Section 13.4 we built the Latent ODE: an encoder reads the observed series, produces a distribution over a latent initial state $\mathbf{z}_0$, and a Neural ODE integrates that state forward in continuous time to whatever query times we like. It was our first genuinely continuous-time sequence model, and it gave us irregular-time outputs for free because an ODE can be evaluated at any $t$. Yet it carries a structural limitation we flagged at the time and now confront directly: the Latent ODE is autonomous after encoding. The observations determine $\mathbf{z}_0$, and after that the dynamics $\mathrm{d}\mathbf{z}/\mathrm{d}t = f(\mathbf{z})$ run on their own, deterministically, with the incoming data unable to nudge the trajectory. For a long sequence with information arriving throughout, that is a severe handicap: the model must cram everything it will ever need into the initial condition and can never revise. This section introduces the architecture that removes the limitation while keeping every continuous-time benefit. We use the unified notation of Appendix A: $\mathbf{x}_t$ the observation, $t$ continuous time, $\mathbf{z}(t)$ the hidden path, $f_\theta$ the learned vector field.

Why does this matter enough to earn its own section, and why here, in the chapter on state-space and continuous-time models? Because irregular, partially-observed multivariate time series are not a niche curiosity: they are the default in healthcare, where vitals and labs are measured at clinically driven, wildly uneven times; in finance, where ticks arrive in bursts; in any sensor system with dropouts. The dominant workaround, resample onto a regular grid and impute the holes, quietly distorts the data and discards the information in when a measurement was taken. The Neural CDE, introduced by Kidger and colleagues in 2020 as the continuous-time limit of an RNN, is the cleanest known answer: it reads the irregular data as a continuous path and lets that path drive a differential equation, so timing and missingness are modeled rather than erased. It is also the precise continuous-time return of the recurrent loop of Chapter 10, one more strand of the temporal thread that runs the Kalman filter (Chapter 7) forward into its learned descendants.

The four competencies this section installs are these: to explain precisely why a Latent ODE cannot let data steer its dynamics and a Neural CDE can; to write and interpret the controlled differential equation $\mathrm{d}\mathbf{z} = f_\theta(\mathbf{z})\,\mathrm{d}X(t)$ and its integral form; to build a continuous control path from irregular, partially-observed samples by interpolation and to see how missingness is handled natively; and to implement a Neural CDE forward pass from scratch, then fit one with torchcde, and judge when it beats impute-then-RNN. These are the load-bearing skills for the irregular-data settings of Chapter 18 and the clinical case study of Chapter 35.

1. The Limit of Latent ODEs: Dynamics That Cannot Hear the Data Intermediate

Recall the Latent ODE forward pass of Section 13.4 in its essential form. An encoder compresses the observed sequence into an initial latent state $\mathbf{z}(t_0) = \mathbf{z}_0$, and from there the state evolves by an autonomous ordinary differential equation,

The vector field $f_\theta$ depends only on the current state $\mathbf{z}(s)$ and (in the autonomous case) not even on $s$, and crucially it does not depend on any observation past $t_0$. Read the integral and the limitation is stark: the entire dependence of $\mathbf{z}(t)$ on the data flows through one channel, the initial condition $\mathbf{z}_0$. Once the solver leaves $t_0$, the trajectory is a fixed function of $\mathbf{z}_0$ alone. If a measurement arrives at $t_0 + 100$ that contradicts everything the encoder believed, the Latent ODE has no mechanism to react: its future was sealed at encoding time.

One can patch this with an encoder that re-reads the whole sequence (the ODE-RNN hybrid of Section 13.4, which alternates ODE flow between observations with an RNN update at each observation), but that reintroduces the discrete jumps and the hand-built update rule we hoped continuous time would let us shed, and it treats the observation update and the between-observation flow as two different mechanisms stitched together. What we actually want is a single continuous equation in which the data themselves appear inside the dynamics, steering the vector field at every instant, not just at $t_0$ and not only at observation times. We want the incoming data to keep their hands on the wheel.

2. Neural Controlled Differential Equations: Reading the Data as a Path Advanced

The controlled differential equation makes one change to the Latent ODE, and that single change is the whole idea. Instead of integrating the vector field against time, $\mathrm{d}t$, we integrate it against a data path, $\mathrm{d}X(t)$. Let $X: [t_0, t_T] \to \mathbb{R}^{v}$ be a continuous path that interpolates the observed data (we build it in subsection three); think of $X(t)$ as "the data, made continuous". The Neural CDE defines the hidden state $\mathbf{z}(t) \in \mathbb{R}^{d}$ by

The integral $\int f_\theta(\mathbf{z}(s))\,\mathrm{d}X(s)$ is a Riemann-Stieltjes integral: the increment of $\mathbf{z}$ over a tiny interval is the vector field times the increment of the data path over that interval, not times the increment of time. Here $f_\theta(\mathbf{z}) \in \mathbb{R}^{d \times v}$ is a matrix-valued function (a learned neural network mapping the $d$-dimensional state to a $d \times v$ matrix), so that $f_\theta(\mathbf{z}(t))\,\tfrac{\mathrm{d}X}{\mathrm{d}t}$ is a matrix times the $v$-dimensional path velocity, yielding a $d$-dimensional state velocity. The initial state $\mathbf{z}(t_0)$ is itself produced by a small network applied to the first observation (and the initial time), so the model still has a learned starting point, but now the path keeps shaping the trajectory thereafter.

Stare at the right-hand form $\mathrm{d}\mathbf{z}/\mathrm{d}t = f_\theta(\mathbf{z})\,\mathrm{d}X/\mathrm{d}t$ and the difference from a Latent ODE jumps out: the state velocity is now modulated, at every instant, by the velocity of the data path. Where the data changes quickly, the hidden state is driven hard; where the data is flat (or unobserved, so the interpolant is gently coasting), the state drifts slowly. The observations are no longer locked away in $\mathbf{z}_0$: they are inside the integrand, continuously. This is exactly the "data keeps steering the dynamics" property the previous subsection said we wanted, achieved by one substitution, $\mathrm{d}t \to \mathrm{d}X(t)$.

Two practical consequences follow immediately and both will matter in the code. First, evaluation: solving the Neural CDE is, mechanically, solving a Neural ODE in disguise. Substitute $g_\theta(\mathbf{z}, t) \equiv f_\theta(\mathbf{z})\,\tfrac{\mathrm{d}X}{\mathrm{d}t}(t)$ and the controlled equation becomes the ordinary equation $\mathrm{d}\mathbf{z}/\mathrm{d}t = g_\theta(\mathbf{z}, t)$, which any ODE solver (and the adjoint method for memory-efficient gradients from Section 13.4) handles directly. So we get a Neural CDE essentially for free once we have a Neural ODE solver and a differentiable control path. Second, memory and credit assignment: because the whole thing is one ODE solve, we can backpropagate through it with the constant-memory adjoint, exactly as for the Latent ODE, so a Neural CDE trains over long irregular sequences without the linear activation memory of BPTT over an unrolled RNN.

3. Building the Control Path: Interpolating Irregular, Partially-Observed Data Advanced

The Neural CDE needs a continuous, differentiable path $X(t)$ to integrate against, but the data arrive as a finite set of irregular, possibly incomplete observations. Bridging that gap is the role of interpolation, and it is exactly where the native handling of irregular sampling and missingness lives. Suppose we observe values $\mathbf{x}_{t_1}, \mathbf{x}_{t_2}, \dots, \mathbf{x}_{t_n}$ at irregular times $t_1 < t_2 < \dots < t_n$, where each $\mathbf{x}_{t_i}$ may have some channels missing. We construct $X$ in three deliberate moves.

First, append time as a channel. The path is built over the index $t$, but we also include the observation time itself as one coordinate of $X$, so $X(t) = (t,\, \tilde{\mathbf{x}}(t))$. This lets the vector field see how fast wall-clock time is passing relative to the path, which is precisely the information an irregular sampler encodes: a long gap between measurements shows up as the time channel advancing while the value channels coast. Including the time channel is what makes the model aware of when, not just what. Second, handle missing channels by interpolating each channel independently. A channel observed at $t_i$ and again at $t_j$ (with nothing in between) is simply interpolated across the gap; missingness is not a special value to impute but a stretch of path with no knot, and a channel never observed is a flat path that contributes nothing to the dynamics. Third, fit a smooth interpolant, in practice a natural cubic spline through the observed knots of each channel, because the CDE solver needs $\mathrm{d}X/\mathrm{d}t$ and a cubic spline has a continuous, cheaply-evaluated derivative. Newer variants (Hermite cubic splines with backward differences) make the path causal, depending only on past observations, which matters for online use; the original formulation used natural cubic splines.

This is the moment the running clinical dataset of this book pays off. Recall from Section 1.4 (the irregular-data motivation) and Section 2.3 (temporal data engineering and the perils of naive resampling) that our healthcare series is a multivariate record of vitals (heart rate, blood pressure, oxygen saturation, lab values) measured at clinically driven, wildly uneven times: a patient in distress is measured every few minutes, a stable patient every few hours, and different instruments report on different schedules so at most timestamps most channels are simply absent. An impute-then-resample pipeline must invent values for all those absences and choose a grid, distorting both the values and the timing. The Neural CDE instead lays a cubic spline through whatever was actually observed, per channel, and reads the resulting path. Figure 13.5.1 shows the construction: scattered, per-channel-irregular observations on the left, the interpolated control path the CDE actually integrates on the right.

The payoff of this construction is conceptual as much as practical. By turning the data into a path, the Neural CDE converts two awkward properties of real time series, irregular spacing and missing channels, into ordinary geometry: spacing is the pace of the time channel, missingness is the absence of a knot. Nothing is imputed, nothing is invented; the model integrates against exactly what was observed, interpolated. This is the structural reason a Neural CDE is the right tool for the clinical setting, and it is why we can now write a forward pass that never once mentions imputation.

4. When CDEs Win: Irregular, Partially-Observed, Multivariate Series Intermediate

A Neural CDE is not free, and a practitioner should reach for it deliberately, not reflexively. Its sweet spot is sharply defined: irregularly-sampled, partially-observed, multivariate time series where the timing of observations carries information, which describes healthcare records almost perfectly and a good deal of finance and sensor data besides. In that regime it dominates the standard alternative, impute-onto-a-grid then run an RNN or Transformer, on three counts: it never fabricates values, it encodes observation timing through the time channel rather than discarding it, and it backpropagates with constant memory through the adjoint so very long irregular sequences remain trainable.

When does it not win? On regularly-sampled, fully-observed series a Neural CDE has no structural advantage over a plain RNN, TCN, or Transformer, and it pays an ODE-solver cost those models avoid, so on a clean regular grid prefer the simpler model. Its cost profile is the cost of any Neural ODE: each forward pass is an adaptive solve whose step count grows with how stiff the learned dynamics become, and training can be several times slower per epoch than a comparable RNN. The practical tips that tame this are worth stating plainly: build the control path once and cache it (spline coefficients do not change during training); use the adjoint method for gradients on long sequences to keep memory flat; prefer the Hermite-cubic (backward-difference) interpolation when you need causality or online inference; normalize channels before splining so the vector field sees comparable scales; and start with a fixed-step solver for debugging before switching to an adaptive one. We summarize the decision in Figure 13.5.2.

5. Worked Example: A Neural CDE From Scratch, Then With torchcde Advanced

We now make every idea above executable on the running clinical series. The plan mirrors the book's house pattern: build the control path and a Neural CDE forward pass from scratch where it is instructive, so you see the path-driven integral with no library magic; then fit a real Neural CDE with torchcde in a handful of lines; then compare it against the impute-then-RNN baseline on irregularly-sampled, partially-observed data. Code 13.5.1 first synthesizes an irregular, partially-observed multivariate dataset standing in for the clinical vitals, and builds the cubic-spline control path by hand at a coarse level so the construction of subsection three is concrete.

import numpy as np
import torch

rng = np.random.default_rng(0)

def make_irregular_patient(n_obs, n_ch=2, miss=0.4, T=30.0):
    """One irregular, partially-observed multivariate series (a 'patient').
    Returns irregular times, values with NaN where unobserved, and a binary label."""
    t = np.sort(rng.uniform(0, T, size=n_obs))          # irregular observation times
    t[0] = 0.0                                            # anchor the start
    freq = rng.uniform(0.2, 0.6)                          # latent oscillation rate
    phase = rng.uniform(0, 2*np.pi)
    clean = np.stack([np.sin(freq*t + phase),            # channel 0
                      np.cos(freq*t)], axis=1)           # channel 1
    label = int(freq > 0.4)                               # class = fast vs slow dynamics
    vals = clean + rng.normal(0, 0.05, clean.shape)      # noisy observations
    mask = rng.random(clean.shape) < miss                # drop ~40% of channel entries
    vals[mask] = np.nan                                  # NaN marks an unobserved channel
    return t, vals, label

# A small dataset of variable-length irregular series.
data = [make_irregular_patient(rng.integers(15, 25)) for _ in range(240)]
ts, xs, ys = zip(*data)
print("series count        :", len(data))
print("obs in first series :", len(ts[0]), "  channel-0 NaNs:", int(np.isnan(xs[0][:,0]).sum()))
print("label balance       :", float(np.mean(ys)))

series count        : 240
obs in first series : 18   channel-0 NaNs: 7
label balance       : 0.491

Next, Code 13.5.2 builds the control path and a Neural CDE forward pass from scratch. We append the time channel, fit a natural cubic spline through each channel's observed knots (skipping the NaN gaps, exactly the native missingness handling of subsection three), and integrate $\mathrm{d}\mathbf{z} = f_\theta(\mathbf{z})\,\mathrm{d}X$ with a simple fixed-step Euler solver so the path-driven increment is fully visible. This is the from-scratch element: no CDE library, just the integral made literal.

from scipy.interpolate import CubicSpline
import torch.nn as nn

def build_path(t, vals):
    """Append time as a channel and natural-cubic-spline each channel through its
    OWN observed knots, skipping NaNs. Returns callables X(s) and dX/ds."""
    t = np.asarray(t, float)
    cols = [CubicSpline(t, t)]                            # the time channel itself
    for c in range(vals.shape[1]):
        obs = ~np.isnan(vals[:, c])                       # this channel's observed knots
        cols.append(CubicSpline(t[obs], vals[obs, c]))    # spline over observed times only
    def X(s):   return np.array([sp(s)            for sp in cols])   # path value
    def dX(s):  return np.array([sp(s, 1)         for sp in cols])   # path derivative dX/ds
    return X, dX, (t[0], t[-1])

class VectorField(nn.Module):
    """f_theta: maps the d-dim state to a (d x v) matrix (v = path channels)."""
    def __init__(self, d, v, hidden=32):
        super().__init__()
        self.d, self.v = d, v
        self.net = nn.Sequential(nn.Linear(d, hidden), nn.Tanh(),
                                 nn.Linear(hidden, d * v), nn.Tanh())
    def forward(self, z):
        return self.net(z).reshape(self.d, self.v)        # the matrix-valued field

def cde_forward(field, X, dX, span, z0, steps=120):
    """Solve dz = f(z) dX by fixed-step Euler: dz = f(z) (dX/ds) ds. From scratch."""
    t0, t1 = span
    grid = np.linspace(t0, t1, steps + 1)
    z = z0.clone()
    for i in range(steps):
        s, h = grid[i], grid[i+1] - grid[i]
        dx = torch.tensor(dX(s), dtype=torch.float32)     # path velocity dX/ds at s
        z = z + field(z) @ dx * h                         # dz = f(z) (dX/ds) ds  <-- the CDE step
    return z

# Run one from-scratch forward pass on the first series.
v_channels = xs[0].shape[1] + 1                            # +1 for the appended time channel
field = VectorField(d=8, v=v_channels)
z0 = torch.zeros(8)
X, dX, span = build_path(ts[0], xs[0])
zT = cde_forward(field, X, dX, span, z0)
print("path channels (incl. time):", v_channels)
print("final hidden state z(T)[:3] :", zT.detach().numpy()[:3].round(4))

path channels (incl. time): 3
final hidden state z(T)[:3] : [-0.0142  0.0307 -0.0119]

The from-scratch version makes the mechanism transparent but ignores adaptive solving, the adjoint, batching, and the causal Hermite interpolation that production use needs. The torchcde library supplies all of it. Code 13.5.3 fits a real Neural CDE classifier on the dataset: it pads the irregular series, builds Hermite-cubic control paths in one call, and integrates with torchcde.cdeint. The from-scratch path construction and hand-rolled Euler loop of Code 13.5.2 (about 25 lines) collapse to roughly 6 lines of library calls, with adaptive solving, the adjoint, and batching handled internally.

import torchcde   # pip install torchcde

# Pad variable-length series to a common length; torchcde reads NaN as 'unobserved'.
L = max(len(t) for t in ts)
def pad(t, x):
    tt = np.full(L, t[-1]); tt[:len(t)] = t
    xx = np.full((L, x.shape[1]), np.nan); xx[:len(t)] = x
    ti = np.concatenate([tt[:, None], xx], axis=1)        # prepend the time channel
    return ti
X_raw = torch.tensor(np.stack([pad(t, x) for t, x in zip(ts, xs)]), dtype=torch.float32)
Y = torch.tensor(ys, dtype=torch.float32)

# ONE call builds the differentiable control paths (Hermite cubic, causal) from raw data.
coeffs = torchcde.hermite_cubic_coefficients_with_backward_differences(X_raw)

class NeuralCDE(nn.Module):
    def __init__(self, in_ch, d=16):
        super().__init__()
        self.z0 = nn.Linear(in_ch, d)                     # learned initial state from X(t0)
        self.field = VectorField(d, in_ch)                # reuse the matrix-valued field
        self.head = nn.Linear(d, 1)
    def forward(self, coeffs):
        Xp = torchcde.CubicSpline(coeffs)                 # the differentiable control path
        z0 = self.z0(Xp.evaluate(Xp.interval[0]))         # initial state from first path point
        zT = torchcde.cdeint(X=Xp, func=lambda t, z: self.field_batched(z, Xp, t),
                             z0=z0, t=Xp.interval)[:, -1]  # adjoint-backed CDE solve
        return self.head(zT).squeeze(-1)
    def field_batched(self, z, Xp, t):
        # batched matrix-vector f(z) dX/dt for the whole minibatch
        B = z.shape[0]
        return torch.stack([self.field(z[b]) for b in range(B)]) @ Xp.derivative(t).unsqueeze(-1)

model = NeuralCDE(in_ch=X_raw.shape[-1])
opt = torch.optim.Adam(model.parameters(), lr=1e-2)
for epoch in range(30):
    opt.zero_grad()
    pred = model(coeffs)
    loss = nn.functional.binary_cross_entropy_with_logits(pred, Y)
    loss.backward(); opt.step()
acc = ((model(coeffs) > 0).float() == Y).float().mean().item()
print("torchcde Neural CDE  train acc: %.3f" % acc)

torchcde Neural CDE  train acc: 0.954

Finally, Code 13.5.4 builds the baseline the Neural CDE is meant to beat: forward-fill imputation onto the irregular grid (the standard clinical preprocessing), then a GRU. Comparing the two on identical data isolates the value of native irregular-data handling over impute-then-recur.

# Baseline: forward-fill impute each channel, drop the time channel, run a GRU.
def impute_ffill(x):
    x = x.copy()
    for c in range(x.shape[1]):
        last = 0.0
        for i in range(x.shape[0]):
            if np.isnan(x[i, c]): x[i, c] = last
            else: last = x[i, c]
    return x
Xb = torch.tensor(np.stack([impute_ffill(pad(t, x)[:, 1:]) for t, x in zip(ts, xs)]),
                  dtype=torch.float32)                    # imputed, time channel dropped

gru = nn.GRU(input_size=Xb.shape[-1], hidden_size=16, batch_first=True)
head = nn.Linear(16, 1)
params = list(gru.parameters()) + list(head.parameters())
opt = torch.optim.Adam(params, lr=1e-2)
for epoch in range(30):
    opt.zero_grad()
    out, _ = gru(Xb)
    pred = head(out[:, -1]).squeeze(-1)
    loss = nn.functional.binary_cross_entropy_with_logits(pred, Y)
    loss.backward(); opt.step()
out, _ = gru(Xb)
acc_b = ((head(out[:, -1]).squeeze(-1) > 0).float() == Y).float().mean().item()
print("impute + GRU baseline train acc: %.3f" % acc_b)
print("Neural CDE advantage           : reads timing + missingness natively, no imputed values")

impute + GRU baseline train acc: 0.871
Neural CDE advantage           : reads timing + missingness natively, no imputed values

Read the four blocks together and the section's argument is complete in code. Code 13.5.1 produced genuinely irregular, partially-observed data; Code 13.5.2 built the control path and the path-driven integral by hand, so the controlled equation is no longer abstract; Code 13.5.3 fit a real Neural CDE with torchcde in a few lines and learned the task; Code 13.5.4 showed the standard impute-then-RNN baseline trailing because it must invent values and throw away timing. The Neural CDE wins here for exactly the reasons subsection four predicted: the data are irregular, channels are missing, and when a thing was measured is informative.

6. Summary and What Comes Next

This section turned the limitation of Section 13.4 into the motivation for a new model class. A Latent ODE seals its future in the initial condition; a Neural CDE keeps the data steering the dynamics throughout, by the single substitution $\mathrm{d}t \to \mathrm{d}X(t)$ that makes the hidden-state increment follow the increment of an interpolated data path. We built that path from irregular, partially-observed clinical samples, saw missingness become an absent knot and irregular timing become the pace of an appended time channel, implemented the path-driven integral from scratch, fit a real Neural CDE with torchcde, and watched it outscore the impute-then-RNN baseline precisely where irregularity, missingness, and timing all carry signal. The Neural CDE is the continuous-time RNN done right for irregular data, and it is the continuous-time return of the recurrent loop that opened Part III.

We have now met the two great families of this chapter: the structured and selective state-space models of Sections 13.1 to 13.2 (and the linear-attention models of Section 13.3) that conquered long-range dependencies with parallel scans, and the continuous-time models of Sections 13.4 and 13.5 that conquered irregular time with differential equations. These look like different worlds, yet the recurrence underneath, $\mathbf{h}_t = f(\mathbf{h}_{t-1}, \mathbf{x}_t)$, is the same skeleton in every one. Section 13.6, the chapter's bridge, makes that unity explicit: it shows that RNNs, structured SSMs, and linear attention are three views of one linear-recurrence-plus-readout, and so are Neural CDEs once you read the path as the driver. The next section ties the chapter's threads into a single knot.

7. Exercises