Optimization and Deep Learning Basics

"I am a loss surface in a thousand dimensions, and I promise you there is a valley. The optimizer cannot see me, only the slope beneath its feet, so it shuffles downhill one noisy step at a time, occasionally borrowing momentum to coast through the flat parts and damping its stride where the ground turns sharp. It will not find the lowest point of me, but it does not need to. It needs a point low enough to generalize, and humble enough to admit it has not seen the future yet."

A Loss Surface Resigned to Being Descended

1. The Learning Problem: Empirical Risk Minimization

Supervised learning starts from data, a model class, and a notion of error. We are given examples $(x_i, y_i)$ drawn from some distribution $\mathcal{D}$, a family of functions $f_\theta$ indexed by parameters $\theta$, and a per-example loss $\ell(f_\theta(x), y)$ that scores a single prediction against its target. The quantity we truly care about is the population risk, the expected loss over the data-generating distribution, which we never observe directly:

$$ R(\theta) = \mathbb{E}_{(x,y)\sim\mathcal{D}}\big[\ell(f_\theta(x), y)\big]. $$

Because $\mathcal{D}$ is hidden, we replace the expectation with an average over a finite training set of $N$ examples. That average is the empirical risk, and minimizing it is the entire training objective:

$$ \hat{R}(\theta) = \frac{1}{N}\sum_{i=1}^{N} \ell\big(f_\theta(x_i), y_i\big), \qquad \theta^\star = \arg\min_\theta\, \hat{R}(\theta). $$

The gap between $\hat{R}$ and $R$ is the generalization gap that Section 4 returns to. Everything else in this appendix is machinery for choosing $\ell$ (this section), driving $\theta$ toward $\theta^\star$ (Section 2), parameterizing $f_\theta$ (Section 3), and keeping the gap small (Section 4).

The choice of loss encodes what "wrong" means for the task, and Building Temporal AI uses a small, recurring set. For real-valued forecasting the default is mean squared error, which penalizes large misses quadratically and corresponds to a Gaussian noise assumption:

$$ \ell_{\text{MSE}}(\hat{y}, y) = (\hat{y} - y)^2, \qquad \ell_{\text{MAE}}(\hat{y}, y) = \lvert \hat{y} - y \rvert. $$

Mean absolute error is the robust alternative, less swayed by outliers, and it is the natural choice when a few extreme observations should not dominate the fit, as in the heavy-tailed financial returns of Chapter 14. For classification, including the next-token framing of sequence models, we use cross-entropy between the true class distribution and the predicted one. For a single example with true class $c$ and predicted class probabilities $p_\theta$, this is the negative log probability of the correct class:

$$ \ell_{\text{CE}}(p_\theta, y) = -\sum_{k} y_k \log p_{\theta,k} = -\log p_{\theta,c}. $$

Probabilistic forecasting needs more than a point estimate, so two further losses recur. The quantile (pinball) loss trains a model to emit a chosen quantile $\tau$ rather than a mean, which is how the multi-quantile forecasters of Chapter 19 produce prediction intervals:

$$ \ell_\tau(\hat{y}, y) = \max\big(\tau(y - \hat{y}),\, (\tau - 1)(y - \hat{y})\big). $$

When the model outputs a full distribution rather than a quantile, we minimize the negative log-likelihood of the observed target under that distribution, which for a predicted Gaussian with mean $\mu_\theta$ and variance $\sigma_\theta^2$ reads

$$ \ell_{\text{NLL}}(y) = \frac{1}{2}\log\big(2\pi\sigma_\theta^2\big) + \frac{(y - \mu_\theta)^2}{2\sigma_\theta^2}. $$

NLL is the training objective behind the DeepAR-style probabilistic forecasters of Chapter 19 and the generative temporal models of Chapter 17. Note that minimizing MSE is exactly maximum likelihood under a fixed-variance Gaussian, so MSE and Gaussian NLL are two views of one assumption.

2. Gradient-Based Optimization

The empirical risk $\hat{R}(\theta)$ is differentiable in $\theta$ for the models we use, so we minimize it by following the negative gradient. Vanilla gradient descent takes the full-dataset gradient and steps against it with a learning rate $\eta$:

$$ \theta_{t+1} = \theta_t - \eta\, \nabla_\theta \hat{R}(\theta_t). $$

Computing the full gradient every step is wasteful on large datasets, so we estimate it from a random minibatch $\mathcal{B}$ of examples. This is stochastic gradient descent (SGD), and the noise in its gradient estimate is a feature, not only a cost: it helps the iterate escape sharp, poorly generalizing minima.

$$ g_t = \frac{1}{\lvert\mathcal{B}\rvert}\sum_{i\in\mathcal{B}} \nabla_\theta\, \ell\big(f_{\theta_t}(x_i), y_i\big), \qquad \theta_{t+1} = \theta_t - \eta\, g_t. $$

Plain SGD crawls through ravines, oscillating across the steep direction while creeping along the shallow one. Momentum fixes this by accumulating an exponentially weighted average of past gradients, a velocity that builds up speed in consistent directions and cancels in oscillating ones:

$$ v_{t+1} = \beta v_t + g_t, \qquad \theta_{t+1} = \theta_t - \eta\, v_{t+1}. $$

A second idea is to scale each coordinate's step by its own recent gradient magnitude, so steep coordinates take small steps and flat ones take large steps. RMSProp keeps a running average of squared gradients and divides by its root:

$$ s_{t+1} = \rho\, s_t + (1-\rho)\, g_t^2, \qquad \theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{s_{t+1}} + \epsilon}\, g_t. $$

Adam combines both ideas: a momentum-like first moment $m_t$ and an RMSProp-like second moment $v_t$, each bias-corrected because they start at zero. This is the default optimizer for almost every model in Part III.

$$ m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t, \qquad v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2, $$

$$ \hat{m}_t = \frac{m_t}{1-\beta_1^t}, \quad \hat{v}_t = \frac{v_t}{1-\beta_2^t}, \qquad \theta_{t+1} = \theta_t - \eta\, \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}. $$

AdamW corrects a subtle flaw: in plain Adam, L2 regularization gets entangled with the adaptive scaling, so the effective weight decay varies per coordinate. AdamW decouples the decay, applying it directly to the weights as a separate shrinkage term, which is why it is the recommended default for Transformers and the choice for the attention models of Chapter 12:

$$ \theta_{t+1} = \theta_t - \eta\Big( \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon} + \lambda\, \theta_t \Big). $$

The learning rate is the single most consequential hyperparameter, and it is usually scheduled rather than held fixed. A warmup ramp raises $\eta$ linearly over the first few hundred steps to avoid early instability when the moment estimates are still noisy, then a cosine schedule anneals it smoothly toward zero over training:

$$ \eta_t = \eta_{\min} + \tfrac{1}{2}\big(\eta_{\max} - \eta_{\min}\big)\Big(1 + \cos\frac{\pi t}{T}\Big). $$

Warmup-then-cosine is the standard recipe for the sequence models of Chapters 12 and 15. One caution frames all of the above: a deep network's loss is non-convex, a landscape of many local minima, saddle points, and flat regions, unlike the convex losses of the classical models in Part II where a unique global minimum exists and any descent method reaches it. We do not expect the global optimum for a neural network; we expect a minimum that is good enough and generalizes, which is exactly why the stochasticity of SGD and the discipline of Section 4 matter.

3. Neural-Network Building Blocks

A neural network is a composition of simple, differentiable layers. The workhorse is the linear (fully connected) layer, an affine map from an input vector to an output vector parameterized by a weight matrix and a bias:

$$ h = W x + b, \qquad W \in \mathbb{R}^{d_{\text{out}}\times d_{\text{in}}},\; b \in \mathbb{R}^{d_{\text{out}}}. $$

Stacking linear layers alone collapses to a single linear map, so each is followed by a nonlinear activation. The book uses four. The rectified linear unit (ReLU) zeroes negatives and passes positives unchanged, cheap and the default for convolutional and feedforward blocks. GELU is a smooth ReLU variant standard in Transformers (Chapter 12). The hyperbolic tangent and the logistic sigmoid are bounded, saturating activations that survive inside recurrent gates (Chapter 10) and as output squashers:

$$ \operatorname{ReLU}(z) = \max(0, z), \qquad \operatorname{GELU}(z) = z\,\Phi(z), $$

$$ \tanh(z) = \frac{e^{z} - e^{-z}}{e^{z} + e^{-z}}, \qquad \sigma(z) = \frac{1}{1 + e^{-z}}. $$

Here $\Phi$ is the standard-normal cumulative distribution function. Depth brings two problems, internal covariate shift and vanishing gradients, and three building blocks address them. Normalization layers rescale activations to zero mean and unit variance, then reapply a learned scale $\gamma$ and shift $\beta$. Batch normalization computes these statistics across the batch dimension and suits fixed-length vision-style inputs; layer normalization computes them across the feature dimension of a single example and is the right choice for variable-length sequences, which is why it dominates the recurrent and attention models of Part III:

$$ \hat{h} = \frac{h - \mu}{\sqrt{\sigma^2 + \epsilon}}, \qquad \text{out} = \gamma\,\hat{h} + \beta. $$

Dropout regularizes by randomly zeroing a fraction $p$ of activations at training time and rescaling the survivors, which prevents co-adaptation and acts like training an ensemble of subnetworks. Residual (skip) connections add a layer's input to its output, $y = x + \mathcal{F}(x)$, giving the gradient a direct path backward and letting networks reach the depths the temporal foundation models of Chapter 15 require.

Training a network means computing the gradient of the loss with respect to every parameter. The forward pass evaluates the composed layers to produce a prediction and the scalar loss; the backward pass applies the chain rule from the loss back through each layer to accumulate $\partial \ell / \partial \theta$ for all $\theta$. Modern frameworks do not ask you to derive these gradients. Automatic differentiation records every operation in the forward pass as a computational graph, then traverses it in reverse, multiplying local derivatives by the chain rule. In PyTorch this is the autograd engine: calling loss.backward() walks the recorded graph and fills each parameter's .grad. The unrolling of this backward pass through time, when the network is recurrent, is backpropagation through time, derived in full in Chapter 9; this appendix supplies the static-graph version it specializes.

4. Generalization: From Fitting to Forecasting

Minimizing $\hat{R}$ is not the goal; minimizing $R$ is, and the difference is generalization. A model's expected error decomposes into bias, the error from a too-simple model class that cannot represent the truth, and variance, the error from a too-flexible class that chases the noise in the particular training sample. Underfitting is high bias, overfitting is high variance, and the practitioner's job is to sit at the trade-off where total error is least:

$$ \mathbb{E}\big[(\hat{y} - y)^2\big] = \underbrace{\big(\mathbb{E}[\hat{y}] - y\big)^2}_{\text{bias}^2} + \underbrace{\operatorname{Var}(\hat{y})}_{\text{variance}} + \sigma^2_{\text{noise}}. $$

Overfitting shows up as a training loss that keeps falling while the validation loss flattens and then rises. Two regularizers fight it directly. Weight decay penalizes large parameters, adding $\lambda\lVert\theta\rVert_2^2$ to the loss, which is the decoupled $\lambda\theta_t$ term already seen in AdamW; it pulls the model toward simpler functions. Early stopping halts training at the validation-loss minimum, before the model has spent its capacity memorizing the training set. Dropout from Section 3 is a third regularizer.

The augmented objective makes weight decay explicit:

$$ \hat{R}_{\text{reg}}(\theta) = \frac{1}{N}\sum_{i=1}^{N} \ell\big(f_\theta(x_i), y_i\big) + \lambda \lVert \theta \rVert_2^2. $$

Estimating $R$ requires data the model never trained on, so we split into train, validation, and test sets: train fits parameters, validation tunes hyperparameters and triggers early stopping, and test is touched once for the final honest estimate. For independent data a random split suffices, but temporal data carries a sharp twist. Future values must never inform a model that will be deployed to predict them, so the split must respect time order: train on the past, validate on a later window, test on the latest. A random shuffle leaks the future into the past and produces validation scores that collapse in deployment, the leakage failure mode that Chapter 2 treats in depth. Whenever you read "train/val/test" in Part III, read it as a strictly time-ordered split.

5. A Minimal PyTorch Training Loop

The five ideas above assemble into one loop, and that loop is the literal template every Part III code listing specializes. It instantiates a model, a loss, and an optimizer, then repeats for each epoch and each minibatch: zero the gradients, run the forward pass, compute the loss, backpropagate, and step the optimizer. Reading it once here means every later chapter can show only the part that changes (the model class, the loss, the data) and trust this scaffold underneath.

import torch
import torch.nn as nn

# A tiny regressor: two linear layers with a ReLU between them.
model = nn.Sequential(
    nn.Linear(in_features, 64),
    nn.ReLU(),
    nn.Linear(64, 1),
)
loss_fn = nn.MSELoss()                                  # point-forecast loss (Section 1)
optimizer = torch.optim.AdamW(model.parameters(),       # decoupled weight decay (Section 2)
                              lr=1e-3, weight_decay=1e-2)

for epoch in range(num_epochs):
    model.train()
    for xb, yb in train_loader:                         # one minibatch at a time (SGD)
        optimizer.zero_grad()                           # clear last step's gradients
        pred = model(xb)                                # forward pass
        loss = loss_fn(pred, yb)                        # scalar empirical risk on the batch
        loss.backward()                                 # autograd fills every .grad (Section 3)
        optimizer.step()                                # one AdamW update of all parameters

    model.eval()                                        # disable dropout, freeze norm stats
    with torch.no_grad():                               # no graph: validation must not train
        val_loss = sum(loss_fn(model(xb), yb) for xb, yb in val_loader) / len(val_loader)
    print(f"epoch {epoch}: val_loss={val_loss:.4f}")    # watch for the overfitting turn (Section 4)

Two lines deserve emphasis because beginners drop them. The optimizer.zero_grad() call is mandatory: PyTorch accumulates gradients across backward() calls, so forgetting to zero them sums this batch's gradient onto the last one and corrupts every step. The model.eval() and torch.no_grad() pair around validation switches dropout and batch-norm into inference behavior and skips building the autograd graph, so that scoring the validation set neither updates parameters nor wastes memory. A single named refinement turns this template into a robust trainer, and it appears throughout Part III.

That loop closes the appendix and opens Part III. With the learning problem defined, an optimizer chosen, the building blocks named, the generalization discipline fixed, and the training loop in hand, the recurrent networks, convolutions, and Transformers that follow are variations on one theme: swap the model class $f_\theta$, occasionally swap the loss, always respect the time-ordered split, and run this same descent. For the framework mechanics the loop assumes (tensors, devices, data loaders, autograd internals), turn to Appendix D: PyTorch for Temporal AI; for the probability behind the NLL and quantile losses, turn back to Appendix B: Probability and Statistics Refresher.