Part 8 · Deep learning foundations · module 2 of 3 · 14 min

How models learn: gradient descent

The optimisation algorithm behind all of it — one and two dimensions, local minima, and fixed versus adaptive learning rates.

The essence

Gradient descent is a loop with one idea: compute which direction increases the loss, step the opposite way, repeat. The gradient is a vector of partial derivatives — one per parameter — pointing in the direction of steepest increase, and the learning rate decides how far you step along it.

Why it matters

This is how every parameter in every model in this curriculum got its value. It is also where most training failures live: a learning rate too high diverges, too low never arrives, and the difference between a model that trains and one that does not is usually this single number.

Topics covered (5)

01Overview of gradient descent
02Local minima and whether they matter
03Gradient descent in one dimension
04Gradient descent in two dimensions
05Fixed vs. dynamic learning rates

The algorithm

Start somewhere. Compute the gradient of the loss with respect to every parameter. Step each parameter a small distance in the negative gradient direction. Repeat until the loss stops improving.

That is all of it. What makes it non-trivial at scale is that the loss surface has millions of dimensions, the gradient is estimated from a mini-batch rather than the full dataset, and the step size has to be chosen well enough to make progress without overshooting.

Gradient descent in one dimension, watched step by step

import numpy as np

f  = lambda x: x ** 2 + 3 * x + 5      # minimum at x = -1.5
fp = lambda x: 2 * x + 3

def descend(x0, lr, steps=25):
    x, path = x0, [x0]
    for _ in range(steps):
        x = x - lr * fp(x)             # MINUS: move against the gradient
        path.append(x)
    return np.array(path)

for lr in [0.01, 0.1, 0.5, 0.9, 1.01]:
    path = descend(4.0, lr)
    end = path[-1]
    verdict = ("diverged" if not np.isfinite(end) or abs(end) > 1e3 else
               "oscillating" if abs(path[-1] - path[-2]) > 1e-3 else
               "converged")
    print(f"lr={lr:<5} final x={end:+10.4f}  {verdict}")

# lr = 0.01  crawls: correct direction, far too slow
# lr = 0.1   converges cleanly
# lr = 0.5   converges fast
# lr = 0.9   oscillates around the minimum without settling
# lr = 1.01  diverges: each step overshoots further than the last
#
# For a quadratic the stability threshold is lr < 2/f''(x) = 1.0 exactly.
# Real losses have no such formula, which is why the rate is tuned empirically.

Two dimensions, and why curvature matters

With two parameters the gradient is a vector and the step moves diagonally downhill. The new phenomenon is anisotropy: if the surface is much steeper in one direction than another, the single learning rate is simultaneously too large for the steep direction and too small for the shallow one.

The result is the characteristic zigzag — rapid oscillation across a narrow valley while creeping slowly along it. This is what momentum fixes, by accumulating a velocity that cancels the oscillating components and reinforces the consistent one. It is also what Adam fixes differently, by scaling each parameter's step by its own gradient history, which effectively gives every direction its own learning rate.

Two dimensions, with and without momentum

import numpy as np

# Deliberately ill-conditioned: 20x steeper in y than in x.
f  = lambda x, y: x ** 2 + 20 * y ** 2
grad = lambda x, y: np.array([2 * x, 40 * y])

def descend_2d(start, lr, steps=60, momentum=0.0):
    p = np.array(start, dtype=float)
    v = np.zeros_like(p)
    path = [p.copy()]
    for _ in range(steps):
        g = grad(*p)
        v = momentum * v + g          # accumulate a velocity
        p = p - lr * v
        path.append(p.copy())
    return np.array(path)

plain = descend_2d([2.0, 1.0], lr=0.02)
with_mom = descend_2d([2.0, 1.0], lr=0.02, momentum=0.9)

print("plain    final loss", f(*plain[-1]))
print("momentum final loss", f(*with_mom[-1]))

# Count sign flips in the y-component: that is the zigzag, measured.
flips = lambda path: int(np.sum(np.diff(np.sign(np.diff(path[:, 1]))) != 0))
print("zigzags — plain:", flips(plain), " momentum:", flips(with_mom))

# Visualise on a contour plot; the zigzag is unmistakable:
# X, Y = np.meshgrid(np.linspace(-2.5, 2.5, 100), np.linspace(-1.5, 1.5, 100))
# ax.contour(X, Y, f(X, Y), levels=30)
# ax.plot(plain[:, 0], plain[:, 1], "o-", ms=3)

Local minima, and why they matter less than expected

The textbook worry is getting stuck in a local minimum — a point that is lowest in its neighbourhood but not globally. In one or two dimensions this is a real and visible problem.

In millions of dimensions it largely is not. For a point to be a local minimum the surface must curve upward in every single direction; if even one direction curves downward, it is a saddle point and there is still an escape route. Since the number of directions is enormous, saddle points vastly outnumber local minima.

Saddles are still a problem, because the gradient is small near them and progress stalls. Momentum carries the parameters through, and the noise inherent in mini-batch gradients also helps — stochasticity is a feature here, not just a computational compromise. Empirically, the many minima large networks find tend to have similar loss values, which is why training is far less seed-sensitive than the low-dimensional picture suggests.

Fixed versus dynamic learning rates

A fixed rate forces one compromise for the whole of training. Early on you want large steps to cover ground; late on you want small ones to settle. No single value does both.

A schedule resolves it. Warm-up ramps the rate up over the first few hundred steps, because at step zero the optimiser's statistics are meaningless and a full-size step is a gamble. Then cosine decay brings it down smoothly to near zero, letting the parameters settle rather than bounce.

Adaptive methods such as Adam are a different mechanism, not a replacement: they give each parameter its own effective rate based on its gradient history. In practice you use both — Adam or AdamW, with a warm-up and cosine schedule on top of its base rate.

Fixed, decayed, and adaptive, compared

import numpy as np

f  = lambda x: x ** 2 + 3 * x + 5
fp = lambda x: 2 * x + 3

def run(x0, lr_fn, steps=100):
    x = x0
    for t in range(steps):
        x = x - lr_fn(t) * fp(x)
    return x, f(x)

fixed_small = lambda t: 0.01
fixed_large = lambda t: 0.5
step_decay  = lambda t: 0.5 * (0.5 ** (t // 25))
cosine      = lambda t: 0.5 * 0.5 * (1 + np.cos(np.pi * t / 100))

def warmup_cosine(t, warmup=10, total=100, peak=0.5):
    if t < warmup:
        return peak * t / warmup
    prog = (t - warmup) / (total - warmup)
    return peak * 0.5 * (1 + np.cos(np.pi * prog))

for name, fn in [("fixed 0.01", fixed_small), ("fixed 0.5", fixed_large),
                 ("step decay", step_decay), ("cosine", cosine),
                 ("warmup+cosine", warmup_cosine)]:
    x, loss = run(4.0, fn)
    print(f"{name:16} x={x:+.6f}  loss={loss:.8f}   (optimum x=-1.5)")

# Adam, implemented, to show what 'adaptive' means concretely.
def adam(x0, lr=0.1, steps=100, b1=0.9, b2=0.999, eps=1e-8):
    x, m, v = x0, 0.0, 0.0
    for t in range(1, steps + 1):
        g = fp(x)
        m = b1 * m + (1 - b1) * g              # running mean of the gradient
        v = b2 * v + (1 - b2) * g ** 2         # running mean of its square
        m_hat = m / (1 - b1 ** t)              # bias correction: m starts at 0
        v_hat = v / (1 - b2 ** t)
        x -= lr * m_hat / (np.sqrt(v_hat) + eps)
    return x

print("adam:", adam(4.0))
# Dividing by sqrt(v_hat) is the adaptivity: parameters with consistently
# large gradients take proportionally smaller steps.

Worth remembering

The update is w ← w − lr·∇w; the minus sign is what makes it descend.
Learning rate is the most consequential hyperparameter — too high diverges, too low crawls.
In high dimensions saddle points, not local minima, are the real obstacle.