The Geometry of Gradient Descent

Gradient descent is the workhorse of modern machine learning, and yet the reason it works is a single, elegant fact from multivariable calculus. This note walks through that fact from the ground up.

The setup

We want to minimize a differentiable loss $L(\theta)$ over parameters $\theta \in \mathbb{R}^n$. Starting from a point $\theta_t$, we take a small step $\Delta$ and ask how the loss changes. A first-order Taylor expansion gives

where $\nabla L(\theta_t)$ is the gradient — the vector of partial derivatives $\left( \tfrac{\partial L}{\partial \theta_1}, \dots, \tfrac{\partial L}{\partial \theta_n} \right)$.

Why the negative gradient?

We want to choose the step direction $\Delta$ that decreases $L$ as fast as possible. Restricting to unit-length directions $\lVert \Delta \rVert = 1$, the change in loss is the inner product $\nabla L(\theta_t)^\top \Delta$. By the Cauchy–Schwarz inequality,

with equality exactly when $\Delta$ points opposite the gradient:

Scaling that direction by a step size $\eta > 0$ gives the familiar update:

The update loop

In code

The whole algorithm is a handful of lines. Everything interesting lives in the grad function you pass in.

import numpy as np
 
def gradient_descent(grad, theta, lr=0.1, steps=100):
    """Minimize a function given its gradient."""
    history = [theta]
    for _ in range(steps):
        theta = theta - lr * grad(theta)
        history.append(theta)
    return theta, np.array(history)

Choosing the learning rate

The step size $\eta$ is the one knob that decides whether the iteration converges at all.

The learning rate is the single hyperparameter that can turn a clean convergence proof into a divergence in practice.

For a convex, $\beta$-smooth loss, any fixed $\eta \leq \tfrac{1}{\beta}$ guarantees the loss never increases — a rare and reassuring guarantee in a field with few of them.