TL;DR — Backpropagation is the chain rule applied to a computational graph of operations. Forward pass computes predictions; backward pass propagates gradients from the loss back to every parameter. Modern frameworks like PyTorch and JAX implement this with reverse-mode automatic differentiation, so you write forward code and get gradients for free.

If you’ve trained a neural network, you’ve used backpropagation — even if you’ve never written a single line of it yourself. Frameworks like PyTorch, TensorFlow, and JAX handle the gradient computation automatically, which is wonderful for productivity but can make the underlying mechanics feel like a black box.

That black box is worth opening. Not because you’ll implement backprop by hand in production (you almost certainly won’t), but because understanding it changes how you debug models, reason about vanishing gradients, and design architectures that train well.

This post walks through backpropagation from first principles, then shows how it maps onto a real autograd engine.

The Intuition: A Chain of Accountability

Imagine a factory assembly line. Station A takes raw materials and produces widgets. Station B takes widgets and assembles gadgets. Station C paints the gadgets. At the end, a quality inspector measures defects and tells the manager: “We have a problem.”

The manager can’t just blame one station. She has to walk backward: “How much did the inspector’s score depend on the paint? How much did the paint depend on the assembly? How much did the assembly depend on the widget quality?” Each station gets feedback proportional to its contribution to the final defect.

Backpropagation is exactly this reverse walk. The loss at the output is the inspector’s verdict. Every weight in the network gets a number telling it: if you nudge me, how much will the loss change? That number is the gradient, and the update rule — typically stochastic gradient descent — uses it to nudge each weight in the direction that reduces loss.

The Math: Chain Rule on a Computational Graph

Let’s make this concrete with the simplest non-trivial network: one input, one hidden neuron, one output, with a squared error loss.

  • Input: x = 2
  • True target: y = 1
  • Hidden weight: w1 = 0.5, hidden bias: b1 = 0.1
  • Output weight: w2 = 0.8, output bias: b2 = 0.2
  • Activation: sigmoid σ(z) = 1/(1+e^-z)
  • Loss: L = (ŷ - y)²

Forward pass:

z1 = w1 * x + b1       = 0.5*2 + 0.1  = 1.1
h1 = σ(z1)              ≈ 0.7503
z2 = w2 * h1 + b2       = 0.8*0.7503 + 0.2 ≈ 0.8002
ŷ  = σ(z2)              ≈ 0.6900
L  = (ŷ - y)²           = (0.69 - 1)²  ≈ 0.0961

Now the backward pass. We want ∂L/∂w1, ∂L/∂w2, ∂L/∂b1, ∂L/∂b2. The chain rule tells us to multiply local derivatives along the path:

∂L/∂ŷ   = 2(ŷ - y)              = -0.62
∂ŷ/∂z2  = ŷ(1 - ŷ)              ≈ 0.2139
∂L/∂z2  = ∂L/∂ŷ * ∂ŷ/∂z2        ≈ -0.1326

∂L/∂w2  = ∂L/∂z2 * h1           ≈ -0.0995
∂L/∂b2  = ∂L/∂z2 * 1            ≈ -0.1326

∂L/∂h1  = ∂L/∂z2 * w2           ≈ -0.1061
∂h1/∂z1 = h1(1 - h1)            ≈ 0.1873
∂L/∂z1  = ∂L/∂h1 * ∂h1/∂z1      ≈ -0.0199

∂L/∂w1  = ∂L/∂z1 * x            ≈ -0.0398
∂L/∂b1  = ∂L/∂z1 * 1            ≈ -0.0199

Each weight now has a gradient. With a learning rate of 0.1:

w1 ← 0.5  - 0.1 * (-0.0398) = 0.5040
w2 ← 0.8  - 0.1 * (-0.0995) = 0.8100

The loss decreases on the next forward pass. That’s the whole algorithm.

Computational Graphs and Reverse-Mode Autodiff

In a real network, you don’t have four parameters — you have millions, and the chain-rule tree explodes. The trick that makes this tractable is the computational graph: every operation is a node, every intermediate value is an edge, and gradients flow backward along the edges.

There are two ways to mechanically apply the chain rule on this graph:

  1. Forward-mode autodiff — propagates derivatives from inputs to outputs alongside the forward pass. Efficient when you have few inputs and many outputs.
  2. Reverse-mode autodiff — does a forward pass first, caching intermediates, then propagates derivatives from the output backward to all inputs. Efficient when you have many inputs and one (or few) outputs.

Neural networks have millions of parameters and one scalar loss. Reverse-mode is dramatically cheaper, with a cost roughly proportional to the forward pass. This is what every modern ML framework implements under the hood, and it’s what we mean by “backpropagation” in practice.

The seminal reference for this view is Baydin, Pearlmutter, and Radul’s “Automatic Differentiation in Machine Learning: a Survey”, which makes clear that backprop is just reverse-mode autodiff specialized to neural network loss functions.

How PyTorch Actually Does It

Let’s see this in PyTorch. You write forward code as ordinary Python; PyTorch records the operations on a dynamic graph called autograd.

import torch

x = torch.tensor([2.0])
y = torch.tensor([1.0])
w1 = torch.tensor([0.5], requires_grad=True)
b1 = torch.tensor([0.1], requires_grad=True)
w2 = torch.tensor([0.8], requires_grad=True)
b2 = torch.tensor([0.2], requires_grad=True)

# Forward pass
z1 = w1 * x + b1
h1 = torch.sigmoid(z1)
z2 = w2 * h1 + b2
y_hat = torch.sigmoid(z2)
loss = (y_hat - y) ** 2

# Backward pass — autograd traverses the graph in reverse
loss.backward()

print(w1.grad, w2.grad, b1.grad, b2.grad)

A few things are happening under the hood that are worth knowing:

  • Each torch.autograd.Function knows two things: its forward formula and its backward formula (the local gradient). When you call backward(), PyTorch walks the graph in reverse, calling each function’s backward and multiplying by upstream gradients via the chain rule.
  • requires_grad=True is opt-in. Parameters with this flag get their gradients accumulated into .grad during the backward pass. Tensors without it are treated as constants — useful for inputs and frozen layers.
  • torch.no_grad() disables graph construction entirely, which is essential for inference and evaluation loops where you want maximum speed and zero memory overhead for autograd.

You can also build custom autograd functions by subclassing torch.autograd.Function and implementing forward and backward static methods — useful when you need a non-standard operation that PyTorch doesn’t ship, such as a custom CUDA kernel with a hand-written backward. The PyTorch autograd documentation walks through this in detail.

Patterns in Production: Why This Matters Beyond Theory

Understanding backpropagation isn’t just academic — it directly informs decisions you make in production ML systems.

1. Vanishing and Exploding Gradients

When you stack many layers, gradients are products of many Jacobian terms. If those terms are consistently smaller than 1, gradients shrink toward zero and early layers stop learning. If they’re consistently larger than 1, gradients explode and training diverges.

This is why activations matter: sigmoid saturates and squashes gradients near 0 and 1, which is part of why ReLU became default for deep networks — its derivative is exactly 1 for positive inputs, neither amplifying nor shrinking the signal. Modern normalization layers like BatchNorm and LayerNorm further stabilize gradient magnitudes by explicitly controlling activation statistics.

2. Memory vs. Compute: The Backward Pass Cost

The forward pass caches every intermediate needed for the backward pass. For a transformer with sequence length 4096 and batch size 32, this is substantial — activations are typically the dominant memory consumer during training, not weights.

Three production patterns address this:

  • Gradient checkpointing (a.k.a. activation recomputation) — trade compute for memory by re-running forward passes during backward. PyTorch’s torch.utils.checkpoint makes this a one-line decorator.
  • Gradient accumulation — simulate large batch sizes on small GPUs by accumulating gradients over multiple forward passes before stepping the optimizer.
  • Mixed precision — compute forward and backward in FP16/BF16, keep a master FP32 copy of weights. Modern GPUs like the NVIDIA A100 and H100 have tensor cores that make this near-free in compute but dramatically reduce memory. See NVIDIA’s mixed precision guide.

3. Debugging with Gradient Checks

If your model isn’t learning, the first sanity check is: are the gradients correct? Finite-difference numerical gradients are slow but trivially correct; comparing them to autograd’s gradients catches bugs in custom ops.

from torch.autograd import gradcheck

# input must be double precision and require grad
input = (torch.randn(3, 3, dtype=torch.double, requires_grad=True),)
test = gradcheck(my_custom_function, input, eps=1e-6, atol=1e-4)
print(test)  # True if gradients match

This is standard practice when implementing new layers or loss functions.

4. Gradient Accumulation and Distributed Training

In data-parallel training — where you split a batch across multiple GPUs — every replica computes its own gradients and then all-reduces them so each replica has the global gradient before the optimizer step. Libraries like PyTorch Distributed Data Parallel (DDP) and Horovod automate this, but the underlying contract is the same: gradients are the synchronization point.

Understanding this matters when you debug hangs, deadlocks, or throughput regressions — most distributed training failures trace back to gradient synchronization.

What Backprop Is Not

A few things backpropagation is commonly mistaken for:

  • Not the same as gradient descent. Backprop is the algorithm for computing gradients efficiently. Gradient descent (or Adam, or Adagrad) is the algorithm for using those gradients to update weights. You can backprop without ever taking an optimizer step (e.g., for ablation studies or second-order methods).
  • Not biologically plausible. Real neurons don’t seem to send error signals backward. This is a famous critique — see “Backpropagation and the brain” for a thoughtful modern take — but it doesn’t affect the engineering utility of backprop in software.
  • Not unique to neural networks. Any differentiable model — logistic regression, Gaussian processes, even physics simulators — can use reverse-mode autodiff. PyTorch was originally pitched as a NumPy replacement with autodiff, and many researchers use it for non-neural work entirely.

Key Takeaways

  • Backpropagation is reverse-mode automatic differentiation applied to a loss function composed of differentiable operations. The forward pass builds a graph; the backward pass traverses it in reverse, multiplying local gradients via the chain rule.
  • Cost is proportional to the forward pass. Reverse-mode autodiff gives you gradients for all parameters for roughly the same compute as one forward pass, which is what makes deep learning tractable.
  • Frameworks handle the math, but you control the architecture. Activation choice, normalization, residual connections, and gradient checkpointing all exist because of how gradients flow through deep graphs.
  • Memory is dominated by activations, not weights, on long forward passes. Production training uses checkpointing, mixed precision, and distributed all-reduce to manage this.
  • Always gradient-check custom ops. When you write a custom autograd.Function, finite-difference verification is a one-liner that catches subtle bugs before they cost you days of debugging.

Further Reading