Backpropagation Demystified: Neural Nets from First Principles

Every modern deep learning framework — PyTorch, TensorFlow, JAX — does one thing brilliantly: it computes gradients for you. Call loss.backward() and millions of parameters update simultaneously. But what's actually happening under the hood? Backpropagation is just the chain rule applied systematically through a computational graph. By the end of this post, you'll build a neural network from scratch — no frameworks, no autograd — and understand exactly how every weight gets updated. We'll start with something even simpler: watching gradient descent fit a line in real time. The algorithm was popularised by Rumelhart, Hinton & Williams (1986) in their landmark Nature paper, though the mathematical foundations trace back to Linnainmaa (1970) and Werbos (1974). Gradient Descent in 30 Seconds Before we build a neural network, let's see what gradient descent actually looks like. We'll fit a line $y = ax + b$ to noisy data: Watch how the line wobbles into place. At each step, we compute t