Three Ways to Think About Backpropagation

One of the coolest parts of writing educational material on topics as you’re learning them is how well you understand it afterwards. I recently wrote a tiny course (completely free!) on how to implement the backpropagation algorithm, and I could talk about what an interesting algorithm it is for a while. There are tons of different interpretations you could have about what it’s doing. Here are three.

1. Credit Assignment

The first perspective, and my personal favorite, is that of credit assignment. Every neuron in the network contributes to the final error in some way. Backpropagation is the process of assigning how much “blame” to give each neuron for that error. Neurons that didn’t contribute much to the error get small updates, and neurons that were highly influential get substantial updates to correct their behavior. In the end, every neuron gets nudged in proportion to how responsible it was for the outcome.

2. Computation

The second perspective is through the lens of computation. It’s simply an efficient algorithm. By storing all the intermediate values and calculating the partial derivatives backwards (dynamic programming), we can take advantage of the chain rule to compute the gradient in just one forward and backwards pass.

3. Math

The third perspective is mathematical. Backpropagation can be written as a recursive mathematical expression where you are propagating the error backwards through the neural network, and the error determines how the model should get updated. The derivations can get extremely messy as the architectures get bigger and more complicated (which is why we don’t derive the equations for models like Claude), but it’s possible to do for any model.

Conclusion

If any of these perspectives clicked for you, you might enjoy exploring backpropagation in our free course, Backpropagation from Scratch. In it, we build a tiny backpropagation engine from the ground up and use it to train a neural network.