Credits

1. “The spelled-out intro to neural networks and backpropagation: building micrograd” by Andrej Karpathy

   Andrej Karpathy’s teaching is well known in the machine learning community, and this textbook wouldn’t have existed without him. When deciding that I wanted to focus on writing Machine Learning resources for Impart Education, I decided to look at how he went about teaching the same material. After seeing that he spent the time to create a backpropagation (or autograd) engine, I decided to as well. The structure of the course and code came from him.

2. “Calculus on Computational Graphs: Backpropagation” by Christopher Olah

   Colah’s post is a foundational explanation of how backpropagation works on computational graphs. It shaped how I went about structuring my explanations in this course, and was a valuable reference while writing the sections on computational graphs in this textbook. In fact, after originally “finishing” this course, I stumbled upon this blog and decided to bring some of it’s insights into this course.

3. “Automatic differentiation from scratch” by Emilio Dorigatti

   When doing research for this textbook, I serendipitously stumbled on this blog. The course banner is inspired by the first graph shown in the blog. It was an amazing reference that helped me decide how I wanted to go about discussing many topics in this textbook.

4. “What is Automatic Differentiation?”

   Provided a useful comparison of numerical and symbolic differentiation to automatic differentiation that was used in an attempt to introduce backpropagation.

5. “Chain Rule + Dynamic Programming = Neural Networks”

   Dynamic programming isn’t talked about much when discussing backpropagation, and this blog made me aware of the connection between the two while doing research on backpropagation.