a whole blogpost[7] and is very well described in textbooks[4], so we suggest you refer to them if you want to understand the internals of it. What's important to remember is that backpropagation lets you update the weights of the neural network by propagating the error (difference between the outputed value and the expected value from the training set) along the gradient of the network. Computing that gradient, by the way, requires your activation function to be differentiable. Backpropagation was invented