Calculating value of derivative function is a part of many applied tasks such

– gradient descent/ascent for finding minimum/maximum of a function

– Newton’s method for finding approximations to the roots (or zeroes) of a real-valued function

etc.

There are three main ways to calculate derivatives

1. Finite differences i.e. calculate approximation using (f(x+Δx)-f(x))/Δx when Δx is close to zero.

2. Symbolic differentiation – generates exact formula for derivative function.

3. Automatic differentiation – generates evaluations (and not formulas) of the derivatives. All intermediate expressions are evaluated as soon as possible.

A brilliant explanation of automatic differentiation may be found here in a paper by Warwick Tucker “One-dimensional, first-order, and Taylor-mode automatic differentiation with programming examples”.

So let’s have a closer look on some technics for derivation in Python/R.

Continue reading