Activation Function

Logistic function

The s-shaped logistic function was used in the early days of neural networks because its shape seemed to match what we thought was happening in animals as a threshold for signals passing between neurons, and also because it is mathematically convenient for calculating gradients.

However, it does have some weaknesses. The main one is that for large values, the gradients get very small, and can effectively disappear. This means that training neural networks, which depend on gradients to correct the link weights, can start to fail when this kind of saturation happens.