Activation Functions
Learn about different activation functions and how to apply them to neurons.
We'll cover the following...
Simple activation function
A function that takes the input signal and generates an output signal, but takes into account some kind of threshold, is called an activation function.
Mathematically speaking, there are many activation functions that could achieve this effect. A simple step function could do this.
We can see that for low input values, the output is zero. However, once the threshold input is reached, the output increases. An artificial neuron behaving like this would be like a real biological neuron. The term used by scientists actually describes this very well. They say that neurons fire when the input reaches the threshold. We can improve the step function with the help of the sigmoid function.
Sigmoid function
The s-shaped function shown below is called the sigmoid function. It is smoother than the step function, and this makes it more natural and realistic.
This smooth, s-shaped sigmoid function is what we’ll use for making our own neural network. Artificial ...