Activation Functions
Explore the role of activation functions in deep learning and understand how they transform outputs to enable neurons to activate appropriately. Learn the characteristics, advantages, and disadvantages of popular activation functions like step, sigmoid, tanh, ReLU, and softmax, and discover how to select the best one for different problems to improve neural network performance.
A neuron calculates the “weighted sum” of its inputs and adds a bias.
Now, the predicted value can be anything from -∞ to +∞. The neuron does not know the bounds of the value. So, how do we decide whether the neuron should fire or not? Predict whether the class label is positive or negative (in this case, we have two classes). Thus, add an activation function to make the neural network decide whether a neuron should be activated or not. Hence the given name, “activation functions”.
Types of activation functions
The following section discusses several different types of activation functions that are used in deep learning:
Step function
The first thing that comes to mind is: How about a threshold-based activation function? We saw this activation function when we learned about perceptrons here.
If the predicted value ( ...