Beginner's Guide to Deep Learning

A neuron calculates the ‘weighted sum’ of its inputs and adds a bias.

Now the predicted value can be anything from -inf to +inf. The neuron really doesn’t know the bounds of the value. So how do we decide whether the neuron should fire or not? In other words, predict whether the class label is positive or negative(in the case we have two classes). Thus we add an activation function to make the neural network decide whether a neuron should be activated or not, hence the name, activation functions. 




# Types of Activation Functions
The following section discusses the several different kinds of activation functions used in deep learning:

## Step function
The first thing that comes to our minds is how about a threshold-based activation function? We saw this activation function when we learned about perceptrons [here](https://www.educative.io/collection/page/4998094290681856/6042323658997760/5568494276968448). 

If the predicted value ($y$) is above a certain value(threshold), declare it activated.


**Advantages**

In a binary classification problem, where you want to specify whether the class is "yes" or "no", a step function would do the job. That is, it would say 1(activate) or 0 (not activate). 

**Disadvantages**

The problem with a step function is that it does not cater to multi-value outputs—for example, it cannot support classifying the inputs into one of several categories.

If the predicted value ($y$) is above a certain value(threshold), declare it activated.


**Advantages**

In a binary classification problem, where you want to specify whether the class is "yes" or "no", a step function would do the job. That is, it would say 1(activate) or 0 (not activate). 

**Disadvantages**

The problem with a step function is that it does not cater to multi-value outputs—for example, it cannot support classifying the inputs into one of several categories.