Introduction to Activation Functions

Activation Function or Transfer Function is a mathematical method used to control the output of an Artificial Neuron. From learning key patterns in the data to speeding up the training, these are the key components for the training and optimization of deep neural networks.

Intuition

Look at this scenario to understand the need of activation functions.

Here the points belonging to two categories can be separable with a linear equation.

m: slope; c: intercept and

x: datapoint

What if those data points are arranged as follows:

We can’t simply fit a linear line through them to separate the data points into their respective categories.

We can try a nonlinear equation to fit the data as follows:

This looks better than the previous one. Here we modified the linear equation  to  a nonlinear functionc).

That is known as an activation function. The activation function induces some non-linearity so as to fit the model by learning some inherent patterns in the data.

Readers who read this also read

Close Menu