Hyperbolic Tangent Activation Function

A hyperbolic tangent function is another type of activation function used in Deep Learning, which is a smoother, zero-centred function.

   x: an input data point

In terms of sigmoid function:

Range: (-1, 1)

# Import Numpy Library
import numpy as np

#Hyperbolic Tangent Function

def tanh(x):
    return 2*(sigmoid(2*x)) - 1 

Use cases:

Used in multi-layered neural networks.

Mostly appears in recurrent neural networks for natural language processing and speech recognition.

 

Pros:

Tanh function gives better results than sigmoid function when dealing with multi-layered neural networks.

Since the range of Tanh function is (-1, 1), the outputs are zero centered and having a higher convergence, the network works on normalized data resulting in faster learning.

 

Cons:

Though Tanh is a zero centered function, it still suffers from vanishing gradient  during backpropagation.

 Produces some dead neurons when the input value is 0.

 

Hard Hyperbolic function is a variant of Tanh function, which reduces the computational speed thus aids in NLP applications.

Share

Share on twitter
Share on facebook
Share on linkedin
Share on whatsapp
Share on reddit
Share on telegram
Share on pinterest
Share on email
Share on facebook

Readers who read this also read

Close Menu