🔵 Tanh / Hyperbolic Tangent Activation Function

The tanh (hyperbolic tangent) function is another important nonlinear activation function used in artificial neural networks.

It is closely related to the sigmoid function, but it has one important advantage: its output is centered around zero.

The tanh function is particularly useful for understanding the evolution from sigmoid-based neural networks to modern activation functions such as ReLU.

---

🧠 1. What Is the Tanh Function?

The hyperbolic tangent function is defined as:

$$\boxed{ \tanh(z)=\frac{e^z-e^{-z}}{e^z+e^{-z}} }$$

It can also be expressed in terms of the sigmoid function:

$$\boxed{ \tanh(z)=2\sigma(2z)-1 }$$

where:

$$\boxed{ \sigma(z)=\frac{1}{1+e^{-z}} }$$

The output of tanh lies between:

−1<tanh(z)<1​

So, conceptually:

Tanh=scaled sigmoid−shift​

📈 2. Shape of the Tanh Function

The tanh function has an S-shaped curve, similar to sigmoid.

There are three important regions:

🔹 Large negative input

$$z\rightarrow-\infty$$

Then:

$$\boxed{\tanh(z)\rightarrow-1}$$

🔹 Zero input

$$z=0$$

Then:

$$\boxed{\tanh(0)=0}$$

🔹 Large positive input

$$z\rightarrow+\infty$$

Then:

$$\boxed{\tanh(z)\rightarrow1}$$

Therefore:

$$\boxed{-1<\tanh(z)<1}$$

---

🔢 3. Numerical Examples

Let's calculate some tanh values.

$z$	$\tanh(z)$
-5	-0.9999
-2	-0.9640
-1	-0.7616
0	0
1	0.7616
2	0.9640
5	0.9999

This shows that tanh compresses an arbitrary real-valued input into:

$$\boxed{(-1,1)}$$

---

⚙️ 4. Tanh in an Artificial Neuron

Just like any other activation function, tanh is applied after calculating the weighted sum.

First:

$$\boxed{ z=\sum_{i=1}^{n}w_ix_i+b }$$

Then:

$$\boxed{ y=\tanh(z) }$$

Therefore, the complete neuron is:

$$\boxed{ y= \tanh\left(\sum_{i=1}^{n}w_ix_i+b\right) }$$

🧮 5. Numerical Example

Suppose:

$$x_1=2,\qquad x_2=3$$

Weights:

$$w_1=0.5,\qquad w_2=0.2$$

Bias:

$$b=-0.4$$

Step 1: Calculate weighted sum

$$z=w_1x_1+w_2x_2+b$$ $$z=(0.5)(2)+(0.2)(3)-0.4$$ $$z=1+0.6-0.4$$ $$\boxed{z=1.2}$$

Step 2: Apply tanh

$$y=\tanh(1.2)$$ $$\boxed{y\approx0.8337}$$

Therefore, the neuron produces:

y≈0.834​

🧠 6. Why Is Tanh Better Than Sigmoid in Some Situations?

This is an important comparison.

Sigmoid:

$$\boxed{ 0<\sigma(z)<1 }$$

Tanh:

$$\boxed{ -1<\tanh(z)<1 }$$

The major difference is that tanh is zero-centered.

For example:

Sigmoid:

0 ──────────────── 1
       ↑
    all outputs
    are positive


Tanh:

-1 ─────── 0 ─────── +1
            ↑
       zero-centered

---

🎯 7. Zero-Centered Property

Suppose the input is positive:

$$z>0$$

Then:

$$\tanh(z)>0$$

If the input is negative:

$$z<0$$

then:

$$\tanh(z)<0$$

And:

$$\tanh(0)=0$$

Therefore, tanh produces both positive and negative activations.

This can make optimization more favorable than sigmoid in some settings.

---

📊8. Sigmoid vs Tanh

Property	Sigmoid	Tanh
Formula	$\displaystyle\frac{1}{1+e^{-z}}$	$\displaystyle\frac{e^z-e^{-z}}{e^z+e^{-z}}$
Output range	$(0,1)$	$(-1,1)$
Shape	S-shaped	S-shaped
Zero-centered	❌ No	✅ Yes
Nonlinear	✅ Yes	✅ Yes
Saturates	✅ Yes	✅ Yes
Vanishing gradients	⚠️ Yes	⚠️ Yes
Binary output	Very useful	Less direct
Historical hidden-layer use	Common	Common
Modern deep hidden layers	Less common	Less common

---

🧮 9. Derivative of Tanh

The derivative of tanh has a particularly simple form.

$$\boxed{ \frac{d}{dz}\tanh(z)=1-\tanh^2(z) }$$

Therefore:

$$\boxed{ \tanh'(z)=1-\tanh^2(z) }$$

This derivative is important during backpropagation.

---

🔍 10. Deriving the Tanh Derivative

Start with:

$$\tanh(z)=\frac{e^z-e^{-z}}{e^z+e^{-z}}$$

Using the quotient rule:

$$\frac{d}{dz}\tanh(z) = \frac{ (e^z+e^{-z})(e^z+e^{-z}) - (e^z-e^{-z})(e^z-e^{-z}) }{ (e^z+e^{-z})^2 }$$

After simplifying:

$$\frac{d}{dz}\tanh(z) = \frac{4}{(e^z+e^{-z})^2}$$

This is equivalent to:

$$\boxed{ \tanh'(z)=1-\tanh^2(z) }$$

The second form is much more convenient when discussing neural-network learning.

---

📈 11. Maximum Gradient of Tanh

We have:

$$\tanh'(z)=1-\tanh^2(z)$$

At:

$$z=0$$

we know:

$$\tanh(0)=0$$

Therefore:

$$\tanh'(0)=1-0^2$$ $$\boxed{\tanh'(0)=1}$$

Thus the maximum derivative of tanh is:

$$\boxed{1}$$

Compare this with sigmoid:

$$\boxed{\sigma'(z)_{\max}=0.25}$$

This is one reason tanh can have stronger gradients around zero than sigmoid.

---

⚠️ 12. Tanh and the Vanishing Gradient Problem

Although tanh has some advantages over sigmoid, it still suffers from vanishing gradients.

Consider a large positive value:

$$z=5$$

We have:

$$\tanh(5)\approx0.9999$$

Therefore:

$$\tanh'(5) = 1-(0.9999)^2$$

which is approximately:

$$\boxed{\tanh'(5)\approx0.00018}$$

The derivative is very small.

Similarly, for a large negative value:

$$z=-5$$ $$\tanh(-5)\approx-0.9999$$

and again:

$$\tanh'(-5)\approx0.00018$$

Thus:

$$\boxed{ \tanh'(z)\rightarrow0 \quad\text{as}\quad |z|\rightarrow\infty }$$

---

🌡️ 13. Saturation of Tanh

The tanh function saturates near $-1$ and $+1$.

Output

 +1 |────────────────────
    |                 ___
    |              __/
  0 |────────────●────────────
    |          __/
    |       __/
 -1 |──────
    +────────────────────→ z

In the saturated regions:

$$|\tanh(z)|\approx1$$

Therefore:

$$\tanh'(z)=1-\tanh^2(z)\approx0$$

So:

$$\boxed{ \text{Saturation} \rightarrow \text{small gradient} \rightarrow \text{slow learning} }$$

---

🔄 14. Tanh and Backpropagation

During backpropagation, gradients are propagated through the network.

The derivative of the activation function contributes to this gradient.

For tanh:

$$\boxed{ \tanh'(z)=1-\tanh^2(z) }$$

The weight update is generally:

$$\boxed{ w_{\text{new}} = w_{\text{old}} - \eta \frac{\partial L}{\partial w} }$$

where:

• $L$ = loss
• $\eta$ = learning rate
• $\frac{\partial L}{\partial w}$ = gradient

If tanh is saturated:

$$\tanh'(z)\approx0$$

then the gradient can become very small.

---

🧩 15. Applications of Tanh

🧠  Hidden Layers of Classical Neural Networks

Tanh was historically a popular activation function for hidden layers.

It is still useful in certain architectures where zero-centered activations are desirable.

It is also used in RNN and LSTM

---

🎛️ 16. Advantages of Tanh

🟢 1. Nonlinear

Tanh allows a neural network to learn complex nonlinear relationships.

🟢 2. Zero-Centered

Its output range is:

$$\boxed{-1<\tanh(z)<1}$$

This is an important advantage over sigmoid.

🟢 3. Stronger Gradient Around Zero

At:

$$z=0$$

we have:

$$\boxed{\tanh'(0)=1}$$

compared with sigmoid's maximum derivative:

$$\boxed{\sigma'(z)_{\max}=0.25}$$

🟢 4. Produces Positive and Negative Activations

This can be useful when the representation naturally requires both directions.

🟢 5. Useful in RNN/LSTM Architectures

Tanh has historically been and remains an important component of recurrent architectures.

---

❌ 19. Disadvantages of Tanh

🔴 1. Vanishing Gradients

For large $|z|$:

$$\boxed{\tanh'(z)\rightarrow0}$$

This can slow training in deep networks.

🔴 2. Saturation

The function approaches:

$$-1$$

and:

$$+1$$

at the extremes.

🔴 3. Computationally More Expensive Than ReLU

Tanh involves exponential calculations:

$$e^z,\qquad e^{-z}$$

whereas ReLU is:

$$\boxed{\operatorname{ReLU}(z)=\max(0,z)}$$

🔴 4. Not Usually the First Choice for Modern Deep Hidden Layers

Modern deep networks often use ReLU-family or other newer activation functions because they can optimize more effectively in many settings.

---

📌 20. Where Should We Use Tanh?

A useful  guideline is:

Location / Problem	Tanh
Classical hidden layers	✅ Used historically
RNN hidden state	✅ Common
LSTM candidate/state	✅ Common
Binary classification output	❌ Usually sigmoid
Multiclass classification output	❌ Usually softmax
Regression output	❌ Usually linear
Modern deep hidden layers	⚠️ Often replaced by ReLU-family functions

---

⭐ 21. Key Points for Students

Remember these six points:

① Formula

$$\boxed{ \tanh(z)=\frac{e^z-e^{-z}}{e^z+e^{-z}} }$$

② Range

$$\boxed{-1<\tanh(z)<1}$$

③ Shape

$$\boxed{\text{S-shaped}}$$

④ Zero-centered

$$\boxed{\tanh(0)=0}$$

⑤ Derivative

$$\boxed{ \tanh'(z)=1-\tanh^2(z) }$$

⑥ Main limitation

$$\boxed{\text{Vanishing Gradient due to saturation}}$$

---

🎓 22. Summary

The tanh (hyperbolic tangent) activation function is a nonlinear activation function that maps a real-valued input to the range $(-1,1)$. Unlike sigmoid, tanh is zero-centered, which can make optimization more favorable in some situations. Its derivative is $1-\tanh^2(z)$, with a maximum value of 1 at $z=0$. However, tanh saturates for large positive and negative inputs, causing its gradient to approach zero and potentially producing the vanishing-gradient problem. Tanh has been widely used in recurrent neural networks and LSTM architectures and was historically common in hidden layers of neural networks.