🔵 ReLU Activation Function

ReLU stands for Rectified Linear Unit. It is one of the most widely used activation functions in modern neural networks, particularly in hidden layers.

---

1. Definition of ReLU

ReLU is defined as:

$$\boxed{\operatorname{ReLU}(z)=\max(0,z)}$$

Equivalently, we can write it as a piecewise function:

$$\boxed{ \operatorname{ReLU}(z)= \begin{cases} 0, & z<0\\ z, & z\geq0 \end{cases} }$$

In simple words:

ReLU keeps positive values and converts negative values to zero.

Examples

$$\operatorname{ReLU}(-5)=0$$ $$\operatorname{ReLU}(-2)=0$$ $$\operatorname{ReLU}(0)=0$$ $$\operatorname{ReLU}(3)=3$$ $$\operatorname{ReLU}(10)=10$$

So:

$z$	ReLU($z$)
-5	0
-2	0
-1	0
0	0
1	1
2	2
5	5

---

2. How ReLU works inside a neuron

As with other activation functions, first calculate the weighted sum:

$$z=\sum_{i=1}^{n}w_ix_i+b$$

Then apply ReLU:

$$a=\operatorname{ReLU}(z)$$

Therefore,

$$\boxed{ a=\max\left(0,\sum_{i=1}^{n}w_ix_i+b\right) }$$

Example

Suppose

$$x_1=2,\qquad x_2=3$$ $$w_1=0.5,\qquad w_2=0.2$$

and

$$b=-2$$

First calculate $z$:

$$z=(0.5)(2)+(0.2)(3)-2$$ $$z=1+0.6-2$$ $$z=-0.4$$

Now apply ReLU:

$$a=\operatorname{ReLU}(-0.4)$$ $$\boxed{a=0}$$

---

3. What happens for positive and negative inputs?

The easiest way to understand ReLU is:

🔴 Negative input

If

$$z<0$$

then

$$\operatorname{ReLU}(z)=0$$

The neuron produces zero output.

🟢 Positive input

If

$$z>0$$

then

$$\operatorname{ReLU}(z)=z$$

The neuron passes the value directly to the next layer.

Therefore:

$$\boxed{ \text{Negative}\rightarrow0 \qquad \text{Positive}\rightarrow\text{unchanged} }$$

---

4. Graph of ReLU

The graph has two very simple regions:

$$y=0 \quad \text{for }x<0$$

and

$$y=x \quad \text{for }x\geq0$$

So it looks like this conceptually:

The important point is that ReLU is nonlinear, even though its positive portion is a straight line.

---

5. Derivative of ReLU

For backpropagation, we need the derivative.

For

$$\operatorname{ReLU}(z)= \begin{cases} 0,&z<0\\ z,&z>0 \end{cases}$$

the derivative is

$$\boxed{ \operatorname{ReLU}'(z)= \begin{cases} 0,&z<0\\ 1,&z>0 \end{cases} }$$

At $z=0$, ReLU is not differentiable in the ordinary sense because the slope changes abruptly from $0$ to $1$.

In practical neural-network implementations, a subgradient convention is chosen, commonly either $0$ or $1$.

---

6. Why is ReLU useful for neural networks?

The most important reason is that ReLU introduces nonlinearity.

Consider:

$$z=w_1x_1+w_2x_2+b$$

Without an activation function, a neuron simply performs a linear transformation.

ReLU gives:

$$a=\max(0,w_1x_1+w_2x_2+b)$$

This allows a neural network with multiple layers to model nonlinear relationships.

For example, a multilayer network can learn complex patterns such as:

• image patterns
• handwritten digits
• speech patterns
• object features
• nonlinear classification boundaries

---

7. Why is ReLU often preferred over sigmoid and tanh in hidden layers?

This is an important comparison for students.

Sigmoid

$$\sigma(z)=\frac{1}{1+e^{-z}}$$

Its derivative is

$$\sigma'(z)=\sigma(z)(1-\sigma(z))$$

The maximum derivative is only

$$\sigma'(0)=0.25$$

For large positive or negative $z$, the derivative becomes very small.

---

Tanh

$$\tanh(z)=\frac{e^z-e^{-z}}{e^z+e^{-z}}$$

Its derivative is

$$\tanh'(z)=1-\tanh^2(z)$$

Its maximum derivative is

$$\tanh'(0)=1$$

but it also suffers from saturation for large $|z|$.

---

ReLU

$$\operatorname{ReLU}(z)=\max(0,z)$$

and

$$\operatorname{ReLU}'(z)= \begin{cases} 0,&z<0\\ 1,&z>0 \end{cases}$$

For positive $z$, the derivative is exactly 1.

Therefore, positive activations do not suffer from the same saturation problem as sigmoid and tanh.

---

8. ReLU and the vanishing-gradient problem

This is one of the major advantages of ReLU.

For sigmoid:

$$\sigma'(z)\rightarrow0$$

when $z$ becomes very large or very negative.

For tanh:

$$\tanh'(z)\rightarrow0$$

when

$$|z|\rightarrow\infty$$

But for ReLU:

$$\operatorname{ReLU}'(z)=1$$

for all positive $z$.

Therefore, when the neuron is active:

$$\boxed{\text{gradient can pass through without being multiplied by a small activation derivative}}$$

This was an important reason ReLU became popular in deep neural networks.

---

9. The "dying ReLU" problem

ReLU has its own important limitation.

For negative inputs:

$$\operatorname{ReLU}(z)=0$$

and

$$\operatorname{ReLU}'(z)=0$$

Suppose a neuron consistently receives negative $z$ values.

Then:

$$a=0$$

and

$$\frac{\partial a}{\partial z}=0$$

Consequently, the neuron may stop receiving useful gradient updates and effectively become inactive.

This is called the:

$$\boxed{\text{Dying ReLU problem}}$$

---

10. Leaky ReLU: a solution to dying ReLU

One solution is Leaky ReLU.

It is defined as:

$$\boxed{ \operatorname{LeakyReLU}(z)= \begin{cases} \alpha z,&z<0\\ z,&z\geq0 \end{cases} }$$

where $\alpha$ is a small positive number, such as

$$\alpha=0.01$$

For example:

$$\operatorname{LeakyReLU}(-5)=0.01(-5)=-0.05$$

instead of

$$\operatorname{ReLU}(-5)=0$$

Thus, Leaky ReLU allows a small gradient for negative inputs.

---

11. Why ReLU is so popular

The main advantages are:

✅ 1. Very simple

$$\operatorname{ReLU}(z)=\max(0,z)$$

There is no exponential calculation.

✅ 2. Computationally efficient

It is much simpler to calculate than sigmoid or tanh.

✅ 3. Introduces nonlinearity

It allows neural networks to learn nonlinear functions.

✅ 4. Better gradient behavior for positive inputs

$$\operatorname{ReLU}'(z)=1$$

for $z>0$.

✅ 5. Sparse activation

Negative inputs produce zero:

$$z<0\Rightarrow a=0$$

Therefore, many neurons may be inactive for a particular input. This creates sparse activations.

---

12. Where is ReLU used?

ReLU is particularly common in the hidden layers of neural networks.

For example:

$$\boxed{ \text{Input} \rightarrow \text{Linear} \rightarrow \text{ReLU} \rightarrow \text{Linear} \rightarrow \text{ReLU} \rightarrow \text{Output} }$$

It is widely used in architectures for:

• 🖼️ Image classification
• 👁️ Computer vision
• 🔊 Speech processing
• 📝 Natural language processing
• 🤖 Deep neural networks
• 🧠 Convolutional neural networks

Modern networks also frequently use ReLU variants such as Leaky ReLU, PReLU, ELU, and GELU depending on the architecture.

---

🎓 Summary

"ReLU is a gate. If the input is negative, it closes the gate and produces 0. If the input is positive, it opens the gate and passes the input unchanged."

Mathematically:

$$\boxed{ z<0\Rightarrow0 \qquad z\geq0\Rightarrow z }$$

🧠 Memory trick

$$\boxed{\text{ReLU = Keep Positive, Remove Negative}}$$

or simply:

$$\boxed{\operatorname{ReLU}(z)=\max(0,z)}$$

This one equation captures the entire basic idea of ReLU.

ReLU (Rectified Linear Unit) is a widely used nonlinear activation function in neural networks, especially in hidden layers. It is defined as $\operatorname{ReLU}(z)=\max(0,z)$, meaning negative inputs are converted to zero while positive inputs are passed unchanged. Its derivative is $0$ for negative inputs and $1$ for positive inputs, which helps reduce the vanishing-gradient problem seen with sigmoid and tanh. ReLU is simple, computationally efficient, and introduces the nonlinearity needed to learn complex patterns. However, neurons can become permanently inactive for negative inputs, known as the dying ReLU problem.