🟢 Sigmoid / Logistic Activation Function

1. 🧠 Introduction

The sigmoid function, also called the logistic function, is one of the classical nonlinear activation functions used in artificial neural networks.

It is particularly useful for binary classification, because it converts any real-valued input into a value between 0 and 1.

The sigmoid function is defined as:

$$\boxed{\sigma(z)=\frac{1}{1+e^{-z}}}$$

where:

• $z$ = input to the activation function
• $e$ = Euler's number, approximately $2.71828$
• $\sigma(z)$ = output of the sigmoid function

---

⚙️ 2. Sigmoid in an Artificial Neuron

Before applying the activation function, a neuron calculates the weighted sum of its inputs and adds a bias.

For $n$ inputs:

$$\boxed{z=\sum_{i=1}^{n}w_i x_i+b}$$

or, in expanded form:

$$z=w_1x_1+w_2x_2+\cdots+w_nx_n+b$$

The sigmoid function is then applied to $z$:

$$\boxed{y=\sigma(z)}$$

Therefore:

$$\boxed{ y=\frac{1}{1+e^{-\left(\sum_{i=1}^{n}w_ix_i+b\right)}} }$$

This is the complete computation performed by a sigmoid neuron.

---

📈 3. Shape of the Sigmoid Function

The sigmoid function has an S-shaped curve.

$$\boxed{\sigma(z)=\frac{1}{1+e^{-z}}}$$

Conceptually:

There are three important regions:

🔹 When $z$ is very negative

$$z\rightarrow-\infty$$

then:

$$\boxed{\sigma(z)\rightarrow0}$$

🔹 When $z=0$

$$\sigma(0)=\frac{1}{1+e^0}$$

Since $e^0=1$:

$$\boxed{\sigma(0)=0.5}$$

🔹 When $z$ is very positive

$$z\rightarrow+\infty$$

then:

$$\boxed{\sigma(z)\rightarrow1}$$

Therefore:

$$\boxed{0<\sigma(z)<1}$$

---

🔢 4. Numerical Examples

Let's calculate the sigmoid for different values of $z$.

Example 1: $z=0$

$$\sigma(0)=\frac{1}{1+e^0}$$ $$=\frac{1}{1+1}$$ $$\boxed{\sigma(0)=0.5}$$

---

Example 2: $z=1$

$$\sigma(1)=\frac{1}{1+e^{-1}}$$ $$\boxed{\sigma(1)\approx0.731}$$

---

Example 3: $z=-1$

$$\sigma(-1)=\frac{1}{1+e^1}$$ $$\boxed{\sigma(-1)\approx0.269}$$

---

Example 4: $z=5$

$$\sigma(5)=\frac{1}{1+e^{-5}}$$ $$\boxed{\sigma(5)\approx0.993}$$

---

Example 5: $z=-5$

$$\sigma(-5)=\frac{1}{1+e^5}$$ $$\boxed{\sigma(-5)\approx0.007}$$

Summary

$z$	$\sigma(z)$
-5	0.007
-2	0.119
-1	0.269
0	0.500
1	0.731
2	0.881
5	0.993

This table clearly shows that sigmoid compresses any real-valued input into the interval (0,1).

---

🎯 5. Why Is Sigmoid Useful for Classification?

Suppose a neural network is predicting whether an email is:

• 0 → Not Spam
• 1 → Spam

The neuron calculates:

$$z=w^Tx+b$$

Suppose:

$$z=2.2$$

Then:

$$\sigma(2.2)=\frac{1}{1+e^{-2.2}}$$ $$\boxed{\sigma(2.2)\approx0.900}$$

The output is approximately 0.90.

This can be interpreted as an estimated probability-like output for the positive class:

$$\boxed{P(y=1|x)\approx0.90}$$

So the model gives a high estimated probability to the positive class.

---

🚦 6. Converting Sigmoid Output into a Class

The sigmoid output itself is continuous.

For example:

$$0.12,\quad0.37,\quad0.51,\quad0.82,\quad0.97$$

To convert this into a binary class, we normally choose a threshold.

A common threshold is:

$$\boxed{T=0.5}$$

The classification rule is:

$$\boxed{ \hat y= \begin{cases} 1 & \text{if }\sigma(z)\geq0.5\\ 0 & \text{if }\sigma(z)<0.5 \end{cases} }$$

Because:

$$\sigma(0)=0.5$$

the same decision rule can also be written as:

$$\boxed{ \hat y= \begin{cases} 1 & z\geq0\\ 0 & z<0 \end{cases} }$$

---

🧮 7. Complete Example of a Sigmoid Neuron

Suppose we have two inputs:

$$x_1=2,\qquad x_2=3$$

Weights:

$$w_1=0.8,\qquad w_2=-0.4$$

Bias:

$$b=-0.5$$

Step 1: Calculate weighted sum

$$z=w_1x_1+w_2x_2+b$$ $$z=(0.8)(2)+(-0.4)(3)-0.5$$ $$z=1.6-1.2-0.5$$ $$\boxed{z=-0.1}$$

Step 2: Apply sigmoid

$$y=\frac{1}{1+e^{-(-0.1)}}$$ $$y=\frac{1}{1+e^{0.1}}$$ $$\boxed{y\approx0.475}$$

Since:

$$0.475<0.5$$

the predicted class is:

$$\boxed{\hat y=0}$$

---

🔄 8. Sigmoid and Logistic Regression

An important connection for students is that logistic regression uses exactly this sigmoid transformation.

First calculate:

$$\boxed{z=w^Tx+b}$$

Then:

$$\boxed{ P(y=1|x)=\frac{1}{1+e^{-z}} }$$

Substituting $z$:

$$\boxed{ P(y=1|x)= \frac{1}{1+e^{-(w^Tx+b)}} }$$

Therefore, a single neuron with a sigmoid activation function has essentially the same mathematical form as logistic regression for binary classification.

This is a very useful connection between classical machine learning and neural networks.

---

📐 9. Derivative of the Sigmoid Function

The derivative of sigmoid is particularly convenient.

Starting with:

$$\sigma(z)=\frac{1}{1+e^{-z}}$$

the derivative is:

$$\boxed{ \frac{d\sigma(z)}{dz} = \sigma(z)(1-\sigma(z)) }$$

This can also be written as:

$$\boxed{ \sigma'(z)=\sigma(z)[1-\sigma(z)] }$$

This derivative is used during backpropagation to calculate gradients.

---

🔄 10. Derivation of the Sigmoid Derivative

Starting with:

$$\sigma(z)=(1+e^{-z})^{-1}$$

Differentiate:

$$\sigma'(z) = -(1+e^{-z})^{-2}(-e^{-z})$$

Therefore:

$$\sigma'(z)=\frac{e^{-z}}{(1+e^{-z})^2}$$

Now observe:

$$\sigma(z)=\frac{1}{1+e^{-z}}$$

and:

$$1-\sigma(z) = 1-\frac{1}{1+e^{-z}}$$ $$=\frac{e^{-z}}{1+e^{-z}}$$

Therefore:

$$\sigma(z)(1-\sigma(z)) = \frac{1}{1+e^{-z}} \frac{e^{-z}}{1+e^{-z}}$$ $$=\frac{e^{-z}}{(1+e^{-z})^2}$$

Hence:

$$\boxed{\sigma'(z)=\sigma(z)(1-\sigma(z))}$$

---

📊 11. Maximum Value of the Gradient

The derivative is:

$$\sigma'(z)=\sigma(z)(1-\sigma(z))$$

Let:

$$\sigma(z)=p$$

Then:

$$\sigma'(z)=p(1-p)$$

This is maximum when:

$$p=0.5$$

Therefore:

$$\sigma'(z)=0.5(1-0.5)$$ $$=0.5\times0.5$$ $$\boxed{\sigma'(z)_{\max}=0.25}$$

Thus, the derivative of sigmoid can never be greater than 0.25.

---

⚠️ 12. Vanishing Gradient Problem

This is the major disadvantage of sigmoid.

Consider a very large positive input:

$$z=10$$

Then:

$$\sigma(10)\approx0.99995$$

The derivative becomes:

$$\sigma'(10) = 0.99995(1-0.99995)$$ $$\boxed{\sigma'(10)\approx0.00005}$$

This is extremely small.

Similarly, for:

$$z=-10$$

we obtain a very small derivative.

Therefore:

$$\boxed{\sigma'(z)\rightarrow0 \quad\text{when}\quad z\rightarrow\pm\infty}$$

---

🧠 13. Why Does This Cause a Problem?

Backpropagation uses gradients to update weights.

The basic gradient-descent update is:

$$\boxed{ w_{\text{new}} = w_{\text{old}} - \eta \frac{\partial L}{\partial w} }$$

where:

• $L$ = loss
• $\eta$ = learning rate
• $\frac{\partial L}{\partial w}$ = gradient

If the gradient becomes very small:

$$\frac{\partial L}{\partial w}\approx0$$

then:

$$w_{\text{new}}\approx w_{\text{old}}$$

Therefore, the weights change very slowly.

This is called the:

$$\boxed{\text{Vanishing Gradient Problem}}$$

---

🌡️ 14. Saturation

Sigmoid saturates at both ends.

For large negative values:

$$z\rightarrow-\infty$$ $$\sigma(z)\rightarrow0$$

For large positive values:

$$z\rightarrow+\infty$$ $$\sigma(z)\rightarrow1$$

The curve becomes almost flat in these regions.

Output
  1 |                         ─────────
    |                    ____/
    |                 __/
0.5 |---------------●
    |            __/
    |         __/
  0 |────────
    +──────────────────────────→ z

A flat curve means a small gradient.

Therefore:

$$\boxed{\text{Saturation}\rightarrow\text{small gradient}\rightarrow \text{slow learning}}$$

---

🎯 15. Applications of Sigmoid

📨 15.1 Binary Classification

The most common application is binary classification.

Examples:

• Spam / Not Spam
• Fraud / Not Fraud
• Pass / Fail
• Disease / No Disease
• Defective / Non-defective
• Churn / No Churn

---

✅ 16. Advantages of Sigmoid

🟢 1. Output is Between 0 and 1

$$\boxed{0<\sigma(z)<1}$$

This makes it naturally suitable for probability-like outputs.

🟢 2. Smooth Function

Sigmoid is continuous and differentiable everywhere.

🟢 3. Simple Derivative

$$\boxed{\sigma'(z)=\sigma(z)(1-\sigma(z))}$$

🟢 4. Excellent for Binary Classification

It provides a natural single-output representation for a binary target.

🟢 5. Historically Important

Sigmoid was extensively used in traditional neural networks and remains appropriate for many binary-output applications.

---

❌ 17. Disadvantages of Sigmoid

🔴 1. Vanishing Gradient

$$\boxed{\sigma'(z)\rightarrow0 \quad\text{for large }|z|}$$

This can make deep networks learn slowly.

🔴 2. Saturation

The output becomes almost constant near 0 and 1.

🔴 3. Not Zero-Centered

The output satisfies:

$$\boxed{0<\sigma(z)<1}$$

so it never produces negative activation values.

🔴 4. Maximum Gradient Is Only 0.25

$$\boxed{\sigma'(z)\leq0.25}$$

This can contribute to shrinking gradients across multiple layers.

🔴 5. More Computationally Expensive Than ReLU

Sigmoid requires an exponential operation:

$$e^{-z}$$

whereas ReLU is simply:

$$\boxed{\operatorname{ReLU}(z)=\max(0,z)}$$

---

⚖️ 18. Sigmoid vs Linear Activation

Property	Linear	Sigmoid
Formula	$f(z)=z$	$\displaystyle f(z)=\frac{1}{1+e^{-z}}$
Shape	Straight line	S-shaped
Output range	$(-\infty,+\infty)$	$(0,1)$
Nonlinear	❌ No	✅ Yes
Binary classification	❌ Generally unsuitable	✅ Suitable
Regression output	✅ Common	❌ Generally unsuitable
Derivative	$1$	$\sigma(z)(1-\sigma(z))$
Saturation	❌ No	✅ Yes
Vanishing gradient	Not from sigmoid saturation	⚠️ Yes
Zero-centered	Yes for $f(z)=z$	❌ No

---

🏆 19. Summary

Students can remember sigmoid using the following 5 points:

🟢 1. Formula

$$\boxed{\sigma(z)=\frac{1}{1+e^{-z}}}$$

🟢 2. Range

$$\boxed{0<\sigma(z)<1}$$

🟢 3. Shape

$$\boxed{\text{S-shaped}}$$

🟢 4. Main Application

$$\boxed{\text{Binary Classification}}$$

🔴 5. Main Limitation

$$\boxed{\text{Vanishing Gradient}}$$

---

🎓

The sigmoid or logistic activation function is a nonlinear function that maps any real-valued input to a value between 0 and 1. It is widely used in the output layer of binary classification neural networks because its output can be interpreted as a probability-like value. Its derivative is $\sigma'(z)=\sigma(z)(1-\sigma(z))$, which makes it suitable for gradient-based learning. However, sigmoid suffers from saturation, is not zero-centered, and can cause vanishing gradients, making it generally unsuitable for the hidden layers of modern deep neural networks.