Logistic Regression

Logistic Regression is a simple yet powerful supervised machine learning algorithm mainly used for binary classification problems.

It predicts the probability of an outcome (such as success/failure or yes/no) based on input features.

Instead of giving a direct class, it outputs a probability (0 to 1), which is then used to assign a class.

It is named ‘Logistic Regression’ because its underlying technique is quite the same as Linear Regression. The term “Logistic” is taken from the Logit function that is used in this method of classification.

The logistic function, also called the sigmoid function was developed by statisticians to describe properties of population growth in ecology, rising quickly and maxing out at the carrying capacity of the environment. It’s an S-shaped curve that can take any real-valued number and map it into a value between 0 and 1, but never exactly at those limits.

🔢 Sigmoid (Logistic) Function

This is the heart of Logistic Regression:

$\sigma (x)=\frac{1}{1+e^{-x}}$

🔹 What it does:

• Takes any real number → converts it into a value between 0 and 1
• Produces an S-shaped curve

🔹 Interpretation:

• Output ≈ 0 → Class 0 (Negative)
• Output ≈ 1 → Class 1 (Positive)

📈 The Core Idea of Logistic Regression

Logistic Regression:

1. First applies a linear model
2. Then passes the result through a sigmoid (logistic) function

---

🧮 Mathematical Model ( Binary )

Step 1: Linear Combination

$t = \beta_0 + \beta_1 x$

• $\beta_0$= intercept
• $\beta_1$ = coefficient
• $x$ = input feature

---

Step 2: Apply Sigmoid Function

$\sigma (x)=\frac{1}{1+e^{-({\beta }_0+{\beta }_{1}x)}}$

👉 This gives a probability value

---

🎯 Decision Boundary

• A threshold (usually 0.5) is used:
  • If probability ≥ 0.5 → Class 1
  • If probability < 0.5 → Class 0

📌 Example:

• 0.8 → “Yes”
• 0.3 → “No”

---

⚖️ Why Logistic Regression Handles Outliers Better

• Linear regression tries to fit a straight line → heavily affected by outliers
• Logistic regression:
  • Uses sigmoid curve
  • Output is bounded (0–1)
  • Extreme values get compressed

👉 So outliers don’t distort predictions as much

🧠 How Training Works in Logistic Regression

Training a logistic regression model means:

Finding the best parameters (weights) so that predicted probabilities match the actual class labels as closely as possible.

---

🔄 Step-by-Step Training Process

1. Start with the Model

We compute a linear combination of inputs:

$$z = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \dots + \beta_n x_n$$

Then apply the sigmoid function:

$\hat{y} = \frac{1}{1 + e^{-z}}$

• $\hat{y}$= predicted probability
• Output is always between 0 and 1

---

❗ Why Not Use Mean Squared Error (MSE)?

In linear regression, we use MSE. But in logistic regression:

• The sigmoid function makes the model non-linear
• MSE leads to a non-convex loss function
• This makes optimization difficult (multiple local minima)

👉 So instead, we use Log Loss (Cross-Entropy Loss)

---

📉 Cost Function in Logistic Regression

🔹 Binary Cross-Entropy (Log Loss)

$J(\theta) = -\frac{1}{m} \sum_{i=1}^{m} \left[ y^{(i)} \log(\hat{y}^{(i)}) + (1 - y^{(i)}) \log(1 - \hat{y}^{(i)}) \right]$

---

🔍 Understanding the Formula

Terms:

• $m$ = number of training examples
• $y$ = actual label (0 or 1)
• $\hat{y}$ = predicted probability

---

📊 Intuition Behind Log Loss

Case 1: When $y = 1$

$$\text{Loss} = -\log(\hat{y})$$

• If $\hat{y} = 1$ → loss = 0 ✅ (perfect)
• If $\hat{y}$ is small → loss becomes very large ❌

---

Case 2: When $y = 0$

$$\text{Loss} = -\log(1 - \hat{y})$$

• If $\widehat{y}=\mathrm{0\; \to \; loss\; =\; 0\; ✅}$
• If $\hat{y}$ is close to 1 → large penalty ❌

---

💡 Key Insight

Log Loss:

• Heavily penalizes wrong confident predictions
• Encourages the model to output accurate probabilities

---

⚙️ How the Model Learns (Optimization)

Goal:

Minimize the cost function $J(\theta )$

Method:

👉 Gradient Descent

---

🔁 Gradient Descent Update Rule

$$\theta = \theta - \alpha \cdot \nabla J(\theta)$$

• $\theta$ = model parameters
• $\alpha$ = learning rate
• $\nabla J(\theta)$ = gradient (direction of steepest increase)

👉 We move in the opposite direction to reduce error

---

🔄 Training Loop

1. Initialize weights randomly
2. Compute predictions using sigmoid
3. Calculate log loss
4. Compute gradients
5. Update weights
6. Repeat until convergence

📊 Real-Life Examples

• 📧 Spam detection → Spam / Not Spam
• 🏥 Tumor detection → Malignant / Benign
• 💳 Fraud detection → Fraud / Genuine

📉 Advantages

✔ Simple and easy to implement
✔ Works well for linearly separable data
✔ Outputs probabilities (useful for decision-making)
✔ Fast and efficient

---

⚠️ Limitations

❌ Cannot capture complex non-linear relationships
❌ Assumes linear relationship between input and log-odds
❌ Sensitive to irrelevant features

🔁 Summary

Input Features → Linear Equation → Sigmoid Function → Probability → Class Label

---

 💡Logistic Regression is not actually “regression” in the traditional sense—it’s a classification algorithm that uses a regression-like equation + sigmoid transformation.

⚖️ Key Differences

Feature	Linear Regression	Sigmoid (Logistic)
Shape	Straight line	S-shaped curve
Output Range	−∞ to +∞	0 to 1
Use Case	Regression	Classification
Handles Probability	❌ No	✅ Yes

Appendix :Deriving the gradient descent update rule

🔹 Cost Function (Log Loss)

$J(\theta) = -\frac{1}{m} \sum_{i=1}^{m} \left[y^{(i)} \log(h_\theta(x^{(i)})) + (1 - y^{(i)}) \log(1 - h_\theta(x^{(i)}))\right]$

• Penalizes wrong predictions heavily
• Ensures convex optimization

---

🔹 Gradient Descent Update Rule

$\theta = \theta - \alpha \cdot \frac{1}{m} X^T (h_\theta(x) - y)$

Where:

• α = learning rate
• m= number of samples

Look at ONE training example

Instead of the full sum, take just one data point:

$$J = -\left[y \log(h) + (1-y)\log(1-h)\right]$$

where:

• $h = \sigma(z)$
  
• $z={\theta }^{T}x$

---

Think of it like layers (very important)

We don’t differentiate everything at once.

We go step by step:

θ → z → h → J

So we use chain rule:

$$\frac{dJ}{d\theta} = \frac{dJ}{dh} \cdot \frac{dh}{dz} \cdot \frac{dz}{d\theta}$$

---

Compute each part (simple pieces)

✅ 1. Derivative of cost w.r.t h

$$\frac{dJ}{dh} = \frac{h - y}{h(1-h)}$$

---

✅ 2. Derivative of sigmoid

$$\frac{dh}{dz} = h(1-h)$$

---

✅ 3. Derivative of z

$$\frac{dz}{d\theta} = x$$

 Multiply them

Now multiply all three:

$$\frac{dJ}{d\theta} = \left(\frac{h - y}{h(1-h)}\right) \times (h(1-h)) \times x$$

---

Simplification

The $h(1-h)$ cancels:

$$\frac{dJ}{d\theta} = (h - y) \cdot x$$

---

For all data points

Average over all samples:

$$\frac{\partial J}{\partial \theta} = \frac{1}{m} X^T (h - y)$$

---

Final Gradient Descent Rule

$\theta =\theta -\alpha \cdot \frac{1}{m}{X}^T(h-y)$