Simple Linear Regression using Gradient Descent

 

Simple Linear Regression using Gradient Descent

---

🔹 1. Model (Hypothesis)

In simple linear regression:

$$\hat{y}_i = \beta_0 + \beta_1 x_i$$

Where:

• $\beta_0$ → intercept
• $\beta_1$ → slope

---

🔹 2. Cost Function (MSE)

We minimize the Mean Squared Error (MSE):

$$J(\beta_0, \beta_1) = \frac{1}{2n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$$

👉 The factor $\frac{1}{2}$ simplifies derivatives.

---

🔹 3. Goal

👉 Find $\beta_0$ and $\beta_1$ that minimize $J$

---

🔹 4. Compute Gradients (Very Important)

We take partial derivatives:

---

🔸 Derivative w.r.t $\beta_0$

$$\frac{\partial J}{\partial \beta_0} = -\frac{1}{n} \sum (y_i - \hat{y}_i)$$

---

🔸 Derivative w.r.t $\beta_1$

$$\frac{\partial J}{\partial \beta_1} = -\frac{1}{n} \sum x_i (y_i - \hat{y}_i)$$

---

🔹 5. Update Rules

Using Gradient Descent:

---

🔸 Update $\beta_0$

$$\beta_0 = \beta_0 - \alpha \left(-\frac{1}{n} \sum (y_i - \hat{y}_i)\right)$$
$$= \beta_0 + \frac{\alpha}{n} \sum (y_i - \hat{y}_i)$$

---

🔸 Update $\beta_1$

$$\beta_1 = \beta_1 - \alpha \left(-\frac{1}{n} \sum x_i (y_i - \hat{y}_i)\right)$$
$$= \beta_1 + \frac{\alpha}{n} \sum x_i (y_i - \hat{y}_i)$$

---

🔹 6. Algorithm Steps

📌 Gradient Descent Procedure

1. Initialize $\beta_0, \beta_1$ (0 or small random values)

2. Compute predictions:

   $$\hat{y}_i = \beta_0 + \beta_1 x_i$$

3. Compute error:

   $$e_i = y_i - \hat{y}_i$$
4. Compute gradients
5. Update parameters
6. Repeat until convergence

---

🔹 7. Intuition

• If prediction is too small → increase parameters
• If prediction is too large → decrease parameters

👉 Gradually moves toward best-fit line

---

🔹 8. Why Use Gradient Descent?

• Works for large datasets
• Avoids matrix inversion
• Scales well

---

🔹 9. Convergence

Stops when:

• Cost function stops decreasing
• Maximum iterations reached

---

🔹 10. Important Notes

• Learning rate $\alpha$ is critical
• Feature scaling improves performance
• Works even when closed-form solution is expensive

---

🔹 11. Summary

Step	Description
Model	$\hat{y} = \beta_0 + \beta_1 x$
Cost	MSE
Method	Iterative optimization
Update	Move opposite gradient