Gradient Descent (GD) 

---

🔹 Definition

Gradient Descent is an iterative optimization algorithm used to find the minimum value of a function.

👉 In Machine Learning, it is used to:

• Minimize the loss (cost) function
• Find optimal model parameters (coefficients)

---

🔹 Key Idea

👉 Move step-by-step in the direction where the function decreases fastest.

• Gradient = slope of the function
• We move in the opposite direction of the gradient

---

🔹 3. Mathematical Update Rule

$$\theta_{new} = \theta_{old} - \alpha \cdot \nabla J(\theta)$$

Where:

• $\theta$ → parameter (coefficient)
• $\alpha$ → learning rate
• $\nabla J(\theta)$ → gradient of cost function

---

🔹Steps of Gradient Descent

1. Initialize parameters (random or zero)
2. Compute cost function
3. Calculate gradient (derivative)

4. Update parameters:

   $$\theta = \theta - \alpha \cdot \text{gradient}$$
   
5. Repeat until minimum is reached

Image credit: towards data science

---

🔹 Conditions for Gradient Descent

Function should be:

• ✅ Differentiable → derivative must exist
• ✅ Convex → ensures global minimum

---

🔹 Learning Rate (α)

👉 Controls step size

Learning Rate	Effect
Too small	Slow convergence
Too large	May diverge
Optimal	Fast convergence

---

🔹Cost Function (Example: Linear Regression)

$$J(\theta) = \sum (y - \hat{y})^2$$

👉 GD minimizes this error

---

🔹 Intuition

• Think of rolling a ball downhill
• It moves step-by-step to reach the lowest point

---

🔹 When to Use

• Large datasets
• When closed-form solution (normal equation) is expensive
• Used in:
  • Linear regression
  • Neural networks
  • Deep learning

---

🔹 Advantages

• Simple and easy to implement
• Works well for large data

---

🔹 Limitations

• Can get stuck in local minima (non-convex case)
• Sensitive to learning rate
• Requires tuning

---

🔹Summary Formula

$$p_{n+1} = p_n - \alpha \nabla f(p_n)$$

---

📝 Quick Revision

• Gradient = slope
• Move opposite direction
• Repeat until minimum
• Learning rate is critical

learn More https://aidsktu.blogspot.com/2025/06/gradient-descent.html