LASSO and Ridge Regression

🧠Regularization

Regularization modifies the learning objective to:

$$\text{Minimize: Loss + Penalty}$$

👉 Purpose:

• Control model complexity
• Prevent overfitting
• Improve generalization

---

🔷 Ridge Regression (L2 Regularization)

---

📐 Objective Function

$\min_{w} \left( \sum_{i=1}^{n} (y_i - w^T x_i)^2 + \lambda \sum_{j=1}^{d} w_j^2 \right)$

---

🔍 Key Idea

• Penalizes square of coefficients
• Larger weights → heavily penalized

---

🎯 Effects

• Shrinks coefficients toward zero
• No coefficient becomes exactly zero
• Keeps all features

---

🧩 Properties

• Produces smooth models
• Handles multicollinearity well
• Reduces model variance

---

📊 Intuition

Think of it as:

“Discouraging large weights, but not eliminating features”

---

🔶 LASSO Regression (L1 Regularization)

Least Absolute Shrinkage and Selection Operator

---

📐 Objective Function

${\min }_w({\sum }_{i=1}^n(y_i-w^T{x}_i{)}^2+\lambda {\sum }_{j=1}^d\mathrm{\mid }{w}_j\mathrm{\mid })$

---

🔍 Key Idea

• Penalizes absolute value of coefficients

---

🎯 Effects

• Some coefficients become exactly zero
• Performs automatic feature selection

---

✂️ Sparsity

👉 Key property:

• Produces sparse models
• Removes irrelevant features

---

📊 Intuition

Think of it as:

“Forcing the model to keep only the most important features”

---

⚖️ Core Differences (Geometric Insight)

---

Aspect	Ridge (L2)	LASSO (L1)
Constraint shape	Circle	Diamond
Boundary	Smooth	Sharp corners
Result	Shrink weights	Zero out weights

👉 Sharp corners in LASSO → sparsity

---

📊 Behavior Comparison

---

🔹 Coefficient Shrinkage

Feature	Ridge	LASSO
x₁	2.2	2.1
x₂	1.6	1.5
x₃	0.8	0
x₄	0.6	0

---

🔹 Key Observations

• Ridge → all features retained
• LASSO → some features removed

---

⚖️Detailed Comparison Table

Criterion	Ridge (L2)	LASSO (L1)
Penalty	$w^2$|	|w|
Sparsity	No	Yes
Feature selection	No	Yes
Stability	High	Lower
Correlated features	Shares weights	Picks one feature
Computation	Closed form	Iterative

---

🧠Bias–Variance Perspective

Model	Bias	Variance
Ridge	Moderate	Low
LASSO	Higher	Low

---

🔍 Effect of Regularization Parameter (λ)

---

🔹 Small λ

• Both behave like ordinary regression

🔹 Medium λ

• Ridge → shrink coefficients
• LASSO → shrink + remove some

🔹 Large λ

• Ridge → all weights very small
• LASSO → most weights = 0

---

🧪 Practical Considerations

---

🔹 Feature Scaling (Important!)

• Required for both methods

---

🔹 Model Selection

• Choose λ using:
  • Validation set
  • Cross-validation

---

🧠Intuitive Analogies

---

Ridge:

“Keep all features, but reduce their influence”

---

LASSO:

“Select only the most important features”

---

🟣When to Use Which?

---

🔷 Use Ridge when:

• Many correlated features
• All features are important
• Need stable model

---

🔶 Use LASSO when:

• Many irrelevant features
• Need feature selection
• Want interpretable model

---

🔗 Connection: Elastic Net

Combines both:

$$\lambda_1 \sum |w| + \lambda_2 \sum w^2$$

👉 Useful when:

• Features are correlated
• Need both sparsity and stability

---

🎯Key Takeaways

• Both methods reduce overfitting
• Ridge → smooth shrinkage
• LASSO → sparse solutions

Use Case	Preferred Method
Many correlated features	Ridge
Feature selection needed	LASSO
Balanced approach	Elastic Net