Regularization

 

🧠Why Do We Need Regularization?

In machine learning, especially with complex models:

• The model may fit training data extremely well
• But perform poorly on new data → overfitting

👉 Regularization is a strategy to control model complexity and improve generalization.

---

📘What is Regularization?

🔹 Definition

Regularization adds a penalty term to the learning objective to discourage overly complex models.

---

📐 Standard Learning Objective

Without regularization:

$$\text{Minimize: Loss Function}$$

With regularization:

$$\text{Minimize: Loss + Penalty}$$

---

🔧 General Form

$$\mathcal{L}(w) = \text{Error}(w) + \lambda \cdot \Omega(w)$$

Where:

• $w$ → model parameters
• $\lambda$ → regularization strength
• $\mathrm{\Omega }(w)$→ penalty function

---

🎯 Role of λ (Lambda)

• $\lambda =\mathrm{0\; \to \; no\; regularization}$
• Small $\lambda$ → slight penalty
• Large $\lambda$ → strong constraint

👉 Controls bias–variance tradeoff

---

⚖️ Effect of Regularization

Effect	Outcome
Reduces variance	Less overfitting
Increases bias	Slight underfitting possible
Shrinks coefficients	Simpler model

⚖️Regularization and Generalization

Regularization helps:

• Reduce variance
• Improve generalization
• Control model complexity

---

🔁 Bias–Variance Tradeoff

Effect	Outcome
Increase regularization	↑ Bias, ↓ Variance
Decrease regularization	↓ Bias, ↑ Variance

👉 Goal: optimal balance

---

📐 Mathematical Framework

🔹 Standard Objective

Without regularization:

$$\min_{w} \; \mathcal{L}(w)$$

---

🔹 With Regularization

$$\min_{w} \; \mathcal{L}(w) + \lambda \cdot \Omega(w)$$

Where:

• $\mathcal{L}(w)$→ loss function
• $\Omega(w)$→ regularization term
• $\mathrm{\lambda \; \to \; regularization\; strength}$

---

🎯 Interpretation of λ

λ Value	Effect
0	No regularization
Small	Mild constraint
Large	Strong constraint (may underfit)

---

🧩Types of Regularization

---

🔶L1 Regularization (LASSO)

📐 Penalty:

$$\sum |w_i|$$

🔹 Characteristics:

• Produces sparse models
• Some coefficients become exactly zero
• Performs feature selection

---

✅ Advantages:

• Interpretable models
• Useful for high-dimensional data

⚠️ Limitations:

• Unstable with correlated features

---

🔷 L2 Regularization (Ridge)

📐 Penalty:

$$\sum w_i^2$$

🔹 Characteristics:

• Shrinks coefficients smoothly
• No exact zeros

---

✅ Advantages:

• Stable
• Handles multicollinearity well

---

🟣  Elastic Net

📐 Combination:

$$\lambda_1 \sum |w_i| + \lambda_2 \sum w_i^2$$

---

🔹 Characteristics:

• Combines L1 + L2
• Balances sparsity and stability

---

✅ Use case:

• When features are correlated
• When LASSO alone is unstable

---

🌳  Regularization in Tree-Based Models

---

🔹 Techniques:

• Max depth restriction
• Minimum samples per leaf
• Pruning

---

🎯 Effect:

• Prevents overly complex trees
• Controls variance

---

🧠 Regularization in Neural Networks

---

🔹 1. Weight Decay

• Equivalent to L2 regularization

---

🔹 2. Dropout

• Randomly deactivate neurons during training

Effect:

• Prevents co-adaptation
• Improves generalization

---

🔹 3. Early Stopping

• Stop training when validation error increases

---

🔹 4. Batch Normalization (Indirect)

• Stabilizes training
• Acts as mild regularizer

---

🧪  Data-Based Regularization

---

🔹 Data Augmentation

• Generate new training samples
• Common in image/text tasks

---

🔹 Noise Injection

• Add noise to inputs or weights

---

📊 Geometric Interpretation

Method	Shape	Effect
L1	Diamond	Sparsity
L2	Circle	Smooth shrinkage

---

🔍 Practical Workflow

1. Split data (train/validation/test)
2. Normalize features
3. Choose regularization type
4. Tune λ using validation or cross-validation
5. Evaluate on test set

---

📚 Theoretical Insight (Advanced Note)

From a probabilistic perspective:

Regularization	Equivalent Prior
L1	Laplace prior
L2	Gaussian prior

👉 Connects to Bayesian learning (important from Bishop)

---

🎯 Role of Regularization in ML Systems

Regularization is critical for:

• High-dimensional datasets
• Deep learning models
• Noisy real-world data
• Preventing model instability

---

📊Summary Table

Method	Key Idea	Effect
L1 (LASSO)	Absolute penalty	Sparse models
L2 (Ridge)	Squared penalty	Smooth shrinkage
Elastic Net	Combination	Balanced approach
Dropout	Random neuron removal	Robust NN
Early stopping	Stop training early	Avoid overfit
Tree pruning	Limit tree growth	Reduce variance

---

🎯Key Takeaways for Students

• Regularization controls model complexity
• Essential for good generalization
• L1 → feature selection
• L2 → stability
• λ must be carefully tuned