Generalization - Overfitting Vs Underfitting - Bias Vs Variance

 

🧠 1. What is Generalization?

🔹 Definition

Generalization is the ability of a machine learning model to perform well on unseen data, not just the data it was trained on.

Formally:

A model generalizes well if its prediction error on new data is low.

---

📊 Training vs Generalization Error

• Training error → error on training data
• Generalization error (test error) → error on unseen data

👉 A good model:

• Low training error ✅
• Low generalization error ✅

---

🎯 Core Objective of Machine Learning

We are NOT trying to minimize training error alone.
We want to minimize:

$$\text{Expected Error on New Data}$$

This is often called:

• True risk
• Expected loss

---

⚠️ 2. What is Overfitting?

🔹 Definition

Overfitting occurs when a model learns:

• the underlying pattern ✅
• AND noise/random fluctuations ❌

---

🔍 Intuition

Imagine fitting a curve:

• Simple model → misses pattern → underfitting
• Moderate model → captures pattern → good generalization
• Very complex model → fits every point → overfitting

---

📉 Key Symptom

Metric	Behavior
Training error	Very low
Validation/Test error	High

---

📈 Conceptual Visualization

• Training error ↓ continuously
• Validation error:
  • ↓ initially
  • then ↑ (after a point)

👉 That turning point = start of overfitting

---

🧩Why Overfitting Happens

🔹 1. Model too complex

• Too many parameters
• High flexibility

Examples:

• Deep neural networks
• High-degree polynomial regression

---

🔹 2. Small dataset

• Not enough data to represent true distribution

---

🔹 3. Noise in data

• Model learns random variations

---

🔹 4. Too much training

• Especially in neural networks

---

🔹 5. Data leakage

• Model accidentally sees test information

🧪Detecting Overfitting

🔍 Using validation set:

• Training accuracy ↑
• Validation accuracy ↓

---

🔍 Learning curves:

Plot:

• Training error
• Validation error

Interpretation:

• Large gap → overfitting
• Both high → underfitting

🧠 3. What is Underfitting?

🔹 Definition

Underfitting occurs when a model is too simple to capture the underlying structure of the data.

The model fails to learn even the basic patterns.

---

📊  Key Characteristics

Metric	Behavior
Training error	High ❌
Validation/Test error	High ❌

👉 Unlike overfitting, the model performs poorly everywhere.

---

🔍 Intuition

Imagine trying to fit a straight line to data that clearly follows a curve:

• The model is too rigid
• It cannot adapt to the pattern
• Result → large errors

---

📉 Conceptual Visualization

Think in terms of model complexity:

Model Complexity	Behavior
Very low	Underfitting
Moderate	Good generalization
Very high	Overfitting

🧩  Causes of Underfitting

---

🔹 1. Model Too Simple

Examples:

• Linear model for nonlinear data
• Shallow decision tree

---

🔹 2. Insufficient Features

• Important variables missing
• Poor feature representation

---

🔹 3. Too Much Regularization

• Model is overly constrained

---

🔹 4. Inadequate Training

• Not trained long enough
• Optimization not converged

---

🔹 5. Poor Data Representation

• Lack of feature engineering
• Raw data not informative enough

---

🧪  Detecting Underfitting

---

🔍 Learning Curves

• Training error → high
• Validation error → high
• Gap between them → small

👉 Key insight:

Model cannot even fit training data

---

🔍 Performance Indicators

• Low accuracy on training data
• High loss values
• Predictions too simplistic

---

🛠️ How to Fix Underfitting

---

🔹 1. Increase Model Complexity

• Use more powerful models:
  • Polynomial regression instead of linear
  • Deeper neural networks

---

🔹 2. Add Better Features

• Feature engineering
• Domain knowledge

---

🔹 3. Reduce Regularization

• Lower penalty strength

---

🔹 4. Train Longer

• Increase epochs (for neural networks)

---

🔹 5. Use Nonlinear Models

• Kernel methods
• Tree-based models

---

🧠Intuitive Analogy (Teaching Friendly)

Student analogy:

• Underfitting = student didn’t study enough
• Cannot answer even basic questions

---

⚖️Bias–Variance Tradeoff

This is the theoretical backbone of generalization.

---

🔹 Bias

• Error due to overly simple assumptions
• High bias → underfitting

🔹 Variance

• Error due to sensitivity to training data
• High variance → overfitting

---

📊 Tradeoff

Model Type	Bias	Variance
Simple	High	Low
Complex	Low	High

---

🎯 Goal:

Find a balance:

Low bias + Low variance = Good generalization

 

---

📐Mathematical View 

The terms bias and variance come from statistics, and the names describe what type of error the model is making.

Bias: Why is it called "bias"?

Bias means a systematic tendency toward a particular direction. A model with high bias consistently makes similar mistakes because of strong assumptions.

Mathematically:

$$Bias = E[\hat{f}(x)] - f(x)$$

where:

• $f(x)$ = true function
• $\hat{f}(x)$ = predicted function
• $E[\hat{f}(x)]$ = average prediction over many training datasets

If the average prediction is consistently away from the true value, the model is biased.

Example

Suppose actual relationship:

$$y=x^2$$

but we force a linear model:

$$y=mx+b$$

No matter how many datasets we use, the model keeps trying to fit a straight line to a curve.

The error occurs because the model is biased toward linearity.

Hence it is called bias.

Think of a dartboard:

• All arrows land together but away from the center.
• Predictions are consistently shifted in one direction.

Center (true value)

    X X X
    X X X      ← tightly grouped
        O      ← true value

The model repeatedly misses the target in the same way.

---

Variance: Why is it called "variance"?

Variance means how much something changes or spreads out.

Mathematically:

$$Variance = E[(\hat{f}(x)-E[\hat{f}(x)])^2]$$

It measures how much predictions vary if we train the model on different datasets.

Example

Suppose we collect slightly different training samples:

Dataset 1 → model predicts 20
Dataset 2 → model predicts 35
Dataset 3 → model predicts 12
Dataset 4 → model predicts 40

Predictions change drastically.

The model is sensitive to small changes in data.

Hence it is called variance because predictions have large variability.

Dartboard analogy:

X       X

      O

   X      X

Predictions are scattered around.

---

Why both terms together?

The total prediction error can be decomposed as:

$$Error = Bias^2 + Variance + Irreducible\ Error$$

High bias:

• Model too simple
• Underfitting
• Consistent errors

High variance:

• Model too complex
• Overfitting
• Sensitive to training data

---

A simple way to remember:

• Bias = wrong assumptions → consistently wrong
• Variance = sensitivity to data → unstable predictions

The names come directly from their statistical meanings: bias = systematic deviation, variance = spread/variability.

---

🔬 Generalization in Practice

🔹 Empirical Risk Minimization (ERM)

Model minimizes:

$$\text{Training Loss}$$

BUT we actually care about:

$$\text{Expected Loss}$$

👉 Overfitting happens when:

$$\text{Training Loss} \ll \text{Test Loss}$$

---

🛠️Techniques to Improve Generalization

---

🔹 1. More Data

• Best solution (if possible)

---

🔹 2. Regularization

Adds penalty to model complexity:

Examples:

• L1 regularization (Lasso)
• L2 regularization (Ridge)

---

🔹 3. Early Stopping

• Stop training when validation error increases

---

🔹 4. Cross-Validation

• More reliable performance estimate

---

🔹 5. Simpler Model

• Reduce parameters

---

🔹 6. Dropout (Neural Networks)

• Randomly disable neurons during training

---

🔹 7. Data Augmentation

• Create more training examples

---

🧠 Intuitive Analogy (Good for Teaching)

Student analogy:

• Generalization → ability to solve new questions
• Overfitting → memorizing past exam answers

👉 A student who memorizes:

• Scores high on practice questions
• Fails new questions

---

📊 Model Complexity vs Performance

Complexity	Training Error	Test Error
Low	High	High
Medium	Low	Lowest
High	Very Low	High

---

🔁 Relation to Training / Validation / Test Sets

• Training set → learning
• Validation set → detect overfitting
• Test set → measure generalization

📊Underfitting vs Overfitting

Aspect	Underfitting	Overfitting
Model complexity	Too low	Too high
Training error	High	Very low
Test error	High	High
Generalization	Poor	Poor

---

🎯 Relationship to Generalization

• Underfitting → poor generalization because model is too simple
• Overfitting → poor generalization because model is too complex

👉 Both are undesirable
👉 Goal: balanced model

🎯 Key Takeaways 

• Underfitting = model too simple
• Cannot capture patterns
• High error everywhere
• Fixed by increasing model capacity or improving features
• Low training error ≠ good model
• Generalization is the real goal
• Overfitting = learning noise
• Test error will be high is the model overfits
• Validation set is essential
• Balance model complexity carefully