Classification-Evaluation Measures

 

📊 Classification Performance Measures

---

🧠 Problem Setup

Let:

• $D$ → Testing dataset containing $n$ data points
• Each point lies in a $d$-dimensional space
• $\{c_1, c_2, \dots, c_k\}$ → Set of $k$ class labels
• $M$ → Classification model (classifier)

---

🔍 For each data point $x_i\in \mathrm{D:}$

• $y_i$ → True class label
• $\hat{y}_i = M(x_i)$→ Predicted class label

---

🎯 Classification Accuracy and Its Limitations

---

✅ Classification Accuracy

🔹 Definition

Classification accuracy is the ratio of correctly predicted instances to the total number of instances.

---

📐 Formula

$\text{Accuracy} = \frac{\text{Number of Correct Predictions}}{\text{Total Number of Predictions}}$

📊 Percentage Form

$\text{Accuracy (\%)} = \left( \frac{\text{Correct Predictions}}{\text{Total Predictions}} \right) \times 100$

---

⚠️ Misclassification Rate (Error Rate)

Accuracy can be converted into error rate by taking its complement.

---

📐 Formula

$\text{Error Rate (\%)} = \left(1 - \frac{\text{Correct Predictions}}{\text{Total Predictions}} \right) \times 100$

---

🔍 Interpretation

• High accuracy → Model makes more correct predictions
• Low error rate → Model makes fewer mistakes

---

📌 Key Insight

$$\text{Error Rate} = 1 - \text{Accuracy}$$

---

🎯 Summary

Measure	Meaning
Accuracy	Proportion of correct predictions
Error Rate	Proportion of incorrect predictions

🧠 Why Evaluation Measures Matter

In classification, accuracy alone is often misleading, especially when:

• Classes are imbalanced
• Costs of errors differ
• We care about specific types of mistakes

👉 Evaluation measures help us understand how well a model performs in different aspects.

📊 Limitations of Classification Accuracy

Classification accuracy is a good starting point for evaluating a model, but it often fails in practical scenarios.

---

⚠️ Main Problem

The key issue with accuracy is that:

It hides important details needed to properly understand model performance.

---

🔍 Situations Where Accuracy Fails

---

1. Multi-Class Classification Problem

When your dataset has more than two classes:

• You might get an accuracy of 80%
• But this does not tell you:
  • Whether all classes are predicted equally well
  • Or if the model is ignoring some classes

👉 Example issue:

• Model performs well on 2 classes
• Performs poorly on 1 class
• Overall accuracy still looks high

---

2. Imbalanced Dataset Problem

When class distribution is uneven:

• Example:
  • 90% of data belongs to one class
  • 10% belongs to another

---

🔹 What happens?

• Model predicts only the majority class
• Achieves 90% accuracy

👉 But:

• Model is useless for detecting the minority class

---

🎯 Key Insight

High accuracy does not necessarily mean a good model.

---

🧠 Why Accuracy is Misleading

• Does not show class-wise performance
• Does not distinguish between types of errors
• Ignores data imbalance

---

📊 Solution: Confusion Matrix

To overcome these limitations, we use:

Confusion Matrix

---

✅ What it provides:

• Detailed breakdown of predictions:
  • True Positives (TP)
  • False Positives (FP)
  • False Negatives (FN)
  • True Negatives (TN)

---

🎯 Benefit

• Helps analyze:
  • Which classes are misclassified
  • Type of errors made by the model
  • Performance per class

📊Confusion Matrix (Foundation)

For binary classification, everything starts with the confusion matrix:

	Predicted Negative	Predicted Positive
Actual Negative	TN (True Negative)	FP (False Positive)
Actual Positives	FN (False Negative)	TP (True Positive)

---

🔍 Meaning

• TP → correctly predicted positives
• TN → correctly predicted negatives
• FP → false alarms
• FN → missed positives

---

🎯 Accuracy

📐 Formula

$$\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}$$

---

✅ Interpretation

• Overall correctness of the model

---

⚠️ Limitation

• Misleading for imbalanced datasets

👉 Example:

• 95% negative data → predicting all negative gives 95% accuracy

📊Error Rate

🔹 Definition

Error rate is the proportion of incorrect predictions made by a classification model.

It measures how often the model is wrong.

---

📐 Formula

$$\text{Error Rate}=\frac{\text{Number of wrong predictions}}{\text{Total number of predictions}}$$

Using confusion matrix:

$\text{Error Rate}=\frac{FP+FN}{TP+TN+FP+FN}$

---

🔍 Relation to Accuracy

$$\text{Accuracy}=\frac{TP+TN}{Total}$$

👉 Therefore:

$$\text{Error Rate}=1-\text{Accuracy}$$

---

🔍 Precision

📐 Formula

$$\text{Precision} = \frac{TP}{TP + FP}$$

---

✅ Interpretation

Of all predicted positives, how many are actually correct?

---

🎯 Use When:

• False positives are costly
• Example: spam detection

---

🔍 Recall (Sensitivity / True Positive Rate)

📐 Formula

$$\text{Recall} = \frac{TP}{TP + FN}$$

---

✅ Interpretation

Of all actual positives, how many did we detect?

---

🎯 Use When:

• Missing positives is costly
• Example: disease detection

---

⚖️Precision vs Recall Tradeoff

• Increasing precision → may reduce recall
• Increasing recall → may reduce precision

👉 Controlled via decision threshold

---

🎯F1 Score

📐 Formula

$$F1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$$

---

✅ Interpretation

• Harmonic mean of precision and recall
• Balances both metrics

---

🎯 Use When:

• Need balance between precision and recall
• Imbalanced datasets

---

📊  Specificity (True Negative Rate)

📐 Formula

$$\text{Specificity} = \frac{TN}{TN + FP}$$

---

✅ Interpretation

Ability to correctly identify negatives

🎓Intuitive Examples

---

Medical Diagnosis

• Focus: Recall
  👉 Missing a disease is dangerous

---

Spam Detection

• Focus: Precision
  👉 Avoid marking real emails as spam

---

📚  Significance in Machine Learning

Evaluation metrics help:

• Compare models objectively
• Tune hyperparameters
• Detect overfitting
• Align model with real-world goals

---

🎯 Key Takeaways

• No single metric is sufficient
• Choice depends on problem context
• Confusion matrix is fundamental
• Precision–Recall tradeoff is critical
• Use multiple metrics for robust evaluation