Linear SVM with Soft Margin 

1. Why Do We Need Soft-Margin SVM?

In Hard-Margin SVM, we assume that the two classes can be perfectly separated.

For example:

Class 0       | Margin |       Class 1

  ●  ●        |        |        ○  ○
  ●           |        |           ○

Every point must be on the correct side of the decision boundary.

But in real-world datasets, we may have:

• noisy data,
• overlapping classes,
• outliers,
• wrongly labeled data.

For example:

                    ○ Class 1
       ● ● ●      ●
                 ↑
              Outlier

-------------------------------
      Decision Boundary

                 ○ ○ ○
              Class 1

A Hard-Margin SVM may not find a solution when the data cannot be perfectly separated.

👉 Therefore, we use Soft-Margin SVM.

---

2. Main Idea of Soft-Margin SVM

Soft-Margin SVM says:

It is acceptable for some points to be inside the margin or even on the wrong side of the decision boundary.

But such violations receive a penalty.

So, Soft-Margin SVM tries to balance two objectives:

Objective 1: Find a large margin

Objective 2: Reduce classification errors

---

3. A Simple Example

Consider the following data:

$X_1$	$X_2$	Class
1	2	0
2	3	0
3	3	0
4	4	1
5	5	1
3	5	1
4	2	0

Suppose the data looks approximately like this:

X₂
↑

6 |

5 |             ○       ○

4 |                  ○

3 |      ●     ●

2 |   ●                 ●

  +------------------------------→ X₁

    ● = Class 0
    ○ = Class 1

Sometimes, one or two points may be close to the other class.

Instead of requiring:

$$\text{Every point must be perfectly separated}$$

Soft-Margin SVM allows some violations.

---

4. Understanding the Three Important Regions

Suppose we have:

Class 0        Margin       Margin       Class 1

   ● ●             |             |          ○ ○
                   |             |
                   | Decision    |
                   | Boundary    |

Mathematically:

Negative margin boundary

$$w^Tx+b=-1$$

Decision boundary

$$\boxed{w^Tx+b=0}$$

Positive margin boundary

$$w^Tx+b=+1$$

---

Case 1: Correctly Classified and Outside Margin

For a Class $+1$ point:

$$y(w^Tx+b)>1$$

Example:

$$y=+1,\qquad f(x)=2$$

Then:

$$y f(x)=2$$

This point is:

✅ Correctly classified
✅ Outside the margin
✅ No penalty

---

Case 2: Correctly Classified but Inside the Margin

Suppose:

$$y=+1$$

and:

$$f(x)=0.5$$

Then:

$$yf(x)=0.5$$

The point is:

✅ Correctly classified

But:

❌ Inside the margin

Therefore, Soft-Margin SVM gives a penalty.

---

Case 3: Incorrectly Classified

Suppose:

$$y=-1$$

but:

$$f(x)=2$$

Then:

$$yf(x)=(-1)(2)=-2$$

This means the point is:

❌ Incorrectly classified.

A larger penalty is given.

---

5. Introducing the Slack Variable $\xi$

Soft-Margin SVM introduces a new variable called the:

$$\boxed{\text{Slack Variable } \xi_i}$$

The Hard-Margin constraint was:

$$\boxed{ y_i(w^Tx_i+b)\geq1 }$$

The Soft-Margin constraint becomes:

$$\boxed{ y_i(w^Tx_i+b)\geq1-\xi_i }$$

where:

$$\boxed{\xi_i\geq0}$$

The value of $\xi_i$ tells us how much the point violates the margin.

---

6. Meaning of Different Values of $\xi$

Slack Variable	Meaning
$\xi_i=0$	Correctly classified and outside/on margin
$0<\xi_i<1$	Correctly classified but inside margin
$\xi_i=1$	On the decision boundary
$\xi_i>1$	Incorrectly classified

This is a very useful table for students.

---

7. Mathematical Formulation of Soft-Margin SVM

The optimization problem is:

$$\boxed{ \text{Minimize } \frac{1}{2}\|w\|^2+ C\sum_{i=1}^{n}\xi_i }$$

Subject to:

$$\boxed{ y_i(w^Tx_i+b)\geq1-\xi_i }$$

and:

$$\boxed{ \xi_i\geq0 }$$

---

8. Understanding the Objective Function

The objective is:

$$\boxed{ \frac{1}{2}\|w\|^2+ C\sum\xi_i }$$

It has two parts.

Part 1

$$\frac12\|w\|^2$$

This tries to:

$$\boxed{\text{maximize the margin}}$$

---

Part 2

$$C\sum\xi_i$$

This tries to:

$$\boxed{\text{reduce margin violations}}$$

Therefore, Soft-Margin SVM balances:

$$\boxed{ \text{Large Margin} \quad\text{vs}\quad \text{Few Errors} }$$

---

9. What is the Role of $C$?

The parameter:

$$\boxed{C}$$

controls how strongly we penalize errors.

---

Large $C$

If:

$$C=1000$$

errors are heavily penalized.

The SVM tries very hard to classify every training point correctly.

Large C

"Don't make mistakes!"

This may result in a smaller margin.

---

Small $C$

If:

$$C=0.1$$

the SVM allows more violations.

Small C

"A few mistakes are acceptable."

This can result in a wider margin.

---

Simple Comparison

Value of $C$	Effect
Large $C$	Fewer training errors, smaller margin
Small $C$	More violations allowed, wider margin

---

10. Simple Numerical Example Using Slack Variable

Suppose we have a point:

$$x=(2,3)$$

with class:

$$y=+1$$

Suppose the SVM decision function gives:

$$w^Tx+b=0.6$$

Then:

$$y(w^Tx+b)$$ $$=(+1)(0.6)$$ $$=0.6$$

The Soft-Margin condition is:

$$y(w^Tx+b)\geq1-\xi$$

Substituting:

$$0.6\geq1-\xi$$

Therefore:

$$\xi\geq0.4$$

So:

$$\boxed{\xi=0.4}$$

This means:

The point is correctly classified, but it violates the margin by $0.4$.

---

11. Example of a Misclassified Point

Suppose:

$$y=-1$$

but:

$$w^Tx+b=0.5$$

Then:

$$y(w^Tx+b)$$ $$=(-1)(0.5)$$ $$=-0.5$$

The constraint is:

$$-0.5\geq1-\xi$$

Therefore:

$$\xi\geq1.5$$

Thus:

$$\boxed{\xi=1.5}$$

Since:

$$\xi>1$$

the point is misclassified.

---

12. Soft Margin and Hinge Loss

Instead of explicitly calculating slack variables, we can use hinge loss.

The hinge loss is:

$$\boxed{ L_i=\max(0,1-y_i(w^Tx_i+b)) }$$

Notice that this is equivalent to the amount of margin violation.

Therefore:

$$\boxed{ \xi_i=\max(0,1-y_i(w^Tx_i+b)) }$$

So the Soft-Margin SVM objective can also be written as:

$$\boxed{ J(w,b) = \frac12\|w\|^2 + C\sum_{i=1}^{n} \max(0,1-y_i(w^Tx_i+b)) }$$

This is the formulation commonly used when implementing SVM using gradient descent.

---

13. Hard Margin vs Soft Margin

Hard-Margin SVM	Soft-Margin SVM
Requires perfectly separable data	Allows some violations
No classification errors allowed	Some errors are allowed
Sensitive to outliers	More robust to outliers
Uses strict constraints	Uses slack variables
Best for ideal datasets	Better for real-world datasets

---

Hard-Margin SVM does not allow violations. Soft-Margin SVM allows violations but penalizes them.

---

Final Summary

Hard Margin

$$\boxed{ y_i(w^Tx_i+b)\geq1 }$$

No violations allowed.

---

Soft Margin

$$\boxed{ y_i(w^Tx_i+b)\geq1-\xi_i }$$

Violations are allowed.

---

Objective Function

$$\boxed{ \min \frac12\|w\|^2+ C\sum_i\xi_i }$$

---

Equivalent Hinge Loss Form

$$\boxed{ \min \frac12\|w\|^2+ C\sum_i \max(0,1-y_i(w^Tx_i+b)) }$$

---

Soft-Margin SVM tries to find the widest possible separating boundary while allowing some points to violate the margin or even be misclassified. The parameter C controls the trade-off between a large margin and classification errors.