Hinge Loss in SVM and Finding the Decision Boundary Using Gradient Descent

The basic idea is:

SVM finds the decision boundary by minimizing a loss function consisting of two parts:

Hinge loss → penalizes classification errors and margin violations.

Regularization term → encourages a large margin.

Let's understand this step by step.

1. Recall the SVM Decision Function

For a Linear SVM:

\boxed{f(x)=w^Tx+b}

For two features:

\boxed{f(x)=w_1x_1+w_2x_2+b}

The decision boundary is where:

\boxed{w^Tx+b=0}

The prediction is:

\boxed{\hat y=\operatorname{sign}(w^Tx+b)}

For SVM mathematics, we normally use:

\boxed{y\in\{-1,+1\}}

2. What is Hinge Loss?

Hinge loss measures whether a point is:

correctly classified and safely outside the margin,
correctly classified but inside the margin,
or incorrectly classified.

The hinge loss is:

\boxed{ L(y,f(x))=\max(0,\;1-yf(x)) }

Since:

f(x)=w^Tx+b

we get:

\boxed{ L=\max(0,\;1-y(w^Tx+b)) }

3. Understanding $y f(x)$

The important quantity is:

\boxed{y(w^Tx+b)}

Let:

m=y(w^Tx+b)

This is sometimes called the functional margin.

There are three important cases.

Case 1: $m\geq1$

y(w^Tx+b)\geq1

The point is:

correctly classified ✓
outside the margin ✓

Therefore:

L=\max(0,1-m)=0

No hinge loss.

Case 2: $0<m<1$

The point is:

correctly classified ✓
but inside the margin ✗

Therefore:

L>0

SVM applies a penalty.

Case 3: $m<0$

The point is:

incorrectly classified ✗

Therefore:

L>1

The penalty becomes even larger.

4. Simple Hinge Loss Table

$y f(x)$	Situation	Hinge Loss
$2$	Correct and outside margin	$0$
$1$	On margin	$0$
$0.5$	Correct but inside margin	$0.5$
$0$	On decision boundary	$1$
$-1$	Misclassified	$2$

5. Example of Calculating Hinge Loss

Suppose:

y=+1

and the SVM gives:

f(x)=0.6

Then:

y f(x)=(+1)(0.6)=0.6

The hinge loss is:

L=\max(0,1-0.6)

\boxed{L=0.4}

The point is correctly classified, but it lies inside the margin.

Another Example

Suppose:

y=-1

and:

f(x)=2

Then:

yf(x)=(-1)(2)=-2

Therefore:

L=\max(0,1-(-2))

\boxed{L=3}

This point is incorrectly classified.

6. Why is Hinge Loss Used?

Remember the hard-margin constraint:

\boxed{ y_i(w^Tx_i+b)\geq1 }

Hard-margin SVM requires every point to satisfy this condition.

But real data may contain noise.

Instead of enforcing this constraint strictly, hinge loss gives a penalty whenever:

y_i(w^Tx_i+b)<1

Thus, hinge loss provides a way to formulate SVM as an optimization problem.

7. The Complete SVM Loss Function

To train an SVM, we use two components.

Part 1: Regularization

\boxed{ \frac12\|w\|^2 }

This encourages a smaller $\|w\|$ .

Since the margin is:

\frac{2}{\|w\|}

minimizing $\|w\|$ means maximizing the margin.

Part 2: Hinge Loss

For $n$ training examples:

\boxed{ \sum_{i=1}^{n} \max(0,1-y_i(w^Tx_i+b)) }

8. Complete SVM Objective Function

A common form is:

\boxed{ J(w,b)= \frac12\|w\|^2+ C\sum_{i=1}^{n} \max(0,1-y_i(w^Tx_i+b)) }

where:

$\frac12\|w\|^2$ → maximizes the margin
hinge loss → penalizes violations
$C$ → controls the importance of classification errors

9. How Does Gradient Descent Find the Decision Boundary?

The main idea is simple.

Initially, we do not know:

w_1,\quad w_2,\quad b

So we start with some initial values.

For example:

w_1=0,\qquad w_2=0,\qquad b=0

Then repeatedly:

Calculate predictions.
Calculate hinge loss.
Calculate the gradient.
Update $w$ and $b$ .
Repeat.

Eventually, the values converge toward a good decision boundary.

10. Gradient Descent Rule

The general gradient descent rule is:

\boxed{ \text{New parameter} = \text{Old parameter} - \eta \frac{\partial J}{\partial \text{parameter}} }

where:

\eta

is the learning rate.

For SVM:

\boxed{ w:=w-\eta\frac{\partial J}{\partial w} }

and:

\boxed{ b:=b-\eta\frac{\partial J}{\partial b} }

11. Gradient of the SVM Objective

Recall:

J(w,b)= \frac12\|w\|^2+ C\sum_i \max(0,1-y_i(w^Tx_i+b))

We consider two cases.

Case 1: Point Correctly Classified Outside the Margin

If:

\boxed{ y_i(w^Tx_i+b)\geq1 }

then:

\text{hinge loss}=0

Only the regularization term contributes.

The gradient with respect to $w$ is:

\boxed{ \frac{\partial J}{\partial w}=w }

The gradient with respect to $b$ is:

\boxed{ \frac{\partial J}{\partial b}=0 }

Therefore, the update is:

\boxed{ w:=w-\eta w }

and:

\boxed{ b:=b }

Case 2: Point is Inside the Margin or Misclassified

If:

\boxed{ y_i(w^Tx_i+b)<1 }

then the hinge loss is active.

The gradient is:

\boxed{ \frac{\partial J}{\partial w} = w-Cy_ix_i }

and:

\boxed{ \frac{\partial J}{\partial b} = -Cy_i }

Therefore:

w:=w-\eta(w-Cy_ix_i)

and:

b:=b-\eta(-Cy_i)

Simplifying:

\boxed{ w:=w-\eta w+\eta C y_ix_i }

\boxed{ b:=b+\eta C y_i }

This update moves the decision boundary in a direction that improves classification.

12.Complete Algorithm for Linear SVM Using Gradient Descent

Step 1: Convert labels

$0\rightarrow-1,\qquad1\rightarrow+1$

Step 2: Initialize

$w=(0,0)$ $b=0$

Step 3: Repeat for several epochs

For each training example $(x_i,y_i)$ :

Calculate:

$m_i=y_i(w^Tx_i+b)$

If:

$m_i\geq1$

update:

$\boxed{ w:=w-\eta w }$

Otherwise:

$m_i<1$

update:

$\boxed{ w:=w-\eta(w-Cy_ix_i) }$

and:

$\boxed{ b:=b+\eta Cy_i }$

Step 4: Final Classifier

After training:

$\boxed{ f(x)=w^Tx+b }$

The decision boundary is:

\boxed{ w^Tx+b=0 }

Final Summary

Hinge Loss

\boxed{ L=\max(0,1-y(w^Tx+b)) }

It penalizes:

incorrectly classified points
correctly classified points inside the margin

SVM Objective

\boxed{ J(w,b)= \frac12\|w\|^2+ C\sum_i\max(0,1-y_i(w^Tx_i+b)) }

Gradient/Subgradient Updates

If:

y_i(w^Tx_i+b)\geq1

\boxed{ w:=w-\eta w }

If:

y_i(w^Tx_i+b)<1

\boxed{ w:=w-\eta(w-Cy_ix_i) }

\boxed{ b:=b+\eta Cy_i }

Final Decision Boundary

After learning

$w$ and $b$ :

w1x1+w2x2+b=0