RBF Kernel in SVM

The RBF (Radial Basis Function) kernel is commonly used in SVM to handle non-linearly separable data.

Instead of using a straight-line decision boundary, the RBF kernel helps SVM create curved and complex decision boundaries.

---

1. RBF Kernel Formula

The RBF kernel is computed as:

$$\boxed{ K(x_i,x_j)=\exp\left(-\gamma\|x_i-x_j\|^2\right) }$$

It calculates the similarity between two data points.

---

2. Parameters and Terms in the Formula

Symbol	Meaning
$x_i$	First data point
$x_j$	Second data point
$\|x_i-x_j\|^2$	Squared Euclidean distance between the points
$\gamma$	Controls the influence range of a data point
$\exp()$	Exponential function $e^{(\cdot)}$
$K(x_i,x_j)$	Kernel similarity value

---

3. How Does the RBF Kernel Compute Similarity?

Let us take two points:

$$x_i=(1,2)$$

and

$$x_j=(3,5)$$

Suppose:

$$\gamma=0.1$$

Step 1: Calculate the Difference

$$x_i-x_j=(1-3,\;2-5)$$ $$=(-2,-3)$$

---

Step 2: Calculate the Squared Euclidean Distance

$$\|x_i-x_j\|^2$$ $$=(-2)^2+(-3)^2$$ $$=4+9$$ $$\boxed{13}$$

---

Step 3: Multiply by $-\gamma$

$$-\gamma\|x_i-x_j\|^2$$ $$=-0.1(13)$$ $$=-1.3$$

---

Step 4: Apply the Exponential Function

$$K(x_i,x_j)=e^{-1.3}$$ $$\boxed{K(x_i,x_j)\approx0.273}$$

Therefore, the similarity between these two points is approximately:

$$\boxed{0.273}$$

---

4. Interpretation of the Kernel Value

The RBF kernel value lies between:

$$\boxed{0<K(x_i,x_j)\leq1}$$

If two points are identical

$$x_i=x_j$$

Then:

$$\|x_i-x_j\|^2=0$$

Therefore:

$$K(x_i,x_j)=e^0$$ $$\boxed{K(x_i,x_j)=1}$$

So, identical points have maximum similarity.

---

If two points are far apart

The distance becomes large:

$$\|x_i-x_j\|^2\rightarrow\text{large}$$

Therefore:

$$e^{-\gamma\|x_i-x_j\|^2}\rightarrow0$$

So:

$$\boxed{\text{Far points have similarity close to }0}$$

---

5. The Role of $\gamma$

The parameter:

$$\boxed{\gamma}$$

controls how quickly similarity decreases as distance increases.

Small $\gamma$

Suppose:

$$\gamma=0.01$$

The similarity decreases slowly.

Therefore, even relatively distant points can influence each other.

Result:

$$\boxed{\text{Smoother and simpler decision boundary}}$$

---

Large $\gamma$

Suppose:

$$\gamma=1$$

The similarity decreases quickly.

Only nearby points strongly influence each other.

Result:

$$\boxed{\text{More complex decision boundary}}$$

---

6. Example Showing the Effect of $\gamma$

Suppose the squared distance between two points is:

$$\|x_i-x_j\|^2=4$$

Case 1: Small $\gamma=0.1$

$$K=e^{-0.1(4)}$$ $$=e^{-0.4}$$ $$\boxed{K\approx0.670}$$

---

Case 2: Large $\gamma=1$

$$K=e^{-1(4)}$$ $$=e^{-4}$$ $$\boxed{K\approx0.018}$$

Comparison

$\gamma$	Kernel Value	Meaning
0.1	0.670	Points still have considerable similarity
1	0.018	Points have very little similarity

Thus:

$$\boxed{\text{Large }\gamma\Rightarrow\text{similarity decreases faster}}$$

---

7. Another Important Parameter: $C$

In an SVM using an RBF kernel, another important parameter is:

$$\boxed{C}$$

The parameter $C$ controls the penalty for classification errors.

Small $C$

The model allows more classification errors.

It generally produces a smoother decision boundary.

Large $C$

The model strongly tries to classify all training points correctly.

It may produce a more complex boundary and can lead to overfitting.

---

8. Difference Between $C$ and $\gamma$

This is an important distinction for students.

Parameter	Controls
$C$	Penalty for classification errors
$\gamma$	Influence range of each data point

Easy way to remember:

$$\boxed{ C\rightarrow\text{Cost of errors} }$$ $$\boxed{ \gamma\rightarrow\text{Geographical range of influence} }$$

---

9. How RBF Kernel is Used in SVM

During SVM training, the kernel function is computed between training points.

Conceptually:

$$K(x_1,x_2)$$ $$K(x_1,x_3)$$ $$K(x_2,x_3)$$

and so on.

These values form a kernel matrix.

For example:

$$K= \begin{bmatrix} K(x_1,x_1) & K(x_1,x_2) & K(x_1,x_3)\\ K(x_2,x_1) & K(x_2,x_2) & K(x_2,x_3)\\ K(x_3,x_1) & K(x_3,x_2) & K(x_3,x_3) \end{bmatrix}$$

The SVM uses these similarity values to find the support vectors and construct a non-linear decision boundary.

---

10. Simple Computational Flow

You can explain the process to students as:

Two data points
      ↓
Calculate distance between them
      ↓
Square the distance
      ↓
Multiply by -γ
      ↓
Apply exponential function
      ↓
Get similarity value K(xi, xj)
      ↓
SVM uses similarities to find
a non-linear decision boundary

---

⭐ Summary

The RBF kernel calculates the similarity between two data points using their distance. Nearby points produce a kernel value close to 1, while distant points produce a value close to 0. The parameter γ controls how quickly this similarity decreases, and C controls how strongly the SVM penalizes classification errors.

The RBF kernel is computed as:

$$\boxed{{\displaystyle K(x_i,x_j)=\exp (-\gamma \mathrm{\parallel }{x}_i-x_j{\mathrm{\parallel }}^2)}}$$

​​​