Multilayer Perceptron (MLP) — A Universal Approximator

 

🧠 Multilayer Perceptron (MLP) 

A Multilayer Perceptron (MLP) is a feedforward neural network consisting of an input layer, one or more hidden layers, and an output layer. Unlike a single perceptron, an MLP can learn nonlinear relationships because the hidden neurons use nonlinear activation functions such as sigmoid, tanh, or ReLU. The input is propagated forward through the network, with each neuron computing a weighted sum and applying an activation function. The hidden layers transform the original input into a new feature space, allowing the output layer to implement complex decision boundaries or nonlinear functions.

For a hidden neuron:

$$\boxed{ z_h=\sigma\left(\sum_{j=1}^{d}w_{hj}x_j+w_{h0}\right) }$$

The output layer then combines the hidden-unit outputs:

$$\boxed{ y_i=\sum_{h=1}^{H}v_{ih}z_h+v_{i0} }$$

For regression, the output layer is usually linear. For binary classification, a sigmoid output is commonly used, while multiclass classification generally uses softmax.

Types of MLPs: Shallow vs Deep

Shallow MLPs: With a single hidden layer, these handle moderately complex tasks.
Deep MLPs: Multiple hidden layers make them ideal for high-dimensional, intricate datasets.

Key idea

$$\boxed{ \text{Input} \rightarrow \text{Nonlinear Hidden Representation} \rightarrow \text{Output} }$$

Thus, the main strength of an MLP is its ability to approximate nonlinear functions and implement nonlinear decision boundaries, which a single perceptron cannot do. Training is generally performed using gradient-based methods and backpropagation.

🧠 MLP as a Universal Approximator

The idea of universal approximation is one of the most important theoretical properties of a Multilayer Perceptron (MLP).

The Universal Approximation Theorem states, roughly, that:

An MLP with at least one hidden layer, a suitable nonlinear activation function, and sufficiently many hidden neurons can approximate a very broad class of continuous functions to arbitrary accuracy on a bounded input region.

This does not mean that the MLP automatically learns the function perfectly. It means that there exists a set of weights that can make the network's output as close as desired to the target function.

---

Why can't a single perceptron do this?

A single perceptron computes essentially:

$$y=f(\mathbf{w}^T\mathbf{x}+b)$$

and produces a relatively simple decision boundary.

For example, a perceptron can implement:

$$\text{AND},\quad \text{OR},\quad \text{NAND}$$

but cannot implement nonlinear functions such as XOR with a single neuron.

The reason is that a single perceptron has only one linear decision boundary.

---

What changes when we add a hidden layer?

Consider:

$$\boxed{ \text{Input} \rightarrow \text{Hidden Layer} \rightarrow \text{Output} }$$

Each hidden neuron applies a nonlinear activation function.

For example:

$$z_h=\sigma(\mathbf{w}_h^T\mathbf{x}+b_h)$$

The output neuron then combines these nonlinear features:

$$\boxed{ y=\sum_{h=1}^{H}v_hz_h+b }$$

Therefore:

$$\boxed{ y= \sum_{h=1}^{H} v_h \sigma(\mathbf{w}_h^T\mathbf{x}+b_h) +b }$$

This is a linear combination of nonlinear basis functions.

That is the key idea behind universal approximation.

---

Hidden neurons create useful regions

A hidden neuron can be thought of as detecting a particular region or pattern in the input space.

For example, consider two inputs:

$$x_1,\quad x_2$$

A hidden neuron can create a boundary such as:

$$w_1x_1+w_2x_2+b=0$$

This divides the input space into two regions.

With many hidden neurons, we can create many such boundaries:

                 x₂
                  ↑
        ┌─────────┼─────────┐
        │    Region 1       │
        │                   │
        ├───────────────────┤
        │    Region 2       │
        │                   │
        └───────────────────┘
                  │
                  └────────────→ x₁

By combining many hidden neurons, the network can create increasingly complex regions.

---

From simple regions to complex functions

Suppose we want an MLP to approximate a complicated function:

$$y=f(x)$$

Instead of trying to learn the entire function at once, the hidden layer can divide the input space into many smaller regions.

For example:

Input space

┌────┬────┬────┬────┐
│ R₁ │ R₂ │ R₃ │ R₄ │
├────┼────┼────┼────┤
│ R₅ │ R₆ │ R₇ │ R₈ │
├────┼────┼────┼────┤
│ R₉ │R₁₀ │R₁₁ │R₁₂ │
└────┴────┴────┴────┘

The network can assign an approximate function value to each region.

For example:

$$R_1\rightarrow0.2$$ $$R_2\rightarrow0.5$$ $$R_3\rightarrow0.8$$

and so on.

The resulting function becomes a piecewise approximation of the desired function.

---

How does this become more accurate?

Suppose the actual function is:

$$y=f(x)$$

A coarse network might approximate it like this:

Actual function:

        ╭──────╮
      ╭─╯      ╰──╮
   ╭──╯           ╰──
───╯

A small number of hidden units may produce a rough approximation:

      ┌──────┐
   ┌──┘      └──┐
───┘             └──

If we increase the number of hidden neurons, we can divide the input space into smaller regions:

More hidden units
       ↓
More regions
       ↓
Finer approximation
       ↓
Closer to target function

Therefore:

$$\boxed{ \text{More suitable hidden units} \Rightarrow \text{greater representational capacity} }$$

---

 Why is nonlinearity essential?

Suppose the hidden neurons use linear activation functions.

Then:

$$h=W_1x+b_1$$

and the output is:

$$y=W_2h+b_2$$

Substituting:

$$y=W_2(W_1x+b_1)+b_2$$

which becomes:

$$y=W_2W_1x+W_2b_1+b_2$$

or simply:

$$\boxed{y=Wx+b}$$

So even though we have multiple layers, the entire network is still just a linear function.

Therefore:

$$\boxed{ \text{Nonlinear activation} \Rightarrow \text{essential for universal approximation} }$$

---

Role of Sigmoid/Tanh/ReLU

The classical universal approximation results are often presented using sigmoid-type nonlinear activation functions.

For example:

$$\sigma(z)=\frac{1}{1+e^{-z}}$$

A hidden neuron computes:

$$h_1=\sigma(w_{11}x_1+w_{12}x_2+b_1)$$

Another computes:

$$h_2=\sigma(w_{21}x_1+w_{22}x_2+b_2)$$

and so on.

The output combines them:

$$\boxed{ y=\sum_{h=1}^{H}v_hh_h+b }$$

Thus the MLP builds a complex function by combining many nonlinear functions.

---

A simple analogy: LEGO blocks 🧱

This is a useful way to explain universal approximation to students.

Imagine the target function is a complicated building.

A single neuron is like one LEGO block.

It cannot build the whole structure.

But if we have many neurons:

$$\boxed{ \text{Many simple nonlinear units} \rightarrow \text{Complex function} }$$

Each hidden neuron contributes a small part of the overall function.

The output layer combines these contributions.

So:

An MLP can construct a very complicated function by combining many simple nonlinear functions.

---

Connection with Taylor approximation

The  refers to a piecewise constant approximation.

Suppose we divide the input space into small regions.

Inside each region, we approximate the function by a constant:

$$f(x)\approx c_i$$

For example:

$$f(x)\approx \begin{cases} 0.2,&x\in R_1\\ 0.5,&x\in R_2\\ 0.8,&x\in R_3\\ 1.1,&x\in R_4 \end{cases}$$

This is similar to keeping only the constant term in a local Taylor approximation.

If we make the regions smaller:

$$\boxed{ \text{smaller regions} \Rightarrow \text{better approximation} }$$

An MLP with enough hidden units can implement increasingly fine approximations.

---

What does "Universal" actually mean?

This word can be misunderstood.

It does not mean:

An MLP can perfectly represent every possible function with a small number of neurons.

Instead, it means approximately:

For a broad class of functions, particularly continuous functions on compact/bounded domains, an MLP with a suitable nonlinear activation and sufficiently many hidden units can approximate the function to an arbitrarily small error.

Mathematically, if $f(x)$ is the target function and $F(x)$ is the MLP approximation, then for a desired accuracy $\epsilon>0$, we can seek:

$$\boxed{ |F(x)-f(x)|<\epsilon }$$

over the specified domain.

The exact theorem has technical assumptions about the activation function, domain, and function class, so this should be presented as the intuition, rather than as "one hidden layer can learn literally every function."

---

 One Hidden Layer Is Theoretically Enough

A particularly important result is that an MLP with:

$$\boxed{\text{one hidden layer + sufficiently many hidden neurons}}$$

can approximate a broad class of nonlinear functions.

This is the result commonly associated with the Universal Approximation Theorem.

Therefore, we don't necessarily need many hidden layers to establish universal approximation in theory.

However, this does not mean one hidden layer is always the best practical architecture.

---

Why use Deep Networks if One Hidden Layer is Enough?

The theorem says that one hidden layer can approximate a function, but it does not say that it will do so efficiently.

A shallow network may require a very large number of hidden neurons.

For some problems, multiple layers can represent the same function much more efficiently.

Conceptually:

$$\boxed{ \text{Shallow network} \rightarrow \text{many neurons} }$$

whereas:

$$\boxed{ \text{Deep network} \rightarrow \text{hierarchical representations} }$$

For example, in image recognition:

$$\text{pixels} \rightarrow \text{edges} \rightarrow \text{shapes} \rightarrow \text{objects}$$

Different layers can learn increasingly complex representations.

---

Universal Approximation — Big Picture

The complete idea can be summarized as:

                 TARGET FUNCTION
                       │
                       ▼
              ┌─────────────────┐
              │      MLP        │
              │                 │
Input ───────►│ Hidden neurons  │
              │ + nonlinear     │
              │ activation      │
              └────────┬────────┘
                       │
                       ▼
                 Approximation
                       │
                       ▼
             Target function
              (approximately)

The hidden neurons act as nonlinear basis functions, and the output layer combines these basis functions:

$$\boxed{ F(x)= \sum_{h=1}^{H} v_h\, \phi(w_h^Tx+b_h) +b }$$

This equation captures the fundamental idea of an MLP as a function approximator.

---

🎓 Summary

A MLP acts as a universal approximator because its hidden neurons create nonlinear transformations of the input, and the output layer combines these nonlinear features to construct complex functions. With a suitable nonlinear activation function and sufficiently many hidden neurons, an MLP can approximate a broad class of continuous functions to arbitrary accuracy. Increasing the number of hidden neurons allows the network to create finer and more detailed approximations. The theorem guarantees the existence of such a solution; it does not tell us how many neurons are needed or guarantee that training will easily find the solution.

$$\boxed{ \text{Many nonlinear neurons} \;+\; \text{learned weights} \;\Rightarrow\; \text{approximation of complex functions} }$$