From Optimization to Probabilistic Modeling

 

🧠 From Optimization to Probability

So far, we wrote regression as an optimization problem:

$$\min_w \; \text{Loss} + \lambda \cdot \text{Penalty}$$

But we can reinterpret this using probabilistic modeling.

---

📘 Maximum Likelihood Estimation (MLE)

Assume a standard linear regression model:

$$y = Xw + \epsilon$$

where:

• $ϵ\sim \mathcal{N}(0,{\sigma }^2)(Gaussian\; noise)$

---

🔹 Likelihood

$$P(y|X,w) \propto \exp\left(-\frac{1}{2\sigma^2} \sum (y_i - w^T x_i)^2 \right)$$

---

🔹 MLE Objective

Maximizing likelihood is equivalent to:

$$\min_w \sum (y_i - w^T x_i)^2$$

👉 This is just ordinary least squares

---

🔁 Maximum A Posteriori (MAP)

Instead of only likelihood, MAP uses:

$$P(w|X,y) \propto P(y|X,w) \cdot P(w)$$

---

🔹 MAP Objective

Maximize posterior:

$$\max_w \; P(y|X,w) \cdot P(w)$$

Take negative log:

$$\min_w \; -\log P(y|X,w) - \log P(w)$$

---

🎯 Key Insight

$$\text{Loss} + \text{Regularization} \;\;\Longleftrightarrow\;\; \text{Likelihood} + \text{Prior}$$

👉 Regularization term = log prior

---

🔷Ridge Regression as MAP (Gaussian Prior)

---

🔹 Assume Prior on Weights

$$w \sim \mathcal{N}(0, \tau^2 I)$$

---

🔹 Prior Probability

$$P(w) \propto \exp\left(-\frac{1}{2\tau^2} \sum w_j^2 \right)$$

---

🔹 Negative Log Prior

$$-\log P(w) \propto \sum w_j^2$$

---

🔹 MAP Objective

$\min_{w} \left( \sum (y_i - w^T x_i)^2 + \lambda \sum w_j^2 \right)$

---

🎯 Conclusion

👉 Ridge regression = MAP with Gaussian prior

---

🧠 Interpretation

• Gaussian prior prefers small weights
• Penalizes large deviations smoothly

---

🔶LASSO as MAP (Laplace Prior)

---

🔹 Assume Prior

$$w \sim \text{Laplace}(0, b)$$

---

🔹 Prior Probability

$$P(w) \propto \exp\left(-\frac{1}{b} \sum |w_j| \right)$$

---

🔹 Negative Log Prior

$$-\log P(w) \propto \sum |w_j|$$

---

🔹 MAP Objective

$\min_{w} \left( \sum (y_i - w^T x_i)^2 + \lambda \sum |w_j| \right)$

---

🎯 Conclusion

👉 LASSO = MAP with Laplace prior

---

🧠 Interpretation

• Laplace prior is peaked at zero
• Encourages exact zeros
• Leads to sparsity

---

⚖️Why Priors Matter (Key Insight)

---

🔷 Gaussian Prior (Ridge)

• Smooth bell-shaped curve
• Penalizes large weights gradually

👉 Result:

• Shrinkage, but no zeros

---

🔶 Laplace Prior (LASSO)

• Sharp peak at zero
• Heavy tails

👉 Result:

• Many weights exactly zero

---

📊Visual Intuition (Conceptual)

Prior	Shape	Effect
Gaussian	Smooth	Small weights
Laplace	Sharp at zero	Sparse weights

---

🔁 Connecting λ to Probability

$$\lambda = \frac{\sigma^2}{\tau^2} \quad (\text{Ridge})$$

👉 Interpretation:

• Noise variance vs prior variance
• Balances data vs prior belief

---

🧠Big Picture Interpretation

---

🔹 Without Regularization (MLE)

• Only data matters
• Risk of overfitting

---

🔹 With Regularization (MAP)

• Data + prior belief
• More robust model

---

🎓 Teaching Analogy

---

MLE:

“Trust only the data”

---

MAP:

“Trust data, but also apply prior knowledge”

---

Ridge:

“Weights should be small”

---

LASSO:

“Most weights should be zero”

---

🎯 Key Takeaways

• Regularization = adding prior knowledge
• Ridge = Gaussian prior → smooth shrinkage
• LASSO = Laplace prior → sparsity
• MAP = unifies optimization and probability