Maximum A Posteriori (MAP) Estimation

 

📘 Maximum A Posteriori (MAP) Estimation

🔹 Definition

MAP estimation is a method for estimating model parameters by maximizing the posterior probability of the parameters given the observed data.

👉 In simple terms:
It finds parameters that are most probable after considering both data and prior knowledge.

🔹 Mathematical Formulation

Using Bayes’ theorem:

P(θ∣D)=P(D∣θ)P(θ)P(D)P(\theta\mid D)=\frac{P(D\mid \theta)P(\theta)}{P(D)}

👉 MAP estimate:

θ^MAP=arg⁡max⁡θP(θ∣D)\hat{\theta}_{MAP} = \arg\max_{\theta} P(\theta \mid D)

🔹Simplified Optimization Form

Since P(D)P(D) is constant:

θ^MAP=arg⁡max⁡θP(D∣θ)⋅P(θ)\hat{\theta}_{MAP} = \arg\max_{\theta} P(D \mid \theta) \cdot P(\theta)

Taking log:

θ^MAP=arg⁡max⁡θ[log⁡P(D∣θ)+log⁡P(θ)]\hat{\theta}_{MAP} = \arg\max_{\theta} \left[ \log P(D \mid \theta) + \log P(\theta) \right]

🔹  Interpretation

MAP combines:

  • Likelihood → how well model fits data
  • Prior → what we believe before seeing data

👉 Final estimate balances both.

🔹 Intuition

👉 MAP answers:

“What is the most probable parameter value given both prior knowledge and observed data?”

📌 Real-life Analogy:

  • A doctor has prior knowledge about disease prevalence
  • Observes symptoms (data)
  • Updates belief accordingly

🔹 Difference Between MLE and MAP

FeatureMLEMAP
Uses prior        ❌ No        ✅ Yes
Objective        Maximize likelihood        Maximize posterior
Overfitting        More likely        Reduced
Data requirement        High        Can work with less data

🔹 Example: Bernoulli Distribution with Prior

📌 Problem:

  • Data: coin tosses
  • Prior belief: p∼Beta(α,β)

Likelihood:

P(D∣p)=pk(1−p)n−kP(D \mid p) = p^k (1-p)^{n-k}

Prior:

P(p)=pα−1(1−p)β−1P(p) = p^{\alpha-1}(1-p)^{\beta-1}

Posterior:

P(p∣D)∝pk+α−1(1−p)n−k+β−1P(p \mid D) \propto p^{k+\alpha-1}(1-p)^{n-k+\beta-1}

MAP Estimate:


Beta Distribution

The probability density function (PDF) of Beta distribution is:

f(p)=pα−1(1−p)β−1,0<p<1f(p) = p^{\alpha - 1}(1 - p)^{\beta - 1}, \quad 0 < p < 1

Where:

  • α>0\alpha > 0, β>0\beta > 0

👉 Why Beta?

  • Conjugate prior for Bernoulli
  • Simplifies computation 

Goal

👉 Find the mode, i.e., the value of pp that maximizes f(p)f(p).

Take Log (Simplification Step)

Instead of maximizing f(p)f(p), maximize log:

ℓ(p)=log⁡f(p)\ell(p) = \log f(p)ℓ(p)=(α−1)log⁡p+(β−1)log⁡(1−p)\ell(p) = (\alpha - 1)\log p + (\beta - 1)\log(1 - p)

Differentiate w.r.t. pp

dℓdp=α−1p−β−11−p\frac{d\ell}{dp} = \frac{\alpha - 1}{p} - \frac{\beta - 1}{1 - p}

Set Derivative = 0

Solve for pp

Cross-multiply:

Expand:

(α−1)−(α−1)p=(β−1)p(\alpha - 1) - (\alpha - 1)p = (\beta - 1)p

Rearrange:

α−1=(α−1)p+(β−1)p\alpha - 1 = (\alpha - 1)p + (\beta - 1)p
α−1=p[(α−1)+(β−1)]\alpha - 1 = p[(\alpha - 1) + (\beta - 1)]

Final Step:

p=α−1α+β−2p = \frac{\alpha - 1}{\alpha + \beta - 2}

p^=k+α−1n+α+β−2\hat{p} = \frac{k + \alpha - 1}{n + \alpha + \beta - 2}

📌 Special Case:

  • If α=1,β=1\alpha = 1, \beta = 1 (uniform prior):

👉 MAP = MLE

🔹Example: Gaussian Prior

 MAP Estimation of Mean (Gaussian Likelihood + Gaussian Prior)

🔹 Problem Setup

Assume:

  • Data: x1,x2,...,xnx_1, x_2, ..., x_n
  • Each xi∼N(μ,σ2)x_i \sim \mathcal{N}(\mu, \sigma^2)

👉 Known:

  • Variance σ2\sigma^2

👉 Unknown:

  • Mean μ\mu

🔹  Likelihood Function

P(D∣μ)=∏i=1nN(xi∣μ,σ2)P(D \mid \mu) = \prod_{i=1}^{n} \mathcal{N}(x_i \mid \mu, \sigma^2)
∝exp⁡(−12σ2∑i=1n(xi−μ)2)\propto \exp\left(-\frac{1}{2\sigma^2}\sum_{i=1}^{n}(x_i - \mu)^2\right)

🔹 Prior Distribution (Gaussian Prior)

Assume:

μ∼N(μ0,τ2)\mu \sim \mathcal{N}(\mu_0, \tau^2)

👉 Prior:

P(μ)∝exp⁡(−12τ2(μ−μ0)2)P(\mu) \propto \exp\left(-\frac{1}{2\tau^2}(\mu - \mu_0)^2\right)

🔹 Posterior Distribution

Using Bayes’ theorem:

P(μ∣D)∝P(D∣μ) P(μ)P(\mu \mid D) \propto P(D \mid \mu)\,P(\mu)
∝exp⁡(−12σ2∑(xi−μ)2)⋅exp⁡(−12τ2(μ−μ0)2)\propto \exp\left(-\frac{1}{2\sigma^2}\sum (x_i - \mu)^2\right) \cdot \exp\left(-\frac{1}{2\tau^2}(\mu - \mu_0)^2\right)

🔹 Log-Posterior

log⁡P(μ∣D)=−12σ2∑(xi−μ)2−12τ2(μ−μ0)2\log P(\mu \mid D) = -\frac{1}{2\sigma^2}\sum (x_i - \mu)^2 -\frac{1}{2\tau^2}(\mu - \mu_0)^2

🔹 MAP Objective

Maximize posterior
👉 Equivalent to minimizing:

∑(xi−μ)2+λ(μ−μ0)2\sum (x_i - \mu)^2 + \lambda (\mu - \mu_0)^2

where:

λ=σ2τ2\lambda = \frac{\sigma^2}{\tau^2}

🔹 Differentiate w.r.t. μ\mu

Expand derivative:

ddμ[∑(xi−μ)2+λ(μ−μ0)2]=0\frac{d}{d\mu} \left[ \sum (x_i - \mu)^2 + \lambda (\mu - \mu_0)^2 \right] = 0

Step-by-step:

ddμ∑(xi−μ)2=−2∑(xi−μ)\frac{d}{d\mu} \sum (x_i - \mu)^2 = -2\sum (x_i - \mu)
ddμλ(μ−μ0)2=2λ(μ−μ0)\frac{d}{d\mu} \lambda (\mu - \mu_0)^2 = 2\lambda (\mu - \mu_0)

🔹Set derivative = 0

−2∑(xi−μ)+2λ(μ−μ0)=0-2\sum (x_i - \mu) + 2\lambda (\mu - \mu_0) = 0

Divide by 2:

−∑(xi−μ)+λ(μ−μ0)=0-\sum (x_i - \mu) + \lambda (\mu - \mu_0) = 0

🔹 Simplify

∑xi−nμ+λμ−λμ0=0\sum x_i - n\mu + \lambda \mu - \lambda \mu_0 = 0

🔹 Solve for μ



 nμ+λμ=∑xi+λμ0n\mu + \lambda \mu = \sum x_i + \lambda \mu_0 μ(n+λ)=∑xi+λμ0\mu (n + \lambda) = \sum x_i + \lambda \mu_0

Final Result

μ^MAP=∑xi+λμ0n+λ\hat{\mu}_{MAP} = \frac{\sum x_i + \lambda \mu_0}{n + \lambda}

🔹 Alternative Form (Important)

Let:

xˉ=1n∑xi\bar{x} = \frac{1}{n}\sum x_i

Then:

μ^MAP=nxˉ+λμ0n+λ\hat{\mu}_{MAP} = \frac{n\bar{x} + \lambda \mu_0}{n + \lambda}

🔹 Interpretation

👉 MAP estimate is a weighted average:

  • Data mean xˉ\bar{x}
  • Prior mean μ0\mu_0

🔹  Intuition

  • If more data → rely on xˉ\bar{x}
  • If strong prior → rely on μ0\mu_0

🔹 Special Cases

📌 Case 1: No Prior (Flat Prior)

If λ=0\lambda = 0:

μ^=xˉ\hat{\mu} = \bar{x}

👉 MAP = MLE

📌 Case 2: Strong Prior

If λ\lambda is large:

μ^≈μ0\hat{\mu} \approx \mu_0

🔹 15. Key Insight

MAP = compromise between data and prior knowledge

📝 Summary (Quick Revision)

  • Likelihood → Gaussian
  • Prior → Gaussian
  • Posterior → Gaussian

Final Answer:

μ^=nxˉ+λμ0n+λ\hat{\mu} = \frac{n\bar{x} + \lambda \mu_0}{n + \lambda}

🔹Advantages of MAP

  • Incorporates prior knowledge
  • Reduces overfitting
  • Works well with small datasets
  • More robust than MLE

🔹Disadvantages

  • Requires prior selection
  • Wrong prior → poor results
  • More computationally complex

🔹 Applications

  • Bayesian Machine Learning
  • Regularized regression (Ridge, Lasso)
  • Naive Bayes classifier
  • Neural networks (weight regularization)

🔹 Key Concepts 

📌 Prior Distribution

  • Encodes initial belief

📌 Posterior Distribution

  • Updated belief after data

📌 Conjugate Priors

  • Makes computation easier
  • Example:
    • Bernoulli ↔ Beta
    • Gaussian ↔ Gaussian

📝 Summary 

  • MAP maximizes posterior probability
  • Combines:
    • Likelihood
    • Prior
  • Formula:
θ^=arg⁡max⁡P(D∣θ)P(θ)\hat{\theta} = \arg\max P(D \mid \theta)P(\theta)
  • More robust than MLE

Comments

Post a Comment

Popular posts from this blog

Machine Learning PCCST503 Semester5 KTU CS 2024 Scheme - Dr Binu V P

About Me  - Dr Binu V P Syllabus Machine Learning PCCST503 KTU 2024 Scheme Model Question Paper Machine Learning PCCST503 Semester 5 KTU CS 2024 Scheme Do the programs to better understand the concept. Machine Learning Lab PCCSL508 Module -I Introduction to Machine Learning Introduction to ML Distinguishing Machine Learning from Traditional Programming Machine Learning Paradigms Supervised UnSupervised Semi-Supervised Reinforcement Stages of Machine Learning: From Data Collection to Deployment Parameter Estimation Introduction to Parameter Estimation in Machine Learning Maximum Likelyhood Estimation.(MLE) Maximum A Posteriori Estimation.(MAP) Numerical Problems MLE and MAP Supervised Learning Feature Representation and Problem Formulation Role of loss functions and optimization  Regression Introduction to Regression   Linear regression with one variable  (  lab experiment ) Example Problems- simple linear regression - University Questions Evaluation Metri...
Read more

Introduction to Machine Learning (ML)

  📘 Introduction to Machine Learning (ML) 🔹 What is Machine Learning? Machine Learning is a subfield of Artificial Intelligence that focuses on developing algorithms that allow computers to learn patterns from data and make decisions or predictions without explicit programming . It focuses on developing algorithms and models that allow machines to: Identify patterns in data Make decisions with minimal human intervention Predict future outcomes based on past experiences By continuously learning from new data, machine learning systems become more accurate and efficient over time. 📌 Formal Definition  A computer program is said to learn from experience E with respect to some task T and performance measure P if its performance at task T , as measured by P , improves with experience E . 🔹 Why Machine Learning? Traditional programming: Input + Program → Output Machine Learning paradigm: Input + Output → Model (learned program) 👉 Instead of writing...
Read more

Distinguishing Machine Learning from Traditional Programming

Image
  📘 Distinguishing Machine Learning from Traditional Programming 🔹 1. Fundamental Idea 🧾 Traditional Programming In conventional programming, developers write explicit instructions for the system to follow. The programmer explicitly writes rules/logic System follows deterministic instructions Input data is processed using predefined rules The output is based strictly on those rules 👉 Flow: Input + Program → Output 🤖 Machine Learning In machine learning, the system learns patterns from data instead of being explicitly programmed. Programmer provides  data + learning algorithm The model is trained using data The system  learns rules automatically from data It makes predictions or decisions based on learned patterns 👉 Flow: Input + Output → Learning Algorithm → Model Model + New Input → Predicted Output 🔹 2. Key Difference in Approach Aspect      Traditional Programming Machine Learning Input method Logic creation   ...
Read more