Maximum Likelihood Estimation (MLE)

 

๐Ÿ“˜ Maximum Likelihood Estimation (MLE)

๐Ÿ”น Definition

Maximum Likelihood Estimation (MLE) is a method for estimating the parameters of a statistical model by maximizing the likelihood of the observed data.

๐Ÿ‘‰ In simple terms:
MLE finds parameter values that make the observed data most probable.

๐Ÿ”น Basic Idea

Assume:

  • Dataset: D={x1,x2,...,xn}D = \{x_1, x_2, ..., x_n\}
  • Model parameters: ฮธ\theta

๐Ÿ‘‰ Likelihood function:

L(ฮธ)=P(D∣ฮธ)L(\theta) = P(D \mid \theta)

๐Ÿ‘‰ Goal:

ฮธ^=arg⁡max⁡ฮธP(D∣ฮธ)\hat{\theta} = \arg\max_{\theta} P(D \mid \theta)

๐Ÿ”น Intuition

MLE answers the question:

“Given this data, what parameter values are most likely to have generated it?”

๐Ÿ”น Likelihood vs Probability

  • Probability: P(Data∣ฮธ)P(Data \mid \theta) → data varies
  • Likelihood: same expression, but viewed as a function of ฮธ\theta

๐Ÿ‘‰ Key difference:

  • Data is fixed
  • Parameters are variable

๐Ÿ”นLog-Likelihood 

Instead of maximizing likelihood, we maximize log-likelihood:

โ„“(ฮธ)=log⁡P(D∣ฮธ)\ell(\theta) = \log P(D \mid \theta)

Why log?

  • Converts products → sums
  • Easier differentiation
  • Numerically stable

๐Ÿ”น General Steps in MLE

  1. Define probabilistic model P(x∣ฮธ)P(x \mid \theta)
  2. Write likelihood function
  3. Take log-likelihood
  4. Differentiate w.r.t. parameters
  5. Set derivative = 0
  6. Solve for parameters

๐Ÿ”นExample 1: Bernoulli Distribution

๐Ÿ“Œ Problem:

Assume we have:

  • Data: x1,x2,...,xnx_1, x_2, ..., x_n
  • Each xi∈{0,1}x_i \in \{0,1\} (Bernoulli trials)

Let:

  • pp = probability of success (i.e., x=1x = 1)

๐Ÿ‘‰ Goal: Estimate pp using Maximum Likelihood Estimation (MLE)

Eg: Coin toss (Heads = 1, Tails = 0)

  • pp = probability of heads

Bernoulli Distribution

For a single observation:

P(xi∣p)=pxi(1−p)1−xiP(x_i \mid p) = p^{x_i}(1-p)^{1-x_i}

Likelihood Function

For independent observations:

L(p)=∏i=1npxi(1−p)1−xiL(p) = \prod_{i=1}^{n} p^{x_i}(1-p)^{1-x_i}

L(p) = \prod p^{x_i}(1-p)^{1-x_i}

Log-Likelihood

Take logarithm:

โ„“(p)=∑i=1n[xilog⁡p+(1−xi)log⁡(1−p)]\ell(p) = \sum_{i=1}^{n} \left[ x_i \log p + (1-x_i)\log(1-p) \right]

Differentiate w.r.t. pp

dโ„“dp=∑i=1n[xip−1−xi1−p]\frac{d\ell}{dp} = \sum_{i=1}^{n} \left[ \frac{x_i}{p} - \frac{1-x_i}{1-p} \right]

Set Derivative = 0

∑i=1n[xip−1−xi1−p]=0\sum_{i=1}^{n} \left[ \frac{x_i}{p} - \frac{1-x_i}{1-p} \right] = 0

Simplify

Split the summation:

1p∑xi=11−p∑(1−xi)\frac{1}{p} \sum x_i = \frac{1}{1-p} \sum (1 - x_i)

Let:

  • ∑xi=k\sum x_i = k (number of successes)
  • ∑(1−xi)=n−k\sum (1-x_i) = n - k

So:

kp=n−k1−p\frac{k}{p} = \frac{n-k}{1-p}

Solve for pp

Cross-multiply:

k(1−p)=(n−k)pk(1-p) = (n-k)p
k−kp=np−kpk - kp = np - kp
k=npk = npp=knp = \frac{k}{n}

Final Result

p^=1n∑i=1nxi\hat{p} = \frac{1}{n} \sum_{i=1}^{n} x_i

๐Ÿ‘‰ This is the sample mean

Interpretation

  • p^\hat{p} = proportion of 1’s in data
  • i.e., fraction of successes

๐Ÿ‘‰ Example:

  • Data: (1, 0, 1, 1, 0)
  • p^=3/5=0.6\hat{p} = 3/5 = 0.6

 Important Insight

๐Ÿ‘‰ For Bernoulli distribution:

MLE estimate of probability = sample mean

 Why This Makes Sense

  • If many 1’s → higher probability
  • If fewer 1’s → lower probability

๐Ÿ‘‰ MLE naturally captures empirical frequency

Variance of Bernoulli Distribution

We know:

Var(X)=p(1−p)\text{Var}(X) = p(1 - p)

MLE of Variance

๐Ÿ‘‰ Substitute MLE of pp:

Var^=p^(1−p^)\widehat{\text{Var}} = \hat{p}(1 - \hat{p})

Final Expression

Var^=(kn)(1−kn)\widehat{\text{Var}} = \left(\frac{k}{n}\right) \left(1 - \frac{k}{n}\right)

✅ Final Result

Var^=kn(1−kn)\boxed{ \widehat{\text{Var}} = \frac{k}{n}\left(1 - \frac{k}{n}\right) }

Alternative Form

Var^=xห‰(1−xห‰)\widehat{\text{Var}} = \bar{x}(1 - \bar{x})

where:

xห‰=1n∑xi\bar{x} = \frac{1}{n}\sum x_i

๐Ÿ”นExample 2: Gaussian Distribution

Problem Setup

Assume:

  • Data: x1,x2,...,xn
  • Each xi∼N(ฮผ,ฯƒ2)

๐Ÿ‘‰ Unknown parameters:

  • Mean ฮผ
  • Variance ฯƒ2

๐Ÿ‘‰ Goal: Estimate ฮผ and ฯƒ2 using MLE

Gaussian Probability Density Function

For one data point:

p(x∣ฮผ,ฯƒ2)=12ฯ€ฯƒ2exp⁡(−(x−ฮผ)22ฯƒ2)p(x \mid \mu, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)

Likelihood Function

For independent data:

L(ฮผ,ฯƒ2)=∏i=1np(xi∣ฮผ,ฯƒ2)L(\mu, \sigma^2) = \prod_{i=1}^{n} p(x_i \mid \mu, \sigma^2)
=∏i=1n12ฯ€ฯƒ2exp⁡(−(xi−ฮผ)22ฯƒ2)= \prod_{i=1}^{n} \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x_i-\mu)^2}{2\sigma^2}\right)

Log-Likelihood

Take logarithm:

โ„“(ฮผ,ฯƒ2)=−n2log⁡(2ฯ€ฯƒ2)−12ฯƒ2∑i=1n(xi−ฮผ)2\ell(\mu, \sigma^2) = -\frac{n}{2}\log(2\pi\sigma^2) - \frac{1}{2\sigma^2}\sum_{i=1}^{n}(x_i-\mu)^2

MLE for Mean ฮผ

Step 1: Differentiate w.r.t. ฮผ

∂โ„“∂ฮผ=1ฯƒ2∑i=1n(xi−ฮผ)\frac{\partial \ell}{\partial \mu} = \frac{1}{\sigma^2}\sum_{i=1}^{n}(x_i - \mu)


Step 2: Set derivative = 0

∑i=1n(xi−ฮผ)=0\sum_{i=1}^{n}(x_i - \mu) = 0

Step 3: Solve

∑xi−nฮผ=0\sum x_i - n\mu = 0
ฮผ=1n∑i=1nxi\mu = \frac{1}{n}\sum_{i=1}^{n} x_i

Result:

ฮผ^=sample mean\hat{\mu} = \text{sample mean}

MLE for Variance ฯƒ2\sigma^2

Step 1: Differentiate w.r.t. ฯƒ2\sigma^2


∂โ„“∂ฯƒ2=−n2ฯƒ2+12ฯƒ4∑i=1n(xi−ฮผ)2\frac{\partial \ell}{\partial \sigma^2} = -\frac{n}{2\sigma^2} + \frac{1}{2\sigma^4}\sum_{i=1}^{n}(x_i - \mu)^2

Step 2: Set derivative = 0

−n2ฯƒ2+12ฯƒ4∑(xi−ฮผ)2=0-\frac{n}{2\sigma^2} + \frac{1}{2\sigma^4}\sum (x_i - \mu)^2 = 0

Step 3: Solve

Multiply both sides:

nฯƒ2=∑(xi−ฮผ)2n\sigma^2 = \sum (x_i - \mu)^2
ฯƒ2=1n∑i=1n(xi−ฮผ)2\sigma^2 = \frac{1}{n}\sum_{i=1}^{n}(x_i - \mu)^2

Result:

ฯƒ^2=sample variance (with denominator n)\hat{\sigma}^2 = \text{sample variance (with denominator } n\text{)}

 Final MLE Estimates

ฮผ^=1n∑xi\hat{\mu} = \frac{1}{n}\sum x_i ฯƒ^2=1n∑(xi−ฮผ)2\hat{\sigma}^2 = \frac{1}{n}\sum (x_i - \mu)^2

๐Ÿ‘‰ These are intuitive:

  • Mean → average
  • Variance → spread

๐Ÿ”น Properties of MLE 

✅ 1. Consistency

  • Estimates approach true value as data increases

✅ 2. Efficiency

  • Has minimum variance (under conditions)

✅ 3. Asymptotic Normality

  • Distribution becomes normal for large samples

❌ 4. Bias

  • Can be biased for small samples

๐Ÿ”น  Advantages

  • Simple and widely applicable
  • Works for many distributions
  • Strong theoretical foundation

๐Ÿ”น Disadvantages

  • Sensitive to outliers
  • Requires correct model assumption
  • Can overfit with small data

๐Ÿ”น Applications of MLE

  • Linear Regression
  • Logistic Regression
  • Naive Bayes
  • Hidden Markov Models
  • Neural Networks (via loss functions)

๐Ÿ”น Connection to Loss Functions

MLE often leads to common loss functions:

Model    Loss Function
Gaussian    Mean Squared Error
Bernoulli    Cross-Entropy
Multinomial    Log-loss

๐Ÿ“ Summary 

  • MLE finds parameters that maximize likelihood of data
  • Uses:
    • Likelihood function
    • Log-likelihood
  • Common results:
    • Mean = average
    • Variance = spread
  • Widely used in ML models

Comments

Post a Comment

Popular posts from this blog

Machine Learning PCCST503 Semester5 KTU CS 2024 Scheme - Dr Binu V P

About Me  - Dr Binu V P Syllabus Machine Learning PCCST503 KTU 2024 Scheme Model Question Paper Machine Learning PCCST503 Semester 5 KTU CS 2024 Scheme Do the programs to better understand the concept. Machine Learning Lab PCCSL508 Module -I Introduction to Machine Learning Introduction to ML Distinguishing Machine Learning from Traditional Programming Machine Learning Paradigms Supervised UnSupervised Semi-Supervised Reinforcement Stages of Machine Learning: From Data Collection to Deployment Parameter Estimation Introduction to Parameter Estimation in Machine Learning Maximum Likelyhood Estimation.(MLE) Maximum A Posteriori Estimation.(MAP) Numerical Problems MLE and MAP Supervised Learning Feature Representation and Problem Formulation Role of loss functions and optimization  Regression Introduction to Regression   Linear regression with one variable  (  lab experiment ) Example Problems- simple linear regression - University Questions Evaluation Metri...
Read more

Introduction to Machine Learning (ML)

  ๐Ÿ“˜ Introduction to Machine Learning (ML) ๐Ÿ”น What is Machine Learning? Machine Learning is a subfield of Artificial Intelligence that focuses on developing algorithms that allow computers to learn patterns from data and make decisions or predictions without explicit programming . It focuses on developing algorithms and models that allow machines to: Identify patterns in data Make decisions with minimal human intervention Predict future outcomes based on past experiences By continuously learning from new data, machine learning systems become more accurate and efficient over time. ๐Ÿ“Œ Formal Definition  A computer program is said to learn from experience E with respect to some task T and performance measure P if its performance at task T , as measured by P , improves with experience E . ๐Ÿ”น Why Machine Learning? Traditional programming: Input + Program → Output Machine Learning paradigm: Input + Output → Model (learned program) ๐Ÿ‘‰ Instead of writing...
Read more

Distinguishing Machine Learning from Traditional Programming

Image
  ๐Ÿ“˜ Distinguishing Machine Learning from Traditional Programming ๐Ÿ”น 1. Fundamental Idea ๐Ÿงพ Traditional Programming In conventional programming, developers write explicit instructions for the system to follow. The programmer explicitly writes rules/logic System follows deterministic instructions Input data is processed using predefined rules The output is based strictly on those rules ๐Ÿ‘‰ Flow: Input + Program → Output ๐Ÿค– Machine Learning In machine learning, the system learns patterns from data instead of being explicitly programmed. Programmer provides  data + learning algorithm The model is trained using data The system  learns rules automatically from data It makes predictions or decisions based on learned patterns ๐Ÿ‘‰ Flow: Input + Output → Learning Algorithm → Model Model + New Input → Predicted Output ๐Ÿ”น 2. Key Difference in Approach Aspect      Traditional Programming Machine Learning Input method Logic creation   ...
Read more