Parameter Estimation in Machine Learning

🔹 Definition

Parameter Estimation is the process of determining the optimal values of model parameters so that the model best fits the given data.

👉 In simple terms:
It is how a machine learning model learns from data.

---

🔹What are Parameters?

Parameters are internal variables of a model that are learned during training.

📌 Examples:

• Linear Regression → weights (w), bias (b)
• Neural Networks → weights and biases
• Gaussian models → mean (μ), variance (σ²)

---

🔹 Objective of Parameter Estimation

👉 Find parameter values that:

• Minimize prediction error
• Maximize model accuracy

---

🔹Common Methods

🟦 1. Maximum Likelihood Estimation (MLE)

• Chooses parameters that maximize the likelihood of observed data

👉 Idea:

$$\text{Choose } \theta \text{ such that } P(Data \mid \theta) \text{ is maximum}$$

---

🟩 2. Maximum A Posteriori (MAP)

• Uses prior knowledge + data

👉 Idea:

$$P(\theta \mid Data)$$

• Combines:
  • Likelihood
  • Prior probability

---

🔹 Example (Simple)

📌 Linear Regression

Model:

$$y = wx + b$$

👉 Parameter estimation finds:

• Best $w$ (slope)
• Best $b$ (intercept)

So that prediction error is minimized.

---

🔹Loss Function

Parameter estimation is done by minimizing a loss function.

Example:

• Mean Squared Error (MSE):

$$\text{Error} = (y_{actual} - y_{predicted})^2$$

👉 Goal:

• Find parameters that minimize total error

---

🔹Optimization Techniques

• Gradient Descent
• Stochastic Gradient Descent (SGD)

👉 Used to update parameters iteratively.

---

🔹 Types of Parameters

📌 Model Parameters

• Learned from data
• Example: weights

📌 Hyperparameters

• Set before training
• Example: learning rate

---

📝 Summary 

• Parameter estimation = finding best model parameters
• Goal: minimize error / maximize likelihood
• Methods:
  • MLE
  • MAP
• Uses optimization techniques like gradient descent