Multivariate Linear Regression using Gradient Descent

 

Multivariate Linear Regression using Gradient Descent

🔹 1. Model (Hypothesis Function)

For multiple features:

y^i=β0+β1xi1+β2xi2+⋯+βpxip\hat{y}_i = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \cdots + \beta_p x_{ip}

🔹2. Matrix Form (Very Important)

Add a column of 1s for intercept:

X=[1x11x12⋯x1p1x21x22⋯x2p⋮⋮⋮⋱⋮1xn1xn2⋯xnp]X = \begin{bmatrix} 1 & x_{11} & x_{12} & \cdots & x_{1p} \\ 1 & x_{21} & x_{22} & \cdots & x_{2p} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & x_{n1} & x_{n2} & \cdots & x_{np} \end{bmatrix}

📌 Parameter Vector

β=[β0β1⋮βp]\beta = \begin{bmatrix} \beta_0 \\ \beta_1 \\ \vdots \\ \beta_p \end{bmatrix}

📌 Model Equation

y=Xβy = X\beta

🔹 3. Cost Function (MSE)

J(β)=12n∑i=1n(yi−y^i)2

👉 Goal: Minimize this cost

🔹 4. Gradient of Cost Function

For all parameters together:

∇J(β)=1nXT(Xβ−y)\nabla J(\beta) = \frac{1}{n} X^T (X\beta - y)

👉 This gives partial derivatives for all coefficients

🔹 5. Gradient Descent Update Rule

β=β−α⋅∇J(β)\beta = \beta - \alpha \cdot \nabla J(\beta)

🔸 Substitute Gradient

β=β−αnXT(Xβ−y)\beta = \beta - \frac{\alpha}{n} X^T (X\beta - y)

🔹 6. Expanded Form (Component-wise)

For each parameter βj\beta_j:

βj=βj−αn∑i=1n(xij)(y^i−yi)\beta_j = \beta_j - \frac{\alpha}{n} \sum_{i=1}^{n} (x_{ij})(\hat{y}_i - y_i)

🔹 7. Algorithm Steps

📌 Gradient Descent Procedure

  1. Initialize β0,β1,...,βp\beta_0, \beta_1, ..., \beta_p

  2. Compute predictions:

    y^=Xβ\hat{y} = X\beta

  3. Compute error:

    (Xβ−y)(X\beta - y)

  4. Compute gradient:

    XT(Xβ−y)X^T (X\beta - y)

  5. Update parameters:

    β=β−αnXT(Xβ−y)\beta = \beta - \frac{\alpha}{n} X^T (X\beta - y)
  6. Repeat until convergence

🔹 8. Intuition

  • Each coefficient is adjusted based on its contribution to error
  • All parameters are updated simultaneously

🔹 9. Role of Learning Rate (α)

Learning RateEffect
Small    Slow convergence
Large    May diverge
Optimal    Fast convergence

🔹 10. Why Use Gradient Descent?

👉 Preferred when:

  • Large number of features
  • Large dataset
  • Matrix inversion is costly

🔹 11. Important Notes

  • Feature scaling is very important
  • Helps faster convergence
  • Works for high-dimensional data

🔹 12. Comparison with Normal Equation

MethodAdvantageLimitation
Normal Equation        Exact solution        Expensive for large data
Gradient Descent        Scalable        Needs tuning

🔹 13. Summary 

β=β−αnXT(Xβ−y)\boxed{ \beta = \beta - \frac{\alpha}{n} X^T (X\beta - y) }

👉 Multivariate GD is just an extension of simple GD:

  • Works on vectors instead of scalars
  • Updates all parameters together

👉 Gradient Descent:

  • Iteratively minimizes error
  • Finds optimal regression coefficients
  • Works efficiently for large-scale problems


Appendix

Goal

We want to derive:

∇J(β)=1nXT(Xβ−y)\nabla J(\beta) = \frac{1}{n} X^T (X\beta - y)

🔹 1. Start with Cost Function

J(β)=12n(Xβ−y)T(Xβ−y)J(\beta) = \frac{1}{2n} (X\beta - y)^T (X\beta - y)

👉 Let:

e=Xβ−ye = X\beta - y

So:

J(β)=12neTeJ(\beta) = \frac{1}{2n} e^T e

🔹 2. Expand the Expression

J(β)=12n(Xβ−y)T(Xβ−y)J(\beta) = \frac{1}{2n} (X\beta - y)^T (X\beta - y)
=12n(βTXTXβ−2yTXβ+yTy)= \frac{1}{2n} \left( \beta^T X^T X \beta - 2y^T X \beta + y^T y \right)

🔹 3. Take Gradient w.r.t β

We differentiate term by term.

🔸 Term 1:

∂∂β(βTXTXβ)=2XTXβ\frac{\partial}{\partial \beta} (\beta^T X^T X \beta) = 2X^T X \beta

🔸 Term 2:

∂∂β(−2yTXβ)=−2XTy\frac{\partial}{\partial \beta} (-2y^T X \beta) = -2X^T y

🔸 Term 3:

∂∂β(yTy)=0\frac{\partial}{\partial \beta} (y^T y) = 0

🔹 4. Combine Results

∇J(β)=12n(2XTXβ−2XTy)\nabla J(\beta) = \frac{1}{2n} (2X^T X \beta - 2X^T y)

🔹 5. Simplify

Cancel 2:

∇J(β)=1n(XTXβ−XTy)\nabla J(\beta) = \frac{1}{n} (X^T X \beta - X^T y)

🔹 6. Factor

∇J(β)=1nXT(Xβ−y)\nabla J(\beta) = \frac{1}{n} X^T (X\beta - y)

✅ Final Result

∇J(β)=1nXT(Xβ−y)\boxed{ \nabla J(\beta) = \frac{1}{n} X^T (X\beta - y) }

Comments

Post a Comment

Popular posts from this blog

Machine Learning PCCST503 Semester5 KTU CS 2024 Scheme - Dr Binu V P

About Me  - Dr Binu V P Syllabus Machine Learning PCCST503 KTU 2024 Scheme Model Question Paper Machine Learning PCCST503 Semester 5 KTU CS 2024 Scheme Do the programs to better understand the concept. Machine Learning Lab PCCSL508 Module -I Introduction to Machine Learning Introduction to ML Distinguishing Machine Learning from Traditional Programming Machine Learning Paradigms Supervised UnSupervised Semi-Supervised Reinforcement Stages of Machine Learning: From Data Collection to Deployment Parameter Estimation Introduction to Parameter Estimation in Machine Learning Maximum Likelyhood Estimation.(MLE) Maximum A Posteriori Estimation.(MAP) Numerical Problems MLE and MAP Supervised Learning Feature Representation and Problem Formulation Role of loss functions and optimization  Regression Introduction to Regression   Linear regression with one variable  (  lab experiment ) Example Problems- simple linear regression - University Questions Evaluation Metri...
Read more

Introduction to Machine Learning (ML)

  📘 Introduction to Machine Learning (ML) 🔹 What is Machine Learning? Machine Learning is a subfield of Artificial Intelligence that focuses on developing algorithms that allow computers to learn patterns from data and make decisions or predictions without explicit programming . It focuses on developing algorithms and models that allow machines to: Identify patterns in data Make decisions with minimal human intervention Predict future outcomes based on past experiences By continuously learning from new data, machine learning systems become more accurate and efficient over time. 📌 Formal Definition  A computer program is said to learn from experience E with respect to some task T and performance measure P if its performance at task T , as measured by P , improves with experience E . 🔹 Why Machine Learning? Traditional programming: Input + Program → Output Machine Learning paradigm: Input + Output → Model (learned program) 👉 Instead of writing...
Read more

Distinguishing Machine Learning from Traditional Programming

Image
  📘 Distinguishing Machine Learning from Traditional Programming 🔹 1. Fundamental Idea 🧾 Traditional Programming In conventional programming, developers write explicit instructions for the system to follow. The programmer explicitly writes rules/logic System follows deterministic instructions Input data is processed using predefined rules The output is based strictly on those rules 👉 Flow: Input + Program → Output 🤖 Machine Learning In machine learning, the system learns patterns from data instead of being explicitly programmed. Programmer provides  data + learning algorithm The model is trained using data The system  learns rules automatically from data It makes predictions or decisions based on learned patterns 👉 Flow: Input + Output → Learning Algorithm → Model Model + New Input → Predicted Output 🔹 2. Key Difference in Approach Aspect      Traditional Programming Machine Learning Input method Logic creation   ...
Read more