Multiple Linear Regression (MLR)

 

 Multiple Linear Regression (MLR)

🔹 Introduction

Multiple Linear Regression is an extension of simple linear regression where:

👉 We use more than one input variable (feature) to predict a single continuous output.

It models the relationship between one dependent variable and two or more independent variables

In real-world problems the response variable is often influenced by multiple predictors

🔹 Problem Formulation

Given dataset:

D={(xi1,xi2,...,xip,yi)}D = \{(x_{i1}, x_{i2}, ..., x_{ip}, y_i)\}
  • xij → j-th feature
  • yiy_i → output

🔹 Model Representation (Hypothesis)

y=β0+β1x1+β2x2+⋯+βpxpy = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_p x_p

Where:

  • β0\beta_0→ intercept
  • βj\beta_j→ coefficient of feature xjx_j

👉 This is a linear combination of inputs

🔹Example

📌 House Price Prediction

Features:

  • x1x_1 → area
  • x2x_2 → number of rooms
  • x3x_3 → location score

Model:

Price=β0+β1(area)+β2(rooms)+β3(location)\text{Price} = \beta_0 + \beta_1(\text{area}) + \beta_2(\text{rooms}) + \beta_3(\text{location})

🔹Objective Function

We minimize squared error:

J(β)=∑(yi−y^i)2J(\beta) = \sum (y_i - \hat{y}_i)^2

👉 Same as simple regression but with multiple variables

🔹Dataset Representation

Assume:

  • nn observations
  • pp features

📌 Feature Matrix XX (size n×p)

X=[x11x12⋯x1px21x22⋯x2p⋮⋮⋱⋮xn1xn2⋯xnp]X = \begin{bmatrix} x_{11} & x_{12} & \cdots & x_{1p} \\ x_{21} & x_{22} & \cdots & x_{2p} \\ \vdots & \vdots & \ddots & \vdots \\ x_{n1} & x_{n2} & \cdots & x_{np} \end{bmatrix}

📌 Response Vector yy

y=[y1y2⋮yn]y = \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{bmatrix}

🔹 Regression Model

For pp features:

yi=β0+β1xi1+β2xi2+⋯+βpxipy_i = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \cdots + \beta_p x_{ip}

🔹 Matrix Form (Very Important)

Add a column of 1s for intercept:

X=[1x11x12⋯x1p1x21x22⋯x2p⋮⋮⋮⋱⋮1xn1xn2⋯xnp]X = \begin{bmatrix} 1 & x_{11} & x_{12} & \cdots & x_{1p} \\ 1 & x_{21} & x_{22} & \cdots & x_{2p} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & x_{n1} & x_{n2} & \cdots & x_{np} \end{bmatrix}

📌 Parameter Vector

β=[β0β1⋮βp]\beta = \begin{bmatrix} \beta_0 \\ \beta_1 \\ \vdots \\ \beta_p \end{bmatrix}

📌 Model Equation

y=Xβy = X\beta

🔹 Least Squares Solution

We estimate parameters using:

β^=(XTX)−1XTy\hat{\beta} = (X^T X)^{-1} X^T y

👉 This is called the Normal Equation

🔹  Final Prediction

y^=Xβ^\hat{y} = X\hat{\beta}

🔹 Important Notes

  • Requires all data in memory
  • Uses matrix operations
  • Efficient for small to medium datasets

🔹 How It Works (Step-by-Step)

  1. Represent data in matrix form
  2. Define hypothesis
  3. Compute error (MSE)
  4. Minimize error
  5. Estimate coefficients

🔹 Geometric Interpretation

  • 1 variable → line
  • 2 variables → plane
  • many variables → hyperplane

🔹Assumptions 

  1. Linearity
  2. Independence of errors
  3. Constant variance (Homoscedasticity)
  4. No multicollinearity
  5. Normal distribution of errors

🔹 Applications

📊 Real-world examples:

  • House price prediction
  • Stock price prediction
  • Medical diagnosis
  • Sales forecasting
  • Risk analysis

🔹Advantages

  • Handles multiple factors
  • More accurate than simple regression
  • Interpretable coefficients

🔹 Limitations

  • Sensitive to multicollinearity
  • Assumes linear relationship
  • Can overfit with many features

🔹 Comparison

Feature    Simple LRMultiple LR
Inputs    One    Multiple
Model    Line    Plane/Hyperplane
Complexity    Low    Higher

📘 Worked Example

🔹 Given Data

X1X_1X2X_2YY
1        1    3.25
12    6.5
22    3.5
01    5

🔹 Step 1: Construct Matrix XX

X=[111112122101]X = \begin{bmatrix} 1 & 1 & 1 \\ 1 & 1 & 2 \\ 1 & 2 & 2 \\ 1 & 0 & 1 \end{bmatrix}

🔹 Step 2: Compute XTXX^T X

XTX=[4464676710]X^T X = \begin{bmatrix} 4 & 4 & 6 \\ 4 & 6 & 7 \\ 6 & 7 & 10 \end{bmatrix}

🔹 Step 3: Compute Inverse

(XTX)−1=[2.750.5−20.51−1−2−12](X^T X)^{-1} = \begin{bmatrix} 2.75 & 0.5 & -2 \\ 0.5 & 1 & -1 \\ -2 & -1 & 2 \end{bmatrix}

🔹 Step 4: Compute XTy

XTy=[18.2516.7528.25]X^T y = \begin{bmatrix} 18.25 \\ 16.75 \\ 28.25 \end{bmatrix}

🔹 Step 5: Compute Coefficients

β^=[2.0625−2.3753.25]\hat{\beta} = \begin{bmatrix} 2.0625 \\ -2.375 \\ 3.25 \end{bmatrix}

✅ Final Regression Model

y^=2.0625−2.375x1+3.25x2\hat{y} = 2.0625 - 2.375x_1 + 3.25x_2


📝 Summary (Quick Revision)

👉 Real-world problems depend on multiple variables, so:

Multiple regression is more practical than simple regression


  • Model:

y=Xβy = X\beta

  • Solution:

β^=(XTX)−1XTy\hat{\beta} = (X^T X)^{-1} X^T y

  • Prediction:

y^=Xβ^\hat{y} = X\hat{\beta}

Comments

Post a Comment

Popular posts from this blog

Machine Learning PCCST503 Semester5 KTU CS 2024 Scheme - Dr Binu V P

About Me  - Dr Binu V P Syllabus Machine Learning PCCST503 KTU 2024 Scheme Model Question Paper Machine Learning PCCST503 Semester 5 KTU CS 2024 Scheme Do the programs to better understand the concept. Machine Learning Lab PCCSL508 Module -I Introduction to Machine Learning Introduction to ML Distinguishing Machine Learning from Traditional Programming Machine Learning Paradigms Supervised UnSupervised Semi-Supervised Reinforcement Stages of Machine Learning: From Data Collection to Deployment Parameter Estimation Introduction to Parameter Estimation in Machine Learning Maximum Likelyhood Estimation.(MLE) Maximum A Posteriori Estimation.(MAP) Numerical Problems MLE and MAP Supervised Learning Feature Representation and Problem Formulation Role of loss functions and optimization  Regression Introduction to Regression   Linear regression with one variable  (  lab experiment ) Example Problems- simple linear regression - University Questions Evaluation Metri...
Read more

Introduction to Machine Learning (ML)

  📘 Introduction to Machine Learning (ML) 🔹 What is Machine Learning? Machine Learning is a subfield of Artificial Intelligence that focuses on developing algorithms that allow computers to learn patterns from data and make decisions or predictions without explicit programming . It focuses on developing algorithms and models that allow machines to: Identify patterns in data Make decisions with minimal human intervention Predict future outcomes based on past experiences By continuously learning from new data, machine learning systems become more accurate and efficient over time. 📌 Formal Definition  A computer program is said to learn from experience E with respect to some task T and performance measure P if its performance at task T , as measured by P , improves with experience E . 🔹 Why Machine Learning? Traditional programming: Input + Program → Output Machine Learning paradigm: Input + Output → Model (learned program) 👉 Instead of writing...
Read more

Distinguishing Machine Learning from Traditional Programming

Image
  📘 Distinguishing Machine Learning from Traditional Programming 🔹 1. Fundamental Idea 🧾 Traditional Programming In conventional programming, developers write explicit instructions for the system to follow. The programmer explicitly writes rules/logic System follows deterministic instructions Input data is processed using predefined rules The output is based strictly on those rules 👉 Flow: Input + Program → Output 🤖 Machine Learning In machine learning, the system learns patterns from data instead of being explicitly programmed. Programmer provides  data + learning algorithm The model is trained using data The system  learns rules automatically from data It makes predictions or decisions based on learned patterns 👉 Flow: Input + Output → Learning Algorithm → Model Model + New Input → Predicted Output 🔹 2. Key Difference in Approach Aspect      Traditional Programming Machine Learning Input method Logic creation   ...
Read more