Simple Linear Regression (One Variable)

 

Simple Linear Regression (One Variable)

🔹 Definition

Simple Linear Regression (SLR) is a supervised learning method used to model the relationship between:

  • One independent variable (input)
  • One dependent variable (output)

👉 It assumes a linear relationship between them

🔹 Problem Formulation

Given dataset:

D={(x1,y1),(x2,y2),...,(xn,yn)}D = \{(x_1,y_1), (x_2,y_2), ..., (x_n,y_n)\}

👉 Goal:
Learn a function:

y=f(x)y = f(x)

🔹Model Representation (Hypothesis)

The model is a straight line:

y=β0+β1xy = \beta_0 + \beta_1 x

Where:

  • β0\beta_0 → intercept
  • β1\beta_1 → slope
  • xx → input
  • yy → predicted output

👉 This represents the best-fit line through the data

🔹 Interpretation of Parameters

📌 Intercept (β0\beta_0)

  • Value of yy when x=0x = 0

📌 Slope (β1\beta_1)

  • Change in yy for a unit change in xx

👉 Example:
If slope = 2 → increasing xx by 1 increases yy by 2

🔹Key Idea: Best-Fit Line

👉 The regression line is chosen such that:

  • It minimizes the error between actual and predicted values
  • Error = vertical distance from data point to line

👉 This is called residual

🔹  Loss Function (Important)

Most commonly used:

L=∑i=1n(yi−y^i)2L = \sum_{i=1}^{n}(y_i - \hat{y}_i)^2

👉 This is Mean Squared Error (MSE)

👉 Goal:

min⁡β0,β1∑(yi−(β0+β1xi))2\min_{\beta_0,\beta_1} \sum (y_i - (\beta_0 + \beta_1 x_i))^2

👉 The best-fit line minimizes squared errors


🔹 Solution (Closed Form)

🔹 Problem Setup

Given data:

(x1,y1),(x2,y2),...,(xn,yn)(x_1,y_1), (x_2,y_2), ..., (x_n,y_n)

Model:

y^i=β0+β1xi\hat{y}_i = \beta_0 + \beta_1 x_i

👉 Goal: Find β0,β1\beta_0, \beta_1 that best fit the data.

🔹  Define Loss Function (Least Squares)

J(β0,β1)=∑i=1n(yi−β0−β1xi)2J(\beta_0, \beta_1) = \sum_{i=1}^{n} (y_i - \beta_0 - \beta_1 x_i)^2

👉 We minimize this error.

🔹 Take Partial Derivatives

We compute:

∂J∂β0=0,∂J∂β1=0\frac{\partial J}{\partial \beta_0} = 0, \quad \frac{\partial J}{\partial \beta_1} = 0

🔹 Derivative w.r.t.

∂J∂β0=−2∑(yi−β0−β1xi)\frac{\partial J}{\partial \beta_0} = -2 \sum (y_i - \beta_0 - \beta_1 x_i)

Set to zero:

∑(yi−β0−β1xi)=0\sum (y_i - \beta_0 - \beta_1 x_i) = 0

Simplify:

∑yi=nβ0+β1∑xi\sum y_i = n\beta_0 + \beta_1 \sum x_i

🔹 Derivative w.r.t.β1

∂J∂β1=−2∑xi(yi−β0−β1xi)\frac{\partial J}{\partial \beta_1} = -2 \sum x_i (y_i - \beta_0 - \beta_1 x_i)

Set to zero:

∑xiyi=β0∑xi+β1∑xi2\sum x_i y_i = \beta_0 \sum x_i + \beta_1 \sum x_i^2

🔹 Solve the Two Equations

We now have:

Equation (1):

∑yi=nβ0+β1∑xi\sum y_i = n\beta_0 + \beta_1 \sum x_i

Equation (2):

∑xiyi=β0∑xi+β1∑xi2\sum x_i y_i = \beta_0 \sum x_i + \beta_1 \sum x_i^2

🔹 Use Mean Values

Let:

xˉ=1n∑xi,yˉ=1n∑yi\bar{x} = \frac{1}{n}\sum x_i,\quad \bar{y} = \frac{1}{n}\sum y_i

From Equation (1):

β0=yˉ−β1xˉ\beta_0 = \bar{y} - \beta_1 \bar{x}

🔹Substitute into Equation (2)

∑xiyi=(yˉ−β1xˉ)∑xi+β1∑xi2\sum x_i y_i = (\bar{y} - \beta_1 \bar{x}) \sum x_i + \beta_1 \sum x_i^2

Expand:

∑xiyi=yˉ∑xi−β1xˉ∑xi+β1∑xi2\sum x_i y_i = \bar{y} \sum x_i - \beta_1 \bar{x} \sum x_i + \beta_1 \sum x_i^2

Rearrange:

∑xiyi−yˉ∑xi=β1(∑xi2−xˉ∑xi)\sum x_i y_i - \bar{y} \sum x_i = \beta_1 \left( \sum x_i^2 - \bar{x} \sum x_i \right)

🔹 Final Simplification

Using:

xˉ=∑xin\bar{x} = \frac{\sum x_i}{n}

Identity 1

∑xiyi−1n(∑xi)(∑yi)=∑(xi−xˉ)(yi−yˉ)\sum x_i y_i - \frac{1}{n}(\sum x_i)(\sum y_i) = \sum (x_i - \bar{x})(y_i - \bar{y})

 Identity 2

∑xi2−1n(∑xi)2=∑(xi−xˉ)2\sum x_i^2 - \frac{1}{n}(\sum x_i)^2 = \sum (x_i - \bar{x})^2

We get:

β1=∑(xi−xˉ)(yi−yˉ)∑(xi−xˉ)2\beta_1 = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})} {\sum (x_i - \bar{x})^2}

From optimization, we get:

β1=∑(xi−xˉ)(yi−yˉ)∑(xi−xˉ)2\beta_1 = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})} {\sum (x_i - \bar{x})^2} β0=yˉ−β1xˉ\beta_0 = \bar{y} - \beta_1 \bar{x}



Key Insight

β1=Cov(x,y)Var(x)\beta_1 = \frac{\text{Cov}(x,y)}{\text{Var}(x)}
Alternate Formula (Using Raw Sums)
β1=n∑xy−(∑x)(∑y)n∑x2−(∑x)2\beta_1 = \frac{n\sum xy - (\sum x)(\sum y)} {n\sum x^2 - (\sum x)^2} β0=∑y−β1∑xn\beta_0 = \frac{\sum y - \beta_1 \sum x}{n}

Alternate Formulas for Simple Linear Regression - Summary

🔹 1. Using Mean Values (Most Important)

Slope β1\beta_1

β1=∑(xi−xˉ)(yi−yˉ)∑(xi−xˉ)2\beta_1 = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})} {\sum (x_i - \bar{x})^2}

Intercept β0\beta_0

β0=yˉ−β1xˉ\beta_0 = \bar{y} - \beta_1 \bar{x}

🔹 2. Equivalent Statistical Form

This is a very elegant form:

β1=Cov(x,y)Var(x)\beta_1 = \frac{\text{Cov}(x,y)}{\text{Var}(x)} β0=yˉ−β1xˉ\beta_0 = \bar{y} - \beta_1 \bar{x}

🔹 3. Alternate Formula (Using Raw Sums)


β1=n∑xy−(∑x)(∑y)n∑x2−(∑x)2\beta_1 = \frac{n\sum xy - (\sum x)(\sum y)} {n\sum x^2 - (\sum x)^2} β0=∑y−β1∑xn\beta_0 = \frac{\sum y - \beta_1 \sum x}{n}

🔹 When to Use Which?

MethodWhen to Use
Mean form    Easier for understanding
Raw sum form    Faster in exams
Covariance form    Conceptual questions

​

🔹 Intuition 

👉 Simple linear regression tries to:

  • Fit a line through data points
  • Capture trend between variables

Assumptions of Simple Linear Regression

  1. Linearity
    • Relationship between xx and yy is linear
  2. Independence
    • Observations are independent
  3. Constant variance (Homoscedasticity)
    • Error variance is constant
  4. Normality of errors
    • Errors are normally distributed

🔹Example

📌 Salary Prediction

  • Input: Years of experience (xx)
  • Output: Salary (yy)

Model:

Salary=β0+β1(Experience)\text{Salary} = \beta_0 + \beta_1 (\text{Experience})

👉 Used widely in real-world ML tasks

🔹 Steps in Applying SLR

  1. Collect data
  2. Plot data (scatter plot)
  3. Choose linear model
  4. Define loss function
  5. Estimate parameters
  6. Evaluate model

🔹 Advantages

  • Simple and interpretable
  • Fast computation
  • Works well for linear relationships

🔹Limitations

  • Cannot model non-linear patterns
  • Sensitive to outliers
  • Assumes linearity

📝 Summary 

  • Model:
y=β0+β1xy = \beta_0 + \beta_1 x
  • Goal:
min⁡∑(yi−y^i)2\min \sum (y_i - \hat{y}_i)^2
  • Output: Continuous value

Comments

Post a Comment

Popular posts from this blog

Machine Learning PCCST503 Semester5 KTU CS 2024 Scheme - Dr Binu V P

About Me  - Dr Binu V P Syllabus Machine Learning PCCST503 KTU 2024 Scheme Model Question Paper Machine Learning PCCST503 Semester 5 KTU CS 2024 Scheme Do the programs to better understand the concept. Machine Learning Lab PCCSL508 Module -I Introduction to Machine Learning Introduction to ML Distinguishing Machine Learning from Traditional Programming Machine Learning Paradigms Supervised UnSupervised Semi-Supervised Reinforcement Stages of Machine Learning: From Data Collection to Deployment Parameter Estimation Introduction to Parameter Estimation in Machine Learning Maximum Likelyhood Estimation.(MLE) Maximum A Posteriori Estimation.(MAP) Numerical Problems MLE and MAP Supervised Learning Feature Representation and Problem Formulation Role of loss functions and optimization  Regression Introduction to Regression   Linear regression with one variable  (  lab experiment ) Example Problems- simple linear regression - University Questions Evaluation Metri...
Read more

Introduction to Machine Learning (ML)

  📘 Introduction to Machine Learning (ML) 🔹 What is Machine Learning? Machine Learning is a subfield of Artificial Intelligence that focuses on developing algorithms that allow computers to learn patterns from data and make decisions or predictions without explicit programming . It focuses on developing algorithms and models that allow machines to: Identify patterns in data Make decisions with minimal human intervention Predict future outcomes based on past experiences By continuously learning from new data, machine learning systems become more accurate and efficient over time. 📌 Formal Definition  A computer program is said to learn from experience E with respect to some task T and performance measure P if its performance at task T , as measured by P , improves with experience E . 🔹 Why Machine Learning? Traditional programming: Input + Program → Output Machine Learning paradigm: Input + Output → Model (learned program) 👉 Instead of writing...
Read more

Distinguishing Machine Learning from Traditional Programming

Image
  📘 Distinguishing Machine Learning from Traditional Programming 🔹 1. Fundamental Idea 🧾 Traditional Programming In conventional programming, developers write explicit instructions for the system to follow. The programmer explicitly writes rules/logic System follows deterministic instructions Input data is processed using predefined rules The output is based strictly on those rules 👉 Flow: Input + Program → Output 🤖 Machine Learning In machine learning, the system learns patterns from data instead of being explicitly programmed. Programmer provides  data + learning algorithm The model is trained using data The system  learns rules automatically from data It makes predictions or decisions based on learned patterns 👉 Flow: Input + Output → Learning Algorithm → Model Model + New Input → Predicted Output 🔹 2. Key Difference in Approach Aspect      Traditional Programming Machine Learning Input method Logic creation   ...
Read more