Supervised Learning: Feature Representation & Problem Formulation

 

Supervised Learning: Feature Representation & Problem Formulation

πŸ”Ή What is Supervised Learning?

Supervised learning involves learning a mapping:

f:X→Yf: X \rightarrow Y

Where:

  • XX → input (features)
  • YY → output (target/label)

πŸ‘‰ Given labeled data:

D={(x1,y1),(x2,y2),...,(xn,yn)}D = \{(x_1,y_1), (x_2,y_2), ..., (x_n,y_n)\}

πŸ”ΉFeature Representation 

πŸ”Ή Definition

Feature representation is the process of converting raw data into a structured numerical format that a machine learning algorithm can use.

πŸ”Ή Feature Vector

Each data point is represented as:

x=[x1,x2,...,xd]x = [x_1, x_2, ..., x_d]

Where:

  • dd = number of features

πŸ‘‰ Called a feature vector

πŸ”ΉTypes of Features

🟦 1. Numerical Features

  • Continuous values
  • Example: height, weight

🟩 2. Categorical Features

  • Discrete categories
  • Example: color = {red, blue, green}

πŸ‘‰ Often converted using:

  • One-hot encoding

🟨 3. Ordinal Features

  • Ordered categories
  • Example: low < medium < high

πŸŸ₯ 4. Binary Features

  • 0 or 1
  • Example: spam/not spam

πŸ”Ή Feature Transformation

πŸ“Œ Why needed?

Raw data is often not directly usable.

Common transformations:

  • Normalization (scale values)
  • Standardization (mean = 0, variance = 1)
  • Encoding categorical variables
  • Log transformation

πŸ”Ή Feature Engineering

πŸ‘‰ Creating new features from existing data

Examples:

  • Age → age group
  • Date → day, month, year
  • Text → word counts (bag-of-words)

πŸ”ΉFeature Selection

πŸ‘‰ Choosing important features

Why?

  • Reduce complexity
  • Improve performance
  • Avoid overfitting

πŸ”Ή Feature Space

All feature vectors form:

Rd\mathbb{R}^d

πŸ‘‰ Called feature space

πŸ“Œ Insight:

  • Each example = point in feature space
  • Learning = finding patterns in this space

πŸ”Ή Problem Formulation (Core Concept)

πŸ”Ή Goal

Learn a function:

y=f(x)y = f(x)

That predicts output for new data.

πŸ”Ή Types of Supervised Problems

🟦 1. Classification

  • Output is discrete
  • Example:
    • Spam / Not spam
    • Disease / No disease

🟩 2. Regression

  • Output is continuous
  • Example:
    • House price
    • Temperature

πŸ”ΉHypothesis Space

Set of all possible models:

H={f(x;ΞΈ)}\mathcal{H} = \{f(x; \theta)\}

πŸ‘‰ Example:

  • Linear models
  • Decision trees
  • Neural networks

πŸ”ΉLoss Function

Measures error between:

  • True value y
  • Predicted value y^

Common Loss Functions:

πŸ“Œ For Regression:

(y−y^)2(MSE)(y - \hat{y})^2 \quad \text{(MSE)}

πŸ“Œ For Classification:

  • 0–1 loss
  • Cross-entropy loss

πŸ”Ή Training vs Testing

  • Training set → learn model
  • Test set → evaluate performance

πŸ”Ή Overfitting vs Underfitting

πŸ“Œ Overfitting

  • Model memorizes data
  • Poor generalization

πŸ“Œ Underfitting

  • Model too simple
  • Cannot capture patterns

πŸ”Ή Generalization

πŸ‘‰ Ability to perform well on unseen data

πŸ”ΉExample 

πŸ“Œ Problem: House Price Prediction

Step 1: Features

  • Area
  • Number of rooms
  • Location

πŸ‘‰ Feature vector:

x=[area,rooms,location]x = [\text{area}, \text{rooms}, \text{location}]

Step 2: Output

y=pricey = \text{price}

Step 3: Model

y=wTx+by = w^T x + b

Step 4: Loss

(y−y^)2(y - \hat{y})^2

Step 5: Learn parameters ( training )

  • Using MLE or gradient descent

Step 6: testing

  • Evaluate the performance

πŸ“ Summary (Quick Revision)

Feature Representation:

  • Converts raw data → numerical form
  • Includes:
    • Feature vectors
    • Encoding
    • Transformation

Problem Formulation:

  • Define:
    • Input X
    • Output Y
    • Model f
    • Model Loss L
    • Learn Parameters (Training- Optimize Loss function)
    • Testing (Evaluate Performance)

Comments

Post a Comment

Popular posts from this blog

Machine Learning PCCST503 Semester5 KTU CS 2024 Scheme - Dr Binu V P

About Me  - Dr Binu V P Syllabus Machine Learning PCCST503 KTU 2024 Scheme Model Question Paper Machine Learning PCCST503 Semester 5 KTU CS 2024 Scheme Do the programs to better understand the concept. Machine Learning Lab PCCSL508 Module -I Introduction to Machine Learning Introduction to ML Distinguishing Machine Learning from Traditional Programming Machine Learning Paradigms Supervised UnSupervised Semi-Supervised Reinforcement Stages of Machine Learning: From Data Collection to Deployment Parameter Estimation Introduction to Parameter Estimation in Machine Learning Maximum Likelyhood Estimation.(MLE) Maximum A Posteriori Estimation.(MAP) Numerical Problems MLE and MAP Supervised Learning Feature Representation and Problem Formulation Role of loss functions and optimization  Regression Introduction to Regression   Linear regression with one variable  (  lab experiment ) Example Problems- simple linear regression - University Questions Evaluation Metri...
Read more

Introduction to Machine Learning (ML)

  πŸ“˜ Introduction to Machine Learning (ML) πŸ”Ή What is Machine Learning? Machine Learning is a subfield of Artificial Intelligence that focuses on developing algorithms that allow computers to learn patterns from data and make decisions or predictions without explicit programming . It focuses on developing algorithms and models that allow machines to: Identify patterns in data Make decisions with minimal human intervention Predict future outcomes based on past experiences By continuously learning from new data, machine learning systems become more accurate and efficient over time. πŸ“Œ Formal Definition  A computer program is said to learn from experience E with respect to some task T and performance measure P if its performance at task T , as measured by P , improves with experience E . πŸ”Ή Why Machine Learning? Traditional programming: Input + Program → Output Machine Learning paradigm: Input + Output → Model (learned program) πŸ‘‰ Instead of writing...
Read more

Distinguishing Machine Learning from Traditional Programming

Image
  πŸ“˜ Distinguishing Machine Learning from Traditional Programming πŸ”Ή 1. Fundamental Idea 🧾 Traditional Programming In conventional programming, developers write explicit instructions for the system to follow. The programmer explicitly writes rules/logic System follows deterministic instructions Input data is processed using predefined rules The output is based strictly on those rules πŸ‘‰ Flow: Input + Program → Output πŸ€– Machine Learning In machine learning, the system learns patterns from data instead of being explicitly programmed. Programmer provides  data + learning algorithm The model is trained using data The system  learns rules automatically from data It makes predictions or decisions based on learned patterns πŸ‘‰ Flow: Input + Output → Learning Algorithm → Model Model + New Input → Predicted Output πŸ”Ή 2. Key Difference in Approach Aspect      Traditional Programming Machine Learning Input method Logic creation   ...
Read more