Bagging & Boosting

5KB
SAS_Studio_Key_Terms_Study_Guide.pdf
PDF
Open

🔁 What Are Bootstrap Samples?

Bootstrap samples are random datasets created by sampling with replacement from the original dataset.

🔍 Key Idea

  • You take the original dataset of size n.

  • You randomly draw n data points, with replacement.

  • This means the same data point can appear more than once, and some may not appear at all.

🎯 Why Bootstrap?

It allows us to:

  • Create multiple versions of the training dataset.

  • Train diverse models (especially useful in ensemble methods like bagging).

  • Estimate uncertainty, variance, and confidence intervals in statistics.

📦 Example (Original Dataset: 5 Points)

Original dataset: [A, B, C, D, E]

Bootstrap sample (random draw with replacement): [A, C, D, D, B] Notice:

  • D appears twice.

  • E is missing.

You could generate multiple such bootstrap samples.

🧠 In Bagging

If you're building 10 trees:

  • Generate 10 bootstrap samples (each the same size as the original data).

  • Train 10 trees—one on each sample.

  • Aggregate their predictions.

📌 Summary

A bootstrap sample is a dataset generated by sampling with replacement from the original dataset. It’s the backbone of the bagging technique to ensure model diversity and robustness.

🔁 Bagging (Bootstrap Aggregating) – Explained with Example

🎯 Goal:

To reduce overfitting and improve accuracy by combining multiple models trained on different versions of the data.

👣 Step-by-Step Example:

📊 Suppose you have a small dataset:

ID

Feature (X)

Label (Y)

1

2

No

2

4

No

3

6

Yes

4

8

Yes

5

10

Yes

🧪 Step 1: Create Bootstrap Samples (with replacement)

Let’s say we want to build 3 models, so we create 3 bootstrap samples.

Bootstrap Sample 1: → Randomly pick 5 rows from the dataset (with replacement) [Row 2, Row 4, Row 4, Row 5, Row 1]

Bootstrap Sample 2: [Row 3, Row 1, Row 2, Row 5, Row 5]

Bootstrap Sample 3: [Row 4, Row 2, Row 3, Row 2, Row 1]

⚠️ Notice: Some rows appear multiple times; some are left out.

🏗 Step 2: Train a model on each sample

Use a weak learner (like a decision tree) to train 3 separate models on each bootstrap sample.

🔮 Step 3: Make Predictions and Aggregate

Suppose you want to predict the label for X = 7.

Each model gives a prediction:

  • Model 1: Yes

  • Model 2: No

  • Model 3: Yes

🧮 Final Bagging Prediction = Majority Vote → Yes

🎁 Key Benefits of Bagging

✅ Advantage

📘 Why It Helps

Reduces Variance

Different models won't overfit the same way

Improves Accuracy

Combines the strengths of all models

Handles Noise

Averaging reduces the impact of outliers

💡 Real-world Analogy:

Imagine asking 5 doctors (models) for a diagnosis (prediction), each trained at different hospitals (bootstrap samples). Instead of relying on one doctor, you take a vote — increasing your chance of a correct outcome.

🧠 Bagging in Practice

The most popular bagging-based algorithm is:

🌲 Random Forest = Bagging + Decision Trees + Random Feature Selection

⚡ Boosting – Intuition & Explanation

🎯 Goal:

To convert weak models (that barely perform better than random guessing) into a strong ensemble by training them sequentially, where each new model learns from the mistakes of the previous ones.

🔍 Key Differences from Bagging:

Bagging

Boosting

Models trained independently

Models trained sequentially

Averages predictions

Adds corrections step-by-step

Reduces variance

Reduces bias

Example: Random Forest

Example: AdaBoost, XGBoost

👣 Step-by-Step Boosting Example (Using AdaBoost)

Let’s say you’re predicting if students pass or fail based on their study hours.

🎓 Small Dataset:

Student

Study Hours

Result

A

1

Fail

B

2

Fail

C

3

Pass

D

4

Pass

E

5

Pass

Step 1: Assign Equal Weights

Each record gets equal weight initially: Weight = 1/5 = 0.2

Step 2: Train First Weak Learner (e.g., a stump)

Let’s say the model says:

  • If Study Hours > 2.5, then Pass, else Fail

It predicts correctly for C, D, E, but incorrectly for A, B.

🟥 Mistakes: A, B 🔁 Boosting increases weights for A and B.

Step 3: Train Second Learner

Now the model focuses more on A and B due to their higher weights.

It tries to correct the previous model's mistakes.

Maybe this time the model uses:

  • If Study Hours > 1.5, then Pass

Now it might still be wrong on some, so the process repeats again.

Step 4: Combine All Models

Each model is given a weight (based on its accuracy), and predictions are combined using weighted majority vote (classification) or weighted average (regression).

🔮 Final Prediction

Instead of using just one model, boosting blends multiple weak learners where:

  • Model 1 learns basic rules.

  • Model 2 corrects Model 1’s errors.

  • Model 3 corrects Model 2’s errors.

  • ...

The final prediction is more accurate than any single model.

💡 Real-world Analogy:

Imagine a student learning math:

  • First tries practice test → gets some questions wrong.

  • Focuses on mistakes → studies those topics.

  • Retakes test → gets better.

  • Repeats this cycle.

Eventually, the student (like the boosted model) becomes an expert.

🔥 Common Boosting Algorithms:

  • AdaBoost – Adjusts weights on misclassified points

  • Gradient Boosting – Learns from residual errors using gradients

  • XGBoost – Optimized Gradient Boosting (fast and powerful)

  • LightGBM, CatBoost – Efficient versions for big data

🧠 Summary:

Boosting builds models sequentially, where each model focuses on the previous model’s errors, leading to a stronger final prediction.

PreviousChapter 08NextLectures

Last updated