Bagging & Boosting

5KB

SAS_Studio_Key_Terms_Study_Guide.pdf

PDF

Open

🔁 What Are Bootstrap Samples?

Bootstrap samples are random datasets created by sampling with replacement from the original dataset.

---

🔍 Key Idea

• You take the original dataset of size n.

• You randomly draw n data points, with replacement.

• This means the same data point can appear more than once, and some may not appear at all.

---

🎯 Why Bootstrap?

It allows us to:

• Create multiple versions of the training dataset.

• Train diverse models (especially useful in ensemble methods like bagging).

• Estimate uncertainty, variance, and confidence intervals in statistics.

---

📦 Example (Original Dataset: 5 Points)

Original dataset: [A, B, C, D, E]

Bootstrap sample (random draw with replacement): [A, C, D, D, B] Notice:

• D appears twice.

• E is missing.

You could generate multiple such bootstrap samples.

---

🧠 In Bagging

If you're building 10 trees:

• Generate 10 bootstrap samples (each the same size as the original data).

• Train 10 trees—one on each sample.

• Aggregate their predictions.

---

📌 Summary

A bootstrap sample is a dataset generated by sampling with replacement from the original dataset. It’s the backbone of the bagging technique to ensure model diversity and robustness.

🔁 Bagging (Bootstrap Aggregating) – Explained with Example

🎯 Goal:

To reduce overfitting and improve accuracy by combining multiple models trained on different versions of the data.

---

👣 Step-by-Step Example:

📊 Suppose you have a small dataset:

ID	Feature (X)	Label (Y)
1	2	No
2	4	No
3	6	Yes
4	8	Yes
5	10	Yes

---

🧪 Step 1: Create Bootstrap Samples (with replacement)

Let’s say we want to build 3 models, so we create 3 bootstrap samples.

Bootstrap Sample 1: → Randomly pick 5 rows from the dataset (with replacement) [Row 2, Row 4, Row 4, Row 5, Row 1]

Bootstrap Sample 2: [Row 3, Row 1, Row 2, Row 5, Row 5]

Bootstrap Sample 3: [Row 4, Row 2, Row 3, Row 2, Row 1]

⚠️ Notice: Some rows appear multiple times; some are left out.

---

🏗 Step 2: Train a model on each sample

Use a weak learner (like a decision tree) to train 3 separate models on each bootstrap sample.

---

🔮 Step 3: Make Predictions and Aggregate

Suppose you want to predict the label for X = 7.

Each model gives a prediction:

• Model 1: Yes

• Model 2: No

• Model 3: Yes

🧮 Final Bagging Prediction = Majority Vote → Yes

---

🎁 Key Benefits of Bagging

✅ Advantage	📘 Why It Helps
Reduces Variance	Different models won't overfit the same way
Improves Accuracy	Combines the strengths of all models
Handles Noise	Averaging reduces the impact of outliers

---

💡 Real-world Analogy:

Imagine asking 5 doctors (models) for a diagnosis (prediction), each trained at different hospitals (bootstrap samples). Instead of relying on one doctor, you take a vote — increasing your chance of a correct outcome.

---

🧠 Bagging in Practice

The most popular bagging-based algorithm is:

🌲 Random Forest = Bagging + Decision Trees + Random Feature Selection

⚡ Boosting – Intuition & Explanation

🎯 Goal:

To convert weak models (that barely perform better than random guessing) into a strong ensemble by training them sequentially, where each new model learns from the mistakes of the previous ones.

---

🔍 Key Differences from Bagging:

Bagging	Boosting
Models trained independently	Models trained sequentially
Averages predictions	Adds corrections step-by-step
Reduces variance	Reduces bias
Example: Random Forest	Example: AdaBoost, XGBoost

---

👣 Step-by-Step Boosting Example (Using AdaBoost)

Let’s say you’re predicting if students pass or fail based on their study hours.

🎓 Small Dataset:

Student	Study Hours	Result
A	1	Fail
B	2	Fail
C	3	Pass
D	4	Pass
E	5	Pass

---

Step 1: Assign Equal Weights

Each record gets equal weight initially: Weight = 1/5 = 0.2

---

Step 2: Train First Weak Learner (e.g., a stump)

Let’s say the model says:

• If Study Hours > 2.5, then Pass, else Fail

It predicts correctly for C, D, E, but incorrectly for A, B.

🟥 Mistakes: A, B 🔁 Boosting increases weights for A and B.

---

Step 3: Train Second Learner

Now the model focuses more on A and B due to their higher weights.

It tries to correct the previous model's mistakes.

Maybe this time the model uses:

• If Study Hours > 1.5, then Pass

Now it might still be wrong on some, so the process repeats again.

---

Step 4: Combine All Models

Each model is given a weight (based on its accuracy), and predictions are combined using weighted majority vote (classification) or weighted average (regression).

---

🔮 Final Prediction

Instead of using just one model, boosting blends multiple weak learners where:

• Model 1 learns basic rules.

• Model 2 corrects Model 1’s errors.

• Model 3 corrects Model 2’s errors.

• ...

The final prediction is more accurate than any single model.

---

💡 Real-world Analogy:

Imagine a student learning math:

• First tries practice test → gets some questions wrong.

• Focuses on mistakes → studies those topics.

• Retakes test → gets better.

• Repeats this cycle.

Eventually, the student (like the boosted model) becomes an expert.

---

🔥 Common Boosting Algorithms:

• AdaBoost – Adjusts weights on misclassified points

• Gradient Boosting – Learns from residual errors using gradients

• XGBoost – Optimized Gradient Boosting (fast and powerful)

• LightGBM, CatBoost – Efficient versions for big data

---

🧠 Summary:

Boosting builds models sequentially, where each model focuses on the previous model’s errors, leading to a stronger final prediction.