Tree Based Models

Random Forest

Introduction

  • Random Forest can be used for

    • Classification
    • Regression

  • A Random Forest model is an ensamble model consisting of many (hundreds or thousands) of slightly different Decision Trees.

  • The majority predictions (majority vote) from all Decision Trees becomes the prediction of the Random Forest for classification»

  • The average predictions from all Decision Trees becomes the prediction of the Random Forest for regression»

The Idea Behind Random Forest

The idea that a combination of weak predictors can lead to a strong prediction is analogous to the wisdom of crowds phenomenon described in Galton 1907:

Visitors at a stock and poultry exhibition in England submitted guesses about the weight of an ox. Although most of the visitors were off with their predictions, surprisingly the mean of all predictions was very close to the real weight of the ox.»

How to ensure that Decision Trees are (slightly) Different

  • Random Subspace Method: Each decision tree uses only a random selection of the predictors.
    • As a rule of thumb: If we have \(m\) predictors, each decision tree uses only \(\sqrt(M)\) predictors (rounded down if needed).
    • In the case here, where we have \(7\) predictors, each decision tree uses only \(2\) randomly chosen predictors (\(\sqrt(7)=2.65\) rounded down to \(2\))

  • Bagging: Each decision tree is presented with slightly different training data: We draw from the original training data with replacement to step-wise generate a new training dataset (Bootstrapping).
    • The new training dataset has the same size as the original dataset.
    • Generate a bootstrap dataset for every decision tree.

Example for Bagging with Bootstapping

Creating 5 Bootstrap Training Datasets DataTrain1, DataTrain2, DataTrain3, DataTrain4, DataTrain5 from original dataset DataTrain.

Original Training Data:

Code 
DataTrain= tibble(ID=seq(1:7),
                  name=c("Adam","Berta","Carlos","Dora","Ernst","Fiola","Gert"),
                  Sex=c("male","female","male","female","male","female","male"),
                  Age=c(22,54,32,21,32,87,18),
                  Weight=c(145,180,167,197,221,146,230))
kbl(DataTrain)
ID name Sex Age Weight
1 Adam male 22 145
2 Berta female 54 180
3 Carlos male 32 167
4 Dora female 21 197
5 Ernst male 32 221
6 Fiola female 87 146
7 Gert male 18 230

1st Bootstrap Sample:

Code 
set.seed(777)
DataTrain1=sample_n(DataTrain, 7, replace = TRUE)
kbl(DataTrain1)
ID name Sex Age Weight
1 Adam male 22 145
2 Berta female 54 180
4 Dora female 21 197
6 Fiola female 87 146
6 Fiola female 87 146
3 Carlos male 32 167
4 Dora female 21 197

2nd Bootstrap sample:

Code 
DataTrain2=sample_n(DataTrain, 7, replace = TRUE)
kbl(DataTrain2)
ID name Sex Age Weight
2 Berta female 54 180
4 Dora female 21 197
1 Adam male 22 145
1 Adam male 22 145
4 Dora female 21 197
4 Dora female 21 197
2 Berta female 54 180

3rd Bootstrap sample:

Code 
DataTrain3=sample_n(DataTrain, 7, replace = TRUE)
kbl(DataTrain3)
ID name Sex Age Weight
7 Gert male 18 230
6 Fiola female 87 146
5 Ernst male 32 221
4 Dora female 21 197
7 Gert male 18 230
7 Gert male 18 230
7 Gert male 18 230

4th Bootstrap sample:

Code 
DataTrain4=sample_n(DataTrain, 7, replace = TRUE)
kbl(DataTrain4)
ID name Sex Age Weight
6 Fiola female 87 146
2 Berta female 54 180
6 Fiola female 87 146
2 Berta female 54 180
7 Gert male 18 230
5 Ernst male 32 221
6 Fiola female 87 146

5th Bootstrap sample:

Code 
DataTrain5=sample_n(DataTrain, 7, replace = TRUE)
kbl(DataTrain5)
ID name Sex Age Weight
2 Berta female 54 180
6 Fiola female 87 146
1 Adam male 22 145
4 Dora female 21 197
1 Adam male 22 145
7 Gert male 18 230
2 Berta female 54 180

Real World Data with a Decision Tree

Predicting vaccination rates in the U.S. based on data from September 2021:

  • Outcome variable: Percentage of fully vaccinated (two shots) people (\(PercVacFull\)).

  • Data from 2,630 continental U.S. counties.»

Real World Data with a Decision Tree — Predictor Variables

  • Race/Ethnicity:

    • Counties’ proportion African Americans (\(PercBlack\)),
    • Counties’ proportion Asian Americans (\(PercAsian\)), and
    • Counties’ proportion Hispanics (\(PercHisp\))

  • Political Affiliation (Presidential election 2020):

    • Counties’ proportion Republican votes (\(PercRep\))

  • Age Groups in Counties:

    • Counties’ proportion young adults (20-25 years); \(PercYoung25\)

    • Counties’ proportion older adults (65 years and older); \(PercOld65\)

  • Income related:

    • Proportion of households receiving food stamps (\(PercFoodSt\))»

Loading the Data and Assigning Training and Testing Data

Code 
DataVax=import("https://ai.lange-analytics.com/data/DataVax.rds") %>%   
        select(County, State, PercVacFull, PercRep,
              PercAsian, PercBlack, PercHisp,
              PercYoung25, PercOld65, PercFoodSt)

set.seed(2021)
Split85=DataVax %>% initial_split(prop = 0.85,
                                 strata = PercVacFull,
                                 breaks = 3)

DataTrain=training(Split85) %>% select(-County, -State)
DataTest=testing(Split85) %>% select(-County, -State)
kbl(head(DataVax))
County State PercVacFull PercRep PercAsian PercBlack PercHisp PercYoung25 PercOld65 PercFoodSt
Baldwin AL 0.504 0.7506689 0.0092 0.0917 0.0456 0.0634185 0.2593257 0.0682006
Barbour AL 0.416 0.4911710 0.0048 0.4744 0.0436 0.0734645 0.2332464 0.2637166
Chambers AL 0.315 0.7063935 0.0112 0.3956 0.0238 0.0646534 0.2595095 0.1517111
Cherokee AL 0.318 0.8319460 0.0025 0.0460 0.0159 0.0525455 0.2786219 0.1626076
Choctaw AL 0.648 0.7445596 0.0013 0.4255 0.0041 0.0612416 0.2802946 0.2118391
Cleburne AL 0.319 0.8402859 0.0002 0.0275 0.0246 0.0594866 0.2486424 0.1378972

Creating Model Design, Recipe, and Fitted Workflow (default hyper-parameters)

Code 
DefaultMtry=floor(sqrt(7)) # floor() rounds down

ModelDesignRandFor=rand_forest(trees=500, min_n=5, mtry = DefaultMtry) %>% 
                    set_engine("ranger") %>% 
                    set_mode("regression")
  
RecipeVax=recipe(PercVacFull~., data=DataTrain)

WfModelVax=workflow() %>% 
               add_model(ModelDesignRandFor) %>% 
               add_recipe(RecipeVax) %>% 
               fit(DataTrain)
Default value for mtry is: 2
══ Workflow [trained] ══════════════════════════════════════════════════════════
Preprocessor: Recipe
Model: rand_forest()

── Preprocessor ────────────────────────────────────────────────────────────────
0 Recipe Steps

── Model ───────────────────────────────────────────────────────────────────────
Ranger result

Call:
 ranger::ranger(x = maybe_data_frame(x), y = y, mtry = min_cols(~DefaultMtry,      x), num.trees = ~500, min.node.size = min_rows(~5, x), num.threads = 1,      verbose = FALSE, seed = sample.int(10^5, 1)) 

Type:                             Regression 
Number of trees:                  500 
Sample size:                      2234 
Number of independent variables:  7 
Mtry:                             2 
Target node size:                 5 
Variable importance mode:         none 
Splitrule:                        variance 
OOB prediction error (MSE):       0.01160306 
R squared (OOB):                  0.4549882

Metrics for Default Random Forest Model

Code 
DataTestWithPred=augment(WfModelVax, new_data=DataTest)
metrics(DataTestWithPred, truth=PercVacFull, estimate=.pred)
# A tibble: 3 × 3
  .metric .estimator .estimate
  <chr>   <chr>          <dbl>
1 rmse    standard      0.109 
2 rsq     standard      0.450 
3 mae     standard      0.0771

Tuning the Random Forest Model

On the next slide you find a link to an R script that you can use to tune the Random Forest model.

Try to beat the predictive quality of the default model we used here.

Tuning the Random Forest Model

ModelDesignRandForTree=rand_forest(trees=500, min_n=tune(), mtry = tune()) %>% 
                    set_engine("ranger") %>% 
                    set_mode("regression")

What to consider for the tuning grid?

  • trees=500: No need to tune because a high number of trees cannot lead to overlearning. This is because the Random Forest prediction is calculated from the average of the trees. However, a high number of trees slows the training (`trees=500’ was set; feel free to change it).

  • mtry=tune() is a hyper-parameter that should be tuned. It determines the number of predictors randomly considered for each decision tree. Recommended is \(\sqrt(7)=2.65\). Maybe you try 2, 3, 5.

  • min_n=tune() is a hyper-parameter that should be tuned. It determines the minimum number of observations inside a node required for a split into child nodes. If this number is not reached the node becomes a terminal node. Therefore, min_n=tune() indirectly determines the tree depth. Default is \(5\). Maybe you try 3, 5, 10, 30.

Below is a link for an R script that you can use to tune the Random Forest model.

See: Script to tune the Random Forest Model