Interpretation of Machine Learning Results

From Blackbox to Intertretation

image goes here

Types of Interpretation

Model specific
- Global
- Local
Model agnostic
- Global
- Local

Loading the Data

Code

library(tidymodels);library(rio);library(vip);library(kableExtra)
DataVaxFull=import("https://www.lange-analytics.com/AIBook/Data/DataVax.rds") %>% 
            mutate(RowNum=row.names(.))

DataVax= DataVaxFull%>%   
                              select(PercVacFull, PercRep,
                                           PercAsian, PercBlack, PercHisp,
                                           PercYoung25, PercOld65, 
                                           PercFoodSt, Population) %>% 
                            mutate(Population=frequency_weights(Population))
set.seed(2021)
Split85=DataVax %>% initial_split(prop = 0.85,
                                 strata = PercVacFull,
                                 breaks = 3)

DataTrain=training(Split85)
DataTest=testing(Split85)

Lime, Shs, and the randomized variable approach (forgot name)

Interpreting Coefficients and t-Values from Linear Regresiion

We use the same training and testing data as created in the prevha

Code

ModelDesignLinRegr= linear_reg() %>% 
  set_engine("lm") %>%   
  set_mode("regression")

RecipeVax=recipe(PercVacFull~., data=DataTrain)

set.seed(2021) 
WfModelVax=workflow() %>% 
          add_model(ModelDesignLinRegr)%>% 
          add_recipe(RecipeVax) %>% 
          add_case_weights(Population) %>% 
          fit(DataTrain)
kbl(tidy(WfModelVax))

term	estimate	std.error	statistic	p.value
(Intercept)	0.9433753	0.0240164	39.2804433	0.0000000
PercRep	-0.5515592	0.0164025	-33.6264704	0.0000000
PercAsian	0.0407740	0.0435763	0.9356931	0.3495327
PercBlack	-0.3771981	0.0207927	-18.1408963	0.0000000
PercHisp	0.0645242	0.0136853	4.7148525	0.0000026
PercYoung25	-0.5999597	0.1060435	-5.6576772	0.0000000
PercOld65	0.1945055	0.0608728	3.1952800	0.0014165
PercFoodSt	-0.1887113	0.0503212	-3.7501339	0.0001813

Variable Importance Permutations:

Use Out-of-Bag data from each tree.
Predict with all variables and record $MSE$ .
Scramble values of first variable and record $MSE$ .
The (standardized; with STD) difference between the $MSE$ and the $MSE$ of the scrambled version is the variable importance for the first variable.
Repeat steps 3 – 4 for all variable.
Plot VIP

Variable Importance Plot – Permutations:

Variable Importance Plots for Tree-Based Models

Code

ModelDesignRandFor=rand_forest(trees=1000, min_n=5, mtry=2) %>% 
  set_engine("ranger", num.threads = NumberOfCores, importance="impurity") %>%   
  set_mode("regression")

RecipeVax=recipe(PercVacFull~., data=DataTrain)

set.seed(2021) 
WfModelVax=workflow() %>% 
          add_model(ModelDesignRandFor)%>% 
          add_recipe(RecipeVax) %>% 
          add_case_weights(Population) %>% 
          fit(DataTrain)
vip(extract_fit_parsnip(WfModelVax))

Shapley Values from a Game Theory Approach

Carsten’s Contribution

Carsten joins last: $\Delta P^{rofit}_{BGC}=5$
Carsten joins second: $\Delta ^{rofit}_{BC}=7$ or $\Delta ^{rofit}_{GC}=8$
Carsten joins first: $\Delta P^{rofit}_{C}=17$
Carsten’s average contribution (Shapley value):
$S^{hap}_C=\frac{1}{3}17+\frac{1}{6}7+\frac{1}{6}8+\frac{1}{3}5$

Shapley Values from a Game Theory Approach

Coalitions and Permutations

Number of coalitions: $N_{coal}= 2^k=2^3=8$
Number of joining scenarios: $k!=3!=1\cdot 2\cdot 3=6$ $BCG$ , $BGC$ ,
$CBG$ , $CGB$ ,
$GBC$ , $GCB$ ,

SHAPley vs SHAP

SHAPley Values estimate contribution of players
SHAP Values estimate contribution of variables
Otherwise, they are the same!

Shap Values for Orange County, CA (different Politic. Impact)

Code

library(DALEX)
ExplainerRandFor <- 
  DALEXtra::explain_tidymodels(
    WfModelVax, 
    data = DataVax, 
    y = DataVax$PercVacFull,
    label = "random forest",
    verbose = FALSE
  )

DataTrainPredictorVar=DataVax %>% select(-PercVacFull, -Population)


i=141
j=137
if (!DraftMode){
set.seed(777)
ShapValues1 = DALEX::predict_parts(
    explainer = ExplainerRandFor, 
    new_observation = DataTrainPredictorVar[i,], 
    type = "shap",
    B = 128)
ShapValues2 = DALEX::predict_parts(
    explainer = ExplainerRandFor, 
    new_observation = DataTrainPredictorVar[j,], 
    type = "shap",
    B = 128)}

County: Orange, CA, Pred. Vac.: 0.72

Pred. Vac. Rate (USA): 0.51

County: Merced, CA, Pred. Vac.: 0.52

Pred. Vac. Rate (USA): 0.51

Shap Values for Two Counties (different Asian Impact)

Code

i=135
j=149
if (!DraftMode){
set.seed(777)
ShapValues1 = DALEX::predict_parts(
    explainer = ExplainerRandFor, 
    new_observation = DataTrainPredictorVar[i,], 
    type = "shap",
    B = 128)
ShapValues2 = DALEX::predict_parts(
    explainer = ExplainerRandFor, 
    new_observation = DataTrainPredictorVar[j,], 
    type = "shap",
    B = 128)}

County: Marin, CA, Pred. Vac.: 0.84

Pred. Vac. Rate (USA): 0.51

County: San Joaquin, CA, Pred. Vac.: 0.84

Pred. Vac. Rate (USA): 0.51

Shap Values by Variable