Interactive Project: Tune k-Nearest Neighbors to Detect Wine Color
Companion Exercises for Practical Machine Learning with R
The interactive R environment and the required packages must be loaded in your browser. This typically takes 1–2 minutes. You can monitor the progress in the bottom-right corner.
After packages are loaded, it still takes some time because the website must run a pre-tuning procedure. Please be patient!!!
When everything is loaded, you get a “Ready to roll” message below. Thank you for your patience!
Libraries already loaded in the background:
- tidyverse
- tidymodels
- kknn
Step 1 - Generating Training and Testing Data:
As before, you use the wine dataset and split the data into training (DataTrain) and testing data (DataTest). Since you use the same value in the set.seed() command, the (random) split will be identical to the one we used before with the \(k=4\) Nearest Neighbor model. The code below has been executed already.
library(tidymodels); library(rio); library(janitor); library(kableExtra)
DataWine=import("https://ai.lange-analytics.com/data/WineData.rds") |>
clean_names("upper_camel") |>
rename(Sulfur=TotalSulfurDioxide) |>
mutate(WineColor=as.factor(WineColor))
set.seed(876)
Split7030=initial_split(DataWine, prop=0.7, strata=WineColor)
DataTrain=training(Split7030)
DataTest=testing(Split7030)
kable(head(DataWine))| WineColor | Acidity | VolatileAcidity | CitricAcid | ResidualSugar | Chlorides | FreeSulfurDioxide | Sulfur | Density | PH | Sulphates | Alcohol | Quality |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| red | 10.8 | 0.320 | 0.44 | 1.6 | 0.063 | 16 | 37 | 0.99850 | 3.22 | 0.78 | 10.00 | 6 |
| white | 6.4 | 0.310 | 0.39 | 7.5 | 0.040 | 57 | 213 | 0.99475 | 3.32 | 0.43 | 10.00 | 5 |
| white | 9.4 | 0.280 | 0.30 | 1.6 | 0.045 | 36 | 139 | 0.99534 | 3.11 | 0.49 | 9.30 | 5 |
| white | 8.2 | 0.220 | 0.36 | 6.8 | 0.034 | 12 | 90 | 0.99440 | 3.01 | 0.38 | 10.50 | 8 |
| white | 6.4 | 0.290 | 0.44 | 3.6 | 0.197 | 75 | 183 | 0.99420 | 3.01 | 0.38 | 9.10 | 5 |
| red | 6.7 | 0.855 | 0.02 | 1.9 | 0.064 | 29 | 38 | 0.99472 | 3.30 | 0.56 | 10.75 | 6 |
Step 2 - Create a Recipe:
Here, you will create a recipe and store it in the R object RecipeWine. Use step_rm() to remove the predictor variable \(Quality\) because it is not related to \(WineColor\), and the command step_normalize() to normalize all remaining predictors. Please, substitute <THESE> placeholders accordingly and execute the code.
Note that the data frame DataTrain has already been loaded in the background:
Here is a hint:
The step to remove a variable such as Quality is called step_rm().
Here is the Solution:
RecipeWine=recipe(WineColor~., data=DataTrain) |>
step_rm(Quality) |>
step_normalize(all_predictors())
print(RecipeWine)
RecipeWine=recipe(WineColor~., data=DataTrain) |>
step_rm(Quality) |>
step_normalize(all_predictors())
print(RecipeWine)Step 3 - Create a Model Design:
Next, you will create the model design and store it into the R object ModelDesignKNN. Since you plan to tune the argument neighbors= (stands for the \(k\) in Nearest Neighbors), you have to add it as an argument into the nearest_neighbor() command by substituting ______ with the argument and its value. Remember that you cannot set neighbors to a specific numerical value because you want to tune the hyper-parameter neighbors later on. Therefore you have to assign the placeholder tune() to the argument neighbors.
Here is a hint:
Instead of assigning a number to the hyper-parameter neighbors use the command tune() as a place holder.
Here is the Solution:
ModelDesignKNN=nearest_neighbor(neighbors=tune()) |>
set_engine("kknn") |>
set_mode("classification")
print(ModelDesignKNN)
ModelDesignKNN=nearest_neighbor(neighbors=tune()) |>
set_engine("kknn") |>
set_mode("classification")
print(ModelDesignKNN)Step 4 - Add the RecipeWine and the Model Design to a Workflow:
The code block below adds the R object Recipe and the model design object ModelDesignKNN to a workflow model named TuneWFModelKNN. The R code has been executed already.
You can see in the printout above that the workflow is not finalized because the number of neighbors has not been set in the model design. The hyper-parameter neighbors is set to tune() instead. Later, in Step 7, tune() will be replaced in a trial and error process with different values for neighbors.
Step 5 - Create a Hyper-Parameter Grid:
When tuning is executed in Step 7, values reaching from 1 –15 for the hyper-parameter neighbors shall be tried out. You need to provide these values in a data frame column that is named the same as the hyper-parameter. Below, replace the ______ to define a column neighbors in the data frameParGrid. The column should contain values from 1 – 15:
Here is a hint:
Use the c() command to create a vector containing the numbers \(1 - 15\) either separated by a comma, or you can use a colon (:) to indicate from/to.
Here is the Solution:
ParGrid=data.frame(neighbors=c(1,2,3,4,5,6,7,8,9,10,11,12,13,14,15))
ParGrid
ParGrid=data.frame(neighbors=c(1,2,3,4,5,6,7,8,9,10,11,12,13,14,15))
ParGridStep 6 - Creating Resamples for Cross-Validation:
The values you have created above for \(k\) (hyper-parameter neighbors) will be evaluated later using five folds (resamples).
Each fold contains the complete training data, but different sections are used for training and assessment in each fold. Please create five folds below by substituting ______ accordingly.
The folds will be saved in the R object FoldsForTuning:
Here is a hint:
The argument v= determines the number of folds to be used for tuning.
Here is the Solution:
set.seed(123)
FoldsForTuning=vfold_cv(DataTrain, v=5, strata=WineColor)
print(FoldsForTuning)
set.seed(123)
FoldsForTuning=vfold_cv(DataTrain, v=5, strata=WineColor)
print(FoldsForTuning)Step 7 - Tune the Workflow and Train All Models:
Now it is time to run the tuning procedure using the tune_grid() command. Be patient because it will take some time to fully execute. Since we have to try out 15 parameters and use five folds for each model, the tune_grid() command has to fit 75 models \((15\cdot 5=75)\).
Please substitute ______ with a list of metrics to be calculated. Use the metrics accuracy, specificity, and sensitivity.
After the tuning is finished, all results are stored in the R object TuneResults, and they can be evaluated by different metrics commands.
For example the command autoplot() provides a diagrammatic overview of the results for all three metrics.
Here is a hint:
The command metric_set takes a comma-separated list of metrics to be evaluated.
Consider that you want to evaluate the metrics accuracy, specificity, sensitivity.
Here is the Solution:
TuneResults=TuneWFModelKNN |> tune_grid(FoldsForTuning, grid=ParGrid ,
metrics=metric_set(accuracy, specificity, sensitivity))
autoplot(TuneResults)
TuneResults=TuneWFModelKNN |> tune_grid(FoldsForTuning, grid=ParGrid ,
metrics=metric_set(accuracy, specificity, sensitivity))
autoplot(TuneResults)The three graphs that you created in the exercise above show for each metric the related accuracy, specificity, and sensitivity for all tried out hyper-parameters (neighbors). The results for the five folds are averaged.
You can see that neighbors values between 1 and 4 produce the best results for accuracy (predicting red and white wines (positive and negative class) correctly) and sensitivity (predicting red wines (positive class) correctly). If you look at specificity (predicting white wines (negative class) correctly), you can see that \(k=5\) produces the best result. However, \(k=5\) also leads to a sharp decrease in sensitivity and a decrease in accuracy as well.
It seems reasonable to use the best result for accuracy, which means choosing a \(k\) between 1 and 4.
Step 8 - Extract the Best Hyper-Parameter(s):
All assessment results for the specified metrics are stored in the tuning object TuneResults. Choose the metric accuracy by substituting ______ accordingly. Afterward, when you execute the code the select_best() command extracts the best hyper-parameter (value for neighbors) for the metric you specified:
Here is a hint:
Substitute ______ with accuracy and make sure to not delete the quotes.
Here is the Solution:
BestHyperPar=select_best(TuneResults, metric="accuracy")
print(BestHyperPar)
BestHyperPar=select_best(TuneResults, metric="accuracy")
print(BestHyperPar)As you will see in the printout after executing the R code, the best value for the hyper-parameter neighbors based on accuracy is \(neighbors=\) 1. It is saved in the data frame BestHyperPar.
You can try other metrics in the code block above and see if the result changes.
Step 9 - Finalize and Train the Best Workflow Model:
The code block below is executed already. The finalize_workflow() command used the value from BestHyperPar to substitute the tune() placeholder in the R object TuneWFModelKNN (created in Steps 2 – 4). The hyper-parameter is now set to \(neighbors=\) 1.
At the end of the printout above, you can see that \((k)\) was set to neighbors=1 rather than tune(). You can also see that WFModelBest is fitted to the training data because the “Minimum missclassification” is reported (based on the training data).
Step 10: Assess Prediction Quality Based on the Testing Data:
Since WFModelBest is a fitted model, you can use it for predictions. In this last step, you use the augment() command to predict WineColor. The augment() command will then add the prediction results as column .pred_class to the testing data.
The conf_mat() command compares the predictions in column .pred_class to the true values to create a confusion matrix. But before, you have to substitute ______ with the variable name (column name) for the outcome variable.
Here is a hint:
The (column) name for the outcome variable is WineColor and the predictions are stored in the column .pred_class.
Here is the Solution:
DataTestWithPredBestModel=augment(WFModelBest,DataTest)
conf_mat(DataTestWithPredBestModel, truth=WineColor, estimate=.pred_class)
DataTestWithPredBestModel=augment(WFModelBest,DataTest)
conf_mat(DataTestWithPredBestModel, truth=WineColor, estimate=.pred_class)You can see in the confusion matrix that based on the testing data from the 480 red wines only 6 were misclassified as being white. Likewise, from the 480 white wines, only 7 were misclassified. Given that the classification was only based on the chemical properties of the wines, the results are excellent.
Most likely, a true wine expert might have reached a similar impressive result, but it would have taken them a long time to classify 960 wines.