Interactive Project: Combine Regularization and Tuning to Optimize Polynomial Regression to Predict House Prices
Companion Exercises for Practical Machine Learning with R
The interactive R environment and the required packages must be loaded in your browser. This typically takes 1–2 minutes. You can monitor the progress in the bottom-right corner.
After packages are loaded, it still takes some time because the website must run a pre-tuning procedure. Please be patient!!!
When everything is loaded, you get a “Ready to roll” message below. Thank you for your patience!
In Section 7.7.3 of Practical Machine Learning with R we already worked with a univariate Polynomial Model to demonstrate regularization and to evaluate overfitting scenarios. In the real world, data scientists usually use multivariate models to include more than one variable in their analysis. Therefore, this interactive exercise will teach you how to use a multivariate approach.
1 The Data
You will again predict house prices using the King County House Sale dataset (Kaggle 2015). In the code block below, the dataset is loaded. Instead of selecting only one variable to predict the price of a house \((Price_i)\), we use the select() command to choose the number of bedrooms \((Bedr_i)\) and the year a house was built \((Year)\), together with the house’s square footage \((Sqft)\). We filter out houses with a price of more than $1,300,000 and square footage 4,500 sqft or more, which are outliers:
Note when creating the training dataset, we use the argument prop=0.005, leading to a training dataset with about 100 observations. This size is small, but in general realistic for a training dataset. However, given a total of 20,699 available observations, we could have used the more common prop=0.7. In that case, we would end up with 14,489 observations for the training dataset. With such a large training dataset, the unregularized model we use later in this section would likely not lead to overfitting, and thus regularization would no longer be needed — which would defeat the purpose of this interactive exercise.
2 Unregularized Benchmark Model
As a benchmark, we begin with an unregularized model. In this case, a multivariate Degree-2 Polynomial Model. It includes the variables \(Sqft\), \(Bedr\), and \(Year\). They are used in their original form, squared, and as various interaction terms:
\[\begin{eqnarray} Price&=& \beta_1 Sqft+\beta_2 Bedr+\beta_3 Year\nonumber \\ &+&\beta_4 Sqft^2+\beta_5 Bedr^2+\beta_6 Year^2\nonumber \\ &+&\beta_7 Sqft Bedr+\beta_8 Sqft Year+\beta_9 Bedr Year\nonumber \\ &+&\beta_{10} Sqft^2 Bedr+\beta_{11} Sqft Bedr^2+\beta_{12} Sqft^2 Bedr^2\nonumber \\ &+&\beta_{13} Sqft^2 Year+\beta_{14} Sqft Year^2+\beta_{15} Sqft^2 Year^2\nonumber \\ &+&\beta_{16} Bedr^2 Year+\beta_{17} Bedr Year^2+\beta_{18} Bedr^2 Year^2 \end{eqnarray}\]As before, we use a recipe to generate and normalize the additional predictors:
Afterward, we create the model design (ModelDesignBenchmark), the workflow model (WFModelBenchmark), and we calibrate WFModelBenchmark to the training data:
When you execute the tidy() command in the code block below, you will see that all predictor variable parameters (\(\beta s\)), except the one for the intercept, are in the multi-million range, and some even in the multi-billion range (see column estimate).
When you execute the code block below, the model’s training and testing data predictions are compared to the true values (\(Price_i\)), and the related metrics are calculated:
After executing the code above, you will see that the mae for the training data is about 111,700 and the mae for the testing data is about 142,800. Such a low training mae, when compared to the testing mae, strongly indicates overfitting.
In the following section, you will use an Elastic-Net model to improve the predictive quality.
3 Regularized Elastic-Net Polynomial Model
When you substitute ______ after mixture= and penalty= in the code further below with values of your choice and execute, the model’s \(\beta\) values will be calibrated and printed. Also, the metrics for the testing data are calculated and printed.
Recall, if mixture=0, you will run a Ridge model; if mixture=1, you will run a Lasso model. For anything in between, you will run a mixed model. Also note, that the argument penalty= represents the penalty factor (\(\lambda\)).
Finding an appropriate value for penalty factor (\(\lambda\)) — the argument penalty= is more tricky. For the Ridge penalty factor, values between \(15{,}000\) and \(154{,}000{,}000\) will work. For the Lasso penalty factor, values between \(15\) and \(154{,}000\) will work.1
Try a few settings and see how the results are changing. For example, use mixture=1 (Lasso) and penalty=100000. Afterward, increase and decrease penalty= and leave mixture=1 constant. How are the results changing? Then try mixture=0 and leave penalty=100000 to see the different results for a Ridge model. You can also set mixture=0.5 or mixture=0.7 to try out different Elastic-Net models. Playing with different hyper-parameter values and observing the resulting \(\beta s\) and metrics will help you to better understand the \(Ridge\), \(Lasso\), and Elastic-Net approaches.
Here is a Hint:
Choose values for the hyper-parameters penalty= and mixture= to choose the intensity (penalty=) and the type of regularization Ridge (mixture=0), Lasso (mixture=1), or a mix betwen the two (e.g., mixture=0.5).
Here is one of many possible solutions. Try out others:
ModelDesignRegularized=linear_reg(penalty=100000 , mixture=1) |>
set_engine("glmnet") |>
set_mode("regression")
WFModelRegularized=workflow() |>
add_model(ModelDesignRegularized) |>
add_recipe(RecipeHousesMultivar) |>
fit(DataTrain)
print("Beta Values (see Column Estimate:)")
tidy(WFModelRegularized)
DataTestWithPredRegularized=augment(WFModelRegularized, DataTest)
print("Metrics for Testing Data:")
metrics(DataTestWithPredRegularized, truth=Price, estimate=.pred)
ModelDesignRegularized=linear_reg(penalty=100000 , mixture=1) |>
set_engine("glmnet") |>
set_mode("regression")
WFModelRegularized=workflow() |>
add_model(ModelDesignRegularized) |>
add_recipe(RecipeHousesMultivar) |>
fit(DataTrain)
print("Beta Values (see Column Estimate:)")
tidy(WFModelRegularized)
DataTestWithPredRegularized=augment(WFModelRegularized, DataTest)
print("Metrics for Testing Data:")
metrics(DataTestWithPredRegularized, truth=Price, estimate=.pred)In the code block above, you changed hyper-parameters and evaluated the effects based on the testing data to better understand regularization. For educational experiments that is perfectly OK.
However, for real research never use the testing data to find the best hyper-parameters. Otherwise, you end up with a model that is highly specialized to the testing data, but when used on new data in the production phase, it would likely perform very poorly — a form of overfitting.
Instead, use validation data derived from the training dataset to evaluate the performance of hyper-parameters. This is the underlying principle of tuning, and in the next section, you will tune the workflow model to improve performance.
4 Tuning the Elastic-Net Polynomial Model
In this section, you will use the 10-Step Template** from Section 6.6 of Practical Machine Learning with R to tune the hyper-parameters penalty and mixture.
Some of the steps are already setup and executed. Other steps you can adjust and execute. Please note, before you can execute Step 7 – Step 10, you need to execute Step 3, Step 4, Step 5, and Step 6. Otherwise you get an error.
- Step 1 - Generate Training and Testing Data:
-
The training and testing data have already been generated above.
- Step 2 - Create a Recipe:
-
The recipe to create the predictors for the multivariate Degree-2 Polynomial model (
RecipeHousesMultivar) was created in connection with the unregularized model and can be reused here. It establishes the predictors with various powers and the interaction terms. - Step 3 - Create a Model Design:
-
The model design from the previous section needs slight modifications. We have to add the
tune()placeholder for the hyper-parameters. Please substitute______in the code block below:
Here is a Hint:
Later, in Step 7 – Step 10, you will try different values for the hyper-parameters penalty and mixture.
Consequently, you cannot enter values here. In lieu of values, you enter a placeholder tune(), which will later be replaced by the values to be tried out.
Here is the Solution:
ModelDesignElastNetTune =linear_reg(penalty=tune(), mixture=tune()) |>
set_engine("glmnet") |>
set_mode("regression")
print(ModelDesignElastNetTune)
ModelDesignElastNetTune =linear_reg(penalty=tune(), mixture=tune()) |>
set_engine("glmnet") |>
set_mode("regression")
print(ModelDesignElastNetTune)- Step 4 - Add the Recipe and the Model Design to a Workflow:
-
As before, we add the recipe and the (modified) model design to the workflow:
- Step 5 - Create a Hyper-Parameter Grid:
-
Previously, when only one hyper-parameter was tuned, you added the values to be tried out in a data frame column named with the same name as the hyper-parameter. When you have more than one hyper-parameter, it gets a little more complicated because you want to try out different combinations of hyper-parameters.
If you decide to try out all combinations of the values from two or more hyper-parameters, you can use the expand.grid() command. The arguments of the expand.grid() command are the hyper-parameters you would like to try out. The assigned values to these arguments are the values you want to try out for each of the hyper-parameters. They are assigned to each argument as R vector objects. The expand.grid() command then generates all possible combinations for the values of the hyper-parameters.
For example, if you leave the code below unchanged (recommended for the first trial) you combine 8 penalty values with three mixture values, and you can use the expand.grid() command to generate 24 possible combinations (\(8 \cdot 3\)):
Hint: Leave the values unchanged for the first trial, but then use other values for subsequent trials. Beware, if the number of combination increases the execution time increases also in Step 7 – Step 10.
The R vector objects assigned to penalty= and mixture= are created with c() commands by listing the related values.
You can see in the printout above that expand.grid() generates all possible combinations of the penalty= values and the mixture= values. For the default values each of these 24 combinations will be tried out when you tune the workflow later on in Step 7 – Step 10.
The expand.grid() command is very helpful when you combine multiple hyper-parameters with several values each. Imagine you want to create a grid with all combinations of five hyper-parameters with ten values each, expand.grid() would create \(100{,}000\) parameter combinations (\(10\cdot 10\cdot 10\cdot 10\cdot 10=10^5\)) for you. Be advised, tuning \(100{,}000\) hyper-parameter combinations would use a lot of computing time.
In general, computing time increases exponentially with the number of hyper-parameters, while computing time increases proportionally when the values for each hyper-parameter increase. These facts must be considered when deciding how many hyper-parameters you want to tune and how many values you like to consider for each hyper-parameter.
- Step 6: - Create Resamples for Cross-Validation:
-
To create resamples for Cross-Validation, we use the
vfold()command. However, although usually recommended, creating ten Cross-Validation folds creates a problem. Given that we have only 101 observations, the validation set of each fold would consist only of about ten observations. Therefore, we will create sets of three folds (v=3), which leaves about \(30\) observation for each validation dataset. To offset the small number of folds, we repeat the process of creating folds ten times (see the argumentrepeats=10) creating a total of \(30\) folds:
Step 7 – Step 10:
Now, you are ready to finalize the tuning process in the interactive project. The training and testing data (from Step 1), the recipe (from Steps 2) have already been loaded in the background.
You need to execute Steps 3 – 6 before you can tune the model!
It takes a while until the tuning is finished. For the default settings in Steps 3 and 6 it takes somewhere between 2 and 7 minutes depending on your computer speed (the maximum allowable cpu time is set to 15 minutes). Enjoy a cup of coffee. For the default model the 24 parameter combinations have to be run for 12 cross validation folds. Meaning the model has to be optimized 288 times. Consider this when changing the parameters in Steps 3 and/or 6.
When R outputs the results, look carefully at the \(\beta\) parameters, the plot generated by autoplot() after Step 7, and the metrics for the testing data. Then change the c() commands that generate the hyper-parameter values for penalty= and mixture=. Afterward, execute the code again and see how the results change.
Please recall, that for this setup Ridge penalty factors between \(15{,}000\) and \(154{,}000{,}000\) will work and Lasso penalty factors between \(15\) and \(154{,}000\) will work.2
Footnotes
These recommendations are based on the 100 \(\lambda\) values that
glmnetanalyzes by default, depending on the model and the training data. Values outside these ranges might produce surprising results.↩︎These recommendations are based on the 100 \(\lambda\) values that
glmnettries by default, depending on the model and the training data. Values outside these ranges might produce surprising results.↩︎