# Polynomial Regression

Overlearning, Training, Testing, and Validation

## Overview

You will learn about:

- Overlearning in detail.

- Circumstances that make overlearning more likely to occur.

- Consequences of overlearning when predicting new data.

- Hyper-parameter tuning to avoid overlearning.

  - Validation
  - Cross Validation»

## Overlearning

If a model performs well when approximating the training data but does not perform well when it faces new data to predict outcomes.

Overlearning is one of the most pressing and still not fully solved problems in machine learning.»

## Circumstances that Can Lead to Overlearning

- If the training dataset does not have a sufficient number of observations.

- If the model considers many variables and thus contains many parameters to calibrate.

- If the underlying machine learning model is highly non-linear.»

## The Data

In what follows we use the Kings County Real Estate dataset.

Code

    library(tidymodels); library(rio); library(janitor)

    DataHousing = import("https://lange-analytics.com/AIBook/Data/HousingData.csv")%>%
      clean_names("upper_camel") %>%
      select(Price, Sqft=SqftLiving)

We want to demonstrate overlearning. Therefore, we ceate conditions that likely trigger overlearning. Consequently, we work only with a very small training dataset (20 observations=0.1% of total observations. All other observations become testing data:

Code

    set.seed(777)
    # initial_split(prop = 0.001, ...) randomly chooses 20 training observations
    Split001=DataHousing %>%
      initial_split(prop = 0.001, strata = Price, breaks = 5)
    DataTrain=training(Split001)
    DataTest=testing(Split001)

## Data Visualization

There seems to be a non-liner trend:

## Data Structure

        Price Sqft
    1  221900 1180
    2  538000 2570
    3  180000  770
    4  604000 1960
    5  510000 1680
    6 1230000 5420

## Polynomial Regression

Regular univariate prediction equation: \[ \widehat{Price}=\beta_1 Sqft+\beta_2 \]

Polynomial univariate prediction equation (degree 5):

\[\begin{eqnarray*} \widehat{Price}&=&\beta_1 Sqft+\beta_2 Sqft^2+\beta_3 Sqft^3 \\ && +\beta_4 Sqft^4+\beta_5 Sqft^5+\beta_6 \end{eqnarray*}\]

## Polynomial Regression

Polynomial univariate prediction equation (degree 5):

\[\begin{eqnarray*} \widehat{Price}&=&\beta_1 Sqft+\beta_2 Sqft^2+\beta_3 Sqft^3\\ && +\beta_4 Sqft^4+\beta_5 Sqft^5+\beta_6 \end{eqnarray*}\]

We create \(Sqft^2\), \(Sqft^3\), \(Sqft^4\), and \(Sqft^5\) as new variables in the data and treat them as they were separate variables in a multivariate regression.

This makes the regression linear in variables but non-linear in data.

Consequently we can OLS to find the optimal \(\beta s\).»

## HOW THE DATA WOULD LOOK LIKE

        Price Sqft    Sqft2        Sqft3        Sqft4        Sqft5
    1  221900 1180  1392400   1643032000 1.938778e+12 2.287758e+15
    2  538000 2570  6604900  16974593000 4.362470e+13 1.121155e+17
    3  180000  770   592900    456533000 3.515304e+11 2.706784e+14
    4  604000 1960  3841600   7529536000 1.475789e+13 2.892547e+16
    5  510000 1680  2822400   4741632000 7.965942e+12 1.338278e+16
    6 1230000 5420 29376400 159220088000 8.629729e+14 4.677313e+18

## Comparing Regular OLS and Polynominal Regression (degree=5)

Code to compare is linked in the footer of this slide.

## Polynomial Regression (degree=5) vs. Regular OLS

### Aproximation of the Training Data

## Polynomial Regression (degree=5) vs. Regular OLS

### Aproximation of the Training Data

## Polynomial Regression (degree=5) vs. Regular OLS

### Training and Testing Data Performance

\[\widehat{Price}=\beta_1 Sqft+\beta_2 Sqft^2+\beta_3 Sqft^3 + +\beta_4 Sqft^4 +\beta_5 Sqft^5 +\beta_{6}\]

## Polynomial Regression (degree=10) vs. Regular OLS

### Training and Testing Data Performance

\[\widehat{Price}=\beta_1 Sqft+\beta_2 Sqft^2+\beta_3 Sqft^3 + \cdots +\beta_{10} Sqft^{10}+\beta_{11}\]

## SUMMARY: POLYNOMIAL REGRESSION

- If we do not have enough data polynomial regression with a high degree might lead to overlearning

- What is the right degree?

- We could try different degrees (e.g., 2, 3, 4, … 10) and see which model performs best.

- Which data are we using to measure performance? Training data (overlearning) and testing data (cannot be used for model optimization) are out.

- We could split off data from the training dataset (validation data). These validation data are not used to calculate the βs. Instead, they are used to find the best setting for the degree of polynomial regression (aka hyper-parameter of polynomial regression).

## Hyper-Parameters

- Hyper-Parameters are parameters other than the \(\beta\) parameters, because they can not be optimized by the optimizer.

- Hyper-Parameters are like settings for a machine learning model such as the number of polynomials (e.g., \(Sqft^N\)) to be considered for polynomial regression. Another example are the number of \(k\) Nearest Neighbors.

- Hyper parameters often make a model more or less complex and thus influence the quality of predicting but also the chance of overlearning.»

## PROBLEMS OF SPLITTING VALIDATION DATA OFF THE TRAINING DATA

- Reduces data left over to train (finding optimal βs).

- If the training dataset is big enough this is no problem. Otherwise, it is a problem!

## CROSS VALIDATION (4-FOLD)

For each hyper-parameter setting:

1. Splits off validation data from training data (e.g. last quarter)

2. Runs the model and calculates metrics based on validation data.

3. Splits off validation data from training data (next quarter)

4. Repeats steps 2 – 3 four times.

We end up with four results for each hyper-parameter setting. We calculate the average of the four results as an result for that specific hyper parameter.

# CROSS VALIDATION FOR POLYNOMIAL REGRESSION AND THE KING COUNTY REALESTATE DATASET

## MORE REALISTIC DATASPLIT: 80% TRAINING, 20% TESTING

    set.seed(987)

    Split80=DataHousing %>%
      initial_split(prop = 0.8, strata = Price, breaks = 5)
    DataTrain=training(Split80)
    DataTest=testing(Split80)

    print(Split80)

    <Training/Testing/Total>
    <17289/4324/21613>

## Crossvalidation — The Idea Behind It

## 10 Steps to Create a Model, Tune it, and Predict

The 10 general steps are:

1. Generating training and testing data with `initial_split()`, `training()`, `testing()`.

2. Create recipe to determine predictor and outcome variables. Optionally add one or more `step_X()` commands.

3. Create model design and mark parameters to be `tune()` ed. without `fit()`

4. Create workflow by `add_recipe()` and `add_model()`

5. Create a hyper-parameter grid containing the hyper-parameter combinations to be validated.

6. Create cross validation datasets (aka resamples) containing the folds (use commands `vfold()`).

7. Tune the machine learning model with `tune_grid()` and track specific metrics defined by `metric_set()`. Runs all hyper-parameter combinations for all folds.

8. Extract the best hyper-parameter combination from the tuning results based on selected metrics (use `select_best()`)

9. Finalize the model by training it with the full set of training data with the best hyper-parameter combination (see `finalize_workflow() %>% fit()`).

10. Assessing predictive quality of the final model by using the testing dataset to predict (see `augment() %>% metrics()`).

## Run all 10 Steps to Tune the Real Estate Model

Code to run all 10 steps is linked in the footer of this slide.

## Exercise from AIBook 🤓

Use k-Nearest Neighbors to estimate the color of a wine

Click the link in the footer of this slide to start the exercise.

## Research Project 🤓

Click the link in the footer of this slide to download a skeleton of the R script for the research project.