supervised_learning_models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Supervised Learning Model Comparison\n",
    "\n",
    "![A piggybank with 401k written beside it](https://imgur.com/2xg0qOu.jpg)\n",
    "*Getty Images*\n",
    "\n",
    "The data science process:\n",
    "\n",
    "1. Define the problem.\n",
    "2. Gather the data.\n",
    "3. Explore the data.\n",
    "4. Model the data.\n",
    "5. Evaluate the model.\n",
    "6. Answer the problem.\n",
    "\n",
    "Today we'll be focused on creating and comparing regression and classification models. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Define the problem.\n",
    "\n",
    "Scenario:\n",
    "\n",
    "We're a data scientist with a financial services company. Specifically, we want to leverage data in order to identify potential customers.\n",
    "\n",
    "\"401(k)s\" or \"IRAs,\" these are two types of retirement accounts. Very broadly speaking:\n",
    "- You can put money for retirement into both of these accounts.\n",
    "- The money in these accounts gets invested and hopefully has a lot more money in it when you retire.\n",
    "- These are a little different from regular bank accounts in that there are certain tax benefits to these accounts. Also, employers frequently match money that you put into a 401k.\n",
    "- If you want to learn more about them, check out [this site](https://www.nerdwallet.com/article/ira-vs-401k-retirement-accounts).\n",
    "\n",
    "We'll tackle one regression problem and one classification problem today.\n",
    "- Regression: What features best predict one's income?\n",
    "- Classification: Predict whether or not one is eligible for a 401k.\n",
    "\n",
    "Check out the data dictionary [here](http://fmwww.bc.edu/ec-p/data/wooldridge2k/401KSUBS.DES).\n",
    "\n",
    "### NOTE: When predicting `inc`, we should pretend as though you do not have access to the `e401k`, the `p401k` variable, and the `pira` variable. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Gather the data.\n",
    "\n",
    "##### Let's read in the data from the repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>e401k</th>\n",
       "      <th>inc</th>\n",
       "      <th>marr</th>\n",
       "      <th>male</th>\n",
       "      <th>age</th>\n",
       "      <th>fsize</th>\n",
       "      <th>nettfa</th>\n",
       "      <th>p401k</th>\n",
       "      <th>pira</th>\n",
       "      <th>incsq</th>\n",
       "      <th>agesq</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>13.170</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>1</td>\n",
       "      <td>4.575</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>173.4489</td>\n",
       "      <td>1600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>61.230</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "      <td>154.000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3749.1130</td>\n",
       "      <td>1225</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>12.858</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>44</td>\n",
       "      <td>2</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>165.3282</td>\n",
       "      <td>1936</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>98.880</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>44</td>\n",
       "      <td>2</td>\n",
       "      <td>21.800</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9777.2540</td>\n",
       "      <td>1936</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>22.614</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>53</td>\n",
       "      <td>1</td>\n",
       "      <td>18.450</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>511.3930</td>\n",
       "      <td>2809</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   e401k     inc  marr  male  age  fsize   nettfa  p401k  pira      incsq  \\\n",
       "0      0  13.170     0     0   40      1    4.575      0     1   173.4489   \n",
       "1      1  61.230     0     1   35      1  154.000      1     0  3749.1130   \n",
       "2      0  12.858     1     0   44      2    0.000      0     0   165.3282   \n",
       "3      0  98.880     1     1   44      2   21.800      0     0  9777.2540   \n",
       "4      0  22.614     0     0   53      1   18.450      0     0   511.3930   \n",
       "\n",
       "   agesq  \n",
       "0   1600  \n",
       "1   1225  \n",
       "2   1936  \n",
       "3   1936  \n",
       "4   2809  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('401ksubs.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9275, 11)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Other variables that, if available, would be helpful to have:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "- Savings\n",
    "- Job type\n",
    "- Credit history\n",
    "- Investments\n",
    "- Rents or owns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Is it ethically sound to put `race` into our model in order to better predict who to target when advertising IRAs and 401(k)s?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "No, because our model will become discriminatory based on race.\n",
    "\n",
    "Discriminating based on race is the complete opposite of acceptable. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Explore the data.\n",
    "\n",
    "##### When attempting to predict income, which feature(s) would we reasonably not use? Why?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "inc, because if we had inc we wouldn't need to predict inc.\n",
    "\n",
    "incsq, for similar reasons."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Note that we already two variables have already been created for us through feature engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "incsq and agesq.\n",
    "\n",
    "this might've been done because age and income exceptionally good indicators for eligibility, so a subject-matter expert might wanted to make age and income more potenent in their model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Looking at the data dictionary, one variable description appears to be an error."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Age's current description is age^2. The correct description should simply be a person's age.\n",
    "\n",
    "Inc's current description has a similar issue. it's current definition is inc^2. It's actual description should be a person's income."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Model the data. (Part 1: Regression Problem)\n",
    "\n",
    "Recall:\n",
    "- Problem: What features best predict one's income?\n",
    "- When predicting `inc`, we should pretend as though you do not have access to the `e401k`, the `p401k` variable, and the `pira` variable.\n",
    "\n",
    "##### Potential models to use for regression problems and whether they're appropriate for solving this specific regression problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "- a multiple linear regression model (Yes, we can understand influence of features.)\n",
    "- a $k$-nearest neighbors model (No, we cannot understand influence of features.)\n",
    "- a decision tree (Yes, we can understand influence of features.)\n",
    "- a set of bagged decision trees (Yes, we can understand influence of features.)\n",
    "- a random forest (Yes, we can understand influence of features.)\n",
    "- a set of extremely randomized trees (Yes, we can understand influence of features.)\n",
    "- an Adaboost model (Yes, we can understand influence of features.)\n",
    "- an XGBoost model (Yes, we can understand influence of features.)\n",
    "- a support vector regressor (Yes, we can understand influence of features.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Let's try a bunch of model types and see what they spit out.\n",
    "\n",
    "Model types we'll attempt:\n",
    "    - a multiple linear regression model\n",
    "    - a k-nearest neighbors model\n",
    "    - a decision tree\n",
    "    - a set of bagged decision trees\n",
    "    - a random forest\n",
    "    - an Adaboost model\n",
    "    - a support vector regressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.neighbors import KNeighborsRegressor\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "from sklearn.ensemble import BaggingRegressor, RandomForestRegressor, AdaBoostRegressor\n",
    "from sklearn.svm import SVR\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.pipeline import Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>e401k</th>\n",
       "      <th>inc</th>\n",
       "      <th>marr</th>\n",
       "      <th>male</th>\n",
       "      <th>age</th>\n",
       "      <th>fsize</th>\n",
       "      <th>nettfa</th>\n",
       "      <th>p401k</th>\n",
       "      <th>pira</th>\n",
       "      <th>incsq</th>\n",
       "      <th>agesq</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.803173</td>\n",
       "      <td>-1.082858</td>\n",
       "      <td>-1.300887</td>\n",
       "      <td>-0.506898</td>\n",
       "      <td>-0.104886</td>\n",
       "      <td>-1.235500</td>\n",
       "      <td>-0.226651</td>\n",
       "      <td>-0.617776</td>\n",
       "      <td>1.712236</td>\n",
       "      <td>-0.648965</td>\n",
       "      <td>-0.216227</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.245062</td>\n",
       "      <td>0.912268</td>\n",
       "      <td>-1.300887</td>\n",
       "      <td>1.972784</td>\n",
       "      <td>-0.590372</td>\n",
       "      <td>-1.235500</td>\n",
       "      <td>2.109561</td>\n",
       "      <td>1.618709</td>\n",
       "      <td>-0.584032</td>\n",
       "      <td>0.542404</td>\n",
       "      <td>-0.634940</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.803173</td>\n",
       "      <td>-1.095810</td>\n",
       "      <td>0.768706</td>\n",
       "      <td>-0.506898</td>\n",
       "      <td>0.283503</td>\n",
       "      <td>-0.580086</td>\n",
       "      <td>-0.298179</td>\n",
       "      <td>-0.617776</td>\n",
       "      <td>-0.584032</td>\n",
       "      <td>-0.651671</td>\n",
       "      <td>0.158941</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.803173</td>\n",
       "      <td>2.475242</td>\n",
       "      <td>0.768706</td>\n",
       "      <td>1.972784</td>\n",
       "      <td>0.283503</td>\n",
       "      <td>-0.580086</td>\n",
       "      <td>0.042656</td>\n",
       "      <td>-0.617776</td>\n",
       "      <td>-0.584032</td>\n",
       "      <td>2.550909</td>\n",
       "      <td>0.158941</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.803173</td>\n",
       "      <td>-0.690807</td>\n",
       "      <td>-1.300887</td>\n",
       "      <td>-0.506898</td>\n",
       "      <td>1.157377</td>\n",
       "      <td>-1.235500</td>\n",
       "      <td>-0.009720</td>\n",
       "      <td>-0.617776</td>\n",
       "      <td>-0.584032</td>\n",
       "      <td>-0.536366</td>\n",
       "      <td>1.133706</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      e401k       inc      marr      male       age     fsize    nettfa  \\\n",
       "0 -0.803173 -1.082858 -1.300887 -0.506898 -0.104886 -1.235500 -0.226651   \n",
       "1  1.245062  0.912268 -1.300887  1.972784 -0.590372 -1.235500  2.109561   \n",
       "2 -0.803173 -1.095810  0.768706 -0.506898  0.283503 -0.580086 -0.298179   \n",
       "3 -0.803173  2.475242  0.768706  1.972784  0.283503 -0.580086  0.042656   \n",
       "4 -0.803173 -0.690807 -1.300887 -0.506898  1.157377 -1.235500 -0.009720   \n",
       "\n",
       "      p401k      pira     incsq     agesq  \n",
       "0 -0.617776  1.712236 -0.648965 -0.216227  \n",
       "1  1.618709 -0.584032  0.542404 -0.634940  \n",
       "2 -0.617776 -0.584032 -0.651671  0.158941  \n",
       "3 -0.617776 -0.584032  2.550909  0.158941  \n",
       "4 -0.617776 -0.584032 -0.536366  1.133706  "
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scaled_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(scaled_df.drop(columns = ['e401k', 'p401k', 'pira', 'inc', 'incsq']),\n",
    "                                                    scaled_df['inc'],\n",
    "                                                    test_size = .2,\n",
    "                                                    random_state = 42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVR()"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = LinearRegression()\n",
    "lr.fit(X_train, y_train)\n",
    "\n",
    "knn_r = KNeighborsRegressor()\n",
    "knn_r.fit(X_train, y_train)\n",
    "\n",
    "dt_r = DecisionTreeRegressor()\n",
    "dt_r.fit(X_train, y_train)\n",
    "\n",
    "bag_r = BaggingRegressor()\n",
    "bag_r.fit(X_train, y_train)\n",
    "\n",
    "rf_r = RandomForestRegressor()\n",
    "rf_r.fit(X_train, y_train)\n",
    "\n",
    "ada_r = AdaBoostRegressor()\n",
    "ada_r.fit(X_train, y_train)\n",
    "\n",
    "svr = SVR()\n",
    "svr.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe_svec_r.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Bootstrapping?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Bootstrapping is random resampling with replacement.\n",
    "\n",
    "Combining the models from bootstrapped give us a better idea as to what we'd see from the whole population than to just getting one model from our original sample."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### A decision tree vs a set of bagged decision trees"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "With a set of bagged decision trees, we have bootstrapped  different samples and grown one decision tree on each bootstrapped sample and aggregate our predictions.\n",
    "\n",
    "A decision tree only uses the original sample and grows just one tree."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### A set of bagged decision trees vs a random forest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "The fundamental difference is that in Random forests, only a subset of features are selected at random out of the total and the best split feature from the subset is used to split each node in a tree, unlike in bagging where all features are considered for splitting a node.\n",
    "\n",
    "With bagged decision trees, we generate many different trees on pretty similar data. These trees are strongly correlated with one another. Because these trees are correlated with one another, they will have high variance. By \"de-correlating\" our trees from one another, we can drastically reduce the variance of our model.\n",
    "\n",
    "Random forest reduces variance (at the expense of a small increase in bias) and thus should greatly improve the overall performance of the final model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Evaluate the model. (Part 1: Regression Problem)\n",
    "\n",
    "##### 13. Using RMSE, evaluate each of the models you fit on both the training and testing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import metrics\n",
    "from sklearn.metrics import mean_squared_error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rmse(model, X_train, X_test, y_train, y_test):\n",
    "    mse_train = mean_squared_error(y_true = y_train,\n",
    "                                   y_pred = model.predict(X_train))\n",
    "    mse_test = mean_squared_error(y_true = y_test, \n",
    "                                  y_pred = model.predict(X_test))\n",
    "    \n",
    "    rmse_train = np.sqrt(mse_train)\n",
    "    rmse_test = np.sqrt(mse_test)               \n",
    "    print('Training RMSE: ', rmse_train)\n",
    "    print('Testing RMSE: ', rmse_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training RMSE:  0.8370830332453493\n",
      "Testing RMSE:  0.8675193605893169\n"
     ]
    }
   ],
   "source": [
    "rmse(lr, X_train, X_test, y_train, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training RMSE:  0.6860819173270954\n",
      "Testing RMSE:  0.837326616404933\n"
     ]
    }
   ],
   "source": [
    "rmse(knn_r, X_train, X_test, y_train, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training RMSE:  0.0939782006070435\n",
      "Testing RMSE:  1.1272071615450177\n"
     ]
    }
   ],
   "source": [
    "rmse(dt_r, X_train, X_test, y_train, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training RMSE:  0.36582653691398165\n",
      "Testing RMSE:  0.8746667054536171\n"
     ]
    }
   ],
   "source": [
    "rmse(bag_r, X_train, X_test, y_train, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training RMSE:  0.32073401874223845\n",
      "Testing RMSE:  0.843648400610395\n"
     ]
    }
   ],
   "source": [
    "rmse(rf_r, X_train, X_test, y_train, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training RMSE:  0.9352864035892462\n",
      "Testing RMSE:  0.9780338648986455\n"
     ]
    }
   ],
   "source": [
    "rmse(ada_r, X_train, X_test, y_train, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training RMSE:  0.7858885593080797\n",
      "Testing RMSE:  0.8206525603025994\n"
     ]
    }
   ],
   "source": [
    "rmse(svr, X_train, X_test, y_train, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### What do we notice?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "All my models are ovefit.\n",
    "\n",
    "Testing RMSE is worse (higher) than our training RMSE. \n",
    "\n",
    "Our models are generalizing poorly to held-out/unseen data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Based on everything we've covered so far, if we had to pick just one model as your final model to use to answer the problem in front of you, which one model should we pick?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "We'd get rid of KNN right away because we can't easily identify which features affect income, which is the point of what we're doing. We're looking for which features best predict income, not which model makes the best predictions.\n",
    "\n",
    "Outside of that, it's pretty much a judgement call on which models overfit the least (inear regression, AdaBoost, and the Support Vector Regressor). \n",
    "\n",
    "Some considerations we might have our:\n",
    "\n",
    "- Do we have time to tune the models to try and eke out better performance?\n",
    "- Is one model substantially better at solving the problem we want to solve?\n",
    "- Do we need something understandable by a lay audience? (i.e. Linear regression is more common and more easily understood than AdaBoost or Support Vector Machines.)\n",
    "\n",
    "If given time, we should try tuning the three remaining models to see if one performs substantially better than the other. If they all have roughly the same performance, we should generally go with the simplest/easiest to understand model. In this case, linear regression."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### What could we do to improve the performance of our models?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "- Use gridsearch\n",
    "- Use polynomial features\n",
    "- Remove outliers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Model the data. (Part 2: Classification Problem)\n",
    "\n",
    "Recall:\n",
    "- Problem: Predict whether or not one is eligible for a 401k.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "##### Why would we shy away from using `p401k` in our model?\n",
    "\n",
    "If you participate in a 401k already, you're eligible by default. that'll just give us a list of everyone with a 401k, which won't necessarily tell us anything about the factors that best determine eligibility. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Potential classification models we can use and whether they're appropriate for solving our problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "- a logistic regression model (Yes, we can predict whether or not one is eligible for a 401(k).)\n",
    "- a $k$-nearest neighbors model (Yes, we can predict whether or not one is eligible for a 401(k).)\n",
    "- a Naive Bayes model (Yes, we can predict whether or not one is eligible for a 401(k).)\n",
    "- a decision tree (Yes, we can predict whether or not one is eligible for a 401(k).)\n",
    "- a set of bagged decision trees (Yes, we can predict whether or not one is eligible for a 401(k).)\n",
    "- a random forest (Yes, we can predict whether or not one is eligible for a 401(k).)\n",
    "- a set of extremely randomized trees (Yes, we can predict whether or not one is eligible for a 401(k).)\n",
    "- an Adaboost model (Yes, we can predict whether or not one is eligible for a 401(k).)\n",
    "- an XGBoost model (Yes, we can predict whether or not one is eligible for a 401(k).)\n",
    "- a support vector classifier (Yes, we can predict whether or not one is eligible for a 401(k).)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Let's attempt to use the following models for our classification problem:\n",
    "    - a logistic regression model\n",
    "    - a k-nearest neighbors model\n",
    "    - a decision tree\n",
    "    - a set of bagged decision trees\n",
    "    - a random forest\n",
    "    - an Adaboost model\n",
    "    - a support vector classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import BaggingClassifier, RandomForestClassifier, AdaBoostClassifier\n",
    "from sklearn.svm import SVC\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(scaled_df.drop(columns = ['e401k', 'p401k']),\n",
    "                                                    [1 if scaled_df['e401k'][i] > 0 else 0 for i in range(scaled_df.shape[0])],\n",
    "                                                    ## I ran the above line because when I scaled e401k, e401k was no longer binary!\n",
    "                                                    ## I needed to turn my Y vector into a discrete variable.\n",
    "                                                    test_size = .2,\n",
    "                                                    random_state = 42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC()"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logreg = LogisticRegression()\n",
    "logreg.fit(X_train, y_train)\n",
    "\n",
    "knn_c = KNeighborsClassifier()\n",
    "knn_c.fit(X_train, y_train)\n",
    "\n",
    "dt_c = DecisionTreeClassifier()\n",
    "dt_c.fit(X_train, y_train)\n",
    "\n",
    "bag_c = BaggingClassifier()\n",
    "bag_c.fit(X_train, y_train)\n",
    "\n",
    "rf_c = RandomForestClassifier()\n",
    "rf_c.fit(X_train, y_train)\n",
    "\n",
    "ada_c = AdaBoostClassifier()\n",
    "ada_c.fit(X_train, y_train)\n",
    "\n",
    "svc = SVC()\n",
    "svc.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Evaluate the model. (Part 2: Classfication Problem)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "##### If our \"positive\" class is someone eligible for a 401(k) are false positives and false negatives are...\n",
    "False positives: someone we predicted to be eligible but who wasn't.\n",
    "\n",
    "False negatives: someone we predicted to be ineligible but who actually is eligible."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### In this specific case, we want to minimize false positives..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Because incorrectly assuming someone is eligible might hurt your company financially. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### If we want to minimize false positives, then we want to minimize specificity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "$$\\text{Specificity} = \\frac{TN}{N} = \\frac{TN}{TN + FP}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### If we wanted to balance our false positives and false negatives we'd probably want to use `f1-score`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "$$\n",
    "\\begin{eqnarray*}\n",
    "F_1 &=& \\frac{2}{\\frac{1}{\\text{precision}} + \\frac{1}{\\text{recall}}} \\\\\n",
    "&=& \\frac{2}{\\frac{1}{\\frac{TP}{TP + FP}} + \\frac{1}{\\frac{TP}{TP + FN}}} \\\\\n",
    "&=& \\frac{2}{\\frac{TP + FP}{TP} + \\frac{TP + FN}{TP}} \\\\\n",
    "&=& \\frac{2}{\\frac{TP + FP + TP + FN}{TP}} \\\\\n",
    "&=& \\frac{2}{2 + \\frac{FP + FN}{TP}}\n",
    "\\end{eqnarray*}\n",
    "$$\n",
    "\n",
    "Since the F1 score is an average of Precision and Recall, it means that the F1 score gives equal weight to Precision and Recall:<br>\n",
    "A model will obtain a high F1 score if both Precision and Recall are high<br>\n",
    "A model will obtain a low F1 score if both Precision and Recall are low<br>\n",
    "A model will obtain a medium F1 score if one of Precision and Recall is low and the other is high<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Let's evaluate our models using `f1-score`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import f1_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f1_scores(model, X_train, X_test, y_train, y_test):\n",
    "    f1_train = f1_score(y_true = y_train,\n",
    "                        y_pred = model.predict(X_train))\n",
    "    f1_test = f1_score(y_true = y_test, \n",
    "                       y_pred = model.predict(X_test))\n",
    "                   \n",
    "    print('Training F1: ', f1_train)\n",
    "    print('Testing F1: ', f1_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training F1:  0.4727870199219552\n",
      "Testing F1:  0.4777870913663035\n"
     ]
    }
   ],
   "source": [
    "f1_scores(logreg, X_train, X_test, y_train, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training F1:  0.653122648607976\n",
      "Testing F1:  0.4977511244377811\n"
     ]
    }
   ],
   "source": [
    "f1_scores(knn_c, X_train, X_test, y_train, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training F1:  1.0\n",
      "Testing F1:  0.4702627939142462\n"
     ]
    }
   ],
   "source": [
    "f1_scores(dt_c, X_train, X_test, y_train, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training F1:  0.9725380444288962\n",
      "Testing F1:  0.49615975422427033\n"
     ]
    }
   ],
   "source": [
    "f1_scores(bag_c, X_train, X_test, y_train, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training F1:  1.0\n",
      "Testing F1:  0.5465465465465464\n"
     ]
    }
   ],
   "source": [
    "f1_scores(rf_c, X_train, X_test, y_train, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training F1:  1.0\n",
      "Testing F1:  0.4702627939142462\n"
     ]
    }
   ],
   "source": [
    "f1_scores(dt_c, X_train, X_test, y_train, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training F1:  0.5621436716077537\n",
      "Testing F1:  0.5688487584650113\n"
     ]
    }
   ],
   "source": [
    "f1_scores(ada_c, X_train, X_test, y_train, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training F1:  0.47162162162162163\n",
      "Testing F1:  0.45207956600361665\n"
     ]
    }
   ],
   "source": [
    "f1_scores(svc, X_train, X_test, y_train, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### We want our $F_1$ score to be as high as possible. Thus, overfitting occurs when we have a high training $F_1$ score and a low testing $F_1$ score."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Models that appear to be overfit are:\n",
    "- knn\n",
    "- decision trees\n",
    "- bagged decision trees\n",
    "- random forest\n",
    "- SVC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Which model should we pick?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Borrowing from earlier...\n",
    "\n",
    "Any models that cannot solve my problem, we remove:\n",
    "    - In this case, all models can solve my problem. I want to maximize my ability to correctly predict whether or not someone is eligible for a 401(k).\n",
    "    \n",
    "Among the models that _can_ solve our problem, we want to find the one that performs the best based on a metric of my choice:\n",
    "    - In this case, the models that seem to overfit the least (i.e. the gap between training and testing $F_1$) are logistic regression and AdaBoost.\n",
    "    \n",
    "Once again, it becomes a judgment call:\n",
    "    - Do we have time to tune the models to try and eke out better performance?\n",
    "    - Is one model substantially better at solving the problem we want to solve?\n",
    "    - Do we need something understandable by a lay audience? (i.e. Logistic regression is more common and more easily understood than AdaBoost.)\n",
    "    \n",
    "If given time, we should try tuning the remaining models to see if one performs substantially better than the other. \n",
    "\n",
    "If they all have roughly the same performance, we generally go with the simplest/easiest to understand model. \n",
    "\n",
    "In this case, that would bne logistic regression."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Answer the problem.\n",
    "\n",
    "- Regression: What features best predict one's income?\n",
    "- Classification: Predict whether or not one is eligible for a 401k."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}