Final_CC_Default.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1><font size = '6'><center>Predicting Default for Credit Card Holders in Taiwanese Market</center></font></H1>\n",
    "\n",
    "# Contents\n",
    "\n",
    "- <a href='#1'>Introduction</a>\n",
    "- <a href='#2'>Business Understanding:</a>\n",
    "- <a href='#3'>Data Sources:</a>\n",
    "    - <a href='#3'>Importing Packages</a>  \n",
    "    - <a href='#3'>Uploading Data</a>\n",
    "    - <a href='#4'>Creating Train, Validation, and Testing Sets</a>  \n",
    "- <a href='#3'>Data Understanding:</a>\n",
    "- <a href='#5'>Data Cleaning</a>  \n",
    "- <a href='#5'>Exploratory Data Analysis</a>\n",
    "- <a href='#6'>Feature Engineering</a>  \n",
    "- <a href='#3'>Data Prepoccessing</a>\n",
    "- <a href='#3'>Modeling</a>\n",
    "- <a href='#3'>Evaluation:</a>\n",
    "- <a href='#3'>Analysis:</a>\n",
    "- <a href='#3'>Next Steps:</a>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "This dataset contains information on default payments, demographic factors, credit data, history of payment, and bill statements of credit card customers in Taiwan from April 2005 to September 2005.\n",
    "\n",
    "\n",
    "Here is the detailed key for the 25 variables used in the dataset:\n",
    "\n",
    "- ID: ID of each client\n",
    "- LIMIT_BAL: Amount of given credit in NT dollars (includes individual and family/supplementary credit\n",
    "- SEX: Gender\n",
    "    - 1=male\n",
    "    - 2=female\n",
    "- EDUCATION:\n",
    "    - 1=graduate school\n",
    "    - 2=university\n",
    "    - 3=high school\n",
    "    - 4=others\n",
    "    - 5=unknown\n",
    "    - 6=unknown\n",
    "- MARRIAGE: Marital status\n",
    "    - 1=married\n",
    "    - 2=single\n",
    "    - 3=others)\n",
    "- AGE: Age in years\n",
    "- PAY_0: Repayment status in September, 2005\n",
    "    - -1=pay duly\n",
    "    - 1=payment delay for one month\n",
    "    - 2=payment delay for two months\n",
    "    ....\n",
    "    - 8=payment delay for eight months\n",
    "    - 9=payment delay for nine months and above\n",
    "- PAY_2: Repayment status in August, 2005 (scale same as above)\n",
    "- PAY_3: Repayment status in July, 2005 (scale same as above)\n",
    "- PAY_4: Repayment status in June, 2005 (scale same as above)\n",
    "- PAY_5: Repayment status in May, 2005 (scale same as above)\n",
    "- PAY_6: Repayment status in April, 2005 (scale same as above)\n",
    "- BILL_AMT1: Amount of bill statement in September,2005 (NT dollar)\n",
    "- BILL_AMT2: Amount of bill statement in August, 2005 (NT dollar)\n",
    "- BILL_AMT3: Amount of bill statement in July, 2005 (NT dollar)\n",
    "- BILL_AMT4: Amount of bill statement in June, 2005 (NT dollar)\n",
    "- BILL_AMT5: Amount of bill statement in May, 2005 (NT dollar)\n",
    "- BILL_AMT6: Amount of bill statement in April, 2005 (NT dollar)\n",
    "- PAY_AMT1: Amount of previous payment in September, 2005 (NT dollar)\n",
    "- PAY_AMT2: Amount of previous payment in August, 2005 (NT dollar)\n",
    "- PAY_AMT3: Amount of previous payment in July, 2005 (NT dollar)\n",
    "- PAY_AMT4: Amount of previous payment in June, 2005 (NT dollar)\n",
    "- PAY_AMT5: Amount of previous payment in May, 2005 (NT dollar)\n",
    "- PAY_AMT6: Amount of previous payment in April, 2005 (NT dollar)\n",
    "- default.payment.next.month: Default payment\n",
    "    - 1=yes\n",
    "    - 0=no"
   ]
  },
  {
   "source": [
    "# Business Understanding:\n",
    "\n",
    "\n",
    "\n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "# Data Sources:"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Importing Packages\n",
    "import numpy as np \n",
    "import pandas as pd\n",
    "import re\n",
    "import json\n",
    "import requests\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "pd.set_option(\"display.max_rows\", 999)\n",
    "pd.set_option(\"display.max_columns\", 999)\n",
    "from collections import Counter\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib import style\n",
    "import seaborn as sns\n",
    "%matplotlib inline\n",
    "plt.style.use(\"fivethirtyeight\")\n",
    "import pickle\n",
    "\n",
    "from sklearn import linear_model\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.discriminant_analysis import LinearDiscriminantAnalysis\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.ensemble import AdaBoostClassifier, GradientBoostingClassifier, RandomForestClassifier, BaggingClassifier\n",
    "from sklearn.model_selection import train_test_split, KFold, cross_val_score, cross_val_predict, GridSearchCV\n",
    "from sklearn.feature_selection import RFECV\n",
    "from sklearn.metrics import accuracy_score, confusion_matrix, precision_score, recall_score, f1_score, roc_auc_score, classification_report,balanced_accuracy_score\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "import xgboost as xgb\n",
    "from xgboost.sklearn import XGBClassifier\n",
    "\n",
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "from utils import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# importing dataset\n",
    "df = pd.read_excel(\"data/default of credit card clients.xls\")\n",
    "new_header = df.iloc[0]\n",
    "df = df[1:] \n",
    "df.columns = new_header\n",
    "df = df.rename(columns={\"default payment next month\": \"default\"}) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Dataset Splits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make \n",
    "X = df.drop([\"default\"], axis=1)\n",
    "y = df[\"default\"]\n",
    "X_train, X_val, y_train, y_val = train_test_split(X, y, train_size=0.8, random_state=42)\n",
    "X_tr, X_tt, y_tr, y_tt = train_test_split(X_train, y_train, train_size=0.875, random_state=42)\n",
    "train = pd.concat([X_tr, y_tr], axis=1)\n",
    "val = pd.concat([X_val, y_val], axis=1)\n",
    "tr = train.drop([\"ID\"], axis=1)\n",
    "val = val.drop([\"ID\"], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Cleaning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The data cleaning process involved converting Taiwanese dollars to American dollars to facilitate understanding of the numbers involved and then converting them into integers.\n",
    "- There were anomalous values in the marriage category, so observations with the value of 0 were converted into 3, which represented other.\n",
    "- There were anomalous values in the education category, so observations with the value of 0, 5, or 6 were all lumped into the 4 or other category.\n",
    "- In the behind1 - behind6 categories, I was originally going to convert all the observations with the negative values into 0, but since there were so many observations with -1 and -2, it couldn't have been an anomaly or mistake.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = 'https://openexchangerates.org/api/latest.json?app_id=c51b1508fb4145259b1c2fade72a2c04'\n",
    "response = requests.get(url)\n",
    "data = response.json()\n",
    "rate = data['rates']['TWD']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [tr, val]\n",
    "for d in data:\n",
    "    d.rename(columns={\"PAY_0\": \"behind1\", \"PAY_2\": \"behind2\", \"PAY_3\": \"behind3\", \"PAY_4\": \"behind4\", \"PAY_5\": \"behind5\", \"PAY_6\": \"behind6\", \"BILL_AMT1\": \"billed1\", \"BILL_AMT2\": \"billed2\", \"BILL_AMT3\": \"billed3\", \"BILL_AMT4\": \"billed4\", \"BILL_AMT5\": \"billed5\", \"BILL_AMT6\": \"billed6\", \"PAY_AMT1\": \"paid1\", \"PAY_AMT2\": \"paid2\", \"PAY_AMT3\": \"paid3\", \"PAY_AMT4\": \"paid4\", \"PAY_AMT5\": \"paid5\", \"PAY_AMT6\": \"paid6\", \"SEX\": \"gender\", \"EDUCATION\": \"education\", \"MARRIAGE\": \"marriage\", \"AGE\": \"age\", \"LIMIT_BAL\": \"limit\"}, inplace=True)\n",
    "    d[['limit']] = d[['limit']]/rate\n",
    "    d[['billed1', 'billed2', 'billed3', 'billed4', 'billed5', 'billed6']] = d[['billed1', 'billed2', 'billed3', 'billed4', 'billed5', 'billed6']].divide(rate, axis=1).astype(int)\n",
    "    d[['paid1', 'paid2', 'paid3', 'paid4', 'paid5', 'paid6']] = d[['paid1', 'paid2', 'paid3', 'paid4', 'paid5', 'paid6']].divide(rate, axis=1).astype(int)\n",
    "    d['limit'] = d['limit'].apply(lambda x: round(x, 2))\n",
    "    d.replace({'marriage': {0:3}}, inplace=True)\n",
    "    d.replace({'education': {5:4, 0:4, 6:4}}, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>limit</th>\n",
       "      <th>gender</th>\n",
       "      <th>education</th>\n",
       "      <th>marriage</th>\n",
       "      <th>age</th>\n",
       "      <th>behind1</th>\n",
       "      <th>behind2</th>\n",
       "      <th>behind3</th>\n",
       "      <th>behind4</th>\n",
       "      <th>behind5</th>\n",
       "      <th>behind6</th>\n",
       "      <th>billed1</th>\n",
       "      <th>billed2</th>\n",
       "      <th>billed3</th>\n",
       "      <th>billed4</th>\n",
       "      <th>billed5</th>\n",
       "      <th>billed6</th>\n",
       "      <th>paid1</th>\n",
       "      <th>paid2</th>\n",
       "      <th>paid3</th>\n",
       "      <th>paid4</th>\n",
       "      <th>paid5</th>\n",
       "      <th>paid6</th>\n",
       "      <th>default</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6191</th>\n",
       "      <td>1788.52</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>44</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1630</td>\n",
       "      <td>1498</td>\n",
       "      <td>1277</td>\n",
       "      <td>799</td>\n",
       "      <td>846</td>\n",
       "      <td>980</td>\n",
       "      <td>107</td>\n",
       "      <td>178</td>\n",
       "      <td>107</td>\n",
       "      <td>107</td>\n",
       "      <td>178</td>\n",
       "      <td>33</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16054</th>\n",
       "      <td>5723.26</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>46</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>890</td>\n",
       "      <td>83</td>\n",
       "      <td>173</td>\n",
       "      <td>147</td>\n",
       "      <td>142</td>\n",
       "      <td>30</td>\n",
       "      <td>83</td>\n",
       "      <td>173</td>\n",
       "      <td>35</td>\n",
       "      <td>142</td>\n",
       "      <td>30</td>\n",
       "      <td>941</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19706</th>\n",
       "      <td>3577.04</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>47</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-2</td>\n",
       "      <td>238</td>\n",
       "      <td>238</td>\n",
       "      <td>0</td>\n",
       "      <td>224</td>\n",
       "      <td>-14</td>\n",
       "      <td>-14</td>\n",
       "      <td>238</td>\n",
       "      <td>0</td>\n",
       "      <td>224</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23128</th>\n",
       "      <td>6080.96</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2829</td>\n",
       "      <td>2238</td>\n",
       "      <td>2264</td>\n",
       "      <td>2285</td>\n",
       "      <td>1556</td>\n",
       "      <td>1573</td>\n",
       "      <td>79</td>\n",
       "      <td>89</td>\n",
       "      <td>92</td>\n",
       "      <td>60</td>\n",
       "      <td>67</td>\n",
       "      <td>75</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28516</th>\n",
       "      <td>5365.56</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>33</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>872</td>\n",
       "      <td>960</td>\n",
       "      <td>1169</td>\n",
       "      <td>1196</td>\n",
       "      <td>994</td>\n",
       "      <td>80</td>\n",
       "      <td>966</td>\n",
       "      <td>1170</td>\n",
       "      <td>1197</td>\n",
       "      <td>994</td>\n",
       "      <td>80</td>\n",
       "      <td>6061</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         limit gender  education  marriage age behind1 behind2 behind3  \\\n",
       "6191   1788.52      2          2         1  44       0       0       0   \n",
       "16054  5723.26      2          3         1  46      -1      -1      -1   \n",
       "19706  3577.04      2          2         1  47      -1      -1      -1   \n",
       "23128  6080.96      2          2         1  29       0       0       0   \n",
       "28516  5365.56      2          1         2  33      -2      -2      -2   \n",
       "\n",
       "      behind4 behind5 behind6  billed1  billed2  billed3  billed4  billed5  \\\n",
       "6191        0       0       0     1630     1498     1277      799      846   \n",
       "16054       0      -1      -1      890       83      173      147      142   \n",
       "19706      -1      -1      -2      238      238        0      224      -14   \n",
       "23128       0       0       0     2829     2238     2264     2285     1556   \n",
       "28516      -2      -2      -2      872      960     1169     1196      994   \n",
       "\n",
       "       billed6  paid1  paid2  paid3  paid4  paid5  paid6 default  \n",
       "6191       980    107    178    107    107    178     33       0  \n",
       "16054       30     83    173     35    142     30    941       0  \n",
       "19706      -14    238      0    224      0      0      0       1  \n",
       "23128     1573     79     89     92     60     67     75       0  \n",
       "28516       80    966   1170   1197    994     80   6061       0  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tr.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>limit</th>\n",
       "      <th>education</th>\n",
       "      <th>marriage</th>\n",
       "      <th>billed1</th>\n",
       "      <th>billed2</th>\n",
       "      <th>billed3</th>\n",
       "      <th>billed4</th>\n",
       "      <th>billed5</th>\n",
       "      <th>billed6</th>\n",
       "      <th>paid1</th>\n",
       "      <th>paid2</th>\n",
       "      <th>paid3</th>\n",
       "      <th>paid4</th>\n",
       "      <th>paid5</th>\n",
       "      <th>paid6</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>21000.000000</td>\n",
       "      <td>21000.000000</td>\n",
       "      <td>21000.000000</td>\n",
       "      <td>21000.000000</td>\n",
       "      <td>21000.00000</td>\n",
       "      <td>21000.000000</td>\n",
       "      <td>21000.000000</td>\n",
       "      <td>21000.000000</td>\n",
       "      <td>21000.000000</td>\n",
       "      <td>21000.000000</td>\n",
       "      <td>21000.000000</td>\n",
       "      <td>21000.000000</td>\n",
       "      <td>21000.000000</td>\n",
       "      <td>21000.000000</td>\n",
       "      <td>21000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>5981.340686</td>\n",
       "      <td>1.842810</td>\n",
       "      <td>1.555333</td>\n",
       "      <td>1827.979905</td>\n",
       "      <td>1758.19519</td>\n",
       "      <td>1679.613952</td>\n",
       "      <td>1543.949857</td>\n",
       "      <td>1443.855667</td>\n",
       "      <td>1393.781810</td>\n",
       "      <td>204.618429</td>\n",
       "      <td>214.418048</td>\n",
       "      <td>188.432952</td>\n",
       "      <td>175.843714</td>\n",
       "      <td>172.090667</td>\n",
       "      <td>184.061095</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>4634.450783</td>\n",
       "      <td>0.746378</td>\n",
       "      <td>0.522538</td>\n",
       "      <td>2628.013848</td>\n",
       "      <td>2545.94903</td>\n",
       "      <td>2489.800493</td>\n",
       "      <td>2299.447009</td>\n",
       "      <td>2182.100061</td>\n",
       "      <td>2134.592138</td>\n",
       "      <td>626.349241</td>\n",
       "      <td>897.321096</td>\n",
       "      <td>667.001026</td>\n",
       "      <td>601.041739</td>\n",
       "      <td>558.702990</td>\n",
       "      <td>631.634456</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>357.700000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-5922.000000</td>\n",
       "      <td>-2495.00000</td>\n",
       "      <td>-5625.000000</td>\n",
       "      <td>-6080.000000</td>\n",
       "      <td>-2909.000000</td>\n",
       "      <td>-7477.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1788.520000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>127.000000</td>\n",
       "      <td>108.00000</td>\n",
       "      <td>98.000000</td>\n",
       "      <td>84.000000</td>\n",
       "      <td>63.000000</td>\n",
       "      <td>46.000000</td>\n",
       "      <td>35.000000</td>\n",
       "      <td>29.000000</td>\n",
       "      <td>13.000000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>4.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>5007.850000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>803.000000</td>\n",
       "      <td>766.00000</td>\n",
       "      <td>717.500000</td>\n",
       "      <td>680.500000</td>\n",
       "      <td>647.000000</td>\n",
       "      <td>612.000000</td>\n",
       "      <td>75.000000</td>\n",
       "      <td>71.000000</td>\n",
       "      <td>64.000000</td>\n",
       "      <td>53.000000</td>\n",
       "      <td>53.000000</td>\n",
       "      <td>53.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>8584.890000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2387.250000</td>\n",
       "      <td>2263.50000</td>\n",
       "      <td>2135.000000</td>\n",
       "      <td>1943.250000</td>\n",
       "      <td>1798.000000</td>\n",
       "      <td>1767.000000</td>\n",
       "      <td>179.000000</td>\n",
       "      <td>178.000000</td>\n",
       "      <td>160.000000</td>\n",
       "      <td>143.000000</td>\n",
       "      <td>144.000000</td>\n",
       "      <td>143.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>35770.370000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>34500.000000</td>\n",
       "      <td>35195.00000</td>\n",
       "      <td>59525.000000</td>\n",
       "      <td>31892.000000</td>\n",
       "      <td>33165.000000</td>\n",
       "      <td>34399.000000</td>\n",
       "      <td>31247.000000</td>\n",
       "      <td>60246.000000</td>\n",
       "      <td>32051.000000</td>\n",
       "      <td>22213.000000</td>\n",
       "      <td>14951.000000</td>\n",
       "      <td>18856.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              limit     education      marriage       billed1      billed2  \\\n",
       "count  21000.000000  21000.000000  21000.000000  21000.000000  21000.00000   \n",
       "mean    5981.340686      1.842810      1.555333   1827.979905   1758.19519   \n",
       "std     4634.450783      0.746378      0.522538   2628.013848   2545.94903   \n",
       "min      357.700000      1.000000      1.000000  -5922.000000  -2495.00000   \n",
       "25%     1788.520000      1.000000      1.000000    127.000000    108.00000   \n",
       "50%     5007.850000      2.000000      2.000000    803.000000    766.00000   \n",
       "75%     8584.890000      2.000000      2.000000   2387.250000   2263.50000   \n",
       "max    35770.370000      4.000000      3.000000  34500.000000  35195.00000   \n",
       "\n",
       "            billed3       billed4       billed5       billed6         paid1  \\\n",
       "count  21000.000000  21000.000000  21000.000000  21000.000000  21000.000000   \n",
       "mean    1679.613952   1543.949857   1443.855667   1393.781810    204.618429   \n",
       "std     2489.800493   2299.447009   2182.100061   2134.592138    626.349241   \n",
       "min    -5625.000000  -6080.000000  -2909.000000  -7477.000000      0.000000   \n",
       "25%       98.000000     84.000000     63.000000     46.000000     35.000000   \n",
       "50%      717.500000    680.500000    647.000000    612.000000     75.000000   \n",
       "75%     2135.000000   1943.250000   1798.000000   1767.000000    179.000000   \n",
       "max    59525.000000  31892.000000  33165.000000  34399.000000  31247.000000   \n",
       "\n",
       "              paid2         paid3         paid4         paid5         paid6  \n",
       "count  21000.000000  21000.000000  21000.000000  21000.000000  21000.000000  \n",
       "mean     214.418048    188.432952    175.843714    172.090667    184.061095  \n",
       "std      897.321096    667.001026    601.041739    558.702990    631.634456  \n",
       "min        0.000000      0.000000      0.000000      0.000000      0.000000  \n",
       "25%       29.000000     13.000000     10.000000      8.000000      4.000000  \n",
       "50%       71.000000     64.000000     53.000000     53.000000     53.000000  \n",
       "75%      178.000000    160.000000    143.000000    144.000000    143.000000  \n",
       "max    60246.000000  32051.000000  22213.000000  14951.000000  18856.000000  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tr.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Observations: **\n",
    "\n",
    "- No missing data\n",
    "- There were anomalous values for education and marriage, and the anomalous values were reassigned under other.\n",
    "- Did not reassign -2 and -1 to 0 for 'behind' features despite being anomalous because they were so many -2 and -1.  There must be so significance to those values."
   ]
  },
  {
   "source": [
    "# Data Understanding (Exploratory Data Analysis):"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# organize features into categorical and continuous\n",
    "categorical = tr[['gender', 'marriage', 'education', 'behind1', 'behind2', 'behind3', 'behind4', 'behind5', 'behind6']]\n",
    "continuous = tr[['limit', 'age', 'billed1', 'billed2', 'billed3', 'billed4', 'billed5', 'billed6', 'paid1', 'paid2', 'paid3', 'paid4', 'paid5', 'paid6']]\n",
    "cat_col = categorical.columns\n",
    "cont_col = continuous.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# display distributions of all the continuous variables\n",
    "\n",
    "con_1 = pd.melt(tr, value_vars = cont_col)\n",
    "sns.set_theme(style=\"darkgrid\", font='serif', context='talk')\n",
    "g = sns.FacetGrid(con_1, col='variable', col_wrap=3, sharex=False, sharey=False, height=4)\n",
    "g = g.map(sns.distplot, 'value', color='r')\n",
    "g.set_xticklabels(rotation=45)\n",
    "g.fig.subplots_adjust(top=0.9)\n",
    "g.fig.suptitle(\"Distributions of Continuous Features\")\n",
    "g.fig.tight_layout()\n",
    "plt.savefig(\"../images/distplot.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/distplot.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Observations: **\n",
    "\n",
    "- It is hard to observe any trends with the paid features. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use bar graphs of the distribution of data for categorical variables\n",
    "\n",
    "cat_1 = pd.melt(tr, value_vars=cat_col)\n",
    "sns.set_theme(style=\"darkgrid\", font='serif', context='talk')\n",
    "g = sns.FacetGrid(cat_1, col='variable', col_wrap=3, sharex=False, sharey=False, height=4)\n",
    "g = g.map(sns.countplot, 'value', color='dodgerblue')\n",
    "g.set_xticklabels()\n",
    "g.fig.subplots_adjust(top=0.9)\n",
    "g.fig.suptitle(\"Distributions of Categorical Features\")\n",
    "g.fig.tight_layout()\n",
    "plt.savefig(\"../images/countplot.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/countplot.png\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "yes = tr.default.sum()\n",
    "no = len(tr)-yes\n",
    "perc_y = round(yes/len(tr)*100, 1)\n",
    "perc_n = round(no/len(tr)*100, 1)\n",
    "\n",
    "plt.figure(figsize=(8,6))\n",
    "sns.set_theme(style=\"darkgrid\", font='serif', context='talk')\n",
    "sns.countplot('default', data=tr)\n",
    "plt.title('Credit Card Baseline Default', size=16)\n",
    "plt.box(False);\n",
    "plt.savefig(\"../images/baseline.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/baseline.png\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Total Non-Defaulters:  4656\n",
      "Number of Defaulters:  16344\n",
      "Percentage of Non-Defaulters:  22.2\n",
      "Percentage of Defaulters:  77.8\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Training Dataset</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Number of Total Non-Defaulters:</th>\n",
       "      <td>4656.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Number of Defaulters:</th>\n",
       "      <td>16344.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Percentage of Non-Defaulters:</th>\n",
       "      <td>22.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Percentage of Defaulters:</th>\n",
       "      <td>77.8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  Training Dataset\n",
       "Number of Total Non-Defaulters:             4656.0\n",
       "Number of Defaulters:                      16344.0\n",
       "Percentage of Non-Defaulters:                 22.2\n",
       "Percentage of Defaulters:                     77.8"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Number of Total Non-Defaulters: \", yes)\n",
    "print(\"Number of Defaulters: \", no)\n",
    "print(\"Percentage of Non-Defaulters: \", perc_y)\n",
    "print(\"Percentage of Defaulters: \", perc_n)\n",
    "\n",
    "pd.DataFrame\n",
    "default = pd.DataFrame(data = {\"Training Dataset\": [yes, no, perc_y, perc_n]}, \n",
    "                       index = [\"Number of Total Non-Defaulters: \", \"Number of Defaulters: \", \"Percentage of Non-Defaulters: \", \"Percentage of Defaulters: \"])\n",
    "default"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "subset = tr[['gender', 'education', 'marriage', 'behind1', 'behind2', 'behind3', 'behind4', 'behind5', 'behind6', 'default']]\n",
    "f, axes = plt.subplots(3, 3, figsize=(15, 12), facecolor='white')\n",
    "sns.set_theme(style=\"darkgrid\", font='serif', context='paper')\n",
    "f.suptitle('Frequency of Categorical Variables', size=16)\n",
    "ax1 = sns.countplot(x=\"gender\", hue=\"default\", data=subset, ax=axes[0,0])\n",
    "ax2 = sns.countplot(x=\"education\", hue=\"default\", data=subset, ax=axes[0,1])\n",
    "ax3 = sns.countplot(x=\"marriage\", hue=\"default\", data=subset, ax=axes[0,2])\n",
    "ax4 = sns.countplot(x=\"behind1\", hue=\"default\", data=subset, ax=axes[1,0])\n",
    "ax5 = sns.countplot(x=\"behind2\", hue=\"default\", data=subset, ax=axes[1,1])\n",
    "ax6 = sns.countplot(x=\"behind3\", hue=\"default\", data=subset, ax=axes[1,2])\n",
    "ax7 = sns.countplot(x=\"behind4\", hue=\"default\", data=subset, ax=axes[2,0])\n",
    "ax8 = sns.countplot(x=\"behind5\", hue=\"default\", data=subset, ax=axes[2,1])\n",
    "ax9 = sns.countplot(x=\"behind6\", hue=\"default\", data=subset, ax=axes[2,2])\n",
    "plt.savefig(\"../images/default_freq_by_cat.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/default_freq_by_cat.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Observations: **\n",
    "\n",
    "- `gender`, `education`, and `marriage` doesn't seem to change with each group in terms of proportions.  Behind seems to have some correlation with default.  That would make sense since being behind in payments would make it more likely that you would default next month. \n",
    "\n",
    "- There isn’t a very clear distinction between the distribution of `default` on any of the demographic data. However, you do see quite some distribution differences of target classes for monthly repayment status (`behind1`-`behind6`).\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>default</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>education</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6013</td>\n",
       "      <td>1424</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7408</td>\n",
       "      <td>2341</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2626</td>\n",
       "      <td>866</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>297</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "default       0     1\n",
       "education            \n",
       "1          6013  1424\n",
       "2          7408  2341\n",
       "3          2626   866\n",
       "4           297    25"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "education = tr.groupby(['education', 'default']).size().unstack(1)\n",
    "education\n",
    "education.plot(kind=\"bar\", stacked=True)\n",
    "plt.title(\"Distribution Count of Educational Level and Default Status\", size=14)\n",
    "plt.savefig(\"../data/stacked_bar2.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/stacked_bar2.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** No clear relationship with education. The proportion doesn't seem to change with each group. **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>default</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>marriage</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7354</td>\n",
       "      <td>2258</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8778</td>\n",
       "      <td>2336</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>212</td>\n",
       "      <td>62</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "default      0     1\n",
       "marriage            \n",
       "1         7354  2258\n",
       "2         8778  2336\n",
       "3          212    62"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "marriage = tr.groupby(['marriage', 'default']).size().unstack(1)\n",
    "marriage\n",
    "marriage.plot(kind=\"bar\", stacked=True)\n",
    "plt.title(\"Distribution of Default Status for Marital Status\", size=14)\n",
    "plt.savefig(\"../images/stacked_bar3.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/stacked_bar3.png\">"
   ]
  },
  {
   "source": [
    "## Tableau Visualizations\n",
    "\n",
    "These visualizations incorporate some of my engineered features since these were completed post-project. I will offer my analysis of the visualizations shortly and create them with Desktop Public for the interactive versions."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "text/plain": "<IPython.core.display.HTML object>",
      "text/html": "<div class='tableauPlaceholder' id='viz1626030024918' style='position: relative'><noscript><a href='#'><img alt='Distribution of Feature Variables by Default ' src='https:&#47;&#47;public.tableau.com&#47;static&#47;images&#47;Di&#47;DistributionofFeaturesforTaiwaneseCCDefault&#47;FeatureDistributions&#47;1_rss.png' style='border: none' /></a></noscript><object class='tableauViz'  style='display:none;'><param name='host_url' value='https%3A%2F%2Fpublic.tableau.com%2F' /> <param name='embed_code_version' value='3' /> <param name='site_root' value='' /><param name='name' value='DistributionofFeaturesforTaiwaneseCCDefault&#47;FeatureDistributions' /><param name='tabs' value='no' /><param name='toolbar' value='yes' /><param name='static_image' value='https:&#47;&#47;public.tableau.com&#47;static&#47;images&#47;Di&#47;DistributionofFeaturesforTaiwaneseCCDefault&#47;FeatureDistributions&#47;1.png' /> <param name='animate_transition' value='yes' /><param name='display_static_image' value='yes' /><param name='display_spinner' value='yes' /><param name='display_overlay' value='yes' /><param name='display_count' value='yes' /><param name='language' value='en-US' /></object></div>                <script type='text/javascript'>                    var divElement = document.getElementById('viz1626030024918');                    var vizElement = divElement.getElementsByTagName('object')[0];                    if ( divElement.offsetWidth > 800 ) { vizElement.style.width='1000px';vizElement.style.height='627px';} else if ( divElement.offsetWidth > 500 ) { vizElement.style.width='1000px';vizElement.style.height='627px';} else { vizElement.style.width='100%';vizElement.style.height='2227px';}                     var scriptElement = document.createElement('script');                    scriptElement.src = 'https://public.tableau.com/javascripts/api/viz_v1.js';                    vizElement.parentNode.insertBefore(scriptElement, vizElement);                </script>\n"
     },
     "metadata": {}
    }
   ],
   "source": [
    "%%html\n",
    "<div class='tableauPlaceholder' id='viz1626030024918' style='position: relative'><noscript><a href='#'><img alt='Distribution of Feature Variables by Default ' src='https:&#47;&#47;public.tableau.com&#47;static&#47;images&#47;Di&#47;DistributionofFeaturesforTaiwaneseCCDefault&#47;FeatureDistributions&#47;1_rss.png' style='border: none' /></a></noscript><object class='tableauViz'  style='display:none;'><param name='host_url' value='https%3A%2F%2Fpublic.tableau.com%2F' /> <param name='embed_code_version' value='3' /> <param name='site_root' value='' /><param name='name' value='DistributionofFeaturesforTaiwaneseCCDefault&#47;FeatureDistributions' /><param name='tabs' value='no' /><param name='toolbar' value='yes' /><param name='static_image' value='https:&#47;&#47;public.tableau.com&#47;static&#47;images&#47;Di&#47;DistributionofFeaturesforTaiwaneseCCDefault&#47;FeatureDistributions&#47;1.png' /> <param name='animate_transition' value='yes' /><param name='display_static_image' value='yes' /><param name='display_spinner' value='yes' /><param name='display_overlay' value='yes' /><param name='display_count' value='yes' /><param name='language' value='en-US' /></object></div>                <script type='text/javascript'>                    var divElement = document.getElementById('viz1626030024918');                    var vizElement = divElement.getElementsByTagName('object')[0];                    if ( divElement.offsetWidth > 800 ) { vizElement.style.width='1000px';vizElement.style.height='627px';} else if ( divElement.offsetWidth > 500 ) { vizElement.style.width='1000px';vizElement.style.height='627px';} else { vizElement.style.width='100%';vizElement.style.height='2227px';}                     var scriptElement = document.createElement('script');                    scriptElement.src = 'https://public.tableau.com/javascripts/api/viz_v1.js';                    vizElement.parentNode.insertBefore(scriptElement, vizElement);                </script>"
   ]
  },
  {
   "source": [
    "%%html\n",
    "<div class='tableauPlaceholder' id='viz1626029808452' style='position: relative'><noscript><a href='#'><img alt='Age&#47;Limit vs. Marriage&#47;Education&#47;Gender by Default ' src='https:&#47;&#47;public.tableau.com&#47;static&#47;images&#47;Bi&#47;BivariateAnalysisofTaiwaneseCCDefault&#47;AgeandLimit&#47;1_rss.png' style='border: none' /></a></noscript><object class='tableauViz'  style='display:none;'><param name='host_url' value='https%3A%2F%2Fpublic.tableau.com%2F' /> <param name='embed_code_version' value='3' /> <param name='site_root' value='' /><param name='name' value='BivariateAnalysisofTaiwaneseCCDefault&#47;AgeandLimit' /><param name='tabs' value='no' /><param name='toolbar' value='yes' /><param name='static_image' value='https:&#47;&#47;public.tableau.com&#47;static&#47;images&#47;Bi&#47;BivariateAnalysisofTaiwaneseCCDefault&#47;AgeandLimit&#47;1.png' /> <param name='animate_transition' value='yes' /><param name='display_static_image' value='yes' /><param name='display_spinner' value='yes' /><param name='display_overlay' value='yes' /><param name='display_count' value='yes' /><param name='language' value='en-US' /></object></div>                <script type='text/javascript'>                    var divElement = document.getElementById('viz1626029808452');                    var vizElement = divElement.getElementsByTagName('object')[0];                    if ( divElement.offsetWidth > 800 ) { vizElement.style.width='1000px';vizElement.style.height='627px';} else if ( divElement.offsetWidth > 500 ) { vizElement.style.width='1000px';vizElement.style.height='627px';} else { vizElement.style.width='100%';vizElement.style.height='1927px';}                     var scriptElement = document.createElement('script');                    scriptElement.src = 'https://public.tableau.com/javascripts/api/viz_v1.js';                    vizElement.parentNode.insertBefore(scriptElement, vizElement);                </script>"
   ],
   "cell_type": "code",
   "metadata": {},
   "execution_count": 1,
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "text/plain": "<IPython.core.display.HTML object>",
      "text/html": "<div class='tableauPlaceholder' id='viz1626029808452' style='position: relative'><noscript><a href='#'><img alt='Age&#47;Limit vs. Marriage&#47;Education&#47;Gender by Default ' src='https:&#47;&#47;public.tableau.com&#47;static&#47;images&#47;Bi&#47;BivariateAnalysisofTaiwaneseCCDefault&#47;AgeandLimit&#47;1_rss.png' style='border: none' /></a></noscript><object class='tableauViz'  style='display:none;'><param name='host_url' value='https%3A%2F%2Fpublic.tableau.com%2F' /> <param name='embed_code_version' value='3' /> <param name='site_root' value='' /><param name='name' value='BivariateAnalysisofTaiwaneseCCDefault&#47;AgeandLimit' /><param name='tabs' value='no' /><param name='toolbar' value='yes' /><param name='static_image' value='https:&#47;&#47;public.tableau.com&#47;static&#47;images&#47;Bi&#47;BivariateAnalysisofTaiwaneseCCDefault&#47;AgeandLimit&#47;1.png' /> <param name='animate_transition' value='yes' /><param name='display_static_image' value='yes' /><param name='display_spinner' value='yes' /><param name='display_overlay' value='yes' /><param name='display_count' value='yes' /><param name='language' value='en-US' /></object></div>                <script type='text/javascript'>                    var divElement = document.getElementById('viz1626029808452');                    var vizElement = divElement.getElementsByTagName('object')[0];                    if ( divElement.offsetWidth > 800 ) { vizElement.style.width='1000px';vizElement.style.height='627px';} else if ( divElement.offsetWidth > 500 ) { vizElement.style.width='1000px';vizElement.style.height='627px';} else { vizElement.style.width='100%';vizElement.style.height='1927px';}                     var scriptElement = document.createElement('script');                    scriptElement.src = 'https://public.tableau.com/javascripts/api/viz_v1.js';                    vizElement.parentNode.insertBefore(scriptElement, vizElement);                </script>\n"
     },
     "metadata": {}
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Vanilla Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logreg = LogisticRegression(solver=\"liblinear\", random_state=42).fit(X_tr_dum, y_tr)\n",
    "y_pred_log_tr = logreg.predict(X_tr_dum)\n",
    "y_pred_log_val = logreg.predict(X_val_dum)\n",
    "\n",
    "rfc = RandomForestClassifier().fit(X_tr, y_tr)\n",
    "y_pred_rfc_tr = rfc.predict(X_tr)\n",
    "y_pred_rfc_val = rfc.predict(X_val)\n",
    "\n",
    "dtc = DecisionTreeClassifier().fit(X_tr, y_tr)\n",
    "y_pred_dtc_tr = dtc.predict(X_tr)\n",
    "y_pred_dtc_val = dtc.predict(X_val)\n",
    "\n",
    "knn = KNeighborsClassifier().fit(X_tr, y_tr)\n",
    "y_pred_knn_tr = knn.predict(X_tr)\n",
    "y_pred_knn_val = knn.predict(X_val)\n",
    "\n",
    "gnb = GaussianNB().fit(X_tr, y_tr)\n",
    "y_pred_gnb_tr = gnb.predict(X_tr)\n",
    "y_pred_gnb_val = gnb.predict(X_val)\n",
    "\n",
    "lda = LinearDiscriminantAnalysis().fit(X_tr, y_tr)\n",
    "y_pred_lda_tr = lda.predict(X_tr)\n",
    "y_pred_lda_val = lda.predict(X_val)\n",
    "\n",
    "abc = AdaBoostClassifier().fit(X_tr, y_tr)\n",
    "y_pred_abc_tr = abc.predict(X_tr)\n",
    "y_pred_abc_val = abc.predict(X_val)\n",
    "\n",
    "gbc = GradientBoostingClassifier().fit(X_tr, y_tr)\n",
    "y_pred_gbc_tr = gbc.predict(X_tr)\n",
    "y_pred_gbc_val = gbc.predict(X_val)\n",
    "\n",
    "xgb = XGBClassifier().fit(X_tr, y_tr)\n",
    "y_pred_xgb_tr = xgb.predict(X_tr)\n",
    "y_pred_xgb_val = xgb.predict(X_val)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>F1 Score</th>\n",
       "      <th>Recall</th>\n",
       "      <th>Precision</th>\n",
       "      <th>PR AUC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Logistic Regression</td>\n",
       "      <td>0.811500</td>\n",
       "      <td>0.360656</td>\n",
       "      <td>0.242955</td>\n",
       "      <td>0.699561</td>\n",
       "      <td>0.486829</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Random Forest Classifier</td>\n",
       "      <td>0.814833</td>\n",
       "      <td>0.461464</td>\n",
       "      <td>0.362529</td>\n",
       "      <td>0.634667</td>\n",
       "      <td>0.513445</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Decision Tree Classifier</td>\n",
       "      <td>0.730667</td>\n",
       "      <td>0.399257</td>\n",
       "      <td>0.408987</td>\n",
       "      <td>0.389978</td>\n",
       "      <td>0.289090</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>K-Nearest Neighbors</td>\n",
       "      <td>0.798000</td>\n",
       "      <td>0.447080</td>\n",
       "      <td>0.373191</td>\n",
       "      <td>0.557452</td>\n",
       "      <td>0.416605</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Gaussian Naive Bayes</td>\n",
       "      <td>0.724000</td>\n",
       "      <td>0.498486</td>\n",
       "      <td>0.626809</td>\n",
       "      <td>0.413776</td>\n",
       "      <td>0.480981</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Linear Discriminant Analysis</td>\n",
       "      <td>0.810333</td>\n",
       "      <td>0.367778</td>\n",
       "      <td>0.252094</td>\n",
       "      <td>0.679671</td>\n",
       "      <td>0.480476</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>AdaBoost Classifier</td>\n",
       "      <td>0.815667</td>\n",
       "      <td>0.425753</td>\n",
       "      <td>0.312262</td>\n",
       "      <td>0.668842</td>\n",
       "      <td>0.523430</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Gradient Boosting Classifier</td>\n",
       "      <td>0.820833</td>\n",
       "      <td>0.468085</td>\n",
       "      <td>0.360244</td>\n",
       "      <td>0.668079</td>\n",
       "      <td>0.545925</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>XGBoost Classifier</td>\n",
       "      <td>0.816833</td>\n",
       "      <td>0.469338</td>\n",
       "      <td>0.370145</td>\n",
       "      <td>0.641161</td>\n",
       "      <td>0.518488</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     Unnamed: 0  Accuracy  F1 Score    Recall  Precision  \\\n",
       "0           Logistic Regression  0.811500  0.360656  0.242955   0.699561   \n",
       "1      Random Forest Classifier  0.814833  0.461464  0.362529   0.634667   \n",
       "2      Decision Tree Classifier  0.730667  0.399257  0.408987   0.389978   \n",
       "3           K-Nearest Neighbors  0.798000  0.447080  0.373191   0.557452   \n",
       "4          Gaussian Naive Bayes  0.724000  0.498486  0.626809   0.413776   \n",
       "5  Linear Discriminant Analysis  0.810333  0.367778  0.252094   0.679671   \n",
       "6           AdaBoost Classifier  0.815667  0.425753  0.312262   0.668842   \n",
       "7  Gradient Boosting Classifier  0.820833  0.468085  0.360244   0.668079   \n",
       "8            XGBoost Classifier  0.816833  0.469338  0.370145   0.641161   \n",
       "\n",
       "     PR AUC  \n",
       "0  0.486829  \n",
       "1  0.513445  \n",
       "2  0.289090  \n",
       "3  0.416605  \n",
       "4  0.480981  \n",
       "5  0.480476  \n",
       "6  0.523430  \n",
       "7  0.545925  \n",
       "8  0.518488  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "baseline = pd.read_csv(\"data/charts/baseline.csv\")\n",
    "baseline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature Engineering\n",
    "\n",
    "- `age_bin`: 1 = young adult, 2 = middle age, 3 = senior\n",
    "- `gen-mar`: interaction between gender and marriage status\n",
    "- `gen-age`: interaction between age and gender\n",
    "- `avail...`: fraction of estimated available balance based on what is billed per month"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [tr, val]\n",
    "\n",
    "# create features for demographic variables\n",
    "for d in data:\n",
    "    d['age_bin'] = 0\n",
    "    d.loc[((d['age'] > 20) & (d['age'] < 30)) , 'age_bin'] = 1\n",
    "    d.loc[((d['age'] >= 30) & (d['age'] < 60)) , 'age_bin'] = 2\n",
    "    d.loc[((d['age'] >= 60) & (d['age'] < 81)) , 'age_bin'] = 3\n",
    "    # create categories for single, married, divorced males and females\n",
    "    d['gen-mar'] = d['gender'] + d['marriage']\n",
    "    # create categories for young, middle age and senior males and females\n",
    "    d['gen-age'] = d['gender'] + d['age_bin']\n",
    "\n",
    "# feature for credit use percentage: fraction of estimated available balance based on what is billed per month\n",
    "# (credit limit - monthly billed amount) / credit limit\n",
    "for d in data:\n",
    "    d['avail6'] = (d.limit - d.billed6) / d.limit\n",
    "    d['avail5'] = (d.limit - d.billed5) / d.limit\n",
    "    d['avail4'] = (d.limit - d.billed4) / d.limit\n",
    "    d['avail3'] = (d.limit - d.billed3) / d.limit\n",
    "    d['avail2'] = (d.limit - d.billed2) / d.limit\n",
    "    d['avail1'] = (d.limit - d.billed1) / d.limit\n",
    "    d['avg_av'] = (d.avail1 + d.avail2 + d.avail3 + d.avail4 + d.avail5 + d.avail6) / 6\n",
    "\n",
    "# create a feature that indicates whether a client has had a delayed payment or not\n",
    "def delayed_payment(d):\n",
    "    if (d.behind1 > 0) or (d.behind2 > 0) or (d.behind3 > 0) or (d.behind4 > 0) or (d.behind5 > 0) or (d.behind6 > 0):\n",
    "        return 1\n",
    "    else:\n",
    "        return 0\n",
    "for d in data:\n",
    "    d['delayed'] = d.apply(delayed_payment, axis=1)\n",
    "\n",
    "# create feature for the total number of months with delayed payment status for a particular client\n",
    "def total_months_with_delayed_payments(d):\n",
    "    count = 0\n",
    "    if (d.behind1 > 0):\n",
    "        count += 1\n",
    "    if (d.behind2 > 0):\n",
    "        count += 1\n",
    "    if (d.behind3 > 0):\n",
    "        count += 1\n",
    "    if (d.behind4 > 0):\n",
    "        count += 1\n",
    "    if (d.behind5 > 0):\n",
    "        count += 1\n",
    "    if (d.behind6 > 0):\n",
    "        count += 1\n",
    "    return count\n",
    "for d in data:\n",
    "    d['latemths'] = d.apply(total_months_with_delayed_payments, axis=1)\n",
    "\n",
    "# the ratio of amount paid and amount billed\n",
    "for d in data:\n",
    "    d['pperb1'] = d.paid1 / d.billed2\n",
    "    d['pperb2'] = d.paid2 / d.billed3\n",
    "    d['pperb3'] = d.paid3 / d.billed4\n",
    "    d['pperb4'] = d.paid4 / d.billed5\n",
    "    d['pperb5'] = d.paid5 / d.billed6\n",
    "\n",
    "# remove any infinity and NaN values\n",
    "datasets = ['pperb1', 'pperb2', 'pperb3', 'pperb4', 'pperb5']\n",
    "for data in datasets:\n",
    "    tr.replace({data: {np.inf: 0, np.nan: 0}}, inplace=True)\n",
    "    val.replace({data: {np.inf: 0, np.nan: 0}}, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plt.style.use(\"fivethirtyeight\")\n",
    "# sns.set_theme(style=\"darkgrid\", font='serif', context='paper')\n",
    "# plt.figure(figsize = (20,16))\n",
    "# plt.title('Pearson Correlation of Features', y = 1.05, size = 20)\n",
    "# g = sns.heatmap(tr.corr(), cmap='RdBu', square=True, linecolor='white', linewidths=0.2)\n",
    "# plt.savefig(\"../images/correlation_matrix_2.png\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/correlation_matrix_2.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** This includes my engineered features.  Default seems to be correlated with two of my engineered features, delayed and latemnths.  Delayed is whether you have had a delayed payment durig the 6 month history or not. latemnths is the total nunber of months you were given a status of behind in payments.  Seems to be correlated with behind1 and limit.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle_in = open(\"data/pickles/training_features.pickle\",\"rb\")\n",
    "train2 = pickle.load(pickle_in)\n",
    "pickle_in = open(\"data/pickles/validate_features.pickle\",\"rb\")\n",
    "validate2 = pickle.load(pickle_in)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train2 = train2.drop([\"default\"], axis=1)\n",
    "y_tr = train2[\"default\"]\n",
    "X_validate2 = validate2.drop([\"default\"], axis=1)\n",
    "y_val = validate2[\"default\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>limit</th>\n",
       "      <th>age</th>\n",
       "      <th>behind1</th>\n",
       "      <th>behind2</th>\n",
       "      <th>behind3</th>\n",
       "      <th>behind4</th>\n",
       "      <th>behind5</th>\n",
       "      <th>behind6</th>\n",
       "      <th>billed1</th>\n",
       "      <th>billed2</th>\n",
       "      <th>billed3</th>\n",
       "      <th>billed4</th>\n",
       "      <th>billed5</th>\n",
       "      <th>billed6</th>\n",
       "      <th>paid1</th>\n",
       "      <th>paid2</th>\n",
       "      <th>paid3</th>\n",
       "      <th>paid4</th>\n",
       "      <th>paid5</th>\n",
       "      <th>paid6</th>\n",
       "      <th>avail6</th>\n",
       "      <th>avail5</th>\n",
       "      <th>avail4</th>\n",
       "      <th>avail3</th>\n",
       "      <th>avail2</th>\n",
       "      <th>avail1</th>\n",
       "      <th>avg_av</th>\n",
       "      <th>delayed</th>\n",
       "      <th>latemths</th>\n",
       "      <th>pperb1</th>\n",
       "      <th>pperb2</th>\n",
       "      <th>pperb3</th>\n",
       "      <th>pperb4</th>\n",
       "      <th>pperb5</th>\n",
       "      <th>sex_2</th>\n",
       "      <th>edu_2</th>\n",
       "      <th>edu_3</th>\n",
       "      <th>edu_4</th>\n",
       "      <th>mar_2</th>\n",
       "      <th>mar_3</th>\n",
       "      <th>agebin_2</th>\n",
       "      <th>agebin_3</th>\n",
       "      <th>sexmar_3</th>\n",
       "      <th>sexmar_4</th>\n",
       "      <th>sexmar_5</th>\n",
       "      <th>sexage_3</th>\n",
       "      <th>sexage_4</th>\n",
       "      <th>sexage_5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1790.26</td>\n",
       "      <td>44</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1631.93</td>\n",
       "      <td>1500.45</td>\n",
       "      <td>1278.35</td>\n",
       "      <td>800.60</td>\n",
       "      <td>847.12</td>\n",
       "      <td>981.81</td>\n",
       "      <td>107.99</td>\n",
       "      <td>179.13</td>\n",
       "      <td>107.42</td>\n",
       "      <td>107.42</td>\n",
       "      <td>179.03</td>\n",
       "      <td>33.08</td>\n",
       "      <td>0.451582</td>\n",
       "      <td>0.526817</td>\n",
       "      <td>0.552802</td>\n",
       "      <td>0.285942</td>\n",
       "      <td>0.161882</td>\n",
       "      <td>0.088440</td>\n",
       "      <td>0.344578</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.071972</td>\n",
       "      <td>0.140126</td>\n",
       "      <td>0.134174</td>\n",
       "      <td>0.126806</td>\n",
       "      <td>0.182347</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5728.83</td>\n",
       "      <td>46</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>891.69</td>\n",
       "      <td>83.71</td>\n",
       "      <td>173.87</td>\n",
       "      <td>147.77</td>\n",
       "      <td>143.04</td>\n",
       "      <td>30.15</td>\n",
       "      <td>83.89</td>\n",
       "      <td>173.87</td>\n",
       "      <td>35.81</td>\n",
       "      <td>143.04</td>\n",
       "      <td>30.15</td>\n",
       "      <td>942.14</td>\n",
       "      <td>0.994737</td>\n",
       "      <td>0.975032</td>\n",
       "      <td>0.974206</td>\n",
       "      <td>0.969650</td>\n",
       "      <td>0.985388</td>\n",
       "      <td>0.844350</td>\n",
       "      <td>0.957227</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.002150</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.242336</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3580.52</td>\n",
       "      <td>47</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-2</td>\n",
       "      <td>238.68</td>\n",
       "      <td>238.68</td>\n",
       "      <td>0.00</td>\n",
       "      <td>224.50</td>\n",
       "      <td>-14.18</td>\n",
       "      <td>-14.18</td>\n",
       "      <td>238.68</td>\n",
       "      <td>0.00</td>\n",
       "      <td>224.50</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.003960</td>\n",
       "      <td>1.003960</td>\n",
       "      <td>0.937300</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.933339</td>\n",
       "      <td>0.933339</td>\n",
       "      <td>0.968650</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.000000</td>\n",
       "      <td>-0.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6086.88</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2831.87</td>\n",
       "      <td>2240.51</td>\n",
       "      <td>2267.08</td>\n",
       "      <td>2288.06</td>\n",
       "      <td>1557.71</td>\n",
       "      <td>1575.25</td>\n",
       "      <td>80.02</td>\n",
       "      <td>89.26</td>\n",
       "      <td>92.56</td>\n",
       "      <td>60.26</td>\n",
       "      <td>68.07</td>\n",
       "      <td>75.58</td>\n",
       "      <td>0.741206</td>\n",
       "      <td>0.744087</td>\n",
       "      <td>0.624100</td>\n",
       "      <td>0.627546</td>\n",
       "      <td>0.631912</td>\n",
       "      <td>0.534758</td>\n",
       "      <td>0.650602</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.035715</td>\n",
       "      <td>0.039372</td>\n",
       "      <td>0.040453</td>\n",
       "      <td>0.038685</td>\n",
       "      <td>0.043212</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5370.78</td>\n",
       "      <td>33</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>-2</td>\n",
       "      <td>873.40</td>\n",
       "      <td>961.26</td>\n",
       "      <td>1170.90</td>\n",
       "      <td>1198.01</td>\n",
       "      <td>995.38</td>\n",
       "      <td>80.96</td>\n",
       "      <td>966.99</td>\n",
       "      <td>1171.37</td>\n",
       "      <td>1198.58</td>\n",
       "      <td>995.67</td>\n",
       "      <td>80.96</td>\n",
       "      <td>6067.73</td>\n",
       "      <td>0.984926</td>\n",
       "      <td>0.814668</td>\n",
       "      <td>0.776939</td>\n",
       "      <td>0.781987</td>\n",
       "      <td>0.821020</td>\n",
       "      <td>0.837379</td>\n",
       "      <td>0.836153</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.005961</td>\n",
       "      <td>1.000401</td>\n",
       "      <td>1.000476</td>\n",
       "      <td>1.000291</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     limit  age  behind1  behind2  behind3  behind4  behind5  behind6  \\\n",
       "0  1790.26   44        0        0        0        0        0        0   \n",
       "1  5728.83   46       -1       -1       -1        0       -1       -1   \n",
       "2  3580.52   47       -1       -1       -1       -1       -1       -2   \n",
       "3  6086.88   29        0        0        0        0        0        0   \n",
       "4  5370.78   33       -2       -2       -2       -2       -2       -2   \n",
       "\n",
       "   billed1  billed2  billed3  billed4  billed5  billed6   paid1    paid2  \\\n",
       "0  1631.93  1500.45  1278.35   800.60   847.12   981.81  107.99   179.13   \n",
       "1   891.69    83.71   173.87   147.77   143.04    30.15   83.89   173.87   \n",
       "2   238.68   238.68     0.00   224.50   -14.18   -14.18  238.68     0.00   \n",
       "3  2831.87  2240.51  2267.08  2288.06  1557.71  1575.25   80.02    89.26   \n",
       "4   873.40   961.26  1170.90  1198.01   995.38    80.96  966.99  1171.37   \n",
       "\n",
       "     paid3   paid4   paid5    paid6    avail6    avail5    avail4    avail3  \\\n",
       "0   107.42  107.42  179.03    33.08  0.451582  0.526817  0.552802  0.285942   \n",
       "1    35.81  143.04   30.15   942.14  0.994737  0.975032  0.974206  0.969650   \n",
       "2   224.50    0.00    0.00     0.00  1.003960  1.003960  0.937300  1.000000   \n",
       "3    92.56   60.26   68.07    75.58  0.741206  0.744087  0.624100  0.627546   \n",
       "4  1198.58  995.67   80.96  6067.73  0.984926  0.814668  0.776939  0.781987   \n",
       "\n",
       "     avail2    avail1    avg_av  delayed  latemths    pperb1    pperb2  \\\n",
       "0  0.161882  0.088440  0.344578        0         0  0.071972  0.140126   \n",
       "1  0.985388  0.844350  0.957227        0         0  1.002150  1.000000   \n",
       "2  0.933339  0.933339  0.968650        0         0  1.000000  0.000000   \n",
       "3  0.631912  0.534758  0.650602        0         0  0.035715  0.039372   \n",
       "4  0.821020  0.837379  0.836153        0         0  1.005961  1.000401   \n",
       "\n",
       "     pperb3    pperb4    pperb5  sex_2  edu_2  edu_3  edu_4  mar_2  mar_3  \\\n",
       "0  0.134174  0.126806  0.182347      1      1      0      0      0      0   \n",
       "1  0.242336  1.000000  1.000000      1      0      1      0      0      0   \n",
       "2  1.000000 -0.000000 -0.000000      1      1      0      0      0      0   \n",
       "3  0.040453  0.038685  0.043212      1      1      0      0      0      0   \n",
       "4  1.000476  1.000291  1.000000      1      0      0      0      1      0   \n",
       "\n",
       "   agebin_2  agebin_3  sexmar_3  sexmar_4  sexmar_5  sexage_3  sexage_4  \\\n",
       "0         1         0         1         0         0         0         1   \n",
       "1         1         0         1         0         0         0         1   \n",
       "2         1         0         1         0         0         0         1   \n",
       "3         0         0         1         0         0         1         0   \n",
       "4         1         0         0         1         0         0         1   \n",
       "\n",
       "   sexage_5  \n",
       "0         0  \n",
       "1         0  \n",
       "2         0  \n",
       "3         0  \n",
       "4         0  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# # Grab indices of columns for creating dummy variables and create dataframe with dummy variables\n",
    "dum_feat = X_train2[['gender', 'education', 'marriage', 'age_bin', 'gen-mar', 'gen-age']]\n",
    "dum_index = dum_feat.columns\n",
    "tr_dum = pd.get_dummies(data=dum_feat, columns=dum_index, drop_first=True, prefix=['sex', 'edu', 'mar', 'agebin', 'sexmar', 'sexage'])\n",
    "cont_feat = X_train2.drop(['gender', 'education', 'marriage', 'age_bin', 'gen-mar', 'gen-age'], axis=1)\n",
    "X_train2_dum = cont_feat.join(tr_dum)\n",
    "X_train2_dum.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>limit</th>\n",
       "      <th>age</th>\n",
       "      <th>behind1</th>\n",
       "      <th>behind2</th>\n",
       "      <th>behind3</th>\n",
       "      <th>behind4</th>\n",
       "      <th>behind5</th>\n",
       "      <th>behind6</th>\n",
       "      <th>billed1</th>\n",
       "      <th>billed2</th>\n",
       "      <th>billed3</th>\n",
       "      <th>billed4</th>\n",
       "      <th>billed5</th>\n",
       "      <th>billed6</th>\n",
       "      <th>paid1</th>\n",
       "      <th>paid2</th>\n",
       "      <th>paid3</th>\n",
       "      <th>paid4</th>\n",
       "      <th>paid5</th>\n",
       "      <th>paid6</th>\n",
       "      <th>avail6</th>\n",
       "      <th>avail5</th>\n",
       "      <th>avail4</th>\n",
       "      <th>avail3</th>\n",
       "      <th>avail2</th>\n",
       "      <th>avail1</th>\n",
       "      <th>avg_av</th>\n",
       "      <th>delayed</th>\n",
       "      <th>latemths</th>\n",
       "      <th>pperb1</th>\n",
       "      <th>pperb2</th>\n",
       "      <th>pperb3</th>\n",
       "      <th>pperb4</th>\n",
       "      <th>pperb5</th>\n",
       "      <th>sex_2</th>\n",
       "      <th>edu_2</th>\n",
       "      <th>edu_3</th>\n",
       "      <th>edu_4</th>\n",
       "      <th>mar_2</th>\n",
       "      <th>mar_3</th>\n",
       "      <th>agebin_2</th>\n",
       "      <th>agebin_3</th>\n",
       "      <th>sexmar_3</th>\n",
       "      <th>sexmar_4</th>\n",
       "      <th>sexmar_5</th>\n",
       "      <th>sexage_3</th>\n",
       "      <th>sexage_4</th>\n",
       "      <th>sexage_5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1074.16</td>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>317.38</td>\n",
       "      <td>360.27</td>\n",
       "      <td>414.66</td>\n",
       "      <td>450.43</td>\n",
       "      <td>491.10</td>\n",
       "      <td>530.92</td>\n",
       "      <td>53.71</td>\n",
       "      <td>71.61</td>\n",
       "      <td>53.71</td>\n",
       "      <td>53.71</td>\n",
       "      <td>53.71</td>\n",
       "      <td>71.61</td>\n",
       "      <td>0.505735</td>\n",
       "      <td>0.542806</td>\n",
       "      <td>0.580668</td>\n",
       "      <td>0.613968</td>\n",
       "      <td>0.664603</td>\n",
       "      <td>0.704532</td>\n",
       "      <td>0.602052</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.149083</td>\n",
       "      <td>0.172696</td>\n",
       "      <td>0.119242</td>\n",
       "      <td>0.109367</td>\n",
       "      <td>0.101164</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5370.78</td>\n",
       "      <td>26</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4895.86</td>\n",
       "      <td>4498.96</td>\n",
       "      <td>4177.89</td>\n",
       "      <td>3637.13</td>\n",
       "      <td>2783.53</td>\n",
       "      <td>2766.45</td>\n",
       "      <td>160.62</td>\n",
       "      <td>151.64</td>\n",
       "      <td>113.18</td>\n",
       "      <td>94.78</td>\n",
       "      <td>95.56</td>\n",
       "      <td>95.56</td>\n",
       "      <td>0.484907</td>\n",
       "      <td>0.481727</td>\n",
       "      <td>0.322793</td>\n",
       "      <td>0.222107</td>\n",
       "      <td>0.162327</td>\n",
       "      <td>0.088427</td>\n",
       "      <td>0.293715</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.035702</td>\n",
       "      <td>0.036296</td>\n",
       "      <td>0.031118</td>\n",
       "      <td>0.034050</td>\n",
       "      <td>0.034542</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2506.36</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2510.73</td>\n",
       "      <td>2473.42</td>\n",
       "      <td>2453.73</td>\n",
       "      <td>2497.52</td>\n",
       "      <td>2510.34</td>\n",
       "      <td>2513.96</td>\n",
       "      <td>87.04</td>\n",
       "      <td>111.43</td>\n",
       "      <td>107.42</td>\n",
       "      <td>87.29</td>\n",
       "      <td>89.51</td>\n",
       "      <td>91.45</td>\n",
       "      <td>-0.003032</td>\n",
       "      <td>-0.001588</td>\n",
       "      <td>0.003527</td>\n",
       "      <td>0.020999</td>\n",
       "      <td>0.013143</td>\n",
       "      <td>-0.001744</td>\n",
       "      <td>0.005217</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.035190</td>\n",
       "      <td>0.045412</td>\n",
       "      <td>0.043011</td>\n",
       "      <td>0.034772</td>\n",
       "      <td>0.035605</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4654.68</td>\n",
       "      <td>49</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>740.38</td>\n",
       "      <td>678.72</td>\n",
       "      <td>579.04</td>\n",
       "      <td>605.04</td>\n",
       "      <td>402.31</td>\n",
       "      <td>248.63</td>\n",
       "      <td>57.65</td>\n",
       "      <td>64.74</td>\n",
       "      <td>251.14</td>\n",
       "      <td>0.97</td>\n",
       "      <td>251.03</td>\n",
       "      <td>157.83</td>\n",
       "      <td>0.946585</td>\n",
       "      <td>0.913569</td>\n",
       "      <td>0.870015</td>\n",
       "      <td>0.875600</td>\n",
       "      <td>0.854185</td>\n",
       "      <td>0.840939</td>\n",
       "      <td>0.883482</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.084939</td>\n",
       "      <td>0.111806</td>\n",
       "      <td>0.415080</td>\n",
       "      <td>0.002411</td>\n",
       "      <td>1.009653</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1790.26</td>\n",
       "      <td>36</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>3373.85</td>\n",
       "      <td>1705.58</td>\n",
       "      <td>1516.74</td>\n",
       "      <td>700.85</td>\n",
       "      <td>726.67</td>\n",
       "      <td>696.02</td>\n",
       "      <td>71.61</td>\n",
       "      <td>53.71</td>\n",
       "      <td>35.81</td>\n",
       "      <td>64.45</td>\n",
       "      <td>0.00</td>\n",
       "      <td>35.81</td>\n",
       "      <td>0.611218</td>\n",
       "      <td>0.594098</td>\n",
       "      <td>0.608521</td>\n",
       "      <td>0.152782</td>\n",
       "      <td>0.047300</td>\n",
       "      <td>-0.884559</td>\n",
       "      <td>0.188227</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.041986</td>\n",
       "      <td>0.035411</td>\n",
       "      <td>0.051095</td>\n",
       "      <td>0.088692</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     limit  age  behind1  behind2  behind3  behind4  behind5  behind6  \\\n",
       "0  1074.16   25        0        0        0        0        0        0   \n",
       "1  5370.78   26        0        0        0        0        0        0   \n",
       "2  2506.36   32        0        0        0        0        0        0   \n",
       "3  4654.68   49        0        0        0        0        0       -1   \n",
       "4  1790.26   36        0        0        0        0        0        2   \n",
       "\n",
       "   billed1  billed2  billed3  billed4  billed5  billed6   paid1   paid2  \\\n",
       "0   317.38   360.27   414.66   450.43   491.10   530.92   53.71   71.61   \n",
       "1  4895.86  4498.96  4177.89  3637.13  2783.53  2766.45  160.62  151.64   \n",
       "2  2510.73  2473.42  2453.73  2497.52  2510.34  2513.96   87.04  111.43   \n",
       "3   740.38   678.72   579.04   605.04   402.31   248.63   57.65   64.74   \n",
       "4  3373.85  1705.58  1516.74   700.85   726.67   696.02   71.61   53.71   \n",
       "\n",
       "    paid3  paid4   paid5   paid6    avail6    avail5    avail4    avail3  \\\n",
       "0   53.71  53.71   53.71   71.61  0.505735  0.542806  0.580668  0.613968   \n",
       "1  113.18  94.78   95.56   95.56  0.484907  0.481727  0.322793  0.222107   \n",
       "2  107.42  87.29   89.51   91.45 -0.003032 -0.001588  0.003527  0.020999   \n",
       "3  251.14   0.97  251.03  157.83  0.946585  0.913569  0.870015  0.875600   \n",
       "4   35.81  64.45    0.00   35.81  0.611218  0.594098  0.608521  0.152782   \n",
       "\n",
       "     avail2    avail1    avg_av  delayed  latemths    pperb1    pperb2  \\\n",
       "0  0.664603  0.704532  0.602052        0         0  0.149083  0.172696   \n",
       "1  0.162327  0.088427  0.293715        0         0  0.035702  0.036296   \n",
       "2  0.013143 -0.001744  0.005217        0         0  0.035190  0.045412   \n",
       "3  0.854185  0.840939  0.883482        0         0  0.084939  0.111806   \n",
       "4  0.047300 -0.884559  0.188227        1         1  0.041986  0.035411   \n",
       "\n",
       "     pperb3    pperb4    pperb5  sex_2  edu_2  edu_3  edu_4  mar_2  mar_3  \\\n",
       "0  0.119242  0.109367  0.101164      0      1      0      0      1      0   \n",
       "1  0.031118  0.034050  0.034542      1      0      0      0      1      0   \n",
       "2  0.043011  0.034772  0.035605      1      0      1      0      0      0   \n",
       "3  0.415080  0.002411  1.009653      0      0      1      0      1      0   \n",
       "4  0.051095  0.088692  0.000000      1      1      0      0      1      0   \n",
       "\n",
       "   agebin_2  agebin_3  sexmar_3  sexmar_4  sexmar_5  sexage_3  sexage_4  \\\n",
       "0         0         0         1         0         0         0         0   \n",
       "1         0         0         0         1         0         1         0   \n",
       "2         1         0         1         0         0         0         1   \n",
       "3         1         0         1         0         0         1         0   \n",
       "4         1         0         0         1         0         0         1   \n",
       "\n",
       "   sexage_5  \n",
       "0         0  \n",
       "1         0  \n",
       "2         0  \n",
       "3         0  \n",
       "4         0  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dum_feat2 = X_validate2[['gender', 'education', 'marriage', 'age_bin', 'gen-mar', 'gen-age']]\n",
    "dum_index2 = dum_feat2.columns\n",
    "val_dum = pd.get_dummies(data=dum_feat2, columns=dum_index2, drop_first=True, prefix=['sex', 'edu', 'mar', 'agebin', 'sexmar', 'sexage'])\n",
    "cont_feat2 = X_validate2.drop(['gender', 'education', 'marriage', 'age_bin', 'gen-mar', 'gen-age'], axis=1)\n",
    "X_validate2_dum = cont_feat2.join(val_dum)\n",
    "X_validate2_dum.head()"
   ]
  },
  {
   "source": [
    "# Modeling\n",
    "\n",
    "- Building a Vanilla Model:\n",
    "    - Logistic Regression, Decision Tree, Random Forest, Gaussian Naive Bayes, Linear Discriminant Analysis, K-Nearest Neighbors, Adaboost, Gradient Boosting, XGBoost\n",
    "- Feature Selection:\n",
    "    - Random Forest Feature Importance\n",
    "    - Decision Tree Feature Importance\n",
    "    - XGBoost Feature Importance\n",
    "    - Recursive Feature Elimination with Cross Validation\n",
    "- Develop Baseline Models:\n",
    "    - Logistic Regression\n",
    "    - Random Forest Classifier\n",
    "    - Adaboost Classifier\n",
    "    - Gradient Boosting Classifier\n",
    "    - XGBoost Classifier\n",
    "- Hyperparameter Tuning:\n",
    "    - Using GridSearchCV\n",
    "- Tuned Models:\n",
    "    - Logistic Regression, Random Forest, Adaboost, Gradient Boosting, XGBoost Classifiers\n",
    "- Class Imbalance Methods:\n",
    "    - Ensemble:  BaggingClassifier, BalancedBaggingClassifier\n",
    "    - Undersampling: TomekLinks, ENN\n",
    "    - Oversampling: SMOTE, ADASYN\n",
    "    - Hybridized: SMOTE-ENN, SMOTE-Tomek"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler = StandardScaler().fit(X_train2_dum)\n",
    "X_tr2_dum = scaler.transform(X_train2_dum)\n",
    "X_val2_dum = scaler.transform(X_validate2_dum)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler2 = StandardScaler().fit(X_train2)\n",
    "X_tr2 = scaler2.transform(X_train2)\n",
    "X_val2 = scaler2.transform(X_validate2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logreg2 = LogisticRegression(solver=\"liblinear\", random_state=42).fit(X_tr2_dum, y_tr)\n",
    "y_pred_log_tr2 = logreg2.predict(X_tr2_dum)\n",
    "y_pred_log_val2 = logreg2.predict(X_val2_dum)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rfc2 = RandomForestClassifier().fit(X_tr2, y_tr)\n",
    "y_pred_rfc_tr2 = rfc2.predict(X_tr2)\n",
    "y_pred_rfc_val2 = rfc2.predict(X_val2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dtc2 = DecisionTreeClassifier().fit(X_tr2, y_tr)\n",
    "y_pred_dtc_tr2 = dtc2.predict(X_tr2)\n",
    "y_pred_dtc_val2 = dtc2.predict(X_val2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "knn2 = KNeighborsClassifier().fit(X_tr2, y_tr)\n",
    "y_pred_knn_tr2 = knn2.predict(X_tr2)\n",
    "y_pred_knn_val2 = knn2.predict(X_val2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gnb2 = GaussianNB().fit(X_tr2, y_tr)\n",
    "y_pred_gnb_tr2 = gnb2.predict(X_tr2)\n",
    "y_pred_gnb_val2 = gnb2.predict(X_val2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lda2 = LinearDiscriminantAnalysis().fit(X_tr2, y_tr)\n",
    "y_pred_lda_tr2 = lda2.predict(X_tr2)\n",
    "y_pred_lda_val2 = lda2.predict(X_val2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "abc2 = AdaBoostClassifier().fit(X_tr2, y_tr)\n",
    "y_pred_abc_tr2 = abc2.predict(X_tr2)\n",
    "y_pred_abc_val2 = abc2.predict(X_val2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gbc2 = GradientBoostingClassifier().fit(X_tr2, y_tr)\n",
    "y_pred_gbc_tr2 = gbc2.predict(X_tr2)\n",
    "y_pred_gbc_val2 = gbc2.predict(X_val2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb2 = XGBClassifier().fit(X_tr2, y_tr)\n",
    "y_pred_xgb_tr2 = xgb2.predict(X_tr2)\n",
    "y_pred_xgb_val2 = xgb2.predict(X_val2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>F1 Score</th>\n",
       "      <th>Recall</th>\n",
       "      <th>Precision</th>\n",
       "      <th>PR AUC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Logistic Regression 2</td>\n",
       "      <td>0.809000</td>\n",
       "      <td>0.399371</td>\n",
       "      <td>0.290175</td>\n",
       "      <td>0.640336</td>\n",
       "      <td>0.500013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Random Forest Classifier 2</td>\n",
       "      <td>0.816500</td>\n",
       "      <td>0.467344</td>\n",
       "      <td>0.367860</td>\n",
       "      <td>0.640584</td>\n",
       "      <td>0.517269</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Decision Tree Classifier 2</td>\n",
       "      <td>0.725167</td>\n",
       "      <td>0.393527</td>\n",
       "      <td>0.407464</td>\n",
       "      <td>0.380512</td>\n",
       "      <td>0.284835</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>K-Nearest Neighbors 2</td>\n",
       "      <td>0.789500</td>\n",
       "      <td>0.425125</td>\n",
       "      <td>0.355674</td>\n",
       "      <td>0.528281</td>\n",
       "      <td>0.405541</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Gaussian Naive Bayes 2</td>\n",
       "      <td>0.278167</td>\n",
       "      <td>0.374042</td>\n",
       "      <td>0.985529</td>\n",
       "      <td>0.230824</td>\n",
       "      <td>0.476403</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Linear Discriminant Analysis 2</td>\n",
       "      <td>0.809000</td>\n",
       "      <td>0.437684</td>\n",
       "      <td>0.339680</td>\n",
       "      <td>0.615172</td>\n",
       "      <td>0.500475</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>AdaBoost Classifier 2</td>\n",
       "      <td>0.818500</td>\n",
       "      <td>0.451385</td>\n",
       "      <td>0.341203</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.525636</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Gradient Boosting Classifier 2</td>\n",
       "      <td>0.820500</td>\n",
       "      <td>0.463378</td>\n",
       "      <td>0.354151</td>\n",
       "      <td>0.670029</td>\n",
       "      <td>0.544927</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>XGBoost Classifier 2</td>\n",
       "      <td>0.813333</td>\n",
       "      <td>0.458937</td>\n",
       "      <td>0.361767</td>\n",
       "      <td>0.627477</td>\n",
       "      <td>0.529126</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       Unnamed: 0  Accuracy  F1 Score    Recall  Precision  \\\n",
       "0           Logistic Regression 2  0.809000  0.399371  0.290175   0.640336   \n",
       "1      Random Forest Classifier 2  0.816500  0.467344  0.367860   0.640584   \n",
       "2      Decision Tree Classifier 2  0.725167  0.393527  0.407464   0.380512   \n",
       "3           K-Nearest Neighbors 2  0.789500  0.425125  0.355674   0.528281   \n",
       "4          Gaussian Naive Bayes 2  0.278167  0.374042  0.985529   0.230824   \n",
       "5  Linear Discriminant Analysis 2  0.809000  0.437684  0.339680   0.615172   \n",
       "6           AdaBoost Classifier 2  0.818500  0.451385  0.341203   0.666667   \n",
       "7  Gradient Boosting Classifier 2  0.820500  0.463378  0.354151   0.670029   \n",
       "8            XGBoost Classifier 2  0.813333  0.458937  0.361767   0.627477   \n",
       "\n",
       "     PR AUC  \n",
       "0  0.500013  \n",
       "1  0.517269  \n",
       "2  0.284835  \n",
       "3  0.405541  \n",
       "4  0.476403  \n",
       "5  0.500475  \n",
       "6  0.525636  \n",
       "7  0.544927  \n",
       "8  0.529126  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features_model = pd.read_csv(\"data/charts/features_model.csv\")\n",
    "features_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Observations: **\n",
    "\n",
    "In imbalanced datasets, we don't want to use accuracy as our gold standard metric since it is easy to get a high accuracy score by simply classifying all observations as the majority class.\n",
    "\n",
    "ROC AUC score is equivalent to calculating the rank correlation between predictions and targets, i.e. how good at ranking predictions the model is. It tells us what is the probability that a positive instance randomly chosen is ranked higher than a negative instance randomly chosen. In ignoring False Negatives, the ROC AUC can be misleading in very imbalanced datasets because we can achieve a high ROC AUC score even though msot of the minority class is misclassified.\n",
    "\n",
    "With the F1 score, it combines both recall and prevision into one metric by calculating the harmonic mean. It is worth noting that the F1 score is a special case of the F-beta score, where 1 indicates that we care about recall and precision equally. For a F2 score, we care about more recall more than precision, in fact, twice as much. Both F1 score and acuracy is calculated on the predicted classes not the prediction scores. We can adjust the threshold to finetune the F1 score, but accuracy also depends on the threshold.\n",
    "\n",
    "PR AUC score shows the tradeoff between precision and recall at every threshold, where a high score or area represents both high recall and precision. High precision relates to a low FPR, and high recall relates to a low FNR. A high score means that the classifier is returning accurate results (high precision), as well as returning a majority of all positive results (high recall).\n",
    "\n",
    "- To be expected, Random Forest and Decision Tree is overfit with high training accuracy and much lower validation accuracy.\n",
    "- Tree-based and ensemble classifiers have the most potential with the highest PR AUC scores: Random Forest, AdaBoost, Gradient Boosting, XGBoost\n",
    "- Decision Tree once again has an exceptionally low PR AUC score\n",
    "- Gaussian Naive Bayes has an exceptionally low accuracy score in the second model\n",
    "- A change in the F1 score between first and second model is accompanied by changes in recall and precision, which is to be expected since there is a tradeoff between recall and precision.\n",
    "- Gaussian Bayes Classifier metrics changes significantly between the two models\n",
    "- For both models, Gradient Boosting, AdaBoost, XGBoost, Random Forest have the highest PR AUC scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature Selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/rf_feature_importance.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/dtc_feature_importance.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/xgb_feature_importance.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/rfecv.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Observations: **\n",
    "\n",
    "Top Features:\n",
    "\n",
    "- Decision Tree Top 10: `behind1`, `age`, `latemnths`, `avail1`, `avail2`, `billed`, `avail5`, `limit`, `paid1`, `avg_av`....\n",
    "- Random Forest Top 10: `behind1`, `age`, `latemnths`, `limit`, `avg_av`, `avail1`, `billed1`, `delayed`, `avail2`, `behind2`....\n",
    "- XGBoost FI: `delayed`, `behind1`. (also `latemths`, `behind2`)\n",
    "- RFECV: `limit`, `behind1`, `paid2`, `delayed`, `latemths`\n",
    "\n",
    "Features to Remove:\n",
    "\n",
    "- `age_bin`, `gender`, `marriage`, `gen-age`, `gen-mar`, `behind6`, `behind5`, `behind4`, `education`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# New Baseline Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle_in = open(\"data/pickles/training_model.pickle\",\"rb\")\n",
    "train3 = pickle.load(pickle_in)\n",
    "pickle_in = open(\"data/pickles/validate_model.pickle\",\"rb\")\n",
    "validate3 = pickle.load(pickle_in)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train3 = train3.drop([\"default\"], axis=1)\n",
    "y_tr = train3[\"default\"]\n",
    "X_validate3 = validate3.drop([\"default\"], axis=1)\n",
    "y_val = validate3[\"default\"]\n",
    "\n",
    "scaler3 = StandardScaler().fit(X_train3)\n",
    "X_tr3 = scaler3.transform(X_train3)\n",
    "X_val3 = scaler3.transform(X_validate3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logreg3 = LogisticRegression(solver=\"liblinear\", random_state=42).fit(X_tr3, y_tr)\n",
    "y_pred_log_tr3 = logreg3.predict(X_tr3)\n",
    "y_pred_log_val3 = logreg3.predict(X_val3)\n",
    "\n",
    "rfc3 = RandomForestClassifier().fit(X_tr3, y_tr)\n",
    "y_pred_rfc_tr3 = rfc3.predict(X_tr3)\n",
    "y_pred_rfc_val3 = rfc3.predict(X_val3)\n",
    "\n",
    "abc3 = AdaBoostClassifier().fit(X_tr3, y_tr)\n",
    "y_pred_abc_tr3 = abc3.predict(X_tr3)\n",
    "y_pred_abc_val3 = abc3.predict(X_val3)\n",
    "\n",
    "gbc3 = GradientBoostingClassifier().fit(X_tr3, y_tr)\n",
    "y_pred_gbc_tr3 = gbc3.predict(X_tr3)\n",
    "y_pred_gbc_val3 = gbc3.predict(X_val3)\n",
    "\n",
    "xgb3 = XGBClassifier().fit(X_tr3, y_tr)\n",
    "y_pred_xgb_tr3 = xgb3.predict(X_tr3)\n",
    "y_pred_xgb_val3 = xgb3.predict(X_val3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>F1 Score</th>\n",
       "      <th>Recall</th>\n",
       "      <th>Precision</th>\n",
       "      <th>PR AUC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Logistic Regression New Baseline</td>\n",
       "      <td>0.807167</td>\n",
       "      <td>0.380952</td>\n",
       "      <td>0.271135</td>\n",
       "      <td>0.640288</td>\n",
       "      <td>0.498666</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Random Forest New Baseline</td>\n",
       "      <td>0.811500</td>\n",
       "      <td>0.461685</td>\n",
       "      <td>0.369383</td>\n",
       "      <td>0.615482</td>\n",
       "      <td>0.492459</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AdaBoost New Baseline</td>\n",
       "      <td>0.820167</td>\n",
       "      <td>0.448646</td>\n",
       "      <td>0.334349</td>\n",
       "      <td>0.681677</td>\n",
       "      <td>0.531694</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Gradient Boosting New Baseline</td>\n",
       "      <td>0.820500</td>\n",
       "      <td>0.466039</td>\n",
       "      <td>0.357959</td>\n",
       "      <td>0.667614</td>\n",
       "      <td>0.542846</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>XGBoost New Baseline</td>\n",
       "      <td>0.814167</td>\n",
       "      <td>0.455300</td>\n",
       "      <td>0.354912</td>\n",
       "      <td>0.634877</td>\n",
       "      <td>0.508630</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         Unnamed: 0  Accuracy  F1 Score    Recall  Precision  \\\n",
       "0  Logistic Regression New Baseline  0.807167  0.380952  0.271135   0.640288   \n",
       "1        Random Forest New Baseline  0.811500  0.461685  0.369383   0.615482   \n",
       "2             AdaBoost New Baseline  0.820167  0.448646  0.334349   0.681677   \n",
       "3    Gradient Boosting New Baseline  0.820500  0.466039  0.357959   0.667614   \n",
       "4              XGBoost New Baseline  0.814167  0.455300  0.354912   0.634877   \n",
       "\n",
       "     PR AUC  \n",
       "0  0.498666  \n",
       "1  0.492459  \n",
       "2  0.531694  \n",
       "3  0.542846  \n",
       "4  0.508630  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_baseline = pd.read_csv(\"data/charts/new_baseline.csv\")\n",
    "new_baseline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Observations: **\n",
    "\n",
    "- Brought it down to top 10 features\n",
    "- Gradient Boosting has the highest PR AUC Score as well as the highest F1 Score, so we are maximizing Recall and Precision in this model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hyperparameter Tuning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logreg = LogisticRegression()\n",
    "params = {'C': [0.001, 0.01, 0.1, 1, 10], \n",
    "          'penalty': ['none', 'l1', 'l2', 'elasticnet'],\n",
    "          'solver': ['liblinear', 'newton-cg', 'lbfgs', 'sag', 'saga']}\n",
    "gslog = GridSearchCV(estimator = logreg,\n",
    "                     param_grid = params,\n",
    "                     scoring = 'average_precision',\n",
    "                     cv = 10,\n",
    "                     n_jobs = -1).fit(X_tr, y_tr)\n",
    "y_pred_gslog_tr = gslog.predict(X_tr)\n",
    "y_pred_gslog_val = gslog.predict(X_val)\n",
    "\n",
    "\n",
    "rfc = RandomForestClassifier()\n",
    "params = {'n_estimators': [100, 200, 400, 600, 1000],\n",
    "          'criterion': ['entropy', 'gini'],\n",
    "          'max_depth': [5, 8, 15, 25, 30],\n",
    "          'min_samples_split': [2, 5, 10, 15, 100],\n",
    "          'min_samples_leaf': [1, 2, 5, 10]}\n",
    "gsrfc = GridSearchCV(estimator = rfc,\n",
    "                     param_grid = params,\n",
    "                     scoring = 'average_precision',\n",
    "                     cv = 5,\n",
    "                     n_jobs = -1).fit(X_tr, y_tr)\n",
    "y_pred_gsrfc_tr = gsrfc.predict(X_tr)\n",
    "y_pred_gsrfc_val = gsrfc.predict(X_val)\n",
    "\n",
    "\n",
    "rfc = RandomForestClassifier()\n",
    "params = {'n_estimators': [100, 200, 400, 600, 1000],\n",
    "          'criterion': ['entropy', 'gini'],\n",
    "          'max_depth': [5, 8, 15, 25, 30],\n",
    "          'min_samples_split': [2, 5, 10, 15, 100],\n",
    "          'min_samples_leaf': [1, 2, 5, 10]}\n",
    "gsrfc = GridSearchCV(estimator = rfc,\n",
    "                     param_grid = params,\n",
    "                     scoring = 'average_precision',\n",
    "                     cv = 5,\n",
    "                     n_jobs = -1).fit(X_tr, y_tr)\n",
    "y_pred_gsrfc_tr = gsrfc.predict(X_tr)\n",
    "y_pred_gsrfc_val = gsrfc.predict(X_val)\n",
    "\n",
    "abc = AdaBoostClassifier()\n",
    "params = {'n_estimators': [10, 50, 100, 200],\n",
    "          'learning_rate': [0.001, 0.01, 0.1, 0.2, 0.5]}\n",
    "gsabc = GridSearchCV(estimator = abc,\n",
    "                     param_grid = params,\n",
    "                     n_jobs = -1,\n",
    "                     cv = 5,\n",
    "                     scoring = 'average_precision').fit(X_tr, y_tr)\n",
    "y_pred_gsabc_tr = gsabc.predict(X_tr)\n",
    "y_pred_gsabc_val = gsabc.predict(X_val)\n",
    "\n",
    "gbc = GradientBoostingClassifier()\n",
    "params = {'n_estimators': [10, 100, 1000],\n",
    "          'learning_rate': [0.001, 0.01, 0.1],\n",
    "          'max_depth': [3, 7, 9]}\n",
    "gsgbc = GridSearchCV(estimator = gbc,\n",
    "                     param_grid = params, \n",
    "                     n_jobs = -1, \n",
    "                     cv = 5, \n",
    "                     scoring = 'average_precision').fit(X_tr, y_tr)\n",
    "y_pred_gsgbc_tr = gsgbc.predict(X_tr)\n",
    "y_pred_gsgbc_val = gsgbc.predict(X_val)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Best Hyperparameters for each model: **\n",
    "\n",
    "- Logistic Regression: # Best: 0.522622 using {'C': 1, 'penalty': 'l2', 'solver': 'newton-cg'}\n",
    "- Random Forest: # Best: 0.565196 using {'criterion': 'gini', 'max_depth': 8, 'min_samples_leaf': 1, 'min_samples_split': 2, 'n_estimators': 400}\n",
    "- AdaBoost: # Best: 0.545818 using {'learning_rate': 0.1, 'n_estimators': 200}\n",
    "- Gradient Boosting: # Best: 0.558390 using {'learning_rate': 0.01, 'max_depth': 3, 'n_estimators': 1000}\n",
    "- XGBoost: # Best: 0.555500 using {'max_depth': 3, 'min_child_weight': 5, 'n_estimators': 50}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>F1 Score</th>\n",
       "      <th>Recall</th>\n",
       "      <th>Precision</th>\n",
       "      <th>PR AUC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Logistic Regression with GridSearchCV</td>\n",
       "      <td>0.807167</td>\n",
       "      <td>0.380952</td>\n",
       "      <td>0.271135</td>\n",
       "      <td>0.640288</td>\n",
       "      <td>0.498658</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Random Forest with GridSearchCV</td>\n",
       "      <td>0.820667</td>\n",
       "      <td>0.464143</td>\n",
       "      <td>0.354912</td>\n",
       "      <td>0.670504</td>\n",
       "      <td>0.545164</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AdaBoost with GridSearchCV</td>\n",
       "      <td>0.820833</td>\n",
       "      <td>0.443870</td>\n",
       "      <td>0.326733</td>\n",
       "      <td>0.691935</td>\n",
       "      <td>0.524433</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Gradient Boosting with GridSearchCV</td>\n",
       "      <td>0.819667</td>\n",
       "      <td>0.462227</td>\n",
       "      <td>0.354151</td>\n",
       "      <td>0.665236</td>\n",
       "      <td>0.543499</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>XGBoost with GridSearchCV</td>\n",
       "      <td>0.818500</td>\n",
       "      <td>0.457939</td>\n",
       "      <td>0.350343</td>\n",
       "      <td>0.660920</td>\n",
       "      <td>0.539395</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              Unnamed: 0  Accuracy  F1 Score    Recall  \\\n",
       "0  Logistic Regression with GridSearchCV  0.807167  0.380952  0.271135   \n",
       "1        Random Forest with GridSearchCV  0.820667  0.464143  0.354912   \n",
       "2             AdaBoost with GridSearchCV  0.820833  0.443870  0.326733   \n",
       "3    Gradient Boosting with GridSearchCV  0.819667  0.462227  0.354151   \n",
       "4              XGBoost with GridSearchCV  0.818500  0.457939  0.350343   \n",
       "\n",
       "   Precision    PR AUC  \n",
       "0   0.640288  0.498658  \n",
       "1   0.670504  0.545164  \n",
       "2   0.691935  0.524433  \n",
       "3   0.665236  0.543499  \n",
       "4   0.660920  0.539395  "
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores3 = pd.read_csv(\"data/charts/scores3.csv\")\n",
    "scores3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Observations: **\n",
    "\n",
    "- GridSearchCV was run with the scoring parameter set to find the highest average precision score, which is the PR AUC score.\n",
    "- Random Forest after hyperparameter tuning has the best accuracy as well as the highest PR AUC score.\n",
    "- We pickle out Random Forest with GridSearchCV tuned parameters as our best model for now before completing class imbalance methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Class Imbalance Methods\n",
    "\n",
    "(Will update soon....)\n",
    "\n",
    "- Initial findings show that there is dramatically improved PR AUC scores without sacrificing any accuracy.\n",
    "- Accuracy has not improved much, but has not decreased."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pickle Out Best Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle_out = open(\"../data/best_model.pickle\",\"wb\")\n",
    "pickle.dump(rfcb, pickle_out)\n",
    "pickle_out.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Holdout Set Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = pd.read_csv('../data/testing.csv')\n",
    "tt = test.drop([\"ID\"], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = 'https://openexchangerates.org/api/latest.json?app_id=c51b1508fb4145259b1c2fade72a2c04'\n",
    "response = requests.get(url)\n",
    "data = response.json()\n",
    "rate = data['rates']['TWD']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [tt]\n",
    "for d in data:    \n",
    "    d.rename(columns={\"PAY_0\": \"behind1\", \n",
    "                        \"PAY_2\": \"behind2\", \n",
    "                        \"PAY_3\": \"behind3\", \n",
    "                        \"PAY_4\": \"behind4\",\n",
    "                        \"PAY_5\": \"behind5\", \n",
    "                        \"PAY_6\": \"behind6\", \n",
    "                        \"BILL_AMT1\": \"billed1\", \n",
    "                        \"BILL_AMT2\": \"billed2\", \n",
    "                        \"BILL_AMT3\": \"billed3\", \n",
    "                        \"BILL_AMT4\": \"billed4\", \n",
    "                        \"BILL_AMT5\": \"billed5\", \n",
    "                        \"BILL_AMT6\": \"billed6\", \n",
    "                        \"PAY_AMT1\": \"paid1\", \n",
    "                        \"PAY_AMT2\": \"paid2\", \n",
    "                        \"PAY_AMT3\": \"paid3\", \n",
    "                        \"PAY_AMT4\": \"paid4\", \n",
    "                        \"PAY_AMT5\": \"paid5\", \n",
    "                        \"PAY_AMT6\": \"paid6\",\n",
    "                        \"SEX\": \"gender\",\n",
    "                        \"EDUCATION\": \"education\",\n",
    "                        \"MARRIAGE\": \"marriage\",\n",
    "                        \"AGE\": \"age\",\n",
    "                        \"LIMIT_BAL\": \"limit\"}, inplace=True)\n",
    "    d[['limit']] = d[['limit']]/rate\n",
    "    d[['billed1', 'billed2', 'billed3', 'billed4', 'billed5', 'billed6']] = d[['billed1', 'billed2', 'billed3', 'billed4', 'billed5', 'billed6']].divide(rate, axis=1)\n",
    "    d[['paid1', 'paid2', 'paid3', 'paid4', 'paid5', 'paid6']] = d[['paid1', 'paid2', 'paid3', 'paid4', 'paid5', 'paid6']].divide(rate, axis=1)\n",
    "    d['limit'] = d['limit'].apply(lambda x: round(x, 2))\n",
    "    d[['billed1', 'billed2', 'billed3', 'billed4', 'billed5', 'billed6']] = d[['billed1', 'billed2', 'billed3', 'billed4', 'billed5', 'billed6']].apply(lambda x: round(x, 2))\n",
    "    d[['paid1', 'paid2', 'paid3', 'paid4', 'paid5', 'paid6']] = d[['paid1', 'paid2', 'paid3', 'paid4', 'paid5', 'paid6']].apply(lambda x: round(x, 2))\n",
    "    d.replace({'marriage': {0:3}}, inplace=True)\n",
    "    d.replace({'education': {5:4, 0:4, 6:4}}, inplace=True)\n",
    "    \n",
    "tt = tt.drop([\"Unnamed: 0\"], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for d in data:\n",
    "    d['avail6'] = (d.limit - d.billed6) / d.limit\n",
    "    d['avail5'] = (d.limit - d.billed5) / d.limit\n",
    "    d['avail4'] = (d.limit - d.billed4) / d.limit\n",
    "    d['avail3'] = (d.limit - d.billed3) / d.limit\n",
    "    d['avail2'] = (d.limit - d.billed2) / d.limit\n",
    "    d['avail1'] = (d.limit - d.billed1) / d.limit\n",
    "    d['avg_av'] = (d.avail1 + d.avail2 + d.avail3 + d.avail4 + d.avail5 + d.avail6) / 6\n",
    "\n",
    "def delayed_payment(d):\n",
    "    if (d.behind1 > 0) or (d.behind2 > 0) or (d.behind3 > 0) or (d.behind4 > 0) or (d.behind5 > 0) or (d.behind6 > 0):\n",
    "        return 1\n",
    "    else:\n",
    "        return 0\n",
    "for d in data:\n",
    "    d['delayed'] = d.apply(delayed_payment, axis=1)\n",
    "\n",
    "def total_months_with_delayed_payments(d):\n",
    "    count = 0\n",
    "    if (d.behind1 > 0):\n",
    "        count += 1\n",
    "    if (d.behind2 > 0):\n",
    "        count += 1\n",
    "    if (d.behind3 > 0):\n",
    "        count += 1\n",
    "    if (d.behind4 > 0):\n",
    "        count += 1\n",
    "    if (d.behind5 > 0):\n",
    "        count += 1\n",
    "    if (d.behind6 > 0):\n",
    "        count += 1\n",
    "    return count\n",
    "for d in data:\n",
    "    d['latemths'] = d.apply(total_months_with_delayed_payments, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_tt = tt[['limit', 'behind1', 'paid2', 'delayed', 'latemths', 'age', 'behind2', 'billed1', 'avg_av', 'avail1']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle_in = open(\"../data/best_model.pickle\",\"rb\")\n",
    "model = pickle.load(pickle_in)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred_tt = model.predict(X_tt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle_out = open(\"../data/final_prediction.pickle\",\"wb\")\n",
    "pickle.dump(y_pred_tt, pickle_out)\n",
    "pickle_out.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analysis\n",
    "\n",
    "There was not a significant difference in the vanilla model, model with all the engineered features, and model after using feature selection methods.  The initial models were selected for the highest accuracy and PR AUC score.\n",
    "\n",
    "Some of the engineered features created seemed to have a stronger correlation than the original variables.  I have to check for collinearity as some of the variables would overlap in context.  I am surprised that the demographic features does not have a greater correlation with default.  It would seem useful for companies to be able to identify certain demographic groups that are more prone to defaulting.\n",
    "\n",
    "The metric I used was the PR AUC score, but with an eye to increasing accuracy and PR AUC score, which is the scoring parameters I used in GridSearchCV for hyperparameter tuning. Hyperparameter tuning improved accuracy to 82% from a baseline of 77%, and the highest PR AUC score at around 54%. My initial analysis of implementing class imbalance methods is that it substantially increases the PR AUC score to almost 90%, but accuracy tops out at 82% on the validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3.9.4 64-bit ('learn-env': conda)"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.4"
  },
  "interpreter": {
   "hash": "80773cdecc613a186b9eac051636553d2ff0f99c57590601666ed713bd5d0256"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}