House Price Analysis-Part One.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# House Price Analysis -- Part One"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### *Acknowledgement*\n",
    "The data set is downloaded as-is from Kaggle [\n",
    "House Prices: Advanced Regression Techniques](https://www.kaggle.com/c/house-prices-advanced-regression-techniques/dataO).<br>\n",
    "\n",
    "With **79 explanatory variables** describing almost every aspect of residential homes in `Ames, Iowa`.<br> \n",
    "The [Ames Housing dataset](http://jse.amstat.org/v19n3/decock.pdf) was compiled by **Dean De Cock** for use in data science education.<br> It's an incredible alternative for data scientists looking for a modernized and expanded version of<br> the often cited Boston Housing dataset[1]."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset Intro\n",
    "\n",
    "**79 Features** \n",
    "1. Quantitative Features \n",
    "    - `Continuous variables`: 19\n",
    "    - `Discrete variables`: 14 \n",
    "2. Qualitative Features\n",
    "    - `Nominal variables`: 23\n",
    "    - `Ordinal variables`: 23\n",
    "    \n",
    "**Numeric Target Variable**: `SalePrice`\n",
    "\n",
    "There are two datasets: `'train.csv'`, and `'test.csv'`. \n",
    "1. `'train.csv'` will be used to build and validate the model.\n",
    "2. `'test.csv'` will be used to predict the final price of each home.\n",
    "\n",
    "The difference between these two datasets is that `'test.csv'` does NOT have the `Target` variable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Purpose of the Project\n",
    "\n",
    "To use `advanced regression techniques` to build models that predict the final price of each property.<br>\n",
    "This project will focus on using functions in Python that can easily replicate the analysis process when using the `test.csv`.<br> The first major part of the analysis is on preprocessing and understanding the data set (House Price Analysis-Part One).<br> Then, various models will be built and evaluated on performance.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part One: Preprocess the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read dataset\n",
    "def readData(name = 'train.csv'):\n",
    "    data_df = pd.read_csv(name, keep_default_na = False).drop(columns=['Id'])\n",
    "    return data_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Street</th>\n",
       "      <th>Alley</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>LotConfig</th>\n",
       "      <th>...</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>PoolQC</th>\n",
       "      <th>Fence</th>\n",
       "      <th>MiscFeature</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>65</td>\n",
       "      <td>8450</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NA</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Inside</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NA</td>\n",
       "      <td>NA</td>\n",
       "      <td>NA</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>208500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>80</td>\n",
       "      <td>9600</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NA</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>FR2</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NA</td>\n",
       "      <td>NA</td>\n",
       "      <td>NA</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2007</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>181500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>68</td>\n",
       "      <td>11250</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NA</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Inside</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NA</td>\n",
       "      <td>NA</td>\n",
       "      <td>NA</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>223500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>70</td>\n",
       "      <td>RL</td>\n",
       "      <td>60</td>\n",
       "      <td>9550</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NA</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Corner</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NA</td>\n",
       "      <td>NA</td>\n",
       "      <td>NA</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2006</td>\n",
       "      <td>WD</td>\n",
       "      <td>Abnorml</td>\n",
       "      <td>140000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>84</td>\n",
       "      <td>14260</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NA</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>FR2</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NA</td>\n",
       "      <td>NA</td>\n",
       "      <td>NA</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>250000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 80 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   MSSubClass MSZoning LotFrontage  LotArea Street Alley LotShape LandContour  \\\n",
       "0          60       RL          65     8450   Pave    NA      Reg         Lvl   \n",
       "1          20       RL          80     9600   Pave    NA      Reg         Lvl   \n",
       "2          60       RL          68    11250   Pave    NA      IR1         Lvl   \n",
       "3          70       RL          60     9550   Pave    NA      IR1         Lvl   \n",
       "4          60       RL          84    14260   Pave    NA      IR1         Lvl   \n",
       "\n",
       "  Utilities LotConfig    ...     PoolArea PoolQC Fence MiscFeature MiscVal  \\\n",
       "0    AllPub    Inside    ...            0     NA    NA          NA       0   \n",
       "1    AllPub       FR2    ...            0     NA    NA          NA       0   \n",
       "2    AllPub    Inside    ...            0     NA    NA          NA       0   \n",
       "3    AllPub    Corner    ...            0     NA    NA          NA       0   \n",
       "4    AllPub       FR2    ...            0     NA    NA          NA       0   \n",
       "\n",
       "  MoSold  YrSold  SaleType  SaleCondition  SalePrice  \n",
       "0      2    2008        WD         Normal     208500  \n",
       "1      5    2007        WD         Normal     181500  \n",
       "2      9    2008        WD         Normal     223500  \n",
       "3      2    2006        WD        Abnorml     140000  \n",
       "4     12    2008        WD         Normal     250000  \n",
       "\n",
       "[5 rows x 80 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = readData()\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The data set has 1460 rows and 80 columns\n"
     ]
    }
   ],
   "source": [
    "def shapeOfDataSet(df):\n",
    "    print('The data set has {} rows and {} columns'.format(df.shape[0],\\\n",
    "                                                           df.shape[1]))\n",
    "shapeOfDataSet(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Understand the Unique Values\n",
    "\n",
    "To ensure there are NO invalid values/typos existing in the columns.<br> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def uniqueValues(df):\n",
    "    for col in df.columns:\n",
    "        if len(df[col].unique()) <= 20:\n",
    "            print('There are {} unique values in {} column'.format(len(df[col].unique()), col))\n",
    "            print(df[col].unique(), '\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 15 unique values in MSSubClass column\n",
      "[ 60  20  70  50 190  45  90 120  30  85  80 160  75 180  40] \n",
      "\n",
      "There are 5 unique values in MSZoning column\n",
      "['RL' 'RM' 'C (all)' 'FV' 'RH'] \n",
      "\n",
      "There are 2 unique values in Street column\n",
      "['Pave' 'Grvl'] \n",
      "\n",
      "There are 3 unique values in Alley column\n",
      "['NA' 'Grvl' 'Pave'] \n",
      "\n",
      "There are 4 unique values in LotShape column\n",
      "['Reg' 'IR1' 'IR2' 'IR3'] \n",
      "\n",
      "There are 4 unique values in LandContour column\n",
      "['Lvl' 'Bnk' 'Low' 'HLS'] \n",
      "\n",
      "There are 2 unique values in Utilities column\n",
      "['AllPub' 'NoSeWa'] \n",
      "\n",
      "There are 5 unique values in LotConfig column\n",
      "['Inside' 'FR2' 'Corner' 'CulDSac' 'FR3'] \n",
      "\n",
      "There are 3 unique values in LandSlope column\n",
      "['Gtl' 'Mod' 'Sev'] \n",
      "\n",
      "There are 9 unique values in Condition1 column\n",
      "['Norm' 'Feedr' 'PosN' 'Artery' 'RRAe' 'RRNn' 'RRAn' 'PosA' 'RRNe'] \n",
      "\n",
      "There are 8 unique values in Condition2 column\n",
      "['Norm' 'Artery' 'RRNn' 'Feedr' 'PosN' 'PosA' 'RRAn' 'RRAe'] \n",
      "\n",
      "There are 5 unique values in BldgType column\n",
      "['1Fam' '2fmCon' 'Duplex' 'TwnhsE' 'Twnhs'] \n",
      "\n",
      "There are 8 unique values in HouseStyle column\n",
      "['2Story' '1Story' '1.5Fin' '1.5Unf' 'SFoyer' 'SLvl' '2.5Unf' '2.5Fin'] \n",
      "\n",
      "There are 10 unique values in OverallQual column\n",
      "[ 7  6  8  5  9  4 10  3  1  2] \n",
      "\n",
      "There are 9 unique values in OverallCond column\n",
      "[5 8 6 7 4 2 3 9 1] \n",
      "\n",
      "There are 6 unique values in RoofStyle column\n",
      "['Gable' 'Hip' 'Gambrel' 'Mansard' 'Flat' 'Shed'] \n",
      "\n",
      "There are 8 unique values in RoofMatl column\n",
      "['CompShg' 'WdShngl' 'Metal' 'WdShake' 'Membran' 'Tar&Grv' 'Roll'\n",
      " 'ClyTile'] \n",
      "\n",
      "There are 15 unique values in Exterior1st column\n",
      "['VinylSd' 'MetalSd' 'Wd Sdng' 'HdBoard' 'BrkFace' 'WdShing' 'CemntBd'\n",
      " 'Plywood' 'AsbShng' 'Stucco' 'BrkComm' 'AsphShn' 'Stone' 'ImStucc'\n",
      " 'CBlock'] \n",
      "\n",
      "There are 16 unique values in Exterior2nd column\n",
      "['VinylSd' 'MetalSd' 'Wd Shng' 'HdBoard' 'Plywood' 'Wd Sdng' 'CmentBd'\n",
      " 'BrkFace' 'Stucco' 'AsbShng' 'Brk Cmn' 'ImStucc' 'AsphShn' 'Stone'\n",
      " 'Other' 'CBlock'] \n",
      "\n",
      "There are 5 unique values in MasVnrType column\n",
      "['BrkFace' 'None' 'Stone' 'BrkCmn' 'NA'] \n",
      "\n",
      "There are 4 unique values in ExterQual column\n",
      "['Gd' 'TA' 'Ex' 'Fa'] \n",
      "\n",
      "There are 5 unique values in ExterCond column\n",
      "['TA' 'Gd' 'Fa' 'Po' 'Ex'] \n",
      "\n",
      "There are 6 unique values in Foundation column\n",
      "['PConc' 'CBlock' 'BrkTil' 'Wood' 'Slab' 'Stone'] \n",
      "\n",
      "There are 5 unique values in BsmtQual column\n",
      "['Gd' 'TA' 'Ex' 'NA' 'Fa'] \n",
      "\n",
      "There are 5 unique values in BsmtCond column\n",
      "['TA' 'Gd' 'NA' 'Fa' 'Po'] \n",
      "\n",
      "There are 5 unique values in BsmtExposure column\n",
      "['No' 'Gd' 'Mn' 'Av' 'NA'] \n",
      "\n",
      "There are 7 unique values in BsmtFinType1 column\n",
      "['GLQ' 'ALQ' 'Unf' 'Rec' 'BLQ' 'NA' 'LwQ'] \n",
      "\n",
      "There are 7 unique values in BsmtFinType2 column\n",
      "['Unf' 'BLQ' 'NA' 'ALQ' 'Rec' 'LwQ' 'GLQ'] \n",
      "\n",
      "There are 6 unique values in Heating column\n",
      "['GasA' 'GasW' 'Grav' 'Wall' 'OthW' 'Floor'] \n",
      "\n",
      "There are 5 unique values in HeatingQC column\n",
      "['Ex' 'Gd' 'TA' 'Fa' 'Po'] \n",
      "\n",
      "There are 2 unique values in CentralAir column\n",
      "['Y' 'N'] \n",
      "\n",
      "There are 6 unique values in Electrical column\n",
      "['SBrkr' 'FuseF' 'FuseA' 'FuseP' 'Mix' 'NA'] \n",
      "\n",
      "There are 4 unique values in BsmtFullBath column\n",
      "[1 0 2 3] \n",
      "\n",
      "There are 3 unique values in BsmtHalfBath column\n",
      "[0 1 2] \n",
      "\n",
      "There are 4 unique values in FullBath column\n",
      "[2 1 3 0] \n",
      "\n",
      "There are 3 unique values in HalfBath column\n",
      "[1 0 2] \n",
      "\n",
      "There are 8 unique values in BedroomAbvGr column\n",
      "[3 4 1 2 0 5 6 8] \n",
      "\n",
      "There are 4 unique values in KitchenAbvGr column\n",
      "[1 2 3 0] \n",
      "\n",
      "There are 4 unique values in KitchenQual column\n",
      "['Gd' 'TA' 'Ex' 'Fa'] \n",
      "\n",
      "There are 12 unique values in TotRmsAbvGrd column\n",
      "[ 8  6  7  9  5 11  4 10 12  3  2 14] \n",
      "\n",
      "There are 7 unique values in Functional column\n",
      "['Typ' 'Min1' 'Maj1' 'Min2' 'Mod' 'Maj2' 'Sev'] \n",
      "\n",
      "There are 4 unique values in Fireplaces column\n",
      "[0 1 2 3] \n",
      "\n",
      "There are 6 unique values in FireplaceQu column\n",
      "['NA' 'TA' 'Gd' 'Fa' 'Ex' 'Po'] \n",
      "\n",
      "There are 7 unique values in GarageType column\n",
      "['Attchd' 'Detchd' 'BuiltIn' 'CarPort' 'NA' 'Basment' '2Types'] \n",
      "\n",
      "There are 4 unique values in GarageFinish column\n",
      "['RFn' 'Unf' 'Fin' 'NA'] \n",
      "\n",
      "There are 5 unique values in GarageCars column\n",
      "[2 3 1 0 4] \n",
      "\n",
      "There are 6 unique values in GarageQual column\n",
      "['TA' 'Fa' 'Gd' 'NA' 'Ex' 'Po'] \n",
      "\n",
      "There are 6 unique values in GarageCond column\n",
      "['TA' 'Fa' 'NA' 'Gd' 'Po' 'Ex'] \n",
      "\n",
      "There are 3 unique values in PavedDrive column\n",
      "['Y' 'N' 'P'] \n",
      "\n",
      "There are 20 unique values in 3SsnPorch column\n",
      "[  0 320 407 130 180 168 140 508 238 245 196 144 182 162  23 216  96 153\n",
      " 290 304] \n",
      "\n",
      "There are 8 unique values in PoolArea column\n",
      "[  0 512 648 576 555 480 519 738] \n",
      "\n",
      "There are 4 unique values in PoolQC column\n",
      "['NA' 'Ex' 'Fa' 'Gd'] \n",
      "\n",
      "There are 5 unique values in Fence column\n",
      "['NA' 'MnPrv' 'GdWo' 'GdPrv' 'MnWw'] \n",
      "\n",
      "There are 5 unique values in MiscFeature column\n",
      "['NA' 'Shed' 'Gar2' 'Othr' 'TenC'] \n",
      "\n",
      "There are 12 unique values in MoSold column\n",
      "[ 2  5  9 12 10  8 11  4  1  7  3  6] \n",
      "\n",
      "There are 5 unique values in YrSold column\n",
      "[2008 2007 2006 2009 2010] \n",
      "\n",
      "There are 9 unique values in SaleType column\n",
      "['WD' 'New' 'COD' 'ConLD' 'ConLI' 'CWD' 'ConLw' 'Con' 'Oth'] \n",
      "\n",
      "There are 6 unique values in SaleCondition column\n",
      "['Normal' 'Abnorml' 'Partial' 'AdjLand' 'Alloca' 'Family'] \n",
      "\n"
     ]
    }
   ],
   "source": [
    "uniqueValues(data) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is notable that feature `MSSubClass` is actually a qualitative variable but encoded as numeric.<br> \n",
    "A closer look at this feature, shown below:\n",
    "\n",
    "`MSSubClass`: Identifies the type of dwelling involved in the sale.\n",
    "\n",
    "        20\t1-STORY 1946 & NEWER ALL STYLES\n",
    "        30\t1-STORY 1945 & OLDER\n",
    "        40\t1-STORY W/FINISHED ATTIC ALL AGES\n",
    "        45\t1-1/2 STORY - UNFINISHED ALL AGES\n",
    "        50\t1-1/2 STORY FINISHED ALL AGES\n",
    "        60\t2-STORY 1946 & NEWER\n",
    "        70\t2-STORY 1945 & OLDER\n",
    "        75\t2-1/2 STORY ALL AGES\n",
    "        80\tSPLIT OR MULTI-LEVEL\n",
    "        85\tSPLIT FOYER\n",
    "        90\tDUPLEX - ALL STYLES AND AGES\n",
    "       120\t1-STORY PUD (Planned Unit Development) - 1946 & NEWER\n",
    "       150\t1-1/2 STORY PUD - ALL AGES\n",
    "       160\t2-STORY PUD - 1946 & NEWER\n",
    "       180\tPUD - MULTILEVEL - INCL SPLIT LEV/FOYER\n",
    "       190\t2 FAMILY CONVERSION - ALL STYLES AND AGES\n",
    "\n",
    "It is obvious that the numerical distance between each set of subsequent categories are **NOT** equal.<br>\n",
    "Therefore, the sensible next step is to understand the data type to further explore and understand the data set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Understand the Data Type in the Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a global variable for quantitative predictors in data set\n",
    "QUANT_VAR_LST = ['LotFrontage','LotArea','OverallQual','OverallCond','YearBuilt',\\\n",
    "                'YearRemodAdd','MasVnrArea','BsmtFinSF1','BsmtFinSF2','BsmtUnfSF',\\\n",
    "                'TotalBsmtSF','1stFlrSF','2ndFlrSF','LowQualFinSF','GrLivArea',\\\n",
    "                'BsmtFullBath','BsmtHalfBath','FullBath','HalfBath','BedroomAbvGr',\\\n",
    "                'KitchenAbvGr','TotRmsAbvGrd','Fireplaces','GarageYrBlt','GarageCars',\\\n",
    "                'GarageArea','WoodDeckSF','OpenPorchSF','EnclosedPorch','3SsnPorch',\\\n",
    "                'ScreenPorch','PoolArea','MiscVal','MoSold','YrSold']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def originalDataType(df):\n",
    "    '''\n",
    "    This function is to uncover the original data type of the data set, \n",
    "    and return the following:\n",
    "    orig_dataType_count - a dataframe showing different dtypes \n",
    "                          and the number of features for each type\n",
    "    dataType_dct - a dictionary with dtype as key and corresponding features as values \n",
    "    dataType_key_lst - a list of different dtypes existing in the data set\n",
    "    dataType_values_lst - a list of corresponding features in each dtype\n",
    "     \n",
    "    Parameter:\n",
    "    df - a pandas DataFrame of the dedicated data set that will be analyzed on\n",
    "    '''\n",
    "    if 'SalePrice' in df.columns:\n",
    "        orig_dataType_count = pd.DataFrame(df.drop(columns=['SalePrice'])\\\n",
    "                                           .dtypes.value_counts(), columns=['Total Amount'])\n",
    "        dataType_dct = {str(k): list(v) for k, v in df.drop(columns=['SalePrice'])\\\n",
    "                        .groupby(df.dtypes, axis=1)}\n",
    "    else:\n",
    "        orig_dataType_count = pd.DataFrame(df.dtypes.value_counts(), columns=['Total Amount'])\n",
    "        dataType_dct = {str(k): list(v) for k, v in df.groupby(df.dtypes, axis=1)}\n",
    "    \n",
    "    dataType_key_lst = list(dataType_dct.keys()) \n",
    "    dataType_values_lst = list(dataType_dct.values()) \n",
    "    \n",
    "    return orig_dataType_count, dataType_dct, dataType_key_lst, dataType_values_lst"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def referenceFeatureLst(df):  \n",
    "    '''\n",
    "    This function will return the following two lists:\n",
    "    qualitative_var_lst - a list of all correct qualitative features \n",
    "    reference_lst - a list of correct quantitative and qualitative features\n",
    "     \n",
    "    Parameter:\n",
    "    df - a pandas DataFrame of the dedicated data set that will be analyzed on\n",
    "    '''\n",
    "    if 'SalePrice' in df.columns:\n",
    "        qualitative_var_lst = [col for col in df.drop(columns=['SalePrice']).columns \\\n",
    "                       if col not in QUANT_VAR_LST]\n",
    "    else:\n",
    "        qualitative_var_lst = [col for col in df.columns if col not in QUANT_VAR_LST]\n",
    "    reference_lst = [QUANT_VAR_LST, qualitative_var_lst]\n",
    "    return qualitative_var_lst, reference_lst"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def incorrectDataType(df):\n",
    "    '''\n",
    "    This function will return a list of features that have incorrect dtypes \n",
    "     \n",
    "    Parameter:\n",
    "    df - a pandas DataFrame of the dedicated data set that will be analyzed on\n",
    "    '''\n",
    "    # Uncover original data type for each feature:\n",
    "    dataType_values_lst = originalDataType(df)[-1]\n",
    "    reference_lst = referenceFeatureLst(df)[-1]\n",
    "    \n",
    "    wrong_feature = [[],[]]\n",
    "    for i in range(len(dataType_values_lst)):\n",
    "        # Extract variables that are assigned incorrectly \n",
    "        diff_lst = [col for col in dataType_values_lst[i] \\\n",
    "                    if col not in reference_lst[i]]\n",
    "        wrong_feature[i].extend(diff_lst)\n",
    "    return wrong_feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def finalDataTypeCheck(df):\n",
    "    '''\n",
    "    This function integrates the above functions to demonstrate the whole process of \n",
    "    analyzing the original dtypes of each feature,\n",
    "    finding inconsistencies by comparing the assigned types with the reference list, \n",
    "    and finally identifying the features that have wrong dtypes.\n",
    "    \n",
    "    Parameter:\n",
    "    df - a pandas DataFrame of the dedicated data set that will be analyzed on\n",
    "    '''\n",
    "    # Uncover original data type related parameters:\n",
    "    orig_dataType_count, dataType_dct,\\\n",
    "    dataType_key_lst, dataType_values_lst = originalDataType(df)\n",
    "    \n",
    "    qualitative_var_lst = referenceFeatureLst(df)[0]\n",
    "    \n",
    "    wrong_feature = incorrectDataType(df)\n",
    "    \n",
    "    # Understandoriginal data types in data set\n",
    "    print('There are {} different data types:\\n'.format(len(orig_dataType_count)),\\\n",
    "          orig_dataType_count, '\\n')\n",
    "    \n",
    "    # Understand correct number of data types\n",
    "    print('There should be {} qualitative variables'.format(len(qualitative_var_lst)))\n",
    "    print('There should be {} quantitative variables\\n'.format(len(QUANT_VAR_LST)))\n",
    "    \n",
    "\n",
    "    for i in range(len(dataType_dct)):    \n",
    "        if len(wrong_feature) == 0:\n",
    "            print('All variables have the correct data type!') \n",
    "        else:\n",
    "            print('The following variables are assigned as data type: {}'\\\n",
    "                  .format(dataType_key_lst[i]))\n",
    "            print(dataType_values_lst[i], '\\n')\n",
    "        \n",
    "            print('However, there are {} variables should NOT have data type: {}'\\\n",
    "                  .format(len(wrong_feature[i]), dataType_key_lst[i]))\n",
    "            print(wrong_feature[i], '\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 2 different data types:\n",
      "         Total Amount\n",
      "object            46\n",
      "int64             33 \n",
      "\n",
      "There should be 44 qualitative variables\n",
      "There should be 35 quantitative variables\n",
      "\n",
      "The following variables are assigned as data type: int64\n",
      "['MSSubClass', 'LotArea', 'OverallQual', 'OverallCond', 'YearBuilt', 'YearRemodAdd', 'BsmtFinSF1', 'BsmtFinSF2', 'BsmtUnfSF', 'TotalBsmtSF', '1stFlrSF', '2ndFlrSF', 'LowQualFinSF', 'GrLivArea', 'BsmtFullBath', 'BsmtHalfBath', 'FullBath', 'HalfBath', 'BedroomAbvGr', 'KitchenAbvGr', 'TotRmsAbvGrd', 'Fireplaces', 'GarageCars', 'GarageArea', 'WoodDeckSF', 'OpenPorchSF', 'EnclosedPorch', '3SsnPorch', 'ScreenPorch', 'PoolArea', 'MiscVal', 'MoSold', 'YrSold'] \n",
      "\n",
      "However, there are 1 variables should NOT have data type: int64\n",
      "['MSSubClass'] \n",
      "\n",
      "The following variables are assigned as data type: object\n",
      "['MSZoning', 'LotFrontage', 'Street', 'Alley', 'LotShape', 'LandContour', 'Utilities', 'LotConfig', 'LandSlope', 'Neighborhood', 'Condition1', 'Condition2', 'BldgType', 'HouseStyle', 'RoofStyle', 'RoofMatl', 'Exterior1st', 'Exterior2nd', 'MasVnrType', 'MasVnrArea', 'ExterQual', 'ExterCond', 'Foundation', 'BsmtQual', 'BsmtCond', 'BsmtExposure', 'BsmtFinType1', 'BsmtFinType2', 'Heating', 'HeatingQC', 'CentralAir', 'Electrical', 'KitchenQual', 'Functional', 'FireplaceQu', 'GarageType', 'GarageYrBlt', 'GarageFinish', 'GarageQual', 'GarageCond', 'PavedDrive', 'PoolQC', 'Fence', 'MiscFeature', 'SaleType', 'SaleCondition'] \n",
      "\n",
      "However, there are 3 variables should NOT have data type: object\n",
      "['LotFrontage', 'MasVnrArea', 'GarageYrBlt'] \n",
      "\n"
     ]
    }
   ],
   "source": [
    "finalDataTypeCheck(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def correctDataType(df):\n",
    "    '''\n",
    "    This function is to correct the wrong dtypes discovered in the above section\n",
    " \n",
    "    Parameter:\n",
    "    df - a pandas DataFrame of the dedicated data set that will be analyzed on\n",
    "    '''\n",
    "    wrong_feature = incorrectDataType(df)\n",
    "    # Covert features that are supposed to be qualitative to 'object'\n",
    "    df[wrong_feature[0]] = df[wrong_feature[0]].astype('object')\n",
    "    # Covert features that are supposed to be quantitative to numeric values\n",
    "    for i in range(len(wrong_feature[1])):\n",
    "        df[wrong_feature[1][i]].replace('NA', np.nan, inplace = True)\n",
    "        df[wrong_feature[1][i]] = pd.to_numeric(df[wrong_feature[1][i]], errors='coerce')\n",
    "    # Print the dtype of the data set again to ensure accuracy\n",
    "    orig_dataType_count = originalDataType(df)[0]\n",
    "    print('After correcting the data type for columns: {},\\nthe dtypes of the dataset is:\\n{}'\\\n",
    "          .format(wrong_feature[1], orig_dataType_count))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "After correcting the data type for columns: ['LotFrontage', 'MasVnrArea', 'GarageYrBlt'],\n",
      "the dtypes of the dataset is:\n",
      "         Total Amount\n",
      "object             44\n",
      "int64              32\n",
      "float64             3\n"
     ]
    }
   ],
   "source": [
    "correctDataType(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion:\n",
    "From the observations, there were some variables incorrectly assgined as `object` that are actually  `numeric features`. <br> This is becuase in Python, some missing values that are represented as strings in a continuous feature can coerce it to <br> read them as object dtypes [2]. For certain columns, having a value of `NA` has meaning (e.g. having no basement).<br> When applying `pd.read_csv()`, the parameter `keep_default_na` was set to False to keep `NA` as a string.<br>A consequence of this, however, is that any legitimate missing values in a numeric column would also contain `NA` as a string,<br> which coereces the entire column to be an `object` dtype. <br>\n",
    "\n",
    "Therefore, after correcting all features to have the right dtype, those numeric columns should have missing values.<br>The sensible next step should be to calcualte the missing values in the dataset. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Understand Missing Values\n",
    "\n",
    "\n",
    "Create a table that contains **`Total Amount`** and **`Percentage`** of missing values for features that have missing values "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def missValueSummary(df):\n",
    "    df = pd.DataFrame([df.isnull().sum(), round(df.isnull().mean(),3)])\n",
    "    df_transpose = df.T  # transpose dataframe, same as df.transpose()\n",
    "    df_transpose.columns = ['Total Amount', 'Percentage']\n",
    "    sum_per_df = df_transpose[df_transpose.Percentage > 0]\\\n",
    "                        .sort_values(by='Percentage', ascending = False)\n",
    "    sum_per_df['Total Amount'] = sum_per_df['Total Amount'].astype('int64')\n",
    "    if sum_per_df.shape[0] > 0:\n",
    "        return sum_per_df\n",
    "    else:\n",
    "        print('There is no missing value in the data set.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Total Amount</th>\n",
       "      <th>Percentage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>LotFrontage</th>\n",
       "      <td>259</td>\n",
       "      <td>0.177</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GarageYrBlt</th>\n",
       "      <td>81</td>\n",
       "      <td>0.055</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MasVnrArea</th>\n",
       "      <td>8</td>\n",
       "      <td>0.005</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Total Amount  Percentage\n",
       "LotFrontage           259       0.177\n",
       "GarageYrBlt            81       0.055\n",
       "MasVnrArea              8       0.005"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "missValueSummary(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pre-processing 'test.csv' using Created Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Street</th>\n",
       "      <th>Alley</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>LotConfig</th>\n",
       "      <th>...</th>\n",
       "      <th>ScreenPorch</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>PoolQC</th>\n",
       "      <th>Fence</th>\n",
       "      <th>MiscFeature</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>20</td>\n",
       "      <td>RH</td>\n",
       "      <td>80</td>\n",
       "      <td>11622</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NA</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Inside</td>\n",
       "      <td>...</td>\n",
       "      <td>120</td>\n",
       "      <td>0</td>\n",
       "      <td>NA</td>\n",
       "      <td>MnPrv</td>\n",
       "      <td>NA</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>2010</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>81</td>\n",
       "      <td>14267</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NA</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Corner</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NA</td>\n",
       "      <td>NA</td>\n",
       "      <td>Gar2</td>\n",
       "      <td>12500</td>\n",
       "      <td>6</td>\n",
       "      <td>2010</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>74</td>\n",
       "      <td>13830</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NA</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Inside</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NA</td>\n",
       "      <td>MnPrv</td>\n",
       "      <td>NA</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2010</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>78</td>\n",
       "      <td>9978</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NA</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Inside</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NA</td>\n",
       "      <td>NA</td>\n",
       "      <td>NA</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>2010</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>120</td>\n",
       "      <td>RL</td>\n",
       "      <td>43</td>\n",
       "      <td>5005</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NA</td>\n",
       "      <td>IR1</td>\n",
       "      <td>HLS</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Inside</td>\n",
       "      <td>...</td>\n",
       "      <td>144</td>\n",
       "      <td>0</td>\n",
       "      <td>NA</td>\n",
       "      <td>NA</td>\n",
       "      <td>NA</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2010</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 79 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   MSSubClass MSZoning LotFrontage  LotArea Street Alley LotShape LandContour  \\\n",
       "0          20       RH          80    11622   Pave    NA      Reg         Lvl   \n",
       "1          20       RL          81    14267   Pave    NA      IR1         Lvl   \n",
       "2          60       RL          74    13830   Pave    NA      IR1         Lvl   \n",
       "3          60       RL          78     9978   Pave    NA      IR1         Lvl   \n",
       "4         120       RL          43     5005   Pave    NA      IR1         HLS   \n",
       "\n",
       "  Utilities LotConfig      ...       ScreenPorch PoolArea PoolQC  Fence  \\\n",
       "0    AllPub    Inside      ...               120        0     NA  MnPrv   \n",
       "1    AllPub    Corner      ...                 0        0     NA     NA   \n",
       "2    AllPub    Inside      ...                 0        0     NA  MnPrv   \n",
       "3    AllPub    Inside      ...                 0        0     NA     NA   \n",
       "4    AllPub    Inside      ...               144        0     NA     NA   \n",
       "\n",
       "  MiscFeature MiscVal  MoSold  YrSold  SaleType  SaleCondition  \n",
       "0          NA       0       6    2010        WD         Normal  \n",
       "1        Gar2   12500       6    2010        WD         Normal  \n",
       "2          NA       0       3    2010        WD         Normal  \n",
       "3          NA       0       6    2010        WD         Normal  \n",
       "4          NA       0       1    2010        WD         Normal  \n",
       "\n",
       "[5 rows x 79 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data1 = readData('test.csv')\n",
    "data1.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 16 unique values in MSSubClass column\n",
      "[ 20  60 120 160  80  30  50  90  85 190  45  70  75 180  40 150] \n",
      "\n",
      "There are 6 unique values in MSZoning column\n",
      "['RH' 'RL' 'RM' 'FV' 'C (all)' 'NA'] \n",
      "\n",
      "There are 2 unique values in Street column\n",
      "['Pave' 'Grvl'] \n",
      "\n",
      "There are 3 unique values in Alley column\n",
      "['NA' 'Pave' 'Grvl'] \n",
      "\n",
      "There are 4 unique values in LotShape column\n",
      "['Reg' 'IR1' 'IR2' 'IR3'] \n",
      "\n",
      "There are 4 unique values in LandContour column\n",
      "['Lvl' 'HLS' 'Bnk' 'Low'] \n",
      "\n",
      "There are 2 unique values in Utilities column\n",
      "['AllPub' 'NA'] \n",
      "\n",
      "There are 5 unique values in LotConfig column\n",
      "['Inside' 'Corner' 'FR2' 'CulDSac' 'FR3'] \n",
      "\n",
      "There are 3 unique values in LandSlope column\n",
      "['Gtl' 'Mod' 'Sev'] \n",
      "\n",
      "There are 9 unique values in Condition1 column\n",
      "['Feedr' 'Norm' 'PosN' 'RRNe' 'Artery' 'RRNn' 'PosA' 'RRAn' 'RRAe'] \n",
      "\n",
      "There are 5 unique values in Condition2 column\n",
      "['Norm' 'Feedr' 'PosA' 'PosN' 'Artery'] \n",
      "\n",
      "There are 5 unique values in BldgType column\n",
      "['1Fam' 'TwnhsE' 'Twnhs' 'Duplex' '2fmCon'] \n",
      "\n",
      "There are 7 unique values in HouseStyle column\n",
      "['1Story' '2Story' 'SLvl' '1.5Fin' 'SFoyer' '2.5Unf' '1.5Unf'] \n",
      "\n",
      "There are 10 unique values in OverallQual column\n",
      "[ 5  6  8  7  4  9  2  3 10  1] \n",
      "\n",
      "There are 9 unique values in OverallCond column\n",
      "[6 5 7 8 2 9 3 4 1] \n",
      "\n",
      "There are 6 unique values in RoofStyle column\n",
      "['Gable' 'Hip' 'Gambrel' 'Flat' 'Mansard' 'Shed'] \n",
      "\n",
      "There are 4 unique values in RoofMatl column\n",
      "['CompShg' 'Tar&Grv' 'WdShake' 'WdShngl'] \n",
      "\n",
      "There are 14 unique values in Exterior1st column\n",
      "['VinylSd' 'Wd Sdng' 'HdBoard' 'Plywood' 'MetalSd' 'CemntBd' 'WdShing'\n",
      " 'BrkFace' 'AsbShng' 'BrkComm' 'Stucco' 'AsphShn' 'NA' 'CBlock'] \n",
      "\n",
      "There are 16 unique values in Exterior2nd column\n",
      "['VinylSd' 'Wd Sdng' 'HdBoard' 'Plywood' 'MetalSd' 'Brk Cmn' 'CmentBd'\n",
      " 'ImStucc' 'Wd Shng' 'AsbShng' 'Stucco' 'CBlock' 'BrkFace' 'AsphShn' 'NA'\n",
      " 'Stone'] \n",
      "\n",
      "There are 5 unique values in MasVnrType column\n",
      "['None' 'BrkFace' 'Stone' 'BrkCmn' 'NA'] \n",
      "\n",
      "There are 4 unique values in ExterQual column\n",
      "['TA' 'Gd' 'Ex' 'Fa'] \n",
      "\n",
      "There are 5 unique values in ExterCond column\n",
      "['TA' 'Gd' 'Fa' 'Po' 'Ex'] \n",
      "\n",
      "There are 6 unique values in Foundation column\n",
      "['CBlock' 'PConc' 'BrkTil' 'Stone' 'Slab' 'Wood'] \n",
      "\n",
      "There are 5 unique values in BsmtQual column\n",
      "['TA' 'Gd' 'Ex' 'Fa' 'NA'] \n",
      "\n",
      "There are 5 unique values in BsmtCond column\n",
      "['TA' 'Po' 'Fa' 'Gd' 'NA'] \n",
      "\n",
      "There are 5 unique values in BsmtExposure column\n",
      "['No' 'Gd' 'Mn' 'Av' 'NA'] \n",
      "\n",
      "There are 7 unique values in BsmtFinType1 column\n",
      "['Rec' 'ALQ' 'GLQ' 'Unf' 'BLQ' 'LwQ' 'NA'] \n",
      "\n",
      "There are 7 unique values in BsmtFinType2 column\n",
      "['LwQ' 'Unf' 'Rec' 'BLQ' 'GLQ' 'ALQ' 'NA'] \n",
      "\n",
      "There are 4 unique values in Heating column\n",
      "['GasA' 'GasW' 'Grav' 'Wall'] \n",
      "\n",
      "There are 5 unique values in HeatingQC column\n",
      "['TA' 'Gd' 'Ex' 'Fa' 'Po'] \n",
      "\n",
      "There are 2 unique values in CentralAir column\n",
      "['Y' 'N'] \n",
      "\n",
      "There are 4 unique values in Electrical column\n",
      "['SBrkr' 'FuseA' 'FuseF' 'FuseP'] \n",
      "\n",
      "There are 15 unique values in LowQualFinSF column\n",
      "[   0  362 1064  431  436  259  312  108  697  512  114  205   80  140\n",
      "  450] \n",
      "\n",
      "There are 5 unique values in BsmtFullBath column\n",
      "['0' '1' '2' '3' 'NA'] \n",
      "\n",
      "There are 4 unique values in BsmtHalfBath column\n",
      "['0' '1' 'NA' '2'] \n",
      "\n",
      "There are 5 unique values in FullBath column\n",
      "[1 2 3 4 0] \n",
      "\n",
      "There are 3 unique values in HalfBath column\n",
      "[0 1 2] \n",
      "\n",
      "There are 7 unique values in BedroomAbvGr column\n",
      "[2 3 4 1 6 5 0] \n",
      "\n",
      "There are 3 unique values in KitchenAbvGr column\n",
      "[1 2 0] \n",
      "\n",
      "There are 5 unique values in KitchenQual column\n",
      "['TA' 'Gd' 'Ex' 'Fa' 'NA'] \n",
      "\n",
      "There are 12 unique values in TotRmsAbvGrd column\n",
      "[ 5  6  7  4 10  8  9  3 12 11 13 15] \n",
      "\n",
      "There are 8 unique values in Functional column\n",
      "['Typ' 'Min2' 'Min1' 'Mod' 'Maj1' 'Sev' 'Maj2' 'NA'] \n",
      "\n",
      "There are 5 unique values in Fireplaces column\n",
      "[0 1 2 3 4] \n",
      "\n",
      "There are 6 unique values in FireplaceQu column\n",
      "['NA' 'TA' 'Gd' 'Po' 'Fa' 'Ex'] \n",
      "\n",
      "There are 7 unique values in GarageType column\n",
      "['Attchd' 'Detchd' 'BuiltIn' 'NA' 'Basment' '2Types' 'CarPort'] \n",
      "\n",
      "There are 4 unique values in GarageFinish column\n",
      "['Unf' 'Fin' 'RFn' 'NA'] \n",
      "\n",
      "There are 7 unique values in GarageCars column\n",
      "['1' '2' '3' '0' '4' '5' 'NA'] \n",
      "\n",
      "There are 5 unique values in GarageQual column\n",
      "['TA' 'NA' 'Fa' 'Gd' 'Po'] \n",
      "\n",
      "There are 6 unique values in GarageCond column\n",
      "['TA' 'NA' 'Fa' 'Gd' 'Po' 'Ex'] \n",
      "\n",
      "There are 3 unique values in PavedDrive column\n",
      "['Y' 'N' 'P'] \n",
      "\n",
      "There are 13 unique values in 3SsnPorch column\n",
      "[  0 224 255 225 360 150 153 174 120 219 176  86 323] \n",
      "\n",
      "There are 7 unique values in PoolArea column\n",
      "[  0 144 368 444 228 561 800] \n",
      "\n",
      "There are 3 unique values in PoolQC column\n",
      "['NA' 'Ex' 'Gd'] \n",
      "\n",
      "There are 5 unique values in Fence column\n",
      "['MnPrv' 'NA' 'GdPrv' 'GdWo' 'MnWw'] \n",
      "\n",
      "There are 4 unique values in MiscFeature column\n",
      "['NA' 'Gar2' 'Shed' 'Othr'] \n",
      "\n",
      "There are 12 unique values in MoSold column\n",
      "[ 6  3  1  4  5  2  7 10  8 11  9 12] \n",
      "\n",
      "There are 5 unique values in YrSold column\n",
      "[2010 2009 2008 2007 2006] \n",
      "\n",
      "There are 10 unique values in SaleType column\n",
      "['WD' 'COD' 'New' 'ConLD' 'Oth' 'Con' 'ConLw' 'ConLI' 'CWD' 'NA'] \n",
      "\n",
      "There are 6 unique values in SaleCondition column\n",
      "['Normal' 'Partial' 'Abnorml' 'Family' 'Alloca' 'AdjLand'] \n",
      "\n"
     ]
    }
   ],
   "source": [
    "uniqueValues(data1) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 2 different data types:\n",
      "         Total Amount\n",
      "object            54\n",
      "int64             25 \n",
      "\n",
      "There should be 44 qualitative variables\n",
      "There should be 35 quantitative variables\n",
      "\n",
      "The following variables are assigned as data type: int64\n",
      "['MSSubClass', 'LotArea', 'OverallQual', 'OverallCond', 'YearBuilt', 'YearRemodAdd', '1stFlrSF', '2ndFlrSF', 'LowQualFinSF', 'GrLivArea', 'FullBath', 'HalfBath', 'BedroomAbvGr', 'KitchenAbvGr', 'TotRmsAbvGrd', 'Fireplaces', 'WoodDeckSF', 'OpenPorchSF', 'EnclosedPorch', '3SsnPorch', 'ScreenPorch', 'PoolArea', 'MiscVal', 'MoSold', 'YrSold'] \n",
      "\n",
      "However, there are 1 variables should NOT have data type: int64\n",
      "['MSSubClass'] \n",
      "\n",
      "The following variables are assigned as data type: object\n",
      "['MSZoning', 'LotFrontage', 'Street', 'Alley', 'LotShape', 'LandContour', 'Utilities', 'LotConfig', 'LandSlope', 'Neighborhood', 'Condition1', 'Condition2', 'BldgType', 'HouseStyle', 'RoofStyle', 'RoofMatl', 'Exterior1st', 'Exterior2nd', 'MasVnrType', 'MasVnrArea', 'ExterQual', 'ExterCond', 'Foundation', 'BsmtQual', 'BsmtCond', 'BsmtExposure', 'BsmtFinType1', 'BsmtFinSF1', 'BsmtFinType2', 'BsmtFinSF2', 'BsmtUnfSF', 'TotalBsmtSF', 'Heating', 'HeatingQC', 'CentralAir', 'Electrical', 'BsmtFullBath', 'BsmtHalfBath', 'KitchenQual', 'Functional', 'FireplaceQu', 'GarageType', 'GarageYrBlt', 'GarageFinish', 'GarageCars', 'GarageArea', 'GarageQual', 'GarageCond', 'PavedDrive', 'PoolQC', 'Fence', 'MiscFeature', 'SaleType', 'SaleCondition'] \n",
      "\n",
      "However, there are 11 variables should NOT have data type: object\n",
      "['LotFrontage', 'MasVnrArea', 'BsmtFinSF1', 'BsmtFinSF2', 'BsmtUnfSF', 'TotalBsmtSF', 'BsmtFullBath', 'BsmtHalfBath', 'GarageYrBlt', 'GarageCars', 'GarageArea'] \n",
      "\n"
     ]
    }
   ],
   "source": [
    "finalDataTypeCheck(data1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "After correcting the data type for columns: ['LotFrontage', 'MasVnrArea', 'BsmtFinSF1', 'BsmtFinSF2', 'BsmtUnfSF', 'TotalBsmtSF', 'BsmtFullBath', 'BsmtHalfBath', 'GarageYrBlt', 'GarageCars', 'GarageArea'],\n",
      "the dtypes of the dataset is:\n",
      "         Total Amount\n",
      "object             44\n",
      "int64              24\n",
      "float64            11\n"
     ]
    }
   ],
   "source": [
    "correctDataType(data1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Total Amount</th>\n",
       "      <th>Percentage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>LotFrontage</th>\n",
       "      <td>227</td>\n",
       "      <td>0.156</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GarageYrBlt</th>\n",
       "      <td>78</td>\n",
       "      <td>0.053</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MasVnrArea</th>\n",
       "      <td>15</td>\n",
       "      <td>0.010</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <td>1</td>\n",
       "      <td>0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BsmtFinSF2</th>\n",
       "      <td>1</td>\n",
       "      <td>0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BsmtUnfSF</th>\n",
       "      <td>1</td>\n",
       "      <td>0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TotalBsmtSF</th>\n",
       "      <td>1</td>\n",
       "      <td>0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BsmtFullBath</th>\n",
       "      <td>2</td>\n",
       "      <td>0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BsmtHalfBath</th>\n",
       "      <td>2</td>\n",
       "      <td>0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GarageCars</th>\n",
       "      <td>1</td>\n",
       "      <td>0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GarageArea</th>\n",
       "      <td>1</td>\n",
       "      <td>0.001</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Total Amount  Percentage\n",
       "LotFrontage            227       0.156\n",
       "GarageYrBlt             78       0.053\n",
       "MasVnrArea              15       0.010\n",
       "BsmtFinSF1               1       0.001\n",
       "BsmtFinSF2               1       0.001\n",
       "BsmtUnfSF                1       0.001\n",
       "TotalBsmtSF              1       0.001\n",
       "BsmtFullBath             2       0.001\n",
       "BsmtHalfBath             2       0.001\n",
       "GarageCars               1       0.001\n",
       "GarageArea               1       0.001"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "missValueSummary(data1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References\n",
    "\n",
    "1. [House Prices: Advanced Regression Techniques](https://www.kaggle.com/c/house-prices-advanced-regression-techniques/overview/description)\n",
    "2. [Handling Categorical Data in Python](https://www.datacamp.com/community/tutorials/categorical-data)\n",
    "3. [Inconsistent result with replace(np.nan, None, inplace=True)](https://github.com/pandas-dev/pandas/issues/17494)\n",
    "4. [SettingwithCopyWarning: How to Fix This Warning in Pandas](https://www.dataquest.io/blog/settingwithcopywarning/)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}