0c - unit 07.ipynb

{
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  },
  "orig_nbformat": 2,
  "kernelspec": {
   "name": "python3710jvsc74a57bd0a970c7e51787c6ef7f3c805442fbb5a30ae2745635e1d2514e62d936ba85c55c",
   "display_name": "Python 3.7.10 64-bit ('ml_crash_course': conda)"
  },
  "metadata": {
   "interpreter": {
    "hash": "5fd7d2d989b956829c9ad45cc6f2d78f0054d35d23dba8942e8fbd9b2870e6f3"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2,
 "cells": [
  {
   "source": [
    "# Exercise: Titanic Dataset - One-Hot Vectors\n",
    "\n",
    "In this Unit we will build a model to predict who survived the Titanic disaster.\n",
    "\n",
    "While doing so, we will practice transforming data between numerical and categorical types, including using one-hot vectors.\n",
    "\n",
    "## Preparing data\n",
    "\n",
    "Let's start by opening and quickly cleaning up our dataset, like we did in the last unit:\n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "   Survived  Pclass     Sex   Age  SibSp  Parch     Fare    Cabin Embarked\n",
       "0         0       3    male  22.0      1      0   7.2500  Unknown        S\n",
       "1         1       1  female  38.0      1      0  71.2833      C85        C\n",
       "2         1       3  female  26.0      0      0   7.9250  Unknown        S\n",
       "3         1       1  female  35.0      1      0  53.1000     C123        S\n",
       "4         0       3    male  35.0      0      0   8.0500  Unknown        S"
      ],
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Survived</th>\n      <th>Pclass</th>\n      <th>Sex</th>\n      <th>Age</th>\n      <th>SibSp</th>\n      <th>Parch</th>\n      <th>Fare</th>\n      <th>Cabin</th>\n      <th>Embarked</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>0</td>\n      <td>3</td>\n      <td>male</td>\n      <td>22.0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>7.2500</td>\n      <td>Unknown</td>\n      <td>S</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1</td>\n      <td>1</td>\n      <td>female</td>\n      <td>38.0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>71.2833</td>\n      <td>C85</td>\n      <td>C</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>1</td>\n      <td>3</td>\n      <td>female</td>\n      <td>26.0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>7.9250</td>\n      <td>Unknown</td>\n      <td>S</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>1</td>\n      <td>1</td>\n      <td>female</td>\n      <td>35.0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>53.1000</td>\n      <td>C123</td>\n      <td>S</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>0</td>\n      <td>3</td>\n      <td>male</td>\n      <td>35.0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>8.0500</td>\n      <td>Unknown</td>\n      <td>S</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {},
     "execution_count": 1
    }
   ],
   "source": [
    "import pandas\n",
    "\n",
    "# Open our dataset from file\n",
    "dataset = pandas.read_csv(\"Data/titanic.csv\", index_col=False, sep=\",\", header=0)\n",
    "\n",
    "# Fill missing cabin information with 'Unknown'\n",
    "dataset[\"Cabin\"].fillna(\"Unknown\", inplace=True)\n",
    "\n",
    "# Remove rows missing Age information\n",
    "dataset.dropna(subset=[\"Age\"], inplace=True)\n",
    "\n",
    "# Remove the Name, PassengerId, and Ticket fields\n",
    "# This is optional and only to make it easier to read our print-outs\n",
    "dataset.drop([\"PassengerId\", \"Name\", \"Ticket\"], axis=1, inplace=True)\n",
    "\n",
    "dataset.head()\n"
   ]
  },
  {
   "source": [
    "## About Our Model\n",
    "\n",
    "We will be training a type of model called Logistic Regression, which will predict who survives the Titanic disaster.\n",
    "\n",
    "You do not need to understand logistic regression to understand this exercise, so we have put the implementation of outside of this notebook in a method called `train_logistic_regression`. If you are curious, you can read this method in our GitHub repository.\n",
    "\n",
    "`train_logistic_regression`:\n",
    "\n",
    "1. Accepts our data frame and a list of features to include in the model. \n",
    "2. Trains the model\n",
    "3. Returns a number stating how well the model performs predicting survival. **Smaller numbers are better.**\n",
    "\n",
    "## Numerical Only\n",
    "\n",
    "Let's create a model, only using the numerical features.\n",
    "\n",
    "First, we will use `Pclass` here as a ordinal feature, rather than a one-hot encoded categorical feature."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Numerical-Only, Log-Loss (cost): 0.6121682789483452\n"
     ]
    }
   ],
   "source": [
    "from m0c_logistic_regression import train_logistic_regression\n",
    "\n",
    "features = [\"Age\", \"Pclass\", \"SibSp\", \"Parch\", \"Fare\"] \n",
    "loss_numerical_only = train_logistic_regression(dataset, features)\n",
    "\n",
    "print(f\"Numerical-Only, Log-Loss (cost): {loss_numerical_only}\")"
   ]
  },
  {
   "source": [
    "We have our starting point. Let's see if we can improve the model using categorical features.\n",
    "\n",
    "## Binary Categorical Features\n",
    "\n",
    "Categorical features that have just two potential values can be encoded in a single column as `0` and `1`.\n",
    "\n",
    "Let's convert `Sex` values into `IsFemale` - a `0` for male and `1` for female - and include that in our model."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "   Survived  Pclass     Sex   Age  SibSp  Parch     Fare    Cabin Embarked  \\\n0         0       3    male  22.0      1      0   7.2500  Unknown        S   \n1         1       1  female  38.0      1      0  71.2833      C85        C   \n2         1       3  female  26.0      0      0   7.9250  Unknown        S   \n3         1       1  female  35.0      1      0  53.1000     C123        S   \n4         0       3    male  35.0      0      0   8.0500  Unknown        S   \n\n   IsFemale  \n0         0  \n1         1  \n2         1  \n3         1  \n4         0  \n\nNumerical + Sex, Log-Loss (cost): 0.4707144737255896\n"
     ]
    }
   ],
   "source": [
    "# Swap male / female with numerical values\n",
    "# We can do this because there are only two categories\n",
    "dataset[\"IsFemale\"] = dataset.Sex.replace({'male':0, 'female':1})\n",
    "\n",
    "# Print out the first few rows of the dataset\n",
    "print(dataset.head())\n",
    "\n",
    "# Run and test the model, also using IsFemale this time\n",
    "features = [\"Age\", \"Pclass\", \"SibSp\", \"Parch\", \"Fare\", \"IsFemale\"] \n",
    "loss_binary_categoricals = train_logistic_regression(dataset, features)\n",
    "\n",
    "print(f\"\\nNumerical + Sex, Log-Loss (cost): {loss_binary_categoricals}\")"
   ]
  },
  {
   "source": [
    "Our loss (error) has decreased! This means this model performs better than the previous.\n",
    "\n",
    "## One-Hot Encoding\n",
    "\n",
    "Ticket class (`Pclass`) is an Ordinal feature. That means that its potential values (1, 2 & 3) are treated as having an order and being equally spaced. It's possible that this even spacing is simply not correct though - in stories we have heard about the Titanic, the third-class passengers were treated much worse than those in 1st and 2nd class.\n",
    "\n",
    "Let's convert `Pclass` into a categorical feature using one-hot encoding:"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Possible values for PClass: [3 1 2]\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "   Survived     Sex   Age  SibSp  Parch     Fare    Cabin Embarked  IsFemale  \\\n",
       "0         0    male  22.0      1      0   7.2500  Unknown        S         0   \n",
       "1         1  female  38.0      1      0  71.2833      C85        C         1   \n",
       "2         1  female  26.0      0      0   7.9250  Unknown        S         1   \n",
       "3         1  female  35.0      1      0  53.1000     C123        S         1   \n",
       "4         0    male  35.0      0      0   8.0500  Unknown        S         0   \n",
       "\n",
       "   Pclass_1  Pclass_2  Pclass_3  Pclass  \n",
       "0         0         0         1       3  \n",
       "1         1         0         0       1  \n",
       "2         0         0         1       3  \n",
       "3         1         0         0       1  \n",
       "4         0         0         1       3  "
      ],
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Survived</th>\n      <th>Sex</th>\n      <th>Age</th>\n      <th>SibSp</th>\n      <th>Parch</th>\n      <th>Fare</th>\n      <th>Cabin</th>\n      <th>Embarked</th>\n      <th>IsFemale</th>\n      <th>Pclass_1</th>\n      <th>Pclass_2</th>\n      <th>Pclass_3</th>\n      <th>Pclass</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>0</td>\n      <td>male</td>\n      <td>22.0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>7.2500</td>\n      <td>Unknown</td>\n      <td>S</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1</td>\n      <td>female</td>\n      <td>38.0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>71.2833</td>\n      <td>C85</td>\n      <td>C</td>\n      <td>1</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>1</td>\n      <td>female</td>\n      <td>26.0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>7.9250</td>\n      <td>Unknown</td>\n      <td>S</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>1</td>\n      <td>female</td>\n      <td>35.0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>53.1000</td>\n      <td>C123</td>\n      <td>S</td>\n      <td>1</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>0</td>\n      <td>male</td>\n      <td>35.0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>8.0500</td>\n      <td>Unknown</td>\n      <td>S</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>3</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {},
     "execution_count": 4
    }
   ],
   "source": [
    "# Get all possible categories for the \"PClass\" column\n",
    "print(f\"Possible values for PClass: {dataset['Pclass'].unique()}\")\n",
    "\n",
    "# Use Pandas to One-Hot encode the PClass category\n",
    "dataset_with_one_hot = pandas.get_dummies(dataset, columns=[\"Pclass\"], drop_first=False)\n",
    "\n",
    "# Add back in the old Pclass column, for learning purposes\n",
    "dataset_with_one_hot[\"Pclass\"] = dataset.Pclass\n",
    "\n",
    "# Print out the first few rows\n",
    "dataset_with_one_hot.head()"
   ]
  },
  {
   "source": [
    "See how `Pclass` has been converted into three values: `Pclass_1`, `Pclass_2` and `Pclass_3`.\n",
    "\n",
    "Rows with `Pclass` of 1 have a value in the `Pclass_1` column. The same pattern is there for values of 2 and 3.\n",
    "\n",
    "Lets now re-run our model treating `Pclass` values as a categorical, rather than ordinal."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "\nNumerical, Sex, Categorical Pclass, Log-Loss (cost): 0.4717298976924814\n"
     ]
    }
   ],
   "source": [
    "# Run and test the model, also using Pclass as a categorical feature this time\n",
    "features = [\"Age\", \"SibSp\", \"Parch\", \"Fare\", \"IsFemale\",\n",
    "            \"Pclass_1\", \"Pclass_2\", \"Pclass_3\"]\n",
    "\n",
    "loss_pclass_categorical = train_logistic_regression(dataset_with_one_hot, features)\n",
    "\n",
    "print(f\"\\nNumerical, Sex, Categorical Pclass, Log-Loss (cost): {loss_pclass_categorical}\")"
   ]
  },
  {
   "source": [
    "This seems to have made things slightly worse!\n",
    "\n",
    "Let's move on.\n",
    "\n",
    "## Including Cabin\n",
    "\n",
    "Recall that many passengers had `Cabin` information. `Cabin` is a categorical feature and should be a good predictor of survival, because people in lower cabins probably had little time to escape during the sinking.\n",
    "\n",
    "Let's encode cabin using one-hot vectors and include it in a model. There are so many cabins this time that we won't print them all out. If you would like to practice printing them out, feel free to edit the code as practice."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "135 cabins found\n\nNumerical, Sex, Categorical Pclass, Cabin, Log-Loss (cost): 0.4600772115551378\n"
     ]
    }
   ],
   "source": [
    "# Use Pandas to One-Hot encode the Cabin and Pclass categories\n",
    "dataset_with_one_hot = pandas.get_dummies(dataset, columns=[\"Pclass\", \"Cabin\"], drop_first=False)\n",
    "\n",
    "# Find cabin column names\n",
    "cabin_column_names = list(c for c in dataset_with_one_hot.columns if c.startswith(\"Cabin_\"))\n",
    "\n",
    "# Print out how many cabins there were\n",
    "print(len(cabin_column_names), \"cabins found\")\n",
    "\n",
    "# Make a list of features\n",
    "features = [\"Age\", \"SibSp\", \"Parch\", \"Fare\", \"IsFemale\",\n",
    "            \"Pclass_1\", \"Pclass_2\", \"Pclass_3\"] + \\\n",
    "            cabin_column_names\n",
    "\n",
    "# Run the model and print the result\n",
    "loss_cabin_categorical = train_logistic_regression(dataset_with_one_hot, features)\n",
    "\n",
    "print(f\"\\nNumerical, Sex, Categorical Pclass, Cabin, Log-Loss (cost): {loss_cabin_categorical}\")"
   ]
  },
  {
   "source": [
    "That's our best result so far!\n",
    "\n",
    "## Improving Power\n",
    "\n",
    "Including very large numbers of categorical classes - such as 135 Cabins - is often not the best way to train a model. This is because the model only has a few examples of each category class to learn from.\n",
    "\n",
    "Models can sometimes be improved by simplifying features. `Cabin` was probably useful because it indicated which floor of the titanic people were probably situated in: those in lower decks would have had their quarters flooded first. \n",
    "\n",
    "Using deck information might be simpler than categorizing people into Cabins. \n",
    "\n",
    "Let's simplify what we have run, replacing the 135 `Cabin` categories with a simpler `Deck` category, that has only 9 values: A - G, T, and U (Unknown)"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Decks:  ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'T', 'U']\n",
      "\n",
      "Simplifying Cabin Into Deck, Log-Loss (cost): 0.45879471810781874\n"
     ]
    }
   ],
   "source": [
    "# We have cabin names, like A31, G45. The letter refers to the deck that\n",
    "# the cabin was on. Extract just the deck and save it to a column. \n",
    "dataset[\"Deck\"] = [c[0] for c in dataset.Cabin]\n",
    "\n",
    "print(\"Decks: \", sorted(dataset.Deck.unique()))\n",
    "\n",
    "# Create one-hot vectors for:\n",
    "# Pclass - the class of ticket. (This could be treated as ordinal or categorical)\n",
    "# Deck - the deck that the cabin was on\n",
    "dataset_with_one_hot = pandas.get_dummies(dataset, columns=[\"Pclass\", \"Deck\"], drop_first=False)\n",
    "\n",
    "# Find the deck names\n",
    "deck_of_cabin_column_names = list(c for c in dataset_with_one_hot.columns if c.startswith(\"Deck_\"))\n",
    " \n",
    "features = [\"Age\", \"IsFemale\", \"SibSp\", \"Parch\", \"Fare\", \n",
    "            \"Pclass_1\", \"Pclass_2\", \"Pclass_3\",\n",
    "            \"Deck_A\", \"Deck_B\", \"Deck_C\", \"Deck_D\", \n",
    "            \"Deck_E\", \"Deck_F\", \"Deck_G\", \"Deck_U\", \"Deck_T\"]\n",
    "\n",
    "loss_deck = train_logistic_regression(dataset_with_one_hot, features)\n",
    "\n",
    "print(f\"\\nSimplifying Cabin Into Deck, Log-Loss (cost): {loss_deck}\")"
   ]
  },
  {
   "source": [
    "## Comparing Models\n",
    "\n",
    "Let's compare the `loss` for these models:"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "                          Dataset  Log-Loss (Low is better)\n",
       "0           Numeric Features Only                  0.612168\n",
       "1            Adding Sex as Binary                  0.470714\n",
       "2  Treating Pclass as Categorical                  0.471730\n",
       "3      Using Cabin as Categorical                  0.460077\n",
       "4    Using Deck rather than Cabin                  0.458795"
      ],
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Dataset</th>\n      <th>Log-Loss (Low is better)</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>Numeric Features Only</td>\n      <td>0.612168</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>Adding Sex as Binary</td>\n      <td>0.470714</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>Treating Pclass as Categorical</td>\n      <td>0.471730</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>Using Cabin as Categorical</td>\n      <td>0.460077</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>Using Deck rather than Cabin</td>\n      <td>0.458795</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {},
     "execution_count": 8
    }
   ],
   "source": [
    "# Use a dataframe to create a comparison table of metrics\n",
    "# Copy metrics from previous Unit\n",
    "\n",
    "l =[[\"Numeric Features Only\", loss_numerical_only],\n",
    "    [\"Adding Sex as Binary\", loss_binary_categoricals],\n",
    "    [\"Treating Pclass as Categorical\", loss_pclass_categorical],\n",
    "    [\"Using Cabin as Categorical\", loss_cabin_categorical],\n",
    "    [\"Using Deck rather than Cabin\", loss_deck]]\n",
    "\n",
    "pandas.DataFrame(l, columns=[\"Dataset\", \"Log-Loss (Low is better)\"])"
   ]
  },
  {
   "source": [
    "We can see that including categorical features can both improve and harm how well a model works. Often, experimentation is the best way to find the best model. \n",
    "\n",
    "## Summary\n",
    "\n",
    "In this unit you learned how to use One-Hot encoding to address categorical data.\n",
    "\n",
    "We also explored how sometimes thinking critically about the problem you are trying can improve a solution better than simply including all possible features in a model."
   ],
   "cell_type": "markdown",
   "metadata": {}
  }
 ]
}