90_minutes_to_machine_learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 90 Minutes To Machine Learning\n",
    "> A brief introduction to the Codeup Data Science program and Machine Learning\n",
    "\n",
    "\n",
    "## Why are we here?\n",
    "1. Intro to the Codeup experience\n",
    "2. Big Picture overview of Data Science\n",
    "3. Intro to Machine Learning concepts and tools including:\n",
    "    - Exposure to Python\n",
    "    - Data acquisition and preparation\n",
    "    - Data visualization\n",
    "    - Building a predictive model w/ Scikit-Learn\n",
    "    - Evaluating how well a predictive model performs\n",
    "\n",
    "## Why Codeup?\n",
    "- Focus on student outcomes\n",
    "- Placement services and quality of network\n",
    "- Immersion works. Full-time, live instruction for 5 months works.\n",
    "- Projects simulate the work environment from real world data to presenting findings to stakeholders\n",
    "\n",
    "## What is Data Science?\n",
    "- Interdisciplinary applied science intersecting programming, statistics, and domain expertise\n",
    "- The application of the scientific method of hypothesis -> experiment -> analyze -> repeat to analyze and infer outcomes from data.\n",
    "- A broad description of approaches ranging from business analysis and visualizations to machine learning and deep neural network analysis.\n",
    "![](drawn_ds_venn_diagram.png)\n",
    "\n",
    "## How Does Data Science Relate to Traditional Software and Data Analysis?\n",
    "![](data_science_venn_diagram_with_overlapping_disciplines.png)\n",
    "\n",
    "## What is Machine Learning?\n",
    "- Machine Learning is the process of using previous data as the fuel for determining rules for making predictions of outcomes from future data.\n",
    "- Classical programming takes business rules and data to produce answers. Ex. TurboTax software.\n",
    "- Machine learning takes in data (and sometimes answers/labels for some data) and produces rules or predictions for future data. The example here is text message autocomplete.\n",
    "\n",
    "<img src=\"classical_programming_vs_machine_learning.jpeg\" width=500>\n",
    "\n",
    "## Where does Machine Learning Fit Into Data Science?\n",
    "\n",
    "![example data science pipepine and product](example_data_science_project.png)\n",
    "\n",
    "## Challenges of Machine Learning\n",
    "- Poor quality data\n",
    "- Insufficient quantity of data\n",
    "- Nonrepresentative data\n",
    "- Bias in, Bias out:\n",
    "    - [Cognitive Biases](https://en.wikipedia.org/wiki/List_of_cognitive_biases) arise from being human.\n",
    "    - [Statistical Biases](https://en.wikipedia.org/wiki/Bias_(statistics)) arise from our methodologies. These have to do with how data is collected, how people respond, etc...\n",
    "- Whatever Machine Learning \"learns\", it will keep doing. There is no cognition or intelligence, only pattern recognition and optimization.\n",
    "\n",
    "## What Machine Learning and other skills are in the Codeup Data Science program?\n",
    "\n",
    "![machine learning methods taught at Codeup](machine_learning_methods.png)\n",
    "\n",
    "\n",
    "## What kind of ML will we doing today?\n",
    "- We'll be using a decision tree classifier to predict whether or not we should expect employees to quit a company.\n",
    "- Classification machine learning is used all the time for such things as:\n",
    "    - Facial recognition\n",
    "    - Handwriting recognition and conversion to typed text\n",
    "    - Recommendation engines for new music or movies \n",
    "- Classification is a \"supervised learning\" type of machine learning. That means we train the algorithm on existing data to learn a rule, a recognized pattern, to apply to future data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Planning Stage\n",
    "\n",
    "Let's play imagination!\n",
    "\n",
    "In this scenario, you are a Data Scientist at a consulting company and your supervisor says:\n",
    "\n",
    "_The Human Resources department needs your help. We know that employee attrition is expensive. It's bad for culture, and it costs a lot of time and money to recruit, hire, and onboard new employees. Here is a spreadsheet export of some of our HR information we have on employees. We need you to determine the predictors of an employee staying or leaving. What are the best predictors of attrition? Are the employee survey responses good indicators if people leave?_\n",
    "\n",
    "### Critical Planning Questions?\n",
    "- What's the business goal here?\n",
    "- Who are the stakeholders?\n",
    "- Why is this important to the business?\n",
    "- What does our input data look like? \n",
    "- What is the thing we're trying to predict?\n",
    "- What is our target variable, exactly?\n",
    "- What are our predictor variables?\n",
    "\n",
    "### Our Plan for this Dataset:\n",
    "- Overall goal: Predict if a particular employee is going to leave the organization.\n",
    "- Once we acquire the data, we'll need to clean/prepare it as much as necessary.\n",
    "- Define the target variable\n",
    "- Split the data to have in-sample and out-of-sample data for our modeling.\n",
    "\n",
    "### Data Dictionary \n",
    "- A data dictionary explains the columns in a dataset and what the values in those columns mean. \n",
    "- In this dataset, here's what things mean:\n",
    "    - `Attrition` specifies if an employee is left the company last year or is still employed.\n",
    "    - `MonthlyIncome` is that employee's monthly salary figure.\n",
    "    - `Education` is a number value meaning the academic background of the employee: \n",
    "        1. No college\n",
    "        2. Some college\n",
    "        3. Bachelor degree\n",
    "        4. Master's degree\n",
    "        5. PhD level\n",
    "    - `WorkLifeBalance` is a number value the employee reported on a company survey meaning:\n",
    "        1. \"Bad\"\n",
    "        2. \"Good\"\n",
    "        3. \"Better\"\n",
    "        4. \"Best\"\n",
    "    - `JobSatisfaction` is a number value the employee reported on a company survey meaning:\n",
    "        1. \"Low\"\n",
    "        2. \"Medium\"\n",
    "        3. \"High\"\n",
    "        4. \"Very High\"\n",
    "    - `PercentSalaryHike` is the percentage increase in salary since last year.\n",
    "    - `BusinessTravel` specifies if the employee travels frequently, rarely, or never"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Show Me The Code!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Data Processing and Data Cleaning Libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# Vizualization Libraries\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "# sklearn preprocessing\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# modeling imports\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "# model evaluation\n",
    "from sklearn.metrics import classification_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We'll use this split function later to create in-sample and out-of-sample datasets for modeling\n",
    "def split(df, stratify_by=None):\n",
    "    \"\"\"\n",
    "    3 way split for train, validate, and test datasets\n",
    "    To stratify, send in a column name\n",
    "    \"\"\"\n",
    "    \n",
    "    if stratify_by == None:\n",
    "        train, test = train_test_split(df, test_size=.2, random_state=123)\n",
    "        train, validate = train_test_split(train, test_size=.3, random_state=123)\n",
    "    else:\n",
    "        train, test = train_test_split(df, test_size=.2, random_state=123, stratify=df[stratify_by])\n",
    "        train, validate = train_test_split(train, test_size=.3, random_state=123, stratify=train[stratify_by])\n",
    "    \n",
    "    return train, validate, test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Acquisition Stage\n",
    "\n",
    "Python and its powerful code libraries like pandas allow us to easily acquire and clean data.\n",
    "\n",
    "When you're acquiring data, make sure you know WHAT you're after and HOW to get it. Consult with your team if possible to confirm you have the right data before moving forward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Acquire our data from a Comma Separated Values text file exported from a spreadsheet in Human Resources.\n",
    "url = \"https://raw.githubusercontent.com/ryanorsinger/90-minutes-to-machine-learning/main/data.csv\"\n",
    "df = pd.read_csv(url, index_col=\"index\")\n",
    "\n",
    "# Get a sense of what each row/observation means\n",
    "# See what columns we have to work with for this\n",
    "# Our columns will be our features\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Preparation Stage\n",
    "- Machine Learning algorithms can only work on numbers, so we need to convert some of our columns into numbers.\n",
    "- For example, the `Attrition` column only has 2 different values, so we will convert those \"Yes\" or \"No\" strings into 0 and 1s to represent True and False and so our Machine Algorithms can operate on them.\n",
    "- The `BusinessTravel` column has 3 values, but \"rarely\" is close enough to \"never\", so we'll make this a binary too.\n",
    "- In practice, real world data is _much_ messier than this example dataset.\n",
    "- In industry, the data preparation stage is _critical_ because machine learning algorithms only work on numbers. When any machine learning is working with words, picture, sound, or video, everything has to be converted into numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We'll map \"Yes\" to True and \"No\" to false for the Attrition column\n",
    "df.Attrition = df.Attrition.apply(lambda x: 1 if x == \"Yes\" else 0)\n",
    "\n",
    "# Clean up other column names\n",
    "df = df.rename(columns={\"BusinessTravel\": \"FrequentTravel\"})\n",
    "df.FrequentTravel = df.FrequentTravel.apply(lambda x: 1 if x == \"Travel_Frequently\" else 0)\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The very last step of the Data Preparation stage is to split our data\n",
    "train, validate, test = split(df)\n",
    "\n",
    "# We now how have one in-sample and two out-of-sample datasets for when we get to our modeling\n",
    "train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploration Phase (Exploratory Data Analysis)\n",
    "- Exploratory Data Analysis (EDA) is where we learn about our data and discover:\n",
    "    - What each of the varibles looks like in detail \n",
    "    - How the variables in our dataset relate to each-other\n",
    "    - Most of our takeaways and insights to deliver to stakeholders\n",
    "- The main tasks in EDA include:\n",
    "    - Exploring descriptive statistics\n",
    "    - inferential statistics and hypothesis testing\n",
    "- Exploratory Data Analysis is where data scientists, statisticians, and analysts overlap a _lot_.\n",
    "- We use Machine Learning to make predictive models for the future, but EDA is where we really and truly get to _know_ our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's get to know the Monthly Income information\n",
    "# .describe on a column gives us a quick glance at useful descriptive statistics\n",
    "train.MonthlyIncome.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's visualize the distribution of employee Monthly Income\n",
    "train.MonthlyIncome.hist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# What is the distribution of employees who leave vs. stay?\n",
    "train.Attrition.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# It looks like attrition is REALLY high at this company!!\n",
    "# around 20% of the employees left last year! That's 1 out of every 5 employees! Gone!\n",
    "# This is probably why HR wants top focus on disvovering how to keep employees from leaving!\n",
    "attrition_rate = 180 / 849\n",
    "attrition_rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# How many employees have which kind of educational level?\n",
    "sns.catplot(x=\"Education\",  data=train, kind=\"count\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# How many employees self report \n",
    "# Remember that self-reporting is a statistical bias\n",
    "sns.catplot(x=\"WorkLifeBalance\",  data=train, kind=\"count\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's see how many employees report for the different job satisfaction levels\n",
    "# Remember that self-reporting is a statistical bias\n",
    "# Looks like most people are 3s and 4s. \n",
    "sns.catplot(x=\"JobSatisfaction\",  data=train, kind=\"count\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def category_percentages_by_another_category(df, category_a, category_b):\n",
    "    (df.groupby(category_b)[category_a].value_counts(normalize=True)\n",
    "    .rename('percent')\n",
    "    .reset_index()\n",
    "    .pipe((sns.catplot, 'data'), x=category_a, y='percent', hue=category_b, kind='bar'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "category_percentages_by_another_category(train, 'JobSatisfaction', 'Attrition')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "category_percentages_by_another_category(train, 'WorkLifeBalance', 'Attrition')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Do employees who stay get paid more? \n",
    "# Let's compare the distributions of Monthly Income between employees who stayed vs. left\n",
    "sns.boxplot(x=\"Attrition\", y=\"MonthlyIncome\", data=train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.catplot(x=\"JobSatisfaction\", y=\"MonthlyIncome\", data=df, hue=\"Attrition\", kind=\"bar\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.catplot(x=\"WorkLifeBalance\", y=\"MonthlyIncome\", data=df, hue=\"Attrition\", kind=\"bar\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.catplot(x=\"Attrition\", y=\"FrequentTravel\", data=train, kind=\"bar\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.boxplot(x=\"Attrition\", y=\"PercentSalaryHike\", data=train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EDA can go very deep and take more time than we have for this workshop!\n",
    "- With only 7 variables, like we have here:\n",
    "    - There are various statistics we should calculate for each variable.\n",
    "    - There are dozens of possible variable pairings for data visualizations and charts.\n",
    "    - There are dozens of groupings for statistical hypothesis testing.\n",
    "- We are skipping a _ton_ of exploration here in order to move forward w/ this workshop.\n",
    "- Let's make a special memory and note that we are rushing through the EDA process on purpose. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Onto the Machine Learning Modeling\n",
    "\n",
    "## How does a decision tree work:\n",
    "- Decision Trees work like playing [20 questions](https://en.wikipedia.org/wiki/Twenty_questions) with our features and our target variable.    \n",
    "- Classification algorithms use training data to measure the distance between points or the distance around boundaries between points.\n",
    "- By \"learning\" the pattern recognition around sets of points, the classifier produces a \"decision rule\" to use to apply to classify new incoming data.\n",
    "\n",
    "#### Consider this diagram of a decision tree used to predict iris species w/ petal measurements \n",
    "![decision tree diagram](decision_tree_diagram.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modeling\n",
    "\n",
    "Once we discover and learn which features are drivers of our target variable, we can begin modeling using those features. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Our baseline here is to assume the most common case\n",
    "baseline_accuracy = (train.Attrition == 0).mean()\n",
    "baseline_accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's define our target variable\n",
    "# This is what we're trying to predict\n",
    "y_train = train[[\"Attrition\"]]\n",
    "y_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = train.drop(['Attrition'],axis=1)\n",
    "X_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The easiest part of the entire Data Science pipeline is fitting the machine learning model...\n",
    "decision_tree_model = DecisionTreeClassifier(max_depth=8)\n",
    "decision_tree_model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Produce a set of Attrition predictions\n",
    "y_pred = decision_tree_model.predict(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating the Model\n",
    "- \"Accuracy\" is calculated as the total number of True Positives + True Negatives divided by the total number of observations\n",
    "- There are [many, many ways](https://en.wikipedia.org/wiki/Sensitivity_and_specificity) to measure how well a model performs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Accuracy = total number of (true positives + number of true negatives) divided by the total numbrer of observations\n",
    "accuracy = decision_tree_model.score(X_train, y_train)\n",
    "accuracy = round(accuracy * 100)\n",
    "print(f'Accuracy of Decision Tree classifier on training set: {accuracy}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now let's see how well this trained model performs on out-of-sample data\n",
    "X_validate = validate.drop(columns=[\"Attrition\"])\n",
    "y_validate = validate[[\"Attrition\"]]\n",
    "\n",
    "accuracy = decision_tree_model.score(X_validate, y_validate)\n",
    "accuracy = round(accuracy * 100)\n",
    "\n",
    "print(f'Accuracy of Decision Tree classifier on training set: {accuracy}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Importance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame({\n",
    "    \"feature\": X_train.columns.tolist(), \n",
    "    \"feature_importance\": decision_tree_model.feature_importances_.round(3)\n",
    "}).set_index(\"feature\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## We have a model. But what does this mean?\n",
    "\n",
    "- Remember that last year's attrition rate was ~ 20%. \n",
    "- Our model performs well on training but not on out-of-sample data.\n",
    "- This behavior in a model is called \"overfit\". \n",
    "- So we're in a space where the model's performance could be improved. \n",
    "- Covering all the ways to evaluate and improve models is something we will do in the Codeup class, not 90 minute intro.\n",
    "\n",
    "# Parting Takeaways\n",
    "\n",
    "Valuable machine learning models that work perfectly the first time are something that only happen in the movies and in tutorials.\n",
    "\n",
    "There is much more to Data Science and Machine Learning than making models!\n",
    "\n",
    "There's _always_ room for more exploraty data analysis. Especially if we get updated, richer information!\n",
    "    \n",
    "We could always use better, higher quality data!\n",
    "\n",
    "Models aren't enough. We need _good, effective models_.\n",
    "\n",
    "\n",
    "### Ways to improve model performance that we teach in Codeup\n",
    "- Hypothesis testing for feature selection in our models\n",
    "- Feature engineering\n",
    "- Handling imbalanced data problems \n",
    "- Identifying and removing outliers\n",
    "- Different types of ML models\n",
    "- Hyperparameters and hyperparameter tuning \n",
    "- Dimensionality reduction\n",
    "- Addressing underfit and overfit models\n",
    "\n",
    "## If you enjoyed what we did together in 90 minutes today, [apply to Codeup](https://codeup.com/) to learn Data Science in less than 6 months!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}