Assignment_Task_10.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Assignment / Task -10.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyPibvfk4lLZF/1XocDvOazC",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/rammya29/Goeduhub-STP/blob/main/Assignment_Task_10.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KcjlKeKMYFn9"
      },
      "source": [
        "#Assignment / Task -10"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Avs7lCMV-1EV"
      },
      "source": [
        "###Discuss the concept of One-Hot-Encoding, Multicollinearity and the Dummy Variable Trap.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F5QFlX8g-2IV"
      },
      "source": [
        "####**OneHot Encoding**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hk0OvYij_YYs"
      },
      "source": [
        "One-Hot Encoding is another popular technique for treating categorical variables. It simply creates additional features based on the number of unique values in the categorical feature. Every unique value in the category will be added as a feature."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GCwR-bni_cDU"
      },
      "source": [
        "####**Multicollinearity**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LKEBk1km_pw1"
      },
      "source": [
        "Multicollinearity is the occurrence of high intercorrelations among two or more independent variables in a multiple regression model. ... In general, multicollinearity can lead to wider confidence intervals that produce less reliable probabilities in terms of the effect of independent variables in a model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yngX5Zwr_lCU"
      },
      "source": [
        "####**Dummy Variable Trap**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hxKb9mBj_2RD"
      },
      "source": [
        "The Dummy Variable trap is a scenario in which the independent variables are multicollinear - a scenario in which two or more variables are highly correlated; in simple terms one variable can be predicted from the others."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rbmI3hby_9dY"
      },
      "source": [
        "###What is Nominal and Ordinal Variables ?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YBrAZIvb__Pk"
      },
      "source": [
        "A ***categorical variable*** (sometimes called a nominal variable) is one that has two or more categories, but there is no intrinsic ordering to the categories.  For example, a binary variable (such as yes/no question) is a categorical variable having two categories (yes or no) and there is no intrinsic ordering to the categories.  \n",
        "\n",
        "\n",
        "An ***ordinal variable*** is similar to a categorical variable.  The difference between the two is that there is a clear ordering of the categories.  For example, suppose you have a variable, economic status, with three categories (low, medium and high)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oL2AVuLiAXki"
      },
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AUevnWLuYI8x"
      },
      "source": [
        "###Salary Dataset of 52 professors having categorical columns. Apply dummy variables concept and one-hot-encoding on categorical columns.\n",
        "\n",
        "Dataset Link- [Click here](https://data.princeton.edu/wws509/datasets/salary.dat)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 205
        },
        "id": "VuDa_21TWldi",
        "outputId": "0c30060b-51bf-4659-9aae-448bf14f47a8"
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "df = pd.read_table('/content/SalartData.txt',delim_whitespace=True)\n",
        "df.head()"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sx</th>\n",
              "      <th>rk</th>\n",
              "      <th>yr</th>\n",
              "      <th>dg</th>\n",
              "      <th>yd</th>\n",
              "      <th>sl</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>male</td>\n",
              "      <td>full</td>\n",
              "      <td>25</td>\n",
              "      <td>doctorate</td>\n",
              "      <td>35</td>\n",
              "      <td>36350</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>male</td>\n",
              "      <td>full</td>\n",
              "      <td>13</td>\n",
              "      <td>doctorate</td>\n",
              "      <td>22</td>\n",
              "      <td>35350</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>male</td>\n",
              "      <td>full</td>\n",
              "      <td>10</td>\n",
              "      <td>doctorate</td>\n",
              "      <td>23</td>\n",
              "      <td>28200</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>female</td>\n",
              "      <td>full</td>\n",
              "      <td>7</td>\n",
              "      <td>doctorate</td>\n",
              "      <td>27</td>\n",
              "      <td>26775</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>male</td>\n",
              "      <td>full</td>\n",
              "      <td>19</td>\n",
              "      <td>masters</td>\n",
              "      <td>30</td>\n",
              "      <td>33696</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       sx    rk  yr         dg  yd     sl\n",
              "0    male  full  25  doctorate  35  36350\n",
              "1    male  full  13  doctorate  22  35350\n",
              "2    male  full  10  doctorate  23  28200\n",
              "3  female  full   7  doctorate  27  26775\n",
              "4    male  full  19    masters  30  33696"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 299
        },
        "id": "9me4Q0xp92wx",
        "outputId": "34a8b1e6-476f-45dc-e6be-cae102da8d0b"
      },
      "source": [
        "df.describe()"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>yr</th>\n",
              "      <th>yd</th>\n",
              "      <th>sl</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>52.000000</td>\n",
              "      <td>52.000000</td>\n",
              "      <td>52.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>7.480769</td>\n",
              "      <td>16.115385</td>\n",
              "      <td>23797.653846</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>5.507536</td>\n",
              "      <td>10.222340</td>\n",
              "      <td>5917.289154</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>15000.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>3.000000</td>\n",
              "      <td>6.750000</td>\n",
              "      <td>18246.750000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>7.000000</td>\n",
              "      <td>15.500000</td>\n",
              "      <td>23719.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>11.000000</td>\n",
              "      <td>23.250000</td>\n",
              "      <td>27258.500000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>25.000000</td>\n",
              "      <td>35.000000</td>\n",
              "      <td>38045.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "              yr         yd            sl\n",
              "count  52.000000  52.000000     52.000000\n",
              "mean    7.480769  16.115385  23797.653846\n",
              "std     5.507536  10.222340   5917.289154\n",
              "min     0.000000   1.000000  15000.000000\n",
              "25%     3.000000   6.750000  18246.750000\n",
              "50%     7.000000  15.500000  23719.000000\n",
              "75%    11.000000  23.250000  27258.500000\n",
              "max    25.000000  35.000000  38045.000000"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ABPnhYjI96_U",
        "outputId": "d1e3cc9a-01d7-4618-e484-cc1af0d8d487"
      },
      "source": [
        "df.info()"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 52 entries, 0 to 51\n",
            "Data columns (total 6 columns):\n",
            " #   Column  Non-Null Count  Dtype \n",
            "---  ------  --------------  ----- \n",
            " 0   sx      52 non-null     object\n",
            " 1   rk      52 non-null     object\n",
            " 2   yr      52 non-null     int64 \n",
            " 3   dg      52 non-null     object\n",
            " 4   yd      52 non-null     int64 \n",
            " 5   sl      52 non-null     int64 \n",
            "dtypes: int64(3), object(3)\n",
            "memory usage: 2.6+ KB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jUqGoBLO-Hh7",
        "outputId": "5087a046-2851-45d4-e068-d9347110603f"
      },
      "source": [
        "df.columns"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Index(['sx', 'rk', 'yr', 'dg', 'yd', 'sl'], dtype='object')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 205
        },
        "id": "PgJxqFAH-KUe",
        "outputId": "8b96c7c8-10ca-4444-eecc-de124cc8b8fa"
      },
      "source": [
        "pd.get_dummies(df, columns = ['sx', 'rk' , 'dg']).head()"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>yr</th>\n",
              "      <th>yd</th>\n",
              "      <th>sl</th>\n",
              "      <th>sx_female</th>\n",
              "      <th>sx_male</th>\n",
              "      <th>rk_assistant</th>\n",
              "      <th>rk_associate</th>\n",
              "      <th>rk_full</th>\n",
              "      <th>dg_doctorate</th>\n",
              "      <th>dg_masters</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>25</td>\n",
              "      <td>35</td>\n",
              "      <td>36350</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>13</td>\n",
              "      <td>22</td>\n",
              "      <td>35350</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>10</td>\n",
              "      <td>23</td>\n",
              "      <td>28200</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>7</td>\n",
              "      <td>27</td>\n",
              "      <td>26775</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>19</td>\n",
              "      <td>30</td>\n",
              "      <td>33696</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   yr  yd     sl  sx_female  ...  rk_associate  rk_full  dg_doctorate  dg_masters\n",
              "0  25  35  36350          0  ...             0        1             1           0\n",
              "1  13  22  35350          0  ...             0        1             1           0\n",
              "2  10  23  28200          0  ...             0        1             1           0\n",
              "3   7  27  26775          1  ...             0        1             1           0\n",
              "4  19  30  33696          0  ...             0        1             0           1\n",
              "\n",
              "[5 rows x 10 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 205
        },
        "id": "jOrzN6DABQRo",
        "outputId": "d2ffa352-5015-4202-db1a-fa2585e74b8b"
      },
      "source": [
        "pd.get_dummies(df, columns = ['sx', 'rk' , 'dg'], drop_first=True).head()"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>yr</th>\n",
              "      <th>yd</th>\n",
              "      <th>sl</th>\n",
              "      <th>sx_male</th>\n",
              "      <th>rk_associate</th>\n",
              "      <th>rk_full</th>\n",
              "      <th>dg_masters</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>25</td>\n",
              "      <td>35</td>\n",
              "      <td>36350</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>13</td>\n",
              "      <td>22</td>\n",
              "      <td>35350</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>10</td>\n",
              "      <td>23</td>\n",
              "      <td>28200</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>7</td>\n",
              "      <td>27</td>\n",
              "      <td>26775</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>19</td>\n",
              "      <td>30</td>\n",
              "      <td>33696</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   yr  yd     sl  sx_male  rk_associate  rk_full  dg_masters\n",
              "0  25  35  36350        1             0        1           0\n",
              "1  13  22  35350        1             0        1           0\n",
              "2  10  23  28200        1             0        1           0\n",
              "3   7  27  26775        0             0        1           0\n",
              "4  19  30  33696        1             0        1           1"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DGoVXoTR8-_O"
      },
      "source": [
        "###OneHot Encoder"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eeWG85WM93e2"
      },
      "source": [
        "import numpy as np"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QY8fY33fBb_d",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e0c0dc8e-6e0f-4fc2-e4f6-9454a6f4dd3f"
      },
      "source": [
        "from sklearn.preprocessing import OneHotEncoder\n",
        "from sklearn.compose import ColumnTransformer\n",
        "\n",
        "col = ColumnTransformer([('encoder', OneHotEncoder(), [0,1,3])], remainder = 'passthrough')\n",
        "data = np.array(col.fit_transform(df), dtype = str)\n",
        "\n",
        "print(data[:7])"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[['0.0' '1.0' '0.0' '0.0' '1.0' '1.0' '0.0' '25.0' '35.0' '36350.0']\n",
            " ['0.0' '1.0' '0.0' '0.0' '1.0' '1.0' '0.0' '13.0' '22.0' '35350.0']\n",
            " ['0.0' '1.0' '0.0' '0.0' '1.0' '1.0' '0.0' '10.0' '23.0' '28200.0']\n",
            " ['1.0' '0.0' '0.0' '0.0' '1.0' '1.0' '0.0' '7.0' '27.0' '26775.0']\n",
            " ['0.0' '1.0' '0.0' '0.0' '1.0' '0.0' '1.0' '19.0' '30.0' '33696.0']\n",
            " ['0.0' '1.0' '0.0' '0.0' '1.0' '1.0' '0.0' '16.0' '21.0' '28516.0']\n",
            " ['1.0' '0.0' '0.0' '0.0' '1.0' '0.0' '1.0' '0.0' '32.0' '24900.0']]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GgA4afPn-ebV"
      },
      "source": [
        "#End Of Assignment"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qDYQAq2r-hPU"
      },
      "source": [
        "**Author** : Rammya Dharshini K\n",
        "\n",
        "**Date** : 10-06-2021\n",
        "\n",
        "**Reg. ID** : GO_STP_1613"
      ]
    }
  ]
}