Modelling Assignment Debayan Notebook 1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "352bc1e2",
   "metadata": {},
   "source": [
    "# Home Credit Default Risk Prediction Using Naive Bayes Model "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f3b6025",
   "metadata": {},
   "source": [
    "## By Debayan Dutta July-09-2023"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bab15125",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e71d07e6",
   "metadata": {},
   "source": [
    "Project Goal: Home Credit will be able to identify if a customer is a safe candidate to lend to, then create a personalized customer loan and repayment plan to be accountable for, resulting in an increase in revenue, improved customer experience, and lower default rates.\n",
    "\n",
    "Business Problem: Home Credit desires to know safe borrowers in a customer base that is unfamiliar with banking and give the customer a plan for successful loan repayment. Lending to those who are more likely to default on loans decreases the profits of Home Credit and results in negative customer experiences.\n",
    "\n",
    "Analytic Problem:\n",
    "\n",
    "The target variable is specificially customers that do have a negative history of repayment to lend to, and postive repayment. Represented in the application_train/test.csv sets of binary where 1 = Not trust worthy borrower (Client with payment difficulties), 0 = Trustworthy borrower (client with good repayment history).\n",
    "\n",
    "Predict which customers will be good borrowers, using a classification method based on customer financial behavior data.\n",
    "\n",
    "Use a machine learning method to see the relationship that a trust worthy customer has to other attributes about them."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d71a770",
   "metadata": {},
   "source": [
    "## Machine Learning Model Choice 4 - Naive Bayes Algorithm\n",
    "\n",
    "- Naive Bayes is a simple yet powerful probabilistic classifier based on Bayes' theorem and the assumption of feature independence.\n",
    "\n",
    "- Naive Bayes assumes that the presence of a particular feature in a class is unrelated to the presence of any other feature. This makes it a very fast and efficient algorithm to train.\n",
    "\n",
    "- It is commonly used for classification tasks and works well with high-dimensional data.\n",
    "\n",
    "- Naive Bayes calculates the probability of a class given the feature values and makes predictions based on the class with the highest probability.\n",
    "\n",
    "- The \"naive\" assumption assumes that features are conditionally independent given the class, which simplifies the calculation of probabilities.\n",
    "\n",
    "- Naive Bayes is computationally efficient and performs well in many real-world scenarios, especially when the independence assumption holds reasonably well."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8f4dc83",
   "metadata": {},
   "source": [
    "# Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4c418ce",
   "metadata": {},
   "source": [
    "## Importing packages and data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "ba9bd482",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Import necessary libraries\n",
    "import os\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "#conda install -c conda-forge imbalanced-learn\n",
    "from imblearn.over_sampling import RandomOverSampler\n",
    "import matplotlib.pyplot as plt\n",
    "from scipy.stats import randint, uniform\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV\n",
    "from sklearn.preprocessing import OneHotEncoder, StandardScaler\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.metrics import accuracy_score, classification_report\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.metrics import confusion_matrix, accuracy_score, classification_report,precision_score, recall_score, f1_score\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.naive_bayes import CategoricalNB\n",
    "from sklearn.naive_bayes import GaussianNB, BernoulliNB\n",
    "from sklearn import metrics\n",
    "import warnings"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4528c558",
   "metadata": {},
   "source": [
    "## Loading Datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "9d08c2a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Load the necessary datasets\n",
    "train_data = pd.read_csv('application_train.csv')\n",
    "test_data = pd.read_csv('application_test.csv')\n",
    "pos_cash_balance = pd.read_csv('POS_CASH_balance.csv')\n",
    "bureau = pd.read_csv('bureau.csv')\n",
    "#bureau_balance = pd.read_csv('bureau_balance.csv')\n",
    "credit_card_balance = pd.read_csv('credit_card_balance.csv')\n",
    "#installments_payments = pd.read_csv('installments_payments.csv')\n",
    "#previous_application = pd.read_csv('previous_application.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8312a96",
   "metadata": {},
   "source": [
    "## Joining in Datasets "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "fa3d7c92",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Calculate average 'SK_DPD' values in credit_card_balance.csv\n",
    "average_sk_dpd = credit_card_balance.groupby('SK_ID_CURR')['SK_DPD'].mean().reset_index()\n",
    "\n",
    "#Merge average_sk_dpd with train_data based on 'SK_ID_CURR'\n",
    "train_data = train_data.merge(average_sk_dpd, on='SK_ID_CURR', how='left')\n",
    "\n",
    "#Merge average_sk_dpd with test_data based on 'SK_ID_CURR'\n",
    "test_data = test_data.merge(average_sk_dpd, on='SK_ID_CURR', how='left')\n",
    "\n",
    "#Fill missing values with 0\n",
    "train_data['SK_DPD'] = train_data['SK_DPD'].fillna(0)\n",
    "test_data['SK_DPD'] = test_data['SK_DPD'].fillna(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "f3191811",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Calculate average 'CREDIT_DAY_OVERDUE' values in bureau.csv\n",
    "average_credit_day_overdue = bureau.groupby('SK_ID_CURR')['CREDIT_DAY_OVERDUE'].mean().reset_index()\n",
    "\n",
    "#Merge average_credit_day_overdue with train_data based on 'SK_ID_CURR'\n",
    "train_data = train_data.merge(average_credit_day_overdue, on='SK_ID_CURR', how='left')\n",
    "\n",
    "#Merge average_credit_day_overdue with test_data based on 'SK_ID_CURR'\n",
    "test_data = test_data.merge(average_credit_day_overdue, on='SK_ID_CURR', how='left')\n",
    "\n",
    "#Fill missing values with 0\n",
    "train_data['CREDIT_DAY_OVERDUE'] = train_data['CREDIT_DAY_OVERDUE'].fillna(0)\n",
    "test_data['CREDIT_DAY_OVERDUE'] = test_data['CREDIT_DAY_OVERDUE'].fillna(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "752e7730",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(307511, 124)\n",
      "(48744, 123)\n"
     ]
    }
   ],
   "source": [
    "#Shape of the joined datasets\n",
    "print(train_data.shape)\n",
    "print(test_data.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b50a72d",
   "metadata": {},
   "source": [
    "## EDA - Treating Missing Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "ecf3f2ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Remove columns with more than 30% null values\n",
    "train_data = train_data.dropna(thresh=len(train_data) * 0.7, axis=1)\n",
    "test_data = test_data.dropna(thresh=len(test_data) * 0.7, axis=1)\n",
    "\n",
    "#Select valid numeric columns\n",
    "numeric_columns = train_data.select_dtypes(include=np.number).columns\n",
    "\n",
    "#Impute mean for numeric columns with less than 30% null values in train_data\n",
    "train_data = train_data.loc[:, numeric_columns].fillna(train_data.loc[:, numeric_columns].mean())\n",
    "\n",
    "#Select valid numeric columns in test\n",
    "test_numeric_columns = test_data.select_dtypes(include=np.number).columns\n",
    "\n",
    "#Impute mean for numeric columns with less than 30% null values in test_data\n",
    "test_data = test_data.loc[:, test_numeric_columns].fillna(test_data.loc[:, test_numeric_columns].mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "262e704b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SK_ID_CURR                    0\n",
      "TARGET                        0\n",
      "CNT_CHILDREN                  0\n",
      "AMT_INCOME_TOTAL              0\n",
      "AMT_CREDIT                    0\n",
      "                             ..\n",
      "AMT_REQ_CREDIT_BUREAU_MON     0\n",
      "AMT_REQ_CREDIT_BUREAU_QRT     0\n",
      "AMT_REQ_CREDIT_BUREAU_YEAR    0\n",
      "SK_DPD                        0\n",
      "CREDIT_DAY_OVERDUE            0\n",
      "Length: 63, dtype: int64\n",
      "(307511, 63)\n",
      "(48744, 62)\n"
     ]
    }
   ],
   "source": [
    "# Check for null values in train_data\n",
    "null_counts = train_data.isnull().sum()\n",
    "\n",
    "print(null_counts)\n",
    "\n",
    "print(train_data.shape)\n",
    "print(test_data.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aac7a5ff",
   "metadata": {},
   "source": [
    "Reviewing the columns with more than 30% null values dropped the number of variables in the train and test sets by 61. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8ac8c6a",
   "metadata": {},
   "source": [
    "## Modeling Creation Processing & Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "id": "31f72056",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.9195323805342829\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# Preprocess the data\n",
    "label_encoder = LabelEncoder()\n",
    "for column in train_data.columns:\n",
    "    if train_data[column].dtype == 'object':\n",
    "        train_data[column] = label_encoder.fit_transform(train_data[column].astype(str))\n",
    "\n",
    "# Separate the features (X) and target variable (y)\n",
    "X = train_data.drop('TARGET', axis=1)\n",
    "y = train_data['TARGET']\n",
    "\n",
    "# Split the data into training and testing sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
    "\n",
    "# Train the Naive Bayes model\n",
    "naive_bayes = GaussianNB()\n",
    "naive_bayes.fit(X_train, y_train)\n",
    "\n",
    "# Make predictions on the test set\n",
    "y_pred1 = naive_bayes.predict(X_test)\n",
    "\n",
    "# Calculate accuracy\n",
    "accuracy1 = accuracy_score(y_test, y_pred)\n",
    "print(\"Accuracy:\", accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "id": "7dd31b92",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.9195323805342829\n"
     ]
    }
   ],
   "source": [
    "#Estimator BernoulliNB\n",
    "\n",
    "label_encoder = LabelEncoder()\n",
    "for column in train_data.columns:\n",
    "    if train_data[column].dtype == 'object':\n",
    "        train_data[column] = label_encoder.fit_transform(train_data[column].astype(str))\n",
    "\n",
    "# Separate the features (X) and target variable (y)\n",
    "X = train_data.drop('TARGET', axis=1)\n",
    "y = train_data['TARGET']\n",
    "\n",
    "# Split the data into training and testing sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
    "\n",
    "# Train the Naive Bayes model\n",
    "naive_bayes_b1 = BernoulliNB()\n",
    "naive_bayes_b1.fit(X_train, y_train)\n",
    "\n",
    "# Make predictions on the test set\n",
    "y_pred2 = naive_bayes_b1.predict(X_test)\n",
    "\n",
    "# Calculate accuracy\n",
    "accuracy2 = accuracy_score(y_test, y_pred)\n",
    "print(\"Accuracy:\", accuracy)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "4dcd31de",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cross-Validation Scores: [0.91774725 0.91813341 0.91849925 0.91790817 0.91798947]\n",
      "Mean Accuracy: 0.9180555095225575\n",
      "Standard Deviation: 0.00025458504357842574\n"
     ]
    }
   ],
   "source": [
    "# Perform cross-validation\n",
    "scores = cross_val_score(naive_bayes, X_train, y_train, cv=5)\n",
    "\n",
    "# Print cross-validation scores\n",
    "print(\"Cross-Validation Scores:\", scores)\n",
    "print(\"Mean Accuracy:\", scores.mean())\n",
    "print(\"Standard Deviation:\", scores.std())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "id": "5b138c11",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Hyperparameters: {'alpha': 0.1, 'binarize': 1.0, 'fit_prior': True}\n",
      "Accuracy: 0.9195323805342829\n"
     ]
    }
   ],
   "source": [
    "# Adjusting hyperparameters for Bernoulli's Estimator \n",
    "# Perform label encoding on object columns\n",
    "label_encoder = LabelEncoder()\n",
    "for column in train_data.columns:\n",
    "    if train_data[column].dtype == 'object':\n",
    "        train_data[column] = label_encoder.fit_transform(train_data[column].astype(str))\n",
    "\n",
    "# Separate the features (X) and target variable (y)\n",
    "X = train_data.drop('TARGET', axis=1)\n",
    "y = train_data['TARGET']\n",
    "\n",
    "# Split the data into training and testing sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
    "\n",
    "# Define the hyperparameters and their values to tune\n",
    "param_grid = {\n",
    "    'alpha': [0.1, 0.5, 1.0, 2.0, 5.0],\n",
    "    'binarize': [0.0, 0.5, 1.0],\n",
    "    'fit_prior': [True, False]\n",
    "}\n",
    "\n",
    "# Create Bernoulli Naive Bayes model\n",
    "naive_bayes_b2 = BernoulliNB()\n",
    "\n",
    "# Perform Grid Search with 5-fold cross-validation\n",
    "grid_search = GridSearchCV(estimator=naive_bayes_b2, param_grid=param_grid, cv=5)\n",
    "grid_search.fit(X_train, y_train)\n",
    "\n",
    "# Get the best hyperparameter values\n",
    "best_params = grid_search.best_params_\n",
    "print(\"Best Hyperparameters:\", best_params)\n",
    "\n",
    "# Train the Naive Bayes model with the best hyperparameters\n",
    "best_naive_bayes = BernoulliNB(**best_params)\n",
    "best_naive_bayes.fit(X_train, y_train)\n",
    "\n",
    "# Make predictions on the test set\n",
    "y_pred3 = best_naive_bayes.predict(X_test)\n",
    "\n",
    "# Calculate accuracy\n",
    "accuracy3 = accuracy_score(y_test, y_pred)\n",
    "print(\"Accuracy:\", accuracy)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06591a09",
   "metadata": {},
   "source": [
    "### Feature Importance "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "529e17e9",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AMT_INCOME_TOTAL: -3.2518738923403843e-06\n",
      "AMT_CREDIT: -2.6014991138678666e-05\n",
      "AMT_ANNUITY: -5.8533730061949286e-05\n",
      "AMT_GOODS_PRICE: -9.755621676976744e-06\n",
      "DAYS_BIRTH: -6.5037477846585645e-06\n",
      "DAYS_EMPLOYED: -3.2518738923403843e-06\n",
      "DAYS_REGISTRATION: 1.95112433539979e-05\n",
      "DAYS_ID_PUBLISH: -1.6259369461657514e-05\n",
      "DAYS_LAST_PHONE_CHANGE: 9.105246898526431e-05\n",
      "SK_DPD: -6.5037477846585645e-06\n",
      "CREDIT_DAY_OVERDUE: 0.0003154317675560625\n"
     ]
    }
   ],
   "source": [
    "from sklearn.inspection import permutation_importance\n",
    "\n",
    "# Calculate permutation importance\n",
    "importance = permutation_importance(naive_bayes, X_test, y_test)\n",
    "\n",
    "# Get feature importances\n",
    "feature_importance = importance.importances_mean\n",
    "\n",
    "# Print feature importances\n",
    "for i, feature_name in enumerate(X.columns):\n",
    "    if feature_importance[i] != 0:\n",
    "        print(f\"{feature_name}: {feature_importance[i]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "id": "cfa83c26",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix:\n",
      "[[56385   169]\n",
      " [ 4923    26]]\n",
      "[[56467    87]\n",
      " [ 4931    18]]\n",
      "[[56554     0]\n",
      " [ 4949     0]]\n"
     ]
    }
   ],
   "source": [
    "#Create confusion matrix\n",
    "# Calculate confusion matrix\n",
    "conf_matrix1 = confusion_matrix(y_test, y_pred1)\n",
    "conf_matrix2 = confusion_matrix(y_test, y_pred2)\n",
    "conf_matrix3 = confusion_matrix(y_test, y_pred3)\n",
    "\n",
    "\n",
    "# Print the confusion matrix\n",
    "print(\"Confusion Matrix:\")\n",
    "print(conf_matrix1)\n",
    "print(conf_matrix2)\n",
    "print(conf_matrix3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "id": "91a4b639",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.9172072907012666\n",
      "Precision: 0.13333333333333333\n",
      "Recall: 0.005253586583148111\n",
      "F1-score: 0.010108864696734058\n"
     ]
    }
   ],
   "source": [
    "#Evaluating confusion matrix values for Naive Bayes [Estimator = GausianNB]\n",
    "TN, FP, FN, TP = conf_matrix1.ravel()\n",
    "\n",
    "#Calculate accuracy\n",
    "accuracy = (TP + TN) / (TP + TN + FP + FN)\n",
    "\n",
    "#Calculate precision\n",
    "precision = TP / (TP + FP)\n",
    "\n",
    "#Calculate recall\n",
    "recall = TP / (TP + FN)\n",
    "\n",
    "#Calculate F1-score\n",
    "f1_score = 2 * (precision * recall) / (precision + recall)\n",
    "\n",
    "# Print the metrics\n",
    "print(\"Accuracy:\", accuracy)\n",
    "print(\"Precision:\", precision)\n",
    "print(\"Recall:\", recall)\n",
    "print(\"F1-score:\", f1_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "id": "0bc8c284",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.9184104840414289\n",
      "Precision: 0.17142857142857143\n",
      "Recall: 0.003637098403717923\n",
      "F1-score: 0.007123070834982192\n"
     ]
    }
   ],
   "source": [
    "#Evaluating confusion matrix values for Naive Bayes [Estimator = BernoulliNB]\n",
    "TN, FP, FN, TP = conf_matrix2.ravel()\n",
    "\n",
    "#Calculate accuracy\n",
    "accuracy = (TP + TN) / (TP + TN + FP + FN)\n",
    "\n",
    "#Calculate precision\n",
    "precision = TP / (TP + FP)\n",
    "\n",
    "#Calculate recall\n",
    "recall = TP / (TP + FN)\n",
    "\n",
    "#Calculate F1-score\n",
    "f1_score = 2 * (precision * recall) / (precision + recall)\n",
    "\n",
    "# Print the metrics\n",
    "print(\"Accuracy:\", accuracy)\n",
    "print(\"Precision:\", precision)\n",
    "print(\"Recall:\", recall)\n",
    "print(\"F1-score:\", f1_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "id": "7271a96a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.9195323805342829\n",
      "Precision: nan\n",
      "Recall: 0.0\n",
      "F1-score: nan\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\u1427957\\AppData\\Local\\Temp\\ipykernel_8268\\1488848716.py:8: RuntimeWarning: invalid value encountered in longlong_scalars\n",
      "  precision = TP / (TP + FP)\n"
     ]
    }
   ],
   "source": [
    "#Evaluating confusion matrix values for Naive Bayes [Estimator = BernoulliNB] with hyperparameter tuning\n",
    "TN, FP, FN, TP = conf_matrix3.ravel()\n",
    "\n",
    "#Calculate accuracy\n",
    "accuracy = (TP + TN) / (TP + TN + FP + FN)\n",
    "\n",
    "#Calculate precision\n",
    "precision = TP / (TP + FP)\n",
    "\n",
    "#Calculate recall\n",
    "recall = TP / (TP + FN)\n",
    "\n",
    "#Calculate F1-score\n",
    "f1_score = 2 * (precision * recall) / (precision + recall)\n",
    "\n",
    "# Print the metrics\n",
    "print(\"Accuracy:\", accuracy)\n",
    "print(\"Precision:\", precision)\n",
    "print(\"Recall:\", recall)\n",
    "print(\"F1-score:\", f1_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96d90ce3",
   "metadata": {},
   "source": [
    "The accuracy score is 0.917, indicating that 91% of the predictions made by the model were correct. A precision score of 0.133 indicates that out of all the cases predicted as positive, only 13.3% were actually true positives. A recall score of 0.005 means that the model identified 5% of the actual positive cases.The F1-score is 0.010, which suggests that the model's overall performance is very low."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "772b10cd",
   "metadata": {},
   "source": [
    "### Prediction on the Test Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "id": "40a401d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_predictions = best_naive_bayes.predict(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "id": "7412f317",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Create a submission DataFrame\n",
    "submission = pd.DataFrame({\n",
    "    \"SK_ID_CURR\": test_data[\"SK_ID_CURR\"], \n",
    "    \"TARGET\": test_predictions\n",
    "})\n",
    "\n",
    "#Remove duplicate SK_ID_CURR values\n",
    "submission = submission.drop_duplicates(subset=\"SK_ID_CURR\", keep=\"first\")\n",
    "\n",
    "#Save the submission DataFrame to a CSV file\n",
    "#submission.to_csv(\"submission.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "id": "e058bd6a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, ..., 0, 0, 0], dtype=int64)"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "id": "d7ea2d27",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SK_ID_CURR</th>\n",
       "      <th>TARGET</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>100001</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>100005</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>100013</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>100028</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>100038</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48739</th>\n",
       "      <td>456221</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48740</th>\n",
       "      <td>456222</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48741</th>\n",
       "      <td>456223</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48742</th>\n",
       "      <td>456224</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48743</th>\n",
       "      <td>456250</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>48744 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       SK_ID_CURR  TARGET\n",
       "0          100001       0\n",
       "1          100005       0\n",
       "2          100013       0\n",
       "3          100028       0\n",
       "4          100038       0\n",
       "...           ...     ...\n",
       "48739      456221       0\n",
       "48740      456222       0\n",
       "48741      456223       0\n",
       "48742      456224       0\n",
       "48743      456250       0\n",
       "\n",
       "[48744 rows x 2 columns]"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "submission"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1701ab5c",
   "metadata": {},
   "source": [
    "Using estimator as GaussianNB we get accuracy of 0.9172 while using estimator as BernoulliNB we get accuracy 0.91841 and the accuracy increases to 0.9195 when we reset the hyperparameters. \n",
    "In case of estimator = BernoulliNB, we have obtained a kaggle score of 0.502"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}