NLP_Disaster_Tweets_v2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "NLP_Disaster_Tweets_v2.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OJ6RjruJue7f"
      },
      "source": [
        "**Kaggle competition**: [Natural Language Processing with Disaster Tweets](https://www.kaggle.com/c/nlp-getting-started)\n",
        "\n",
        "- The metric used is \"F1-score\".\n",
        "\n",
        "- The algortihm developed in this Notebook achieved a F1-score of about **82%** on the test set (The score is calculated by the Kaggle platform)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JLDanyNywIeW"
      },
      "source": [
        "# Download libraries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "88BEke4gjLxv",
        "outputId": "5907458b-ab87-4644-8958-747adf4e723a"
      },
      "source": [
        "# A dependency of the preprocessing for BERT inputs\n",
        "!pip install -q -U tensorflow-text\n",
        "\n",
        "# AdamW optimizer\n",
        "!pip install -q tf-models-official\n",
        "\n",
        "# Optuna: model optimization lib\n",
        "!pip install -q optuna"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[K     |████████████████████████████████| 4.3 MB 8.7 MB/s \n",
            "\u001b[K     |████████████████████████████████| 1.6 MB 7.7 MB/s \n",
            "\u001b[K     |████████████████████████████████| 43 kB 2.1 MB/s \n",
            "\u001b[K     |████████████████████████████████| 679 kB 47.9 MB/s \n",
            "\u001b[K     |████████████████████████████████| 636 kB 44.5 MB/s \n",
            "\u001b[K     |████████████████████████████████| 37.1 MB 87 kB/s \n",
            "\u001b[K     |████████████████████████████████| 90 kB 10.2 MB/s \n",
            "\u001b[K     |████████████████████████████████| 211 kB 59.4 MB/s \n",
            "\u001b[K     |████████████████████████████████| 99 kB 7.6 MB/s \n",
            "\u001b[K     |████████████████████████████████| 1.2 MB 17.0 MB/s \n",
            "\u001b[K     |████████████████████████████████| 352 kB 55.8 MB/s \n",
            "\u001b[?25h  Building wheel for py-cpuinfo (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building wheel for seqeval (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xKYgcpeBwPJt"
      },
      "source": [
        "# Import libraries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t_XHZnIanA9O"
      },
      "source": [
        "import re\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "import string\n",
        "import spacy\n",
        "from spacy.lang.en.stop_words import STOP_WORDS\n",
        "from spacy.lang.en import English\n",
        "\n",
        "from sklearn.model_selection import StratifiedKFold\n",
        "from sklearn.metrics import f1_score, jaccard_score, precision_score, recall_score\n",
        "\n",
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "import tensorflow_text as text\n",
        "from official.nlp import optimization  # to create AdamW optimizer\n",
        "import tensorflow_addons as tfa\n",
        "\n",
        "import optuna\n",
        "\n",
        "nlp = spacy.load(\"en_core_web_sm\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "om3BfVBDwSEu"
      },
      "source": [
        "# Set the configuration of this project"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MGBMRkaueows"
      },
      "source": [
        "AUTOTUNE = tf.data.AUTOTUNE\n",
        "\n",
        "CONFIGURATION = dict (\n",
        "    seed = 0,\n",
        "    nbr_classes = 1,\n",
        "    nbr_folds = 5,\n",
        "    batch_size = 32,\n",
        "    learning_rate_min = 1e-7,\n",
        "    epochs = 20\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jd_gRzl6wUsF"
      },
      "source": [
        "Check if a GPU is available"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FNGNzZqpem5h",
        "outputId": "9a936398-c47f-4dd1-b206-c8379c946762"
      },
      "source": [
        "gpus = tf.config.list_physical_devices('GPU')\n",
        "\n",
        "if gpus:\n",
        "  try:    \n",
        "    # Currently, memory growth needs to be the same across GPUs\n",
        "    \n",
        "    for gpu in gpus:\n",
        "      tf.config.experimental.set_memory_growth(gpu, True)\n",
        "    \n",
        "    logical_gpus = tf.config.experimental.list_logical_devices('GPU')\n",
        "    \n",
        "    print(len(gpus), \"Physical GPUs,\", len(logical_gpus), \"Logical GPUs\")\n",
        "    \n",
        "  except RuntimeError as e:\n",
        "    # Memory growth must be set before GPUs have been initialized\n",
        "    print(e)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1 Physical GPUs, 1 Logical GPUs\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_FPKY6BAmwRY"
      },
      "source": [
        "# Acces to Google drive"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qzlekBbQm0cA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "006a2ea9-096c-439f-cecc-446db9e9759e"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "26Exl2ba8RDV",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1ad003fa-1a5a-4432-c5ea-e6488a58dccb"
      },
      "source": [
        "# Data is saved in a zip file\n",
        "\n",
        "zip_path = '/content/drive/MyDrive/Gaetan_Travail/ML/Projets_perso/NLP/Kaggle/Disaster_Tweets/nlp-getting-started.zip'\n",
        "\n",
        "!cp {zip_path} /content/\n",
        "!unzip /content/nlp-getting-started.zip -d /content/\n",
        "!rm /content/nlp-getting-started.zip"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Archive:  /content/nlp-getting-started.zip\n",
            "  inflating: /content/sample_submission.csv  \n",
            "  inflating: /content/test.csv       \n",
            "  inflating: /content/train.csv      \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k3kTWrIJnC7g"
      },
      "source": [
        "# Check the DataFrames"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jeQAzuLkUAiQ",
        "outputId": "8e784302-aec1-49fe-9db9-d8aba553e9d0"
      },
      "source": [
        "df_train = pd.read_csv(\"/content/train.csv\")\n",
        "df_test = pd.read_csv(\"/content/test.csv\")\n",
        "\n",
        "print(f\"df_train shape = {df_train.shape}\\ndf_test shape = {df_test.shape}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "df_train shape = (7613, 5)\n",
            "df_test shape = (3263, 4)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "NRZ0T0Z4UM28",
        "outputId": "56e602fe-6026-45cb-fef0-132d78135969"
      },
      "source": [
        "df_train.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>keyword</th>\n",
              "      <th>location</th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Our Deeds are the Reason of this #earthquake M...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>4</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Forest fire near La Ronge Sask. Canada</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>5</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>All residents asked to 'shelter in place' are ...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>6</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>13,000 people receive #wildfires evacuation or...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>7</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Just got sent this photo from Ruby #Alaska as ...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   id keyword  ...                                               text target\n",
              "0   1     NaN  ...  Our Deeds are the Reason of this #earthquake M...      1\n",
              "1   4     NaN  ...             Forest fire near La Ronge Sask. Canada      1\n",
              "2   5     NaN  ...  All residents asked to 'shelter in place' are ...      1\n",
              "3   6     NaN  ...  13,000 people receive #wildfires evacuation or...      1\n",
              "4   7     NaN  ...  Just got sent this photo from Ruby #Alaska as ...      1\n",
              "\n",
              "[5 rows x 5 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 147
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "C_r3uM6gVeEA",
        "outputId": "842a45ce-e8d3-46da-b660-f6386ff35366"
      },
      "source": [
        "df_train.info()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 7613 entries, 0 to 7612\n",
            "Data columns (total 5 columns):\n",
            " #   Column    Non-Null Count  Dtype \n",
            "---  ------    --------------  ----- \n",
            " 0   id        7613 non-null   int64 \n",
            " 1   keyword   7552 non-null   object\n",
            " 2   location  5080 non-null   object\n",
            " 3   text      7613 non-null   object\n",
            " 4   target    7613 non-null   int64 \n",
            "dtypes: int64(2), object(3)\n",
            "memory usage: 297.5+ KB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZYNxenhsVeGo",
        "outputId": "21d54ff2-3dc6-4731-ef61-b3be14186d18"
      },
      "source": [
        "df_train.isna().sum()/len(df_train)*100"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "id           0.000000\n",
              "keyword      0.801261\n",
              "location    33.272035\n",
              "text         0.000000\n",
              "target       0.000000\n",
              "dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        },
        "id": "wzg6cbbadPd4",
        "outputId": "edc07f20-de6f-4aa4-8b05-afd7d74d6ec2"
      },
      "source": [
        "df_train_clean = df_train[[\"text\", \"target\"]].reset_index(drop=True)\n",
        "print(f\"df_train_clean shape = {df_train_clean.shape}\")\n",
        "df_train_clean.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "df_train_clean shape = (7613, 2)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Our Deeds are the Reason of this #earthquake M...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Forest fire near La Ronge Sask. Canada</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>All residents asked to 'shelter in place' are ...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>13,000 people receive #wildfires evacuation or...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Just got sent this photo from Ruby #Alaska as ...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                text  target\n",
              "0  Our Deeds are the Reason of this #earthquake M...       1\n",
              "1             Forest fire near La Ronge Sask. Canada       1\n",
              "2  All residents asked to 'shelter in place' are ...       1\n",
              "3  13,000 people receive #wildfires evacuation or...       1\n",
              "4  Just got sent this photo from Ruby #Alaska as ...       1"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 79
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fujJ2qaGnlgh"
      },
      "source": [
        "# Define functions for the pipeline"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KdTuRu_aA7R5"
      },
      "source": [
        "# https://www.kaggle.com/alifvianmarco/nlp-disaster-tweets-classification\n",
        "\n",
        "abbreviations = {\n",
        "    \"$\" : \" dollar \",\n",
        "    \"€\" : \" euro \",\n",
        "    \"4ao\" : \"for adults only\",\n",
        "    \"a.m\" : \"before midday\",\n",
        "    \"a3\" : \"anytime anywhere anyplace\",\n",
        "    \"aamof\" : \"as a matter of fact\",\n",
        "    \"acct\" : \"account\",\n",
        "    \"adih\" : \"another day in hell\",\n",
        "    \"afaic\" : \"as far as i am concerned\",\n",
        "    \"afaict\" : \"as far as i can tell\",\n",
        "    \"afaik\" : \"as far as i know\",\n",
        "    \"afair\" : \"as far as i remember\",\n",
        "    \"afk\" : \"away from keyboard\",\n",
        "    \"app\" : \"application\",\n",
        "    \"approx\" : \"approximately\",\n",
        "    \"apps\" : \"applications\",\n",
        "    \"asap\" : \"as soon as possible\",\n",
        "    \"asl\" : \"age, sex, location\",\n",
        "    \"atk\" : \"at the keyboard\",\n",
        "    \"ave.\" : \"avenue\",\n",
        "    \"aymm\" : \"are you my mother\",\n",
        "    \"ayor\" : \"at your own risk\", \n",
        "    \"b&b\" : \"bed and breakfast\",\n",
        "    \"b+b\" : \"bed and breakfast\",\n",
        "    \"b.c\" : \"before christ\",\n",
        "    \"b2b\" : \"business to business\",\n",
        "    \"b2c\" : \"business to customer\",\n",
        "    \"b4\" : \"before\",\n",
        "    \"b4n\" : \"bye for now\",\n",
        "    \"b@u\" : \"back at you\",\n",
        "    \"bae\" : \"before anyone else\",\n",
        "    \"bak\" : \"back at keyboard\",\n",
        "    \"bbbg\" : \"bye bye be good\",\n",
        "    \"bbc\" : \"british broadcasting corporation\",\n",
        "    \"bbias\" : \"be back in a second\",\n",
        "    \"bbl\" : \"be back later\",\n",
        "    \"bbs\" : \"be back soon\",\n",
        "    \"be4\" : \"before\",\n",
        "    \"bfn\" : \"bye for now\",\n",
        "    \"blvd\" : \"boulevard\",\n",
        "    \"bout\" : \"about\",\n",
        "    \"brb\" : \"be right back\",\n",
        "    \"bros\" : \"brothers\",\n",
        "    \"brt\" : \"be right there\",\n",
        "    \"bsaaw\" : \"big smile and a wink\",\n",
        "    \"btw\" : \"by the way\",\n",
        "    \"bwl\" : \"bursting with laughter\",\n",
        "    \"c/o\" : \"care of\",\n",
        "    \"cet\" : \"central european time\",\n",
        "    \"cf\" : \"compare\",\n",
        "    \"cia\" : \"central intelligence agency\",\n",
        "    \"csl\" : \"can not stop laughing\",\n",
        "    \"cu\" : \"see you\",\n",
        "    \"cul8r\" : \"see you later\",\n",
        "    \"cv\" : \"curriculum vitae\",\n",
        "    \"cwot\" : \"complete waste of time\",\n",
        "    \"cya\" : \"see you\",\n",
        "    \"cyt\" : \"see you tomorrow\",\n",
        "    \"dae\" : \"does anyone else\",\n",
        "    \"dbmib\" : \"do not bother me i am busy\",\n",
        "    \"diy\" : \"do it yourself\",\n",
        "    \"dm\" : \"direct message\",\n",
        "    \"dwh\" : \"during work hours\",\n",
        "    \"e123\" : \"easy as one two three\",\n",
        "    \"eet\" : \"eastern european time\",\n",
        "    \"eg\" : \"example\",\n",
        "    \"embm\" : \"early morning business meeting\",\n",
        "    \"encl\" : \"enclosed\",\n",
        "    \"encl.\" : \"enclosed\",\n",
        "    \"etc\" : \"and so on\",\n",
        "    \"faq\" : \"frequently asked questions\",\n",
        "    \"fawc\" : \"for anyone who cares\",\n",
        "    \"fb\" : \"facebook\",\n",
        "    \"fc\" : \"fingers crossed\",\n",
        "    \"fig\" : \"figure\",\n",
        "    \"fimh\" : \"forever in my heart\", \n",
        "    \"ft.\" : \"feet\",\n",
        "    \"ft\" : \"featuring\",\n",
        "    \"ftl\" : \"for the loss\",\n",
        "    \"ftw\" : \"for the win\",\n",
        "    \"fwiw\" : \"for what it is worth\",\n",
        "    \"fyi\" : \"for your information\",\n",
        "    \"g9\" : \"genius\",\n",
        "    \"gahoy\" : \"get a hold of yourself\",\n",
        "    \"gal\" : \"get a life\",\n",
        "    \"gcse\" : \"general certificate of secondary education\",\n",
        "    \"gfn\" : \"gone for now\",\n",
        "    \"gg\" : \"good game\",\n",
        "    \"gl\" : \"good luck\",\n",
        "    \"glhf\" : \"good luck have fun\",\n",
        "    \"gmt\" : \"greenwich mean time\",\n",
        "    \"gmta\" : \"great minds think alike\",\n",
        "    \"gn\" : \"good night\",\n",
        "    \"g.o.a.t\" : \"greatest of all time\",\n",
        "    \"goat\" : \"greatest of all time\",\n",
        "    \"goi\" : \"get over it\",\n",
        "    \"gps\" : \"global positioning system\",\n",
        "    \"gr8\" : \"great\",\n",
        "    \"gratz\" : \"congratulations\",\n",
        "    \"gyal\" : \"girl\",\n",
        "    \"h&c\" : \"hot and cold\",\n",
        "    \"hp\" : \"horsepower\",\n",
        "    \"hr\" : \"hour\",\n",
        "    \"hrh\" : \"his royal highness\",\n",
        "    \"ht\" : \"height\",\n",
        "    \"ibrb\" : \"i will be right back\",\n",
        "    \"ic\" : \"i see\",\n",
        "    \"icq\" : \"i seek you\",\n",
        "    \"icymi\" : \"in case you missed it\",\n",
        "    \"idc\" : \"i do not care\",\n",
        "    \"idgadf\" : \"i do not give a damn fuck\",\n",
        "    \"idgaf\" : \"i do not give a fuck\",\n",
        "    \"idk\" : \"i do not know\",\n",
        "    \"ie\" : \"that is\",\n",
        "    \"i.e\" : \"that is\",\n",
        "    \"ifyp\" : \"i feel your pain\",\n",
        "    \"IG\" : \"instagram\",\n",
        "    \"iirc\" : \"if i remember correctly\",\n",
        "    \"ilu\" : \"i love you\",\n",
        "    \"ily\" : \"i love you\",\n",
        "    \"imho\" : \"in my humble opinion\",\n",
        "    \"imo\" : \"in my opinion\",\n",
        "    \"imu\" : \"i miss you\",\n",
        "    \"iow\" : \"in other words\",\n",
        "    \"irl\" : \"in real life\",\n",
        "    \"j4f\" : \"just for fun\",\n",
        "    \"jic\" : \"just in case\",\n",
        "    \"jk\" : \"just kidding\",\n",
        "    \"jsyk\" : \"just so you know\",\n",
        "    \"l8r\" : \"later\",\n",
        "    \"lb\" : \"pound\",\n",
        "    \"lbs\" : \"pounds\",\n",
        "    \"ldr\" : \"long distance relationship\",\n",
        "    \"lmao\" : \"laugh my ass off\",\n",
        "    \"lmfao\" : \"laugh my fucking ass off\",\n",
        "    \"lol\" : \"laughing out loud\",\n",
        "    \"ltd\" : \"limited\",\n",
        "    \"ltns\" : \"long time no see\",\n",
        "    \"m8\" : \"mate\",\n",
        "    \"mf\" : \"motherfucker\",\n",
        "    \"mfs\" : \"motherfuckers\",\n",
        "    \"mfw\" : \"my face when\",\n",
        "    \"mofo\" : \"motherfucker\",\n",
        "    \"mph\" : \"miles per hour\",\n",
        "    \"mr\" : \"mister\",\n",
        "    \"mrw\" : \"my reaction when\",\n",
        "    \"ms\" : \"miss\",\n",
        "    \"mte\" : \"my thoughts exactly\",\n",
        "    \"nagi\" : \"not a good idea\",\n",
        "    \"nbc\" : \"national broadcasting company\",\n",
        "    \"nbd\" : \"not big deal\",\n",
        "    \"nfs\" : \"not for sale\",\n",
        "    \"ngl\" : \"not going to lie\",\n",
        "    \"nhs\" : \"national health service\",\n",
        "    \"nrn\" : \"no reply necessary\",\n",
        "    \"nsfl\" : \"not safe for life\",\n",
        "    \"nsfw\" : \"not safe for work\",\n",
        "    \"nth\" : \"nice to have\",\n",
        "    \"nvr\" : \"never\",\n",
        "    \"nyc\" : \"new york city\",\n",
        "    \"oc\" : \"original content\",\n",
        "    \"og\" : \"original\",\n",
        "    \"ohp\" : \"overhead projector\",\n",
        "    \"oic\" : \"oh i see\",\n",
        "    \"omdb\" : \"over my dead body\",\n",
        "    \"omg\" : \"oh my god\",\n",
        "    \"omw\" : \"on my way\",\n",
        "    \"p.a\" : \"per annum\",\n",
        "    \"p.m\" : \"after midday\",\n",
        "    \"pm\" : \"prime minister\",\n",
        "    \"poc\" : \"people of color\",\n",
        "    \"pov\" : \"point of view\",\n",
        "    \"pp\" : \"pages\",\n",
        "    \"ppl\" : \"people\",\n",
        "    \"prw\" : \"parents are watching\",\n",
        "    \"ps\" : \"postscript\",\n",
        "    \"pt\" : \"point\",\n",
        "    \"ptb\" : \"please text back\",\n",
        "    \"pto\" : \"please turn over\",\n",
        "    \"qpsa\" : \"what happens\", #\"que pasa\",\n",
        "    \"ratchet\" : \"rude\",\n",
        "    \"rbtl\" : \"read between the lines\",\n",
        "    \"rlrt\" : \"real life retweet\", \n",
        "    \"rofl\" : \"rolling on the floor laughing\",\n",
        "    \"roflol\" : \"rolling on the floor laughing out loud\",\n",
        "    \"rotflmao\" : \"rolling on the floor laughing my ass off\",\n",
        "    \"rt\" : \"retweet\",\n",
        "    \"ruok\" : \"are you ok\",\n",
        "    \"sfw\" : \"safe for work\",\n",
        "    \"sk8\" : \"skate\",\n",
        "    \"smh\" : \"shake my head\",\n",
        "    \"sq\" : \"square\",\n",
        "    \"srsly\" : \"seriously\", \n",
        "    \"ssdd\" : \"same stuff different day\",\n",
        "    \"tbh\" : \"to be honest\",\n",
        "    \"tbs\" : \"tablespooful\",\n",
        "    \"tbsp\" : \"tablespooful\",\n",
        "    \"tfw\" : \"that feeling when\",\n",
        "    \"thks\" : \"thank you\",\n",
        "    \"tho\" : \"though\",\n",
        "    \"thx\" : \"thank you\",\n",
        "    \"tia\" : \"thanks in advance\",\n",
        "    \"til\" : \"today i learned\",\n",
        "    \"tl;dr\" : \"too long i did not read\",\n",
        "    \"tldr\" : \"too long i did not read\",\n",
        "    \"tmb\" : \"tweet me back\",\n",
        "    \"tntl\" : \"trying not to laugh\",\n",
        "    \"ttyl\" : \"talk to you later\",\n",
        "    \"u\" : \"you\",\n",
        "    \"u2\" : \"you too\",\n",
        "    \"u4e\" : \"yours for ever\",\n",
        "    \"utc\" : \"coordinated universal time\",\n",
        "    \"w/\" : \"with\",\n",
        "    \"w/o\" : \"without\",\n",
        "    \"w8\" : \"wait\",\n",
        "    \"wassup\" : \"what is up\",\n",
        "    \"wb\" : \"welcome back\",\n",
        "    \"wtf\" : \"what the fuck\",\n",
        "    \"wtg\" : \"way to go\",\n",
        "    \"wtpa\" : \"where the party at\",\n",
        "    \"wuf\" : \"where are you from\",\n",
        "    \"wuzup\" : \"what is up\",\n",
        "    \"wywh\" : \"wish you were here\",\n",
        "    \"yd\" : \"yard\",\n",
        "    \"ygtr\" : \"you got that right\",\n",
        "    \"ynk\" : \"you never know\",\n",
        "    \"zzz\" : \"sleeping bored and tired\"\n",
        "}\n",
        "\n",
        "def word_abbrev(word):\n",
        "    return abbreviations[word.lower()] if word.lower() in abbreviations.keys() else word\n",
        "\n",
        "# Replace all abbreviations\n",
        "def replace_abbrev(text):\n",
        "    string = \"\"\n",
        "    for word in text.split():\n",
        "        string += word_abbrev(word) + \" \"        \n",
        "    return string"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j8HwV_D0grxw"
      },
      "source": [
        "def keep_text_column(df, train=False):\n",
        "  if train:\n",
        "    return df[[\"text\", \"target\"]].reset_index(drop=True)\n",
        "  else:\n",
        "    return df[[\"text\"]].reset_index(drop=True)\n",
        "\n",
        "def delete_special_char(df_column):\n",
        "  # df_column = df_column.apply(lambda x: re.sub(r'^RT[\\s]+', '', x))\n",
        "  #remove urls\n",
        "  df_column = df_column.apply(lambda x: re.sub(r'https?:\\/\\/.*[\\r\\n]*', '', x))\n",
        "  #remove hashtags\n",
        "  df_column = df_column.apply(lambda x: re.sub(r'#', '', x))\n",
        "  # Remove mentions and characters that not in the English alphabets\n",
        "  df_column = df_column.apply(lambda x: re.sub(r'@\\w+',' ', x))\n",
        "  df_column = df_column.apply(lambda x: re.sub(r'[^A-Za-z0-9 ]+', '', x))\n",
        "  return df_column\n",
        "\n",
        "def spacy_tokenizer(sentence):\n",
        "  # Parser for reviews\n",
        "  parser = English()\n",
        "  mytokens = parser(\" \".join([str(x) for x in nlp(sentence) if x.pos_ != \"NUM\" and x.pos_ != \"-PRON-\"]))\n",
        "  mytokens = [word.lemma_.lower().strip() for word in mytokens]\n",
        "  mytokens = [word for word in mytokens if word not in list(STOP_WORDS) and word not in string.punctuation]\n",
        "  mytokens = \" \".join([i for i in mytokens])\n",
        "\n",
        "  return mytokens\n",
        "\n",
        "def pipeline_for_disater_tweets(df, train=False):\n",
        "  df_clean = keep_text_column(df, train)\n",
        "  df_clean.text = df_clean.text.apply(replace_abbrev)\n",
        "  df_clean.text = delete_special_char(df_clean.text)\n",
        "  df_clean.text = df_clean.text.apply(lambda x: spacy_tokenizer(x))\n",
        "  \n",
        "  return df_clean"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xi1yN7wa7UhF"
      },
      "source": [
        "new_df = pipeline_for_disater_tweets(df_train, train=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        },
        "id": "KZOMkPWdRT2a",
        "outputId": "1a8b18f3-daad-4d78-b9e6-86bd898a5d69"
      },
      "source": [
        "print(f\"new df shape = {new_df.shape}\")\n",
        "new_df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "new df shape = (7613, 2)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>deeds reason earthquake allah forgive</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>forest fire near la ronge sask canada</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>residents asked shelter place notified officer...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>people receive wildfires evacuation orders cal...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>got sent photo ruby alaska smoke wildfires pou...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                text  target\n",
              "0              deeds reason earthquake allah forgive       1\n",
              "1              forest fire near la ronge sask canada       1\n",
              "2  residents asked shelter place notified officer...       1\n",
              "3  people receive wildfires evacuation orders cal...       1\n",
              "4  got sent photo ruby alaska smoke wildfires pou...       1"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 172
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "g7_MPmBYKsmN",
        "outputId": "d0a602e2-2cfe-4c00-b1b6-b2f17699e949"
      },
      "source": [
        "new_df.info()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 7613 entries, 0 to 7612\n",
            "Data columns (total 2 columns):\n",
            " #   Column  Non-Null Count  Dtype \n",
            "---  ------  --------------  ----- \n",
            " 0   text    7613 non-null   object\n",
            " 1   target  7613 non-null   int64 \n",
            "dtypes: int64(1), object(1)\n",
            "memory usage: 119.1+ KB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3KEBBbBtWvkr"
      },
      "source": [
        "# saving this new df\n",
        "new_df.to_csv(\"/content/drive/MyDrive/Gaetan_Travail/ML/Projets_perso/NLP/Kaggle/Disaster_Tweets/df_cleaned_v2.csv\", index=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RY_ofVfprQYq"
      },
      "source": [
        "# Imbalance of the dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        },
        "id": "EDnXY7xUrUHn",
        "outputId": "b3a95be4-93b6-4d95-88a2-67b7651b20b0"
      },
      "source": [
        "df_clean = pd.read_csv(\"/content/drive/MyDrive/Gaetan_Travail/ML/Projets_perso/NLP/Kaggle/Disaster_Tweets/df_cleaned_v2.csv\")\n",
        "df_clean = df_clean[df_clean.text.isnull() == False].reset_index(drop=True)\n",
        "print(f\"new df shape = {df_clean.shape}\")\n",
        "df_clean.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "new df shape = (7551, 2)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>deeds reason earthquake allah forgive</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>forest fire near la ronge sask canada</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>residents asked shelter place notified officer...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>people receive wildfires evacuation orders cal...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>got sent photo ruby alaska smoke wildfires pou...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                text  target\n",
              "0              deeds reason earthquake allah forgive       1\n",
              "1              forest fire near la ronge sask canada       1\n",
              "2  residents asked shelter place notified officer...       1\n",
              "3  people receive wildfires evacuation orders cal...       1\n",
              "4  got sent photo ruby alaska smoke wildfires pou...       1"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 194
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "anBCGhTGrSgV",
        "outputId": "5033d3c4-a763-4acd-ad5e-436fcdce50df"
      },
      "source": [
        "# Estimate class weights based on the imbalance of the data set\n",
        "\n",
        "neg = df_clean.target.value_counts()[0]\n",
        "pos = df_clean.target.value_counts()[1]\n",
        "\n",
        "print(\"neg = {}\\npos = {}\\n\".format(neg, pos))\n",
        "\n",
        "# weigts to correct imbalance:\n",
        "total = neg + pos\n",
        "\n",
        "# Scaling by total/2 helps keep the loss to a similar magnitude.\n",
        "# The sum of the weights of all examples stays the same.\n",
        "weight_for_0 = (1 / neg) * (total / 2.0)\n",
        "weight_for_1 = (1 / pos) * (total / 2.0)\n",
        "\n",
        "class_weights = {0: weight_for_0, 1: weight_for_1}\n",
        "\n",
        "print('Weight for class 0: {:.2f}'.format(weight_for_0))\n",
        "print('Weight for class 1: {:.2f}\\n'.format(weight_for_1))\n",
        "\n",
        "# Initial bias:\n",
        "\n",
        "initial_bias = np.log([pos/neg])\n",
        "print(\"initial_bias: {:.2f}\".format(initial_bias[0]))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "neg = 4309\n",
            "pos = 3242\n",
            "\n",
            "Weight for class 0: 0.88\n",
            "Weight for class 1: 1.16\n",
            "\n",
            "initial_bias: -0.28\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tLCpKtehRUCa"
      },
      "source": [
        "# Cross validation (to obtain train and validation sets)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cfLm_Jagd9iY",
        "outputId": "66e30aef-bd83-4b4e-996f-c52a97da9e9c"
      },
      "source": [
        "kfold = StratifiedKFold(n_splits=CONFIGURATION[\"nbr_folds\"], shuffle=True, random_state=CONFIGURATION[\"seed\"])\n",
        "\n",
        "for n, (train_index, val_index) in enumerate(kfold.split(df_clean, df_clean['target'])):\n",
        "  df_clean.loc[val_index, 'fold'] = int(n)\n",
        "\n",
        "df_clean['fold'] = df_clean['fold'].astype(int)\n",
        "\n",
        "df_clean.groupby(['fold', 'target']).size()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "fold  target\n",
              "0     0         862\n",
              "      1         649\n",
              "1     0         861\n",
              "      1         649\n",
              "2     0         862\n",
              "      1         648\n",
              "3     0         862\n",
              "      1         648\n",
              "4     0         862\n",
              "      1         648\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 196
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "hGhPBz8VfCb5",
        "outputId": "d89c4d8b-b081-4f3a-d86f-72a8cdd49653"
      },
      "source": [
        "df_clean.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "      <th>fold</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>deeds reason earthquake allah forgive</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>forest fire near la ronge sask canada</td>\n",
              "      <td>1</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>residents asked shelter place notified officer...</td>\n",
              "      <td>1</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>people receive wildfires evacuation orders cal...</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>got sent photo ruby alaska smoke wildfires pou...</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                text  target  fold\n",
              "0              deeds reason earthquake allah forgive       1     1\n",
              "1              forest fire near la ronge sask canada       1     4\n",
              "2  residents asked shelter place notified officer...       1     4\n",
              "3  people receive wildfires evacuation orders cal...       1     1\n",
              "4  got sent photo ruby alaska smoke wildfires pou...       1     0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 197
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bi7WBUt2m66K",
        "outputId": "778a1c51-2e32-46b8-ef20-a5f3d09513c7"
      },
      "source": [
        "df_clean.info()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 7551 entries, 0 to 7550\n",
            "Data columns (total 3 columns):\n",
            " #   Column  Non-Null Count  Dtype \n",
            "---  ------  --------------  ----- \n",
            " 0   text    7551 non-null   object\n",
            " 1   target  7551 non-null   int64 \n",
            " 2   fold    7551 non-null   int64 \n",
            "dtypes: int64(2), object(1)\n",
            "memory usage: 177.1+ KB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6kPCDUZ_R39K"
      },
      "source": [
        "# Creating Datasets"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RTNWMwmaR3nq"
      },
      "source": [
        "# Dataloaders for the train set and the validation set\n",
        "\n",
        "def get_dataloaders(train_df, valid_df):\n",
        "    trainloader = tf.data.Dataset.from_tensor_slices((train_df.text.values, train_df.target.values))\n",
        "    validloader = tf.data.Dataset.from_tensor_slices((valid_df.text.values, valid_df.target.values))\n",
        "    \n",
        "    trainloader = (\n",
        "        trainloader\n",
        "        .shuffle(1024)\n",
        "        .batch(CONFIGURATION['batch_size'])\n",
        "        .prefetch(AUTOTUNE)\n",
        "    )\n",
        "\n",
        "    validloader = (\n",
        "        validloader\n",
        "        .batch(CONFIGURATION['batch_size'])\n",
        "        .prefetch(AUTOTUNE)\n",
        "    )\n",
        "    \n",
        "    return trainloader, validloader"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IEef5_VR0a0d"
      },
      "source": [
        "def get_dataloader_test(test_df):\n",
        "    testloader = tf.data.Dataset.from_tensor_slices(test_df.text.values)\n",
        "    \n",
        "    testloader = (\n",
        "        testloader\n",
        "        .batch(CONFIGURATION['batch_size'])\n",
        "        .prefetch(AUTOTUNE)\n",
        "    )\n",
        "    \n",
        "    return testloader"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W7sv3es4HfMM"
      },
      "source": [
        "# Embedding - BERT"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UepFBJreibx5",
        "outputId": "d77cfc90-b6e0-432d-a342-6acf49bd3f03"
      },
      "source": [
        "bert_model_name = 'small_bert/bert_en_uncased_L-4_H-512_A-8'\n",
        "\n",
        "map_name_to_handle = {\n",
        "    'bert_en_uncased_L-12_H-768_A-12':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/3',\n",
        "    'bert_en_cased_L-12_H-768_A-12':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_cased_L-12_H-768_A-12/3',\n",
        "    'bert_multi_cased_L-12_H-768_A-12':\n",
        "        'https://tfhub.dev/tensorflow/bert_multi_cased_L-12_H-768_A-12/3',\n",
        "    'small_bert/bert_en_uncased_L-2_H-128_A-2':\n",
        "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-2_H-128_A-2/1',\n",
        "    'small_bert/bert_en_uncased_L-2_H-256_A-4':\n",
        "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-2_H-256_A-4/1',\n",
        "    'small_bert/bert_en_uncased_L-2_H-512_A-8':\n",
        "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-2_H-512_A-8/1',\n",
        "    'small_bert/bert_en_uncased_L-2_H-768_A-12':\n",
        "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-2_H-768_A-12/1',\n",
        "    'small_bert/bert_en_uncased_L-4_H-128_A-2':\n",
        "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-4_H-128_A-2/1',\n",
        "    'small_bert/bert_en_uncased_L-4_H-256_A-4':\n",
        "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-4_H-256_A-4/1',\n",
        "    'small_bert/bert_en_uncased_L-4_H-512_A-8':\n",
        "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-4_H-512_A-8/1',\n",
        "    'small_bert/bert_en_uncased_L-4_H-768_A-12':\n",
        "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-4_H-768_A-12/1',\n",
        "    'small_bert/bert_en_uncased_L-6_H-128_A-2':\n",
        "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-6_H-128_A-2/1',\n",
        "    'small_bert/bert_en_uncased_L-6_H-256_A-4':\n",
        "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-6_H-256_A-4/1',\n",
        "    'small_bert/bert_en_uncased_L-6_H-512_A-8':\n",
        "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-6_H-512_A-8/1',\n",
        "    'small_bert/bert_en_uncased_L-6_H-768_A-12':\n",
        "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-6_H-768_A-12/1',\n",
        "    'small_bert/bert_en_uncased_L-8_H-128_A-2':\n",
        "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-8_H-128_A-2/1',\n",
        "    'small_bert/bert_en_uncased_L-8_H-256_A-4':\n",
        "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-8_H-256_A-4/1',\n",
        "    'small_bert/bert_en_uncased_L-8_H-512_A-8':\n",
        "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-8_H-512_A-8/1',\n",
        "    'small_bert/bert_en_uncased_L-8_H-768_A-12':\n",
        "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-8_H-768_A-12/1',\n",
        "    'small_bert/bert_en_uncased_L-10_H-128_A-2':\n",
        "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-10_H-128_A-2/1',\n",
        "    'small_bert/bert_en_uncased_L-10_H-256_A-4':\n",
        "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-10_H-256_A-4/1',\n",
        "    'small_bert/bert_en_uncased_L-10_H-512_A-8':\n",
        "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-10_H-512_A-8/1',\n",
        "    'small_bert/bert_en_uncased_L-10_H-768_A-12':\n",
        "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-10_H-768_A-12/1',\n",
        "    'small_bert/bert_en_uncased_L-12_H-128_A-2':\n",
        "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-12_H-128_A-2/1',\n",
        "    'small_bert/bert_en_uncased_L-12_H-256_A-4':\n",
        "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-12_H-256_A-4/1',\n",
        "    'small_bert/bert_en_uncased_L-12_H-512_A-8':\n",
        "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-12_H-512_A-8/1',\n",
        "    'small_bert/bert_en_uncased_L-12_H-768_A-12':\n",
        "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-12_H-768_A-12/1',\n",
        "    'albert_en_base':\n",
        "        'https://tfhub.dev/tensorflow/albert_en_base/2',\n",
        "    'electra_small':\n",
        "        'https://tfhub.dev/google/electra_small/2',\n",
        "    'electra_base':\n",
        "        'https://tfhub.dev/google/electra_base/2',\n",
        "    'experts_pubmed':\n",
        "        'https://tfhub.dev/google/experts/bert/pubmed/2',\n",
        "    'experts_wiki_books':\n",
        "        'https://tfhub.dev/google/experts/bert/wiki_books/2',\n",
        "    'talking-heads_base':\n",
        "        'https://tfhub.dev/tensorflow/talkheads_ggelu_bert_en_base/1',\n",
        "}\n",
        "\n",
        "map_model_to_preprocess = {\n",
        "    'bert_en_uncased_L-12_H-768_A-12':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'bert_en_cased_L-12_H-768_A-12':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_cased_preprocess/3',\n",
        "    'small_bert/bert_en_uncased_L-2_H-128_A-2':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'small_bert/bert_en_uncased_L-2_H-256_A-4':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'small_bert/bert_en_uncased_L-2_H-512_A-8':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'small_bert/bert_en_uncased_L-2_H-768_A-12':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'small_bert/bert_en_uncased_L-4_H-128_A-2':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'small_bert/bert_en_uncased_L-4_H-256_A-4':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'small_bert/bert_en_uncased_L-4_H-512_A-8':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'small_bert/bert_en_uncased_L-4_H-768_A-12':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'small_bert/bert_en_uncased_L-6_H-128_A-2':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'small_bert/bert_en_uncased_L-6_H-256_A-4':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'small_bert/bert_en_uncased_L-6_H-512_A-8':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'small_bert/bert_en_uncased_L-6_H-768_A-12':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'small_bert/bert_en_uncased_L-8_H-128_A-2':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'small_bert/bert_en_uncased_L-8_H-256_A-4':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'small_bert/bert_en_uncased_L-8_H-512_A-8':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'small_bert/bert_en_uncased_L-8_H-768_A-12':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'small_bert/bert_en_uncased_L-10_H-128_A-2':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'small_bert/bert_en_uncased_L-10_H-256_A-4':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'small_bert/bert_en_uncased_L-10_H-512_A-8':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'small_bert/bert_en_uncased_L-10_H-768_A-12':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'small_bert/bert_en_uncased_L-12_H-128_A-2':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'small_bert/bert_en_uncased_L-12_H-256_A-4':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'small_bert/bert_en_uncased_L-12_H-512_A-8':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'small_bert/bert_en_uncased_L-12_H-768_A-12':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'bert_multi_cased_L-12_H-768_A-12':\n",
        "        'https://tfhub.dev/tensorflow/bert_multi_cased_preprocess/3',\n",
        "    'albert_en_base':\n",
        "        'https://tfhub.dev/tensorflow/albert_en_preprocess/3',\n",
        "    'electra_small':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'electra_base':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'experts_pubmed':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'experts_wiki_books':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'talking-heads_base':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "}\n",
        "\n",
        "tfhub_handle_encoder = map_name_to_handle[bert_model_name]\n",
        "tfhub_handle_preprocess = map_model_to_preprocess[bert_model_name]\n",
        "\n",
        "print(f'BERT model selected           : {tfhub_handle_encoder}')\n",
        "print(f'Preprocess model auto-selected: {tfhub_handle_preprocess}')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "BERT model selected           : https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-4_H-512_A-8/1\n",
            "Preprocess model auto-selected: https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yLxzmx4HRhET"
      },
      "source": [
        "# Model optimization with Optuna"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PeAN6HbtGjKr"
      },
      "source": [
        "def objective(trial):\n",
        "  # Clear clutter from previous Keras session graphs.\n",
        "  tf.keras.backend.clear_session()\n",
        "\n",
        "  text_input = tf.keras.layers.Input(shape=(), dtype=tf.string, name='text')\n",
        "  preprocessing_layer = hub.KerasLayer(tfhub_handle_preprocess, name='preprocessing')\n",
        "  encoder_inputs = preprocessing_layer(text_input)\n",
        "  encoder = hub.KerasLayer(tfhub_handle_encoder, trainable=True, name='BERT_encoder')\n",
        "  outputs = encoder(encoder_inputs)\n",
        "\n",
        "  x = outputs['pooled_output']\n",
        "  x = tf.keras.layers.Dropout(\n",
        "      rate = trial.suggest_float('dropout_rate', 0.1, 0.5)\n",
        "  )(x)      \n",
        "\n",
        "  output_ = tf.keras.layers.Dense(\n",
        "      CONFIGURATION['nbr_classes'],\n",
        "      activation=\"sigmoid\",\n",
        "      name=\"optimized_model\",\n",
        "      bias_initializer=tf.keras.initializers.Constant(initial_bias)  \n",
        "  )(x)\n",
        "\n",
        "  model = tf.keras.Model(text_input, output_)\n",
        "\n",
        "  #----------------------#\n",
        "\n",
        "  # Prepare train and valid df\n",
        "  fold = 0\n",
        "  df = df_clean.copy()\n",
        "  train_df = df.loc[df.fold != fold].reset_index(drop=True)\n",
        "  valid_df = df.loc[df.fold == fold].reset_index(drop=True)\n",
        "  trainloader, validloader = get_dataloaders(train_df, valid_df)\n",
        "\n",
        "  #----------------------#\n",
        "\n",
        "  steps_per_epoch = tf.data.experimental.cardinality(trainloader).numpy()\n",
        "  num_train_steps = steps_per_epoch * CONFIGURATION[\"epochs\"]\n",
        "  num_warmup_steps = int(0.1*num_train_steps)\n",
        "\n",
        "  lr = trial.suggest_loguniform('lr', 1e-6, 1e-4)\n",
        "  optimizer_ = optimization.create_optimizer(init_lr=lr,\n",
        "                                            num_train_steps=num_train_steps,\n",
        "                                            num_warmup_steps=num_warmup_steps,\n",
        "                                            optimizer_type='adamw')\n",
        "  \n",
        "  # We compile our model with a sampled learning rate.  \n",
        "  \n",
        "  model.compile(\n",
        "      optimizer = optimizer_,\n",
        "      loss = tf.keras.losses.BinaryCrossentropy(),\n",
        "      metrics = tf.metrics.BinaryAccuracy()\n",
        "  )\n",
        "\n",
        "  #----------------------#\n",
        "\n",
        "  # Model training\n",
        "  model.fit(\n",
        "    trainloader,\n",
        "    epochs=CONFIGURATION[\"epochs\"],\n",
        "    batch_size=CONFIGURATION[\"batch_size\"],\n",
        "    validation_data=validloader,\n",
        "    class_weight=class_weights,\n",
        "    callbacks=[\n",
        "        tf.keras.callbacks.EarlyStopping(patience=3, restore_best_weights=True)\n",
        "    ]\n",
        "  )\n",
        "\n",
        "  #----------------------#\n",
        "\n",
        "  # Evaluate the model accuracy on the validation set.\n",
        "  score = model.evaluate(validloader)\n",
        "  return score[1]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BQ1QyG-EGjNG",
        "outputId": "7715bb19-07ee-4860-f527-58f049e95359"
      },
      "source": [
        "# Create a study object and optimize the objective function.\n",
        "study = optuna.create_study(direction=\"maximize\")\n",
        "study.optimize(objective, n_trials=20, timeout=600)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-08-16 14:26:19,755]\u001b[0m A new study created in memory with name: no-name-9846440d-00cc-496e-a174-a4e731bbc299\u001b[0m\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/20\n",
            "189/189 [==============================] - 47s 213ms/step - loss: 0.5936 - binary_accuracy: 0.6836 - val_loss: 0.4767 - val_binary_accuracy: 0.7942\n",
            "Epoch 2/20\n",
            "189/189 [==============================] - 40s 210ms/step - loss: 0.4706 - binary_accuracy: 0.8018 - val_loss: 0.4629 - val_binary_accuracy: 0.8134\n",
            "Epoch 3/20\n",
            "189/189 [==============================] - 40s 210ms/step - loss: 0.3780 - binary_accuracy: 0.8465 - val_loss: 0.6094 - val_binary_accuracy: 0.7756\n",
            "Epoch 4/20\n",
            "189/189 [==============================] - 40s 210ms/step - loss: 0.2879 - binary_accuracy: 0.8924 - val_loss: 0.7207 - val_binary_accuracy: 0.7670\n",
            "Epoch 5/20\n",
            "189/189 [==============================] - 40s 210ms/step - loss: 0.2135 - binary_accuracy: 0.9250 - val_loss: 0.7573 - val_binary_accuracy: 0.7915\n",
            "48/48 [==============================] - 6s 116ms/step - loss: 0.4629 - binary_accuracy: 0.8134\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-08-16 14:30:06,113]\u001b[0m Trial 0 finished with value: 0.8133686184883118 and parameters: {'dropout_rate': 0.39168283760237466, 'lr': 7.74060918356966e-05}. Best is trial 0 with value: 0.8133686184883118.\u001b[0m\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/20\n",
            "189/189 [==============================] - 47s 213ms/step - loss: 0.7557 - binary_accuracy: 0.5369 - val_loss: 0.5541 - val_binary_accuracy: 0.7247\n",
            "Epoch 2/20\n",
            "189/189 [==============================] - 40s 209ms/step - loss: 0.5537 - binary_accuracy: 0.7293 - val_loss: 0.5290 - val_binary_accuracy: 0.7644\n",
            "Epoch 3/20\n",
            "189/189 [==============================] - 40s 210ms/step - loss: 0.4891 - binary_accuracy: 0.7868 - val_loss: 0.4717 - val_binary_accuracy: 0.7902\n",
            "Epoch 4/20\n",
            "189/189 [==============================] - 40s 210ms/step - loss: 0.4478 - binary_accuracy: 0.8094 - val_loss: 0.4821 - val_binary_accuracy: 0.7929\n",
            "Epoch 5/20\n",
            "189/189 [==============================] - 40s 210ms/step - loss: 0.4076 - binary_accuracy: 0.8320 - val_loss: 0.4938 - val_binary_accuracy: 0.7929\n",
            "Epoch 6/20\n",
            "189/189 [==============================] - 39s 209ms/step - loss: 0.3852 - binary_accuracy: 0.8432 - val_loss: 0.5160 - val_binary_accuracy: 0.7849\n",
            "48/48 [==============================] - 6s 118ms/step - loss: 0.4717 - binary_accuracy: 0.7902\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-08-16 14:34:31,670]\u001b[0m Trial 1 finished with value: 0.790205180644989 and parameters: {'dropout_rate': 0.4963480362076478, 'lr': 1.152995463262945e-05}. Best is trial 0 with value: 0.8133686184883118.\u001b[0m\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/20\n",
            "189/189 [==============================] - 48s 213ms/step - loss: 0.5728 - binary_accuracy: 0.6940 - val_loss: 0.5206 - val_binary_accuracy: 0.7730\n",
            "Epoch 2/20\n",
            "189/189 [==============================] - 40s 210ms/step - loss: 0.4467 - binary_accuracy: 0.8051 - val_loss: 0.4656 - val_binary_accuracy: 0.8008\n",
            "Epoch 3/20\n",
            "189/189 [==============================] - 40s 209ms/step - loss: 0.3726 - binary_accuracy: 0.8457 - val_loss: 0.4628 - val_binary_accuracy: 0.8147\n",
            "Epoch 4/20\n",
            "189/189 [==============================] - 40s 210ms/step - loss: 0.2955 - binary_accuracy: 0.8841 - val_loss: 0.5852 - val_binary_accuracy: 0.7856\n",
            "Epoch 5/20\n",
            "189/189 [==============================] - 40s 210ms/step - loss: 0.2297 - binary_accuracy: 0.9167 - val_loss: 0.6789 - val_binary_accuracy: 0.7935\n",
            "Epoch 6/20\n",
            "189/189 [==============================] - 39s 209ms/step - loss: 0.1708 - binary_accuracy: 0.9391 - val_loss: 0.8812 - val_binary_accuracy: 0.7710\n",
            "48/48 [==============================] - 6s 116ms/step - loss: 0.4628 - binary_accuracy: 0.8147\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2021-08-16 14:39:34,256]\u001b[0m Trial 2 finished with value: 0.8146922588348389 and parameters: {'dropout_rate': 0.13355663396579953, 'lr': 4.779455018530521e-05}. Best is trial 2 with value: 0.8146922588348389.\u001b[0m\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JrKf7ruGEbRD",
        "outputId": "cdc2c40a-5be9-4391-b4cb-8ed007841662"
      },
      "source": [
        "print(f\"Number of finished trials: {len(study.trials)}\")\n",
        "\n",
        "trial = study.best_trial\n",
        "print(\"Best trial:\")  \n",
        "print(f\"\\tValue: {trial.value}\")\n",
        "print(\"\\tParams: \")\n",
        "for key, value in trial.params.items():\n",
        "  print(f\"\\t\\t{key}: {value}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of finished trials: 3\n",
            "Best trial:\n",
            "\tValue: 0.8146922588348389\n",
            "\tParams: \n",
            "\t\tdropout_rate: 0.13355663396579953\n",
            "\t\tlr: 4.779455018530521e-05\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jpIpT9q4w0qe"
      },
      "source": [
        "# Model with the best hyperparameters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4Y5-8R5AQ3Dn"
      },
      "source": [
        "def build_classifier_model(model_name):\n",
        "  text_input = tf.keras.layers.Input(shape=(), dtype=tf.string, name='text')\n",
        "  preprocessing_layer = hub.KerasLayer(tfhub_handle_preprocess, name='preprocessing')\n",
        "  encoder_inputs = preprocessing_layer(text_input)\n",
        "  encoder = hub.KerasLayer(tfhub_handle_encoder, trainable=True, name='BERT_encoder')\n",
        "  outputs = encoder(encoder_inputs)\n",
        "\n",
        "  x = outputs['pooled_output']\n",
        "  x = tf.keras.layers.Dropout(\n",
        "      trial.params[\"dropout_rate\"]\n",
        "  )(x)\n",
        "\n",
        "  output_ = tf.keras.layers.Dense(\n",
        "      CONFIGURATION['nbr_classes'],\n",
        "      activation=\"sigmoid\",\n",
        "      name=model_name,\n",
        "      bias_initializer=tf.keras.initializers.Constant(initial_bias)  \n",
        "  )(x)\n",
        "\n",
        "  return tf.keras.Model(text_input, output_)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "teV2130AQ3GP"
      },
      "source": [
        "def model_cv(model_name, df, fold):\n",
        "  \n",
        "  print(f\"Model trained: {model_name}\")\n",
        "\n",
        "  #----------------------#\n",
        "\n",
        "  print(\"\\n******************\\n\")\n",
        "\n",
        "  print(f\"Fold {fold}:\\n\")\n",
        "\n",
        "  # Prepare train and valid df\n",
        "  train_df = df.loc[df.fold != fold].reset_index(drop=True)\n",
        "  valid_df = df.loc[df.fold == fold].reset_index(drop=True)\n",
        "  trainloader, validloader = get_dataloaders(train_df, valid_df)\n",
        "\n",
        "  #----------------------#\n",
        "\n",
        "  # Creating the model\n",
        "  tf.keras.backend.clear_session()\n",
        "  model = build_classifier_model(model_name)\n",
        "\n",
        "  #----------------------#\n",
        "\n",
        "  steps_per_epoch = tf.data.experimental.cardinality(trainloader).numpy()\n",
        "  num_train_steps = steps_per_epoch * CONFIGURATION[\"epochs\"]\n",
        "  num_warmup_steps = int(0.1 * num_train_steps)\n",
        "\n",
        "  optimizer_ = optimization.create_optimizer(\n",
        "      init_lr=trial.params[\"lr\"],\n",
        "      num_train_steps=num_train_steps,\n",
        "      num_warmup_steps=num_warmup_steps,\n",
        "      optimizer_type='adamw'\n",
        "  )\n",
        "  \n",
        "  # compiling the model\n",
        "  model.compile(\n",
        "      optimizer = optimizer_,\n",
        "      loss = tf.keras.losses.BinaryCrossentropy(),\n",
        "      metrics = tf.metrics.BinaryAccuracy()\n",
        "  ) \n",
        "\n",
        "  #----------------------#\n",
        "        \n",
        "  # Model training\n",
        "  model.fit(\n",
        "    trainloader,\n",
        "    epochs=CONFIGURATION[\"epochs\"],\n",
        "    batch_size=CONFIGURATION[\"batch_size\"],\n",
        "    validation_data=validloader,\n",
        "    class_weight=class_weights,\n",
        "    callbacks=[\n",
        "        tf.keras.callbacks.EarlyStopping(patience=3, restore_best_weights=True)\n",
        "    ]\n",
        "  )\n",
        "\n",
        "  #----------------------#\n",
        "\n",
        "  # Saving the model\n",
        "  # model.save(f'/content/drive/MyDrive/Gaetan_Travail/ML/Projets_perso/NLP/Kaggle/Disaster_Tweets/Models/{model_name}_{fold}.h5')\n",
        "\n",
        "  return model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hQq7eapARNlF",
        "outputId": "af8fbd5c-6e6e-489a-f772-a27e61d1e3f7"
      },
      "source": [
        "list_models = []\n",
        "\n",
        "for fold_ in range(CONFIGURATION[\"nbr_folds\"]):\n",
        "  model = model_cv(model_name=\"model_optimized\", df=df_clean, fold=fold_)\n",
        "  list_models.append(model)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model trained: model_optimized\n",
            "\n",
            "******************\n",
            "\n",
            "Fold 0:\n",
            "\n",
            "Epoch 1/20\n",
            "189/189 [==============================] - 47s 212ms/step - loss: 0.5666 - binary_accuracy: 0.7086 - val_loss: 0.4819 - val_binary_accuracy: 0.7756\n",
            "Epoch 2/20\n",
            "189/189 [==============================] - 40s 209ms/step - loss: 0.4444 - binary_accuracy: 0.8060 - val_loss: 0.4706 - val_binary_accuracy: 0.8028\n",
            "Epoch 3/20\n",
            "189/189 [==============================] - 40s 210ms/step - loss: 0.3700 - binary_accuracy: 0.8508 - val_loss: 0.5539 - val_binary_accuracy: 0.7869\n",
            "Epoch 4/20\n",
            "189/189 [==============================] - 39s 209ms/step - loss: 0.3002 - binary_accuracy: 0.8806 - val_loss: 0.5259 - val_binary_accuracy: 0.8127\n",
            "Epoch 5/20\n",
            "189/189 [==============================] - 40s 210ms/step - loss: 0.2276 - binary_accuracy: 0.9167 - val_loss: 0.7139 - val_binary_accuracy: 0.7942\n",
            "Model trained: model_optimized\n",
            "\n",
            "******************\n",
            "\n",
            "Fold 1:\n",
            "\n",
            "Epoch 1/20\n",
            "189/189 [==============================] - 47s 211ms/step - loss: 0.5793 - binary_accuracy: 0.6908 - val_loss: 0.4276 - val_binary_accuracy: 0.8113\n",
            "Epoch 2/20\n",
            "189/189 [==============================] - 39s 209ms/step - loss: 0.4582 - binary_accuracy: 0.7987 - val_loss: 0.4289 - val_binary_accuracy: 0.8139\n",
            "Epoch 3/20\n",
            "189/189 [==============================] - 39s 208ms/step - loss: 0.3807 - binary_accuracy: 0.8429 - val_loss: 0.4752 - val_binary_accuracy: 0.8185\n",
            "Epoch 4/20\n",
            "189/189 [==============================] - 39s 209ms/step - loss: 0.3091 - binary_accuracy: 0.8795 - val_loss: 0.5564 - val_binary_accuracy: 0.8086\n",
            "Model trained: model_optimized\n",
            "\n",
            "******************\n",
            "\n",
            "Fold 2:\n",
            "\n",
            "Epoch 1/20\n",
            "189/189 [==============================] - 53s 214ms/step - loss: 0.5620 - binary_accuracy: 0.7186 - val_loss: 0.4463 - val_binary_accuracy: 0.8079\n",
            "Epoch 2/20\n",
            "189/189 [==============================] - 40s 211ms/step - loss: 0.4521 - binary_accuracy: 0.8027 - val_loss: 0.4992 - val_binary_accuracy: 0.7662\n",
            "Epoch 3/20\n",
            "189/189 [==============================] - 40s 211ms/step - loss: 0.3695 - binary_accuracy: 0.8475 - val_loss: 0.4361 - val_binary_accuracy: 0.8272\n",
            "Epoch 4/20\n",
            "189/189 [==============================] - 40s 211ms/step - loss: 0.2935 - binary_accuracy: 0.8861 - val_loss: 0.5197 - val_binary_accuracy: 0.8000\n",
            "Epoch 5/20\n",
            "189/189 [==============================] - 40s 210ms/step - loss: 0.2172 - binary_accuracy: 0.9197 - val_loss: 0.6517 - val_binary_accuracy: 0.8119\n",
            "Epoch 6/20\n",
            "189/189 [==============================] - 39s 209ms/step - loss: 0.1661 - binary_accuracy: 0.9401 - val_loss: 0.7321 - val_binary_accuracy: 0.7715\n",
            "Model trained: model_optimized\n",
            "\n",
            "******************\n",
            "\n",
            "Fold 3:\n",
            "\n",
            "Epoch 1/20\n",
            "189/189 [==============================] - 47s 212ms/step - loss: 0.5823 - binary_accuracy: 0.7019 - val_loss: 0.4550 - val_binary_accuracy: 0.7980\n",
            "Epoch 2/20\n",
            "189/189 [==============================] - 40s 211ms/step - loss: 0.4511 - binary_accuracy: 0.8062 - val_loss: 0.4216 - val_binary_accuracy: 0.8166\n",
            "Epoch 3/20\n",
            "189/189 [==============================] - 40s 210ms/step - loss: 0.3733 - binary_accuracy: 0.8485 - val_loss: 0.4709 - val_binary_accuracy: 0.7974\n",
            "Epoch 4/20\n",
            "189/189 [==============================] - 40s 211ms/step - loss: 0.2951 - binary_accuracy: 0.8886 - val_loss: 0.5290 - val_binary_accuracy: 0.8093\n",
            "Epoch 5/20\n",
            "189/189 [==============================] - 40s 212ms/step - loss: 0.2229 - binary_accuracy: 0.9179 - val_loss: 0.7727 - val_binary_accuracy: 0.7795\n",
            "Model trained: model_optimized\n",
            "\n",
            "******************\n",
            "\n",
            "Fold 4:\n",
            "\n",
            "Epoch 1/20\n",
            "189/189 [==============================] - 47s 213ms/step - loss: 0.5742 - binary_accuracy: 0.6802 - val_loss: 0.4633 - val_binary_accuracy: 0.8013\n",
            "Epoch 2/20\n",
            "189/189 [==============================] - 40s 211ms/step - loss: 0.4523 - binary_accuracy: 0.7982 - val_loss: 0.4249 - val_binary_accuracy: 0.8139\n",
            "Epoch 3/20\n",
            "189/189 [==============================] - 40s 211ms/step - loss: 0.3785 - binary_accuracy: 0.8427 - val_loss: 0.4838 - val_binary_accuracy: 0.8106\n",
            "Epoch 4/20\n",
            "189/189 [==============================] - 40s 210ms/step - loss: 0.2964 - binary_accuracy: 0.8802 - val_loss: 0.5042 - val_binary_accuracy: 0.8099\n",
            "Epoch 5/20\n",
            "189/189 [==============================] - 40s 210ms/step - loss: 0.2171 - binary_accuracy: 0.9181 - val_loss: 0.6276 - val_binary_accuracy: 0.7894\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DNBMsoeWFKd0"
      },
      "source": [
        "# Checking F1 score and threshold to use"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XJuIg_JIZ3_F"
      },
      "source": [
        "# The max (for F1 and Jaccard scores) is near to a threshold ~ 0.9\n",
        "def max_scores(target, y_pred, min_range=10, max_range=90):\n",
        "\n",
        "    result_threshold = []\n",
        "    result_f1 = []\n",
        "    result_jaccard = []\n",
        "    result_prec = []\n",
        "    result_recall = []\n",
        "\n",
        "    for i in range(min_range, max_range, 1):\n",
        "      threshold = i/100\n",
        "\n",
        "      y_pred_threshold = [1 if x > threshold else 0 for x in y_pred]\n",
        "                                                           \n",
        "      result_threshold.append(threshold)\n",
        "      result_jaccard.append(jaccard_score(target, y_pred_threshold, average='weighted'))\n",
        "      result_f1.append(f1_score(target, y_pred_threshold, average='weighted'))        \n",
        "      result_prec.append(precision_score(target, y_pred_threshold, average='weighted'))\n",
        "      result_recall.append(recall_score(target, y_pred_threshold, average='weighted'))\n",
        "\n",
        "    # we want to maximize the F1 score:\n",
        "    index = result_f1.index(max(result_f1))\n",
        "    \n",
        "    print(\"Threshold = \", result_threshold[index])\n",
        "    print(\"\\nScores (average == weighted):\\n\")\n",
        "    print(\"Jaccard score    = {:.4f}\".format(result_jaccard[index]))\n",
        "    print(\"F1 score         = {:.4f}\".format(result_f1[index]))\n",
        "    print(\"Precision score  = {:.4f}\".format(result_prec[index]))\n",
        "    print(\"Recall score     = {:.4f}\".format(result_recall[index]))\n",
        "  \n",
        "    return result_threshold[index]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FUMm5l323emD",
        "outputId": "98aed1eb-7a2f-4994-dec7-8c2816608419"
      },
      "source": [
        "list_threshold = []\n",
        "\n",
        "for fold, model in enumerate(list_models):\n",
        "\n",
        "  print(f\"Fold {fold}:\")\n",
        "\n",
        "  train_df = df_clean.loc[df_clean.fold != fold].reset_index(drop=True)\n",
        "  valid_df = df_clean.loc[df_clean.fold == fold].reset_index(drop=True)\n",
        "  trainloader, validloader = get_dataloaders(train_df, valid_df)\n",
        "\n",
        "  valid_df[\"y_pred\"] = model.predict(validloader)\n",
        "\n",
        "  list_threshold.append(max_scores(valid_df.target, valid_df.y_pred, min_range=10, max_range=90))\n",
        "  \n",
        "  print()\n",
        "  results = model.evaluate(validloader, verbose=2)\n",
        "  \n",
        "  print(\"\\n******************\\n\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Fold 0:\n",
            "Threshold =  0.3\n",
            "\n",
            "Scores (average == weighted):\n",
            "\n",
            "Jaccard score    = 0.6790\n",
            "F1 score         = 0.8080\n",
            "Precision score  = 0.8081\n",
            "Recall score     = 0.8087\n",
            "\n",
            "48/48 - 5s - loss: 0.4706 - binary_accuracy: 0.8028\n",
            "\n",
            "******************\n",
            "\n",
            "Fold 1:\n",
            "Threshold =  0.47\n",
            "\n",
            "Scores (average == weighted):\n",
            "\n",
            "Jaccard score    = 0.6831\n",
            "F1 score         = 0.8104\n",
            "Precision score  = 0.8160\n",
            "Recall score     = 0.8132\n",
            "\n",
            "48/48 - 5s - loss: 0.4276 - binary_accuracy: 0.8113\n",
            "\n",
            "******************\n",
            "\n",
            "Fold 2:\n",
            "Threshold =  0.51\n",
            "\n",
            "Scores (average == weighted):\n",
            "\n",
            "Jaccard score    = 0.7077\n",
            "F1 score         = 0.8278\n",
            "Precision score  = 0.8314\n",
            "Recall score     = 0.8298\n",
            "\n",
            "48/48 - 5s - loss: 0.4361 - binary_accuracy: 0.8272\n",
            "\n",
            "******************\n",
            "\n",
            "Fold 3:\n",
            "Threshold =  0.42\n",
            "\n",
            "Scores (average == weighted):\n",
            "\n",
            "Jaccard score    = 0.6966\n",
            "F1 score         = 0.8204\n",
            "Precision score  = 0.8207\n",
            "Recall score     = 0.8212\n",
            "\n",
            "48/48 - 5s - loss: 0.4216 - binary_accuracy: 0.8166\n",
            "\n",
            "******************\n",
            "\n",
            "Fold 4:\n",
            "Threshold =  0.32\n",
            "\n",
            "Scores (average == weighted):\n",
            "\n",
            "Jaccard score    = 0.6993\n",
            "F1 score         = 0.8222\n",
            "Precision score  = 0.8228\n",
            "Recall score     = 0.8232\n",
            "\n",
            "48/48 - 5s - loss: 0.4249 - binary_accuracy: 0.8139\n",
            "\n",
            "******************\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hOsSHbbzxCVG"
      },
      "source": [
        "The best model (i.e. higher F1-score) used the fold 2 for the validation set, and the others for the train set.\n",
        "\n",
        "It achieved a F1-score of about 83% on the validation set."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QiGW_Z2GxnSo"
      },
      "source": [
        "# Test set forecasting\n",
        "\n",
        "Predict the test set for a Kaggle submission."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EvXsqTLm-AMV"
      },
      "source": [
        "new_df_test = pipeline_for_disater_tweets(df_test, train=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c-5jre9Y5BDJ"
      },
      "source": [
        "# Best model -> Fold 2\n",
        "fold_ = 2\n",
        "\n",
        "testloader = get_dataloader_test(new_df_test)\n",
        "\n",
        "y_pred_test = list_models[fold_].predict(testloader)\n",
        "\n",
        "submit = pd.DataFrame({\n",
        "    'id': df_test.id,\n",
        "    'target': [1 if x > list_threshold[fold_] else 0 for x in y_pred_test]\n",
        "})\n",
        "\n",
        "name_ = f\"submit_v2_fold_{fold_}\"\n",
        "submit.to_csv(f\"/content/drive/MyDrive/Gaetan_Travail/ML/Projets_perso/NLP/Kaggle/Disaster_Tweets/{name_}.csv\", index=False)"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}