text_sentiment_BERT_classifier.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "text_sentiment_BERT_classifier.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "n4a4pNXUEatn"
      ],
      "machine_shape": "hm"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.9"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r67y9UpchZ38"
      },
      "source": [
        "# Transfer learning with pre-trained BERT for text sentiment classification\n",
        "\n",
        "In this notebook, we will use the Twitter dataset for sentiment classificatoin using pre-trained BERT with a binary classifier on top of it. Given a sentence, we are going to classify whether this sentence has negative meaning. Negative meaning will have label == 0, otherwise will have label == 1."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FdYR4CpYBkX8"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bx1oSm8NBpTc"
      },
      "source": [
        "# Dependencies of the preprocessing for BERT inputs and optimizing BERT\n",
        "!pip install -q tensorflow-text\n",
        "!pip install -q tf-models-official\n",
        "!pip install transformers"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zxMj1LNnAsDf"
      },
      "source": [
        "import os\n",
        "import datetime\n",
        "\n",
        "from pathlib import Path\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "import tensorflow_text as text\n",
        "from official.nlp import optimization\n",
        "\n",
        "from transformers import pipeline\n",
        "\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "AUTOTUNE = tf.data.experimental.AUTOTUNE\n",
        "\n",
        "tf.get_logger().setLevel('ERROR')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SkOXr7KDsBJn"
      },
      "source": [
        "path_prefix = Path.cwd()\n",
        "print(path_prefix)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ByS_MtY53U6F"
      },
      "source": [
        "data_path = path_prefix.joinpath('data/')\n",
        "data_path.mkdir(exist_ok = True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9YrAlczfM_w6"
      },
      "source": [
        "## Download Dataset\n",
        "[Dataset](https://www.kaggle.com/c/ml2020spring-hw4)\n",
        "\n",
        "There are three .txt files -- training_label.txt、training_nolabel.txt、testing_data.txt\n",
        "\n",
        "- training_label.txt: training data with labels（0 or 1)\n",
        "    - +++$+++ is separating symbols\n",
        "    - e.g., 1 +++$+++ are wtf ... awww thanks !\n",
        "\n",
        "- training_nolabel.txt：training data without labels\n",
        "    - We will use this training data for semi-supervised learning\n",
        "    - ex: hates being this burnt !! ouch\n",
        "\n",
        "- testing_data.txt： Predict with testing data \n",
        "\n",
        "    >id,text\n",
        "\n",
        "    >0,my dog ate our dinner . no , seriously ... he ate it .\n",
        "\n",
        "    >1,omg last day sooon n of primary noooooo x im gona be swimming out of school wif the amount of tears am gona cry\n",
        "\n",
        "    >2,stupid boys .. they ' re so .. stupid !"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HSyhKF2iwrcl"
      },
      "source": [
        "### Download dataset if not have any"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x2gwKORmuViJ"
      },
      "source": [
        "%cd $data_path\n",
        "\n",
        "if not os.path.exists('training_label.txt') or\\\n",
        "    not os.path.exists('training_nolabel.txt') or\\\n",
        "    not os.path.exists('testing_data.txt'):\n",
        "    print(\"Dataset is incompleted . Downloading\")\n",
        "    # Method1\n",
        "    !wget --no-check-certificate 'https://drive.google.com/uc?export=download&id=1dPHIl8ZnfDz_fxNd2ZeBYedTat2lfxcO' -O 'training_label.txt'\n",
        "    !wget --no-check-certificate 'https://drive.google.com/uc?export=download&id=1x1rJOX_ETqnOZjdMAbEE2pqIjRNa8xcc' -O 'training_nolabel.txt'\n",
        "    !wget --no-check-certificate 'https://drive.google.com/uc?export=download&id=16CtnQwSDCob9xmm6EdHHR7PNFNiOrQ30' -O 'testing_data.txt'\n",
        "\n",
        "    # Method2\n",
        "    # !gdown --id '1lz0Wtwxsh5YCPdqQ3E3l_nbfJT1N13V8' --output data.zip\n",
        "    # !unzip data.zip\n",
        "    # !ls\n",
        "else:\n",
        "    print(\"data is all set\")\n",
        "   \n",
        "%cd $path_prefix"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8hDIokoP6464"
      },
      "source": [
        "# this is for filtering the warnings\n",
        "import warnings\n",
        "warnings.filterwarnings('ignore')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tPvWPi4G3U6L"
      },
      "source": [
        "## Preprocess dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wvPmzWoY3U6M"
      },
      "source": [
        "### Setup paths and configurations"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EztIWqCmlZof"
      },
      "source": [
        "# Preset the paths to dataset\n",
        "train_with_label = os.path.join(path_prefix, 'data/training_label.txt')\n",
        "train_no_label = os.path.join(path_prefix, 'data/training_nolabel.txt')\n",
        "testing_data = os.path.join(path_prefix, 'data/testing_data.txt')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8X2R4_jQ3U6N"
      },
      "source": [
        "### Read dataset from folder"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n4a4pNXUEatn"
      },
      "source": [
        "#### Helping functions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "byD7fqKCEOwj"
      },
      "source": [
        "def load_data(path, separator = '+++$+++'):\n",
        "    \"\"\"\n",
        "    Reading dataset.\n",
        "    \n",
        "    Data format:\n",
        "        With label:\n",
        "            label, separator, text\n",
        "        Without label:\n",
        "            text\n",
        "        Testing:\n",
        "            id, text\n",
        "        \n",
        "    Inputs:\n",
        "    - path: str. The path of the dataset\n",
        "    - separator: str. The string separating label and text\n",
        "\n",
        "    Outputs:\n",
        "    - x: List of str.\n",
        "    - y: List of int.\n",
        "    \"\"\"\n",
        "\n",
        "    if 'nolabel' in path:\n",
        "        return load_non_labelled_data(path, separator)\n",
        "    elif 'test' in path:\n",
        "        return load_testing_data(path, separator)\n",
        "    \n",
        "    return load_labelled_data(path, separator)\n",
        "\n",
        "\n",
        "def load_labelled_data(path, separator):\n",
        "    \"\"\"\n",
        "    Reading dataset with label.\n",
        "\n",
        "    Data format:\n",
        "        label, separator, text\n",
        "        \n",
        "    Inputs:\n",
        "    - path: str. The path of the dataset\n",
        "    - separator: str. The string separating label and text\n",
        "\n",
        "    Outputs:\n",
        "    - x: List of str.\n",
        "    - y: List of int\n",
        "    \"\"\"\n",
        "\n",
        "    with open(path, 'r') as f:\n",
        "        lines = f.readlines()\n",
        "        lines = [line.strip('\\n') for line in lines]\n",
        "\n",
        "    y = [int(line[0]) for line in lines]\n",
        "    # Skip the separator\n",
        "    x = [line[line.find(separator) + len(separator) + 1 : ] for line in lines]\n",
        "\n",
        "    return x, y\n",
        "\n",
        "\n",
        "def load_non_labelled_data(path, separator):\n",
        "    \"\"\"\n",
        "    Reading dataset without label.\n",
        "\n",
        "    Data format:\n",
        "        text\n",
        "        \n",
        "    Inputs:\n",
        "    - path: str. The path of the dataset\n",
        "    - separator: str. The string separating label and text\n",
        "\n",
        "    Outputs:\n",
        "    - x: List of str.\n",
        "    \"\"\"\n",
        "    with open(path, 'r') as f:\n",
        "        lines = f.readlines()\n",
        "        x = [line.strip('\\n') for line in lines]\n",
        "\n",
        "    return x\n",
        "\n",
        "\n",
        "def load_testing_data(path, separator):\n",
        "    \"\"\"\n",
        "    Reading testing set.\n",
        "    \n",
        "    Data format:\n",
        "        title\n",
        "        id1,text1\n",
        "        id2,text2\n",
        "            .\n",
        "            .\n",
        "\n",
        "    Inputs:\n",
        "    - path: str. The path of the dataset\n",
        "\n",
        "    Outputs:\n",
        "    - X: List of str.\n",
        "    \"\"\"\n",
        "    with open(path, 'r') as f:\n",
        "        lines = f.readlines()\n",
        "        X = [line.strip('\\n').split(separator, maxsplit = 1)[1] for line in lines[1:]]\n",
        "\n",
        "    return X"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FCZF4CGrGDdq"
      },
      "source": [
        "#### Load dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ep2cyTH43U6N"
      },
      "source": [
        "# Read 'training_label.txt' and 'training_nolabel.txt'\n",
        "print(\"loading data ...\")\n",
        "X_train_label, y_train_label = load_data(train_with_label)\n",
        "X_train, X_val, y_train, y_val = train_test_split(X_train_label, \n",
        "                                                  y_train_label, \n",
        "                                                  test_size = 0.1)\n",
        "\n",
        "X_train_no_label = load_data(train_no_label)\n",
        "\n",
        "X_test = load_data(testing_data, ',')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1_NyN4Zy3U6O"
      },
      "source": [
        "print(f\"Total number of the training data with label: {len(X_train)}\")\n",
        "print(f\"Total number of the training data without label: {len(X_train_no_label)}\")\n",
        "print(f\"Total number of the validation data: {len(X_val)}\")\n",
        "print(f\"Total number of the validation data: {len(X_test)}\")\n",
        "\n",
        "print(f\"Positive rate in training dataset: {np.sum(y_train) / len(y_train)}\")\n",
        "print(f\"Positive rate in validation dataset: {np.sum(y_val) / len(y_val)}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gd8rKGiX3U6R"
      },
      "source": [
        "### Preprocess training and validation datasets"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ldarFO2G3U6R"
      },
      "source": [
        "# Configuration\n",
        "seed = 42\n",
        "batch_size = 32\n",
        "\n",
        "with tf.device('/cpu:0'):\n",
        "    # Training dataset\n",
        "    train_dataset = tf.data.Dataset.from_tensor_slices((X_train, y_train))\n",
        "\n",
        "    train_dataset = train_dataset.shuffle(len(X_train))\n",
        "    train_dataset = train_dataset.batch(batch_size)\n",
        "    train_dataset = train_dataset.cache().prefetch(AUTOTUNE)\n",
        "\n",
        "    # Validation dataset\n",
        "    val_dataset = tf.data.Dataset.from_tensor_slices((X_val, y_val))\n",
        "\n",
        "    val_dataset = val_dataset.shuffle(len(X_val))\n",
        "    val_dataset = val_dataset.batch(batch_size)\n",
        "    val_dataset = val_dataset.cache().prefetch(AUTOTUNE)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fw0a84FU3U6S"
      },
      "source": [
        "for x_batch, y_batch in train_dataset.take(1):\n",
        "    print(f\"x_batch shape: {x_batch.shape}\")\n",
        "    print(f\"y_batch shape: {y_batch.shape}\")\n",
        "    print(f\"{x_batch[0]}: {y_batch[0]}\")  "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vETpFcdJ3U6T"
      },
      "source": [
        "## Loading BERT from TensorFlow Hub\n",
        "\n",
        "Here we are going to use small BERT or original BERT from Tensorflow Hub. More versions can be found in [here](https://tfhub.dev/google/collections/bert/1)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s_fpcD4tK3xl"
      },
      "source": [
        "# Choose a BERT model to fine-tune\n",
        "\n",
        "bert_model_name = 'expert/bert_wiki_books_sst2'\n",
        "\n",
        "map_name_to_handle = {\n",
        "    'bert_en_uncased_L-12_H-768_A-12':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/3',\n",
        "    'small_bert/bert_en_uncased_L-2_H-128_A-2':\n",
        "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-2_H-128_A-2/1',\n",
        "    'expert/bert_wiki_books_sst2':\n",
        "        'https://tfhub.dev/google/experts/bert/wiki_books/sst2/2'\n",
        "}\n",
        "\n",
        "map_model_to_preprocess = {\n",
        "    'bert_en_uncased_L-12_H-768_A-12':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'small_bert/bert_en_uncased_L-2_H-128_A-2':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "    'expert/bert_wiki_books_sst2':\n",
        "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
        "}\n",
        "\n",
        "tfhub_handle_encoder = map_name_to_handle[bert_model_name]\n",
        "tfhub_handle_preprocess = map_model_to_preprocess[bert_model_name]\n",
        "\n",
        "print(f'BERT model selected           : {tfhub_handle_encoder}')\n",
        "print(f'Preprocess model auto-selected: {tfhub_handle_preprocess}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gwiM-plbMA6R"
      },
      "source": [
        "### The preprocessing model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZlC4LEuuMAj-"
      },
      "source": [
        "bert_preprocess_model = hub.KerasLayer(tfhub_handle_preprocess)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KyVMkoeMNJ0H"
      },
      "source": [
        "for x_batch, y_batch in train_dataset.take(1):\n",
        "    text = x_batch[0].numpy().decode('utf8')\n",
        "    print(text)\n",
        "    text_test = [text]\n",
        "    text_preprocessed = bert_preprocess_model(text_test)\n",
        "\n",
        "    print(f'Keys       : {list(text_preprocessed.keys())}')\n",
        "    print(f'Shape      : {text_preprocessed[\"input_word_ids\"].shape}')\n",
        "    print(f'Word Ids   : {text_preprocessed[\"input_word_ids\"][0, :]}')\n",
        "    print(f'Input Mask : {text_preprocessed[\"input_mask\"][0, :]}')\n",
        "    print(f'Type Ids   : {text_preprocessed[\"input_type_ids\"][0, :]}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oZ8JcVywPFnM"
      },
      "source": [
        "### The BERT model\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "boz2nhHVPO2g"
      },
      "source": [
        "## Build the sentiment classifier with pre-trained BERT"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6cczXlNsPmAr"
      },
      "source": [
        "def buildClassifierModel():\n",
        "    text_input = tf.keras.layers.Input(shape=(), dtype=tf.string, name='text')\n",
        "    preprocessing_layer = hub.KerasLayer(tfhub_handle_preprocess, name='preprocessing')\n",
        "    encoder_inputs = preprocessing_layer(text_input)\n",
        "    encoder = hub.KerasLayer(tfhub_handle_encoder, trainable=False, name='BERT_encoder')\n",
        "    outputs = encoder(encoder_inputs)\n",
        "    net = outputs['pooled_output']\n",
        "    net = tf.keras.layers.Dropout(0.1)(net)\n",
        "    net = tf.keras.layers.Dense(128, activation='gelu', name='dense')(net)\n",
        "    net = tf.keras.layers.Dropout(0.1)(net)\n",
        "    net = tf.keras.layers.Dense(1, activation=None, name='classifier')(net)\n",
        "    return tf.keras.Model(text_input, net)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NdgjUVpHP79w"
      },
      "source": [
        "Plain check of the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UOEaCqAwPmmp"
      },
      "source": [
        "model = buildClassifierModel()\n",
        "bert_raw_result = model(tf.constant(text_test))\n",
        "print(tf.sigmoid(bert_raw_result))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u9C8hIY63U6V"
      },
      "source": [
        "## Train the model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LyobkwXgRXIA"
      },
      "source": [
        "### Loss function"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FPlqjF9mRb1q"
      },
      "source": [
        "loss_fn = tf.keras.losses.BinaryCrossentropy(from_logits=True)\n",
        "metrics = tf.metrics.BinaryAccuracy()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "beke4iD1Rcca"
      },
      "source": [
        "### Optimizer\n",
        "\n",
        "For fine-tuning, we use the same optimizer that BERT was originally trained with: the \"Adaptive Moments\" (Adam). This optimizer minimizes the prediction loss and does regularization by weight decay (not using moments), which is also known as AdamW.\n",
        "\n",
        "For the learning rate (init_lr), we use the same schedule as BERT pre-training: linear decay of a notional initial learning rate, prefixed with a linear warm-up phase over the first 10% of training steps (num_warmup_steps). In line with the BERT paper, the initial learning rate is smaller for fine-tuning (best of 5e-5, 3e-5, 2e-5)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I-0RceyoRrOd"
      },
      "source": [
        "epochs = 5\n",
        "steps_per_epoch = tf.data.experimental.cardinality(train_dataset).numpy()\n",
        "num_train_steps = steps_per_epoch * epochs\n",
        "num_warmup_steps = int(0.1 * num_train_steps)\n",
        "\n",
        "init_lr = 3e-5\n",
        "optimizer = optimization.create_optimizer(init_lr=init_lr,\n",
        "                                          num_train_steps=num_train_steps,\n",
        "                                          num_warmup_steps=num_warmup_steps,\n",
        "                                          optimizer_type='adamw')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZITRgEo9SBhl"
      },
      "source": [
        "### Training"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "g4-x4HZVSEDO"
      },
      "source": [
        "model.compile(optimizer=optimizer,\n",
        "              loss=loss_fn,\n",
        "              metrics=metrics)\n",
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2-qB6rqB3U6V"
      },
      "source": [
        "# Checkpoint\n",
        "checkpoint_filepath = os.path.join(path_prefix, 'first_train_ckpt/')\n",
        "\n",
        "!rm -rf checkpoint_filepath\n",
        "\n",
        "model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_filepath,\n",
        "                                                               save_weights_only=True,\n",
        "                                                               save_best_only=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "Ppz1vW5UZ7Gd"
      },
      "source": [
        "#@title\n",
        "# Tiny training dataset - for debugging\n",
        "# tiny_train_dataset = tf.data.Dataset.from_tensor_slices((X_train[:1000], y_train[:1000]))\n",
        "\n",
        "# tiny_train_dataset = tiny_train_dataset.batch(batch_size)\n",
        "# tiny_train_dataset = tiny_train_dataset.cache().prefetch(AUTOTUNE)\n",
        "\n",
        "# # Tiny validation dataset\n",
        "# tiny_val_dataset = tf.data.Dataset.from_tensor_slices((X_val[:1000], y_val[:1000]))\n",
        "\n",
        "# tiny_val_dataset = tiny_val_dataset.batch(batch_size)\n",
        "# tiny_val_dataset = tiny_val_dataset.cache().prefetch(AUTOTUNE)\n",
        "\n",
        "\n",
        "# print(f'Training model with {tfhub_handle_encoder}')\n",
        "# history = model.fit(tiny_train_dataset, \n",
        "#                     validation_data = tiny_val_dataset, \n",
        "#                     epochs = epochs, \n",
        "#                     callbacks=[model_checkpoint_callback])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eXQn733F3U6W"
      },
      "source": [
        "print(f'Training model with {tfhub_handle_encoder}')\n",
        "history = model.fit(train_dataset, \n",
        "                    validation_data=val_dataset, \n",
        "                    epochs = epochs, \n",
        "                    callbacks=[model_checkpoint_callback])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZwJajcKtTiUB"
      },
      "source": [
        "### Evaluate"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xrTtAwDTQOVt"
      },
      "source": [
        "best_model = buildClassifierModel()\n",
        "best_model.load_weights(checkpoint_filepath)\n",
        "best_model.compile(optimizer=optimizer,\n",
        "                    loss=loss_fn,\n",
        "                    metrics=metrics)\n",
        "\n",
        "loss, accuracy = best_model.evaluate(val_dataset)\n",
        "\n",
        "print(f'Loss: {loss}')\n",
        "print(f'Accuracy: {accuracy}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z7CBwER5TreR"
      },
      "source": [
        "### See the training history"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zfcv-HifVKAP"
      },
      "source": [
        "history_dict = history.history\n",
        "print(history_dict.keys())\n",
        "\n",
        "acc = history_dict['binary_accuracy']\n",
        "val_acc = history_dict['val_binary_accuracy']\n",
        "loss = history_dict['loss']\n",
        "val_loss = history_dict['val_loss']\n",
        "\n",
        "epochs_range = range(1, len(acc) + 1)\n",
        "fig = plt.figure(figsize=(10, 6))\n",
        "fig.tight_layout()\n",
        "\n",
        "plt.subplot(2, 1, 1)\n",
        "# \"bo\" is for \"blue dot\"\n",
        "plt.plot(epochs_range, loss, 'r', label='Training loss')\n",
        "# b is for \"solid blue line\"\n",
        "plt.plot(epochs_range, val_loss, 'b', label='Validation loss')\n",
        "plt.title('Training and validation loss')\n",
        "# plt.xlabel('Epochs')\n",
        "plt.ylabel('Loss')\n",
        "plt.legend()\n",
        "\n",
        "plt.subplot(2, 1, 2)\n",
        "plt.plot(epochs_range, acc, 'r', label='Training acc')\n",
        "plt.plot(epochs_range, val_acc, 'b', label='Validation acc')\n",
        "plt.title('Training and validation accuracy')\n",
        "plt.xlabel('Epochs')\n",
        "plt.ylabel('Accuracy')\n",
        "plt.legend(loc='lower right')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OdDXZ3Kf3U6c"
      },
      "source": [
        "## Semi-supervised Learning \n",
        "We can further train the model using the training data without label"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m_jPFadH3U6c"
      },
      "source": [
        "### Semi-supervised Learning\n",
        "Here we use simple self-learning strategy to hard label the data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iMmaiVgcIhOH"
      },
      "source": [
        "# Checkpoint\n",
        "checkpoint_filepath_final = os.path.join(path_prefix, 'final_train_ckpt/')\n",
        "\n",
        "!rm -rf checkpoint_filepath_final\n",
        "\n",
        "model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_filepath_final,\n",
        "                                                               save_weights_only=True,\n",
        "                                                               save_best_only=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rFkQdvwr3U6c"
      },
      "source": [
        "threshold = 0.8\n",
        "num_samples = 50000"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NZVSN3e1P0lI"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OxzaNkMgFWZz"
      },
      "source": [
        "used = [False] * len(X_train_no_label)\n",
        "\n",
        "for _ in range(epochs):\n",
        "    new_X = []\n",
        "    new_y = []\n",
        "    rand_idx = np.arange(len(X_train_no_label))\n",
        "    np.random.shuffle(rand_idx)\n",
        "\n",
        "    for idx in rand_idx:\n",
        "        if used[idx]:\n",
        "            continue\n",
        "\n",
        "        pred_prob = tf.sigmoid(best_model.predict(X_train_no_label[idx : idx + 1]))\n",
        "\n",
        "        if pred_prob > threshold or pred_prob < (1 - threshold):\n",
        "            label = 1 if pred_prob > 0.5 else 0\n",
        "            new_X.append(X_train_no_label[idx])\n",
        "            new_y.append(label)\n",
        "            used[idx] = True\n",
        "\n",
        "        if len(new_X) >= num_samples:\n",
        "            break\n",
        "\n",
        "    with tf.device('/cpu:0'):\n",
        "        semi_dataset = tf.data.Dataset.from_tensor_slices((new_X, new_y))\n",
        "        semi_dataset = semi_dataset.batch(batch_size)\n",
        "        semi_dataset = semi_dataset.cache().prefetch(AUTOTUNE)\n",
        "\n",
        "        # Augment dataset\n",
        "        train_dataset = train_dataset.concatenate(semi_dataset)\n",
        "\n",
        "        num_total_batch = len( list(train_dataset.as_numpy_iterator()) )\n",
        "\n",
        "        train_dataset = train_dataset.shuffle(num_total_batch)\n",
        "        train_dataset = train_dataset.cache().prefetch(AUTOTUNE)\n",
        "\n",
        "    # Train with the new augmented dataset\n",
        "    history = best_model.fit(train_dataset, \n",
        "                                validation_data = val_dataset, \n",
        "                                epochs = 1, \n",
        "                                callbacks=[model_checkpoint_callback])\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "grA8WM-l3U6g"
      },
      "source": [
        "best_model = buildClassifierModel()\n",
        "best_model.load_weights(checkpoint_filepath_final)\n",
        "best_model.compile(optimizer=optimizer,\n",
        "                    loss=loss_fn,\n",
        "                    metrics=metrics)\n",
        "\n",
        "loss, accuracy = best_model.evaluate(val_dataset)\n",
        "\n",
        "print(f'Loss: {loss}')\n",
        "print(f'Accuracy: {accuracy}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TGJ6MOrk3U6Y"
      },
      "source": [
        "## Testing"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8fQeaQNeNm3L"
      },
      "source": [
        "### Preprocess test dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xthR037S3U6Z"
      },
      "source": [
        "with tf.device('/cpu:0'):\n",
        "    test_dataset = tf.data.Dataset.from_tensor_slices(X_test)\n",
        "    test_dataset = test_dataset.batch(batch_size)\n",
        "    test_dataset = test_dataset.cache().prefetch(AUTOTUNE)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dQhS6j5c3U6a"
      },
      "source": [
        "### Load the best model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y2HSoakr3U6a"
      },
      "source": [
        "print('\\nload model ...')\n",
        "best_model = buildClassifierModel()\n",
        "best_model.load_weights(checkpoint_filepath_final)\n",
        "best_model.compile(optimizer=optimizer,\n",
        "                    loss=loss_fn,\n",
        "                    metrics=metrics)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "52JUpkOc3U6h"
      },
      "source": [
        "### Make prediction"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MQnXkBX_5J73"
      },
      "source": [
        "def testing(model, dataset):\n",
        "    outputs = model.predict(dataset)\n",
        "\n",
        "    outputs_prob = tf.math.sigmoid(outputs.reshape(-1))\n",
        "\n",
        "    res = np.array([1 if prob > 0.5 else 0 for prob in outputs_prob])\n",
        "\n",
        "    return res"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z7CUCdRv3U6h"
      },
      "source": [
        "outputs = testing(model, test_dataset)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uxv3QF6J3U6i"
      },
      "source": [
        "# Write the result to a CSV file\n",
        "tmp = pd.DataFrame({\"id\":[str(i) for i in range(len(X_test))],\"label\":outputs})\n",
        "print(\"save csv ...\")\n",
        "tmp.to_csv(os.path.join(path_prefix, 'predict.csv'), index=False)\n",
        "print(\"Finish Predicting\")"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}