notebooks-labs/Session3_FineTuning_BERTandGPT.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/iPoetDev/ibm-skills-ai-colab-sessions/blob/main/notebooks-labs/Session3_FineTuning_BERTandGPT.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Wyyh4uo1sxrX"
   },
   "source": [
    "# <ins>Session 3</ins>.2: **IBM Skills Build: Generative AI Live Technical Lab** (Part 2)\n",
    "\n",
    "> #### **Objective**: *Understand the theory and hands-on implementation of*: <br>  1️⃣ Transformers: BERT (HuggingFace)\n",
    ">> - Exploring pre-trained language models like *BERT or GPT* and\n",
    ">> - Fine-tuning them for specific NLP tasks\n",
    "\n",
    "- **URL**: [https://skills.yourlearning.ibm.com/activity/PLAN-CB1CC0D21AFB](https://skills.yourlearning.ibm.com/activity/PLAN-CB1CC0D21AFB \"Programme for Artifical Intelligence: eLearning on IBM.com (Login required)\")<small><sup><strong>eLearning, Login</strong></sup></small><br>\n",
    "- **Share**: [Introduction to Generative AI](https://skills.yourlearning.ibm.com/activity/MDL-388 \"eLearning on IBM.com (Login required\") <small><sup><strong>eLearning, Login</strong></sup></small>\n",
    "- **Recording**: [Recording: Live Technical Session 3](https://skills.yourlearning.ibm.com/activity/URL-6BF19B3CC379 \"Video: IBM's Box.com (Login required\")\n",
    "- **CoLab: Source Notebook**: [https://colab.research.google.com/drive/1FW5-OGD2jegulfkF8afRptkZ3cEakL--?usp=sharing](https://colab.research.google.com/drive/1FW5-OGD2jegulfkF8afRptkZ3cEakL--?usp=sharing \"Authors: Marty Bradly's Session 3 Tune Transformers\")\n",
    "  - Original by author: Marty Bradly: [LinkedIn](https://www.linkedin.com/in/martybradley/), [Website](https://www.evergreen-ai.com/), [GitHub @marty916](https://github.com/marty916 \"Marty Bradly [July, 2024], Last accessed: July 2024\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EDL8AlgfzVIa"
   },
   "source": [
    "## Steps\n",
    "\n",
    "1.   [Setup / Install](#scrollTo=bnhqc5UADi9O&line=1&uniqifier=1)\n",
    "2.   [Load Target Dataset](#scrollTo=Zl9x1VIbgs6x&line=1&uniqifier=1)\n",
    "3.   [Load BERT Pretrained Model](#scrollTo=j0tWoINggxXx&line=1&uniqifier=1)\n",
    "4.   [Training Arguments](#scrollTo=kQEYGBAXg2lv&line=4&uniqifier=1)\n",
    "5.   [Trainer Definition](#scrollTo=2HFVqLsqg84g&line=3&uniqifier=1)\n",
    "6.   [Model Training](#scrollTo=K93oznlqg_0v&line=1&uniqifier=1)\n",
    "7.   [Model Evaluation](#scrollTo=yYL0MDlQhagQ&line=1&uniqifier=1)\n",
    "8.   [Run Predictions](#scrollTo=zT7Ev3gRhkYK&line=3&uniqifier=1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "K6qhc4oCxVcm"
   },
   "source": [
    "---\n",
    "> <hr>\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bnhqc5UADi9O"
   },
   "source": [
    "## 1. <ins>Setup / Install</ins>\n",
    "\n",
    "- Hugging Face\n",
    "    - HuggingFace's Accelerate\n",
    "    - HuggingFace's Transformers\n",
    "    - HuggingFace's Datasets\n",
    "- PyTorch\n",
    "    - Torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1u38QPj8qSA0"
   },
   "outputs": [],
   "source": [
    "# Install, by upgrade, to latest version.\n",
    "! pip install -U accelerate\n",
    "! pip install -U transformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "JanfVEI3nbk3"
   },
   "outputs": [],
   "source": [
    "# Import: HuggingFace's Acclerate\n",
    "import accelerate\n",
    "# Version Check\n",
    "print(\"Using accelerate\", accelerate.__version__)\n",
    "# Import: HuggingFace's Acclerate\n",
    "import transformers\n",
    "# Version Check\n",
    "print(\"Using transformers\", transformers.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iJbXj4hF0wF2"
   },
   "source": [
    "> <hr>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "PTGHvOO0RxMk"
   },
   "outputs": [],
   "source": [
    "# Install, shell command, quietly\n",
    "!pip install -q datasets torch\n",
    "# Import\n",
    "import torch\n",
    "# Version Check\n",
    "print(\"Using torch\", torch.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "fAWQL0-LlJQ6"
   },
   "outputs": [],
   "source": [
    "# Import BERT modules and transformers trainers\n",
    "from transformers import BertTokenizer,\n",
    "                        BertForSequenceClassification,\n",
    "                        Trainer,\n",
    "                        TrainingArguments\n",
    "# Import Datasets\n",
    "from datasets import load_dataset,\n",
    "                     load_metric\n",
    "# Version Check\n",
    "print(\"Using datasets\", datasets.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PMm8ibbtswxe"
   },
   "source": [
    "> <hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Zl9x1VIbgs6x"
   },
   "source": [
    "## 2. <ins>Load the IMDB dataset</ins>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8TS622moWFfD"
   },
   "outputs": [],
   "source": [
    "# Load the dataset\n",
    "dataset = load_dataset(\"imdb\")\n",
    "tokenizer = BertTokenizer.\\\n",
    "                from_pretrained(\"bert-base-uncased\")\n",
    "\n",
    "# To convert them into numerical tokens, padding to length, truncating\n",
    "def tokenize_function(examples):\n",
    "    return tokenizer(examples[\"text\"],\n",
    "                     padding=\"max_length\",\n",
    "                     truncation=True)\n",
    "\n",
    "# Tokenise the Dataset\n",
    "tokenized_datasets = dataset.map(tokenize_function,\n",
    "                                 batched=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CE9HGkX701hw"
   },
   "source": [
    "> <hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "j0tWoINggxXx"
   },
   "source": [
    "## 3. <ins>Load BERT pretrained model</ins>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "MkQtil0_WxnV"
   },
   "outputs": [],
   "source": [
    "# Load pre-trained BERT model\n",
    "model = BertForSequenceClassification.\\\n",
    "            from_pretrained(\"bert-base-uncased\",\n",
    "                            num_labels=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eI749dxy0zgl"
   },
   "source": [
    "> <hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kQEYGBAXg2lv"
   },
   "source": [
    "## 4. <ins>Define the training arguments</ins>\n",
    "\n",
    "> Defines the hyperparameters for training a model using the Hugging Face Transformers library.<br>\n",
    "> - These settings control the training process,\n",
    "> - Affecting the model's performance and convergence.\n",
    "\n",
    "1. Constants for cleaner code.\n",
    "2. Assign Training Args to control the training:\n",
    "    - Tuning performance and convergence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cCk_-JaR1zhc"
   },
   "outputs": [],
   "source": [
    "# Define constants for training arguments\n",
    "OUTPUT_DIR = \"./results\"\n",
    "EVAL_STRATEGY = \"epoch\"         # Evaluate after every epoch/pass\n",
    "LEARNING_RATE = 2e-5            # Learning rate's step sizing: weights updated.\n",
    "TRAIN_BATCH_SIZE = 8    \t    # Forward/Backward pass training examples\n",
    "EVAL_BATCH_SIZE = 8             # Forward/Backward pass evaluation examples\n",
    "NUM_TRAIN_EPOCHS = 3            # Number of epochs to train\n",
    "WEIGHT_DECAY = 0.01             # Regularization: prevents (penalty) overfitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "3IEMt0CGXDOm"
   },
   "outputs": [],
   "source": [
    "# Training arguments\n",
    "training_args = TrainingArguments(\n",
    "    output_dir=OUTPUT_DIR,\n",
    "    eval_strategy=EVAL_STRATEGY,\n",
    "    learning_rate=LEARNING_RATE,\n",
    "    per_device_train_batch_size=TRAIN_BATCH_SIZE,\n",
    "    per_device_eval_batch_size=EVAL_BATCH_SIZE,\n",
    "    num_train_epochs=NUM_TRAIN_EPOCHS,\n",
    "    weight_decay=WEIGHT_DECAY,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jmU9thEz3Kzm"
   },
   "source": [
    "> <hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2HFVqLsqg84g"
   },
   "source": [
    "## 5. <ins>Define the trainer</ins>\n",
    "\n",
    "> This code initializes a Trainer object from the Hugging Face Transformers library, which is used to streamline the training and evaluation of your BERT model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "AYwVEaAVXNpQ"
   },
   "outputs": [],
   "source": [
    "# Define Trainer\n",
    "trainer = Trainer(\n",
    "    model=model,                    # the pre-trained BERT model for fine tuning\n",
    "    args=training_args,             # args, inc hyperparameters.\n",
    "    train_dataset=\\\n",
    "        tokenized_datasets[\"train\"],    # tokenised training dataset\n",
    "    eval_dataset=\\\n",
    "        tokenized_datasets[\"test\"],     # tokenised test/eval dataset\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "flis1EoB3NUP"
   },
   "source": [
    "> <hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "K93oznlqg_0v"
   },
   "source": [
    "## 6. <ins>Train the model</ins>\n",
    "A100 - 36 min\n",
    "other - 5h+\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8bbAaT1vXQCc"
   },
   "outputs": [],
   "source": [
    "# Train the model\n",
    "trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rjYPCH1o7GGv"
   },
   "source": [
    "> <hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yYL0MDlQhagQ"
   },
   "source": [
    "## 7. <ins>Evaluate the model</ins>\n",
    "A100 - 4-5min"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "fDfwon_HXT-A"
   },
   "outputs": [],
   "source": [
    "# Evaluate the model\n",
    "results = trainer.evaluate()\n",
    "print(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EuLAqo4v7Lrn"
   },
   "source": [
    "> <hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zT7Ev3gRhkYK"
   },
   "source": [
    "## 8. <ins>Make predictions</ins>\n",
    "\n",
    "- Constants for cleaner code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "tfhfDz6981ya"
   },
   "outputs": [],
   "source": [
    "TENSOR_FORMAT = \"pt\"    # PyTorch tensors\n",
    "SPECIAL_DEVICE = \"cuda\" # GPU\n",
    "DEFAULT_DEVICE = \"cpu\"  # CPU\n",
    "STORE_DIMENSIONS = -1   # the last dimension, typically holds the class score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9dVHK5nD-m5M"
   },
   "source": [
    "\n",
    "\n",
    "1.  Supply Inputs\n",
    "2.  Tokenises the Inputs according to Tensor formats\n",
    "3.  Move inputs / device to model\n",
    "4.  Move the model for the available device\n",
    "5.  Move input tensors to the same device\n",
    "6.  Model predicitions\n",
    "7.  Extract the predicted class label\n",
    "8.  Display the predicition\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Gj95Hg06XWqG",
    "outputId": "b25d1f6d-b0a0-48df-9558-2232e8b0f9b3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicted label: 0\n"
     ]
    }
   ],
   "source": [
    "# Make predictions\n",
    "text = \"This is a great movie!\"         # the input text to classify\n",
    "# text = \"This is a terrible movie!\"    # the alt text to classify\n",
    "\n",
    "inputs = tokenizer(                         # tokenizes: same BERT tokenizer\n",
    "                    text,                   # Inputs\n",
    "                    return_tensors=TENSOR_FORMAT)\n",
    "                         # Output in PyTorch tenors format\n",
    "\n",
    "# Move the inputs to the same device as the model\n",
    "device = torch.device(\n",
    "                    SPECIAL_DEVICE if torch.cuda.is_available()\n",
    "                            else DEFAULT_DEVICE)\n",
    "# Move the model for the available device\n",
    "model.to(device)\n",
    "# Move input tensors to the same device as the model\n",
    "inputs = {key: value.to(device) for key, value in inputs.items()}\n",
    "\n",
    "# Get predictions from the model\n",
    "outputs = model(**inputs)\n",
    "\n",
    "# Extract the predicted class label (index with highest score)\n",
    "predictions = torch.argmax(outputs.logits,\n",
    "                           dim=STORE_DIMENSIONS)\n",
    "# Display the predicition.\n",
    "print(f\"Predicted label: {predictions.item()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "v7UIN97GAqCT"
   },
   "source": [
    "---\n",
    "> <center> ~ # ~ </center>\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7kUZerNvi2-4"
   },
   "source": [
    "## Author\n",
    "\n",
    "[![LinkedIn](https://img.shields.io/badge/Author-Charles%20J%20Fowler-0077B5?logo=gmail&logoColor=white)](mailto:ipoetdev-github-no-reply@outlook.com \"Contact CJ on GItHub email: ipoetdev-github-no-reply@outlook.com\") <sup>|</sup> [![LinkedIn](https://img.shields.io/badge/Charles%20J%20Fowler-LinkedIn-0077B5?logo=linkedin&logoColor=white)](https://ie.linkedin.com/in/charlesjfowler \"@CharlesJFowler @Linkedin.com\") <sup>|</sup> [![LinkedIn](https://img.shields.io/badge/iPoetDev-GitHub-0077B5?logo=GitHub&logoColor=white)](https://github.com/ipoetdev \"@iPoetDev @GitHub\")\n",
    "\n",
    "## ChangeLog\n",
    "\n",
    "| Date<sup>1</sup> | Version | Changed By | Change | Activity | From |\n",
    "| :--- | :--- | :--- | :--- | :--- | :--- |\n",
    "| 2024-07-16  | 0.1 | Charles J Fowler  | Source uploaded | Uploaded  | [Source Notebook]( https://colab.research.google.com/drive/1FW5-OGD2jegulfkF8afRptkZ3cEakL--?usp=sharing \"Author: Marty Bradly\") |\n",
    "| 2024-07-26  | 0.2 | Charles J Fowler  | Draft Portfolio version | Modify  | --- |  \n",
    "<sup>1</sup>: `YYYY-MM-DD"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "A100",
   "include_colab_link": true,
   "machine_shape": "hm",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}