l06c03_exercise_flowers_with_transfer_learning_solution.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "l06c03_exercise_flowers_with_transfer_learning_solution.ipynb",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "W_tvPdyfA-BL"
      },
      "source": [
        "##### Copyright 2019 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "colab_type": "code",
        "id": "0O_LFhwSBCjm",
        "colab": {}
      },
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "9-3Pry4jh1-E"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/examples/blob/master/courses/udacity_intro_to_tensorflow_for_deep_learning/l06c03_exercise_flowers_with_transfer_learning_solution.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/examples/blob/master/courses/udacity_intro_to_tensorflow_for_deep_learning/l06c03_exercise_flowers_with_transfer_learning_solution.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "NxjpzKTvg_dd"
      },
      "source": [
        "# TensorFlow Hub"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "crU-iluJIEzw"
      },
      "source": [
        "[TensorFlow Hub](http://tensorflow.org/hub) is an online repository of already trained TensorFlow models that you can use.\n",
        "These models can either be used as is, or they can be used for Transfer Learning.\n",
        "\n",
        "Transfer learning is a process where you take an existing trained model, and extend it to do additional work. This involves leaving the bulk of the model unchanged, while adding and retraining the final layers, in order to get a different set of possible outputs.\n",
        "\n",
        "Here, you can see all the models available in [TensorFlow Module Hub](https://tfhub.dev/).\n",
        "\n",
        "Before starting this Colab, you should reset the Colab environment by selecting `Runtime -> Reset all runtimes...` from menu above."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "7RVsYZLEpEWs"
      },
      "source": [
        "# Imports\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ZUCEcRdhnyWn"
      },
      "source": [
        "Some normal imports we've seen before. The new one is importing tensorflow_hub which this Colab will make heavy use of."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "RxwCQNZWIL8y",
        "colab": {}
      },
      "source": [
        "import tensorflow as tf"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "ivDUkUNdINH2",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "import tensorflow_hub as hub\n",
        "import tensorflow_datasets as tfds\n",
        "\n",
        "from tensorflow.keras import layers"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "NU3IAZ02G6VA",
        "colab": {}
      },
      "source": [
        "import logging\n",
        "logger = tf.get_logger()\n",
        "logger.setLevel(logging.ERROR)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "amfzqn1Oo7Om"
      },
      "source": [
        "# TODO: Download the Flowers Dataset using TensorFlow Datasets"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Z93vvAdGxDMD"
      },
      "source": [
        "In the cell below you will download the Flowers dataset using TensorFlow Datasets. If you look at the [TensorFlow Datasets documentation](https://www.tensorflow.org/datasets/datasets#tf_flowers) you will see that the name of the Flowers dataset is `tf_flowers`. You can also see that this dataset is only split into a TRAINING set. You will therefore have to use `tfds.splits` to split this training set into to a `training_set` and a `validation_set`. Do a `[70, 30]` split such that 70 corresponds to the `training_set` and 30 to the `validation_set`. Then load the `tf_flowers` dataset using `tfds.load`. Make sure the `tfds.load` function uses the all the parameters you need, and also make sure it returns the dataset info, so we can retrieve information about the datasets.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "oXiJjX0jfx1o",
        "colab": {}
      },
      "source": [
        "(training_set, validation_set), dataset_info = tfds.load(\n",
        "    'tf_flowers',\n",
        "    split=['train[:70%]', 'train[70%:]'],\n",
        "    with_info=True,\n",
        "    as_supervised=True,\n",
        ")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "X0p1sOEHf0JF"
      },
      "source": [
        "# TODO: Print Information about the Flowers Dataset\n",
        "\n",
        "Now that you have downloaded the dataset, use the dataset info to print the number of classes in the dataset, and also write some code that counts how many images we have in the training and validation sets. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "DrIUV3V0xDL_",
        "colab": {}
      },
      "source": [
        "num_classes = dataset_info.features['label'].num_classes\n",
        "\n",
        "num_training_examples = 0\n",
        "num_validation_examples = 0\n",
        "\n",
        "for example in training_set:\n",
        "  num_training_examples += 1\n",
        "\n",
        "for example in validation_set:\n",
        "  num_validation_examples += 1\n",
        "\n",
        "print('Total Number of Classes: {}'.format(num_classes))\n",
        "print('Total Number of Training Images: {}'.format(num_training_examples))\n",
        "print('Total Number of Validation Images: {} \\n'.format(num_validation_examples))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "UlFZ_hwjCLgS"
      },
      "source": [
        "The images in the Flowers dataset are not all the same size."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "W4lDPkn2cpWZ",
        "colab": {}
      },
      "source": [
        "for i, example in enumerate(training_set.take(5)):\n",
        "  print('Image {} shape: {} label: {}'.format(i+1, example[0].shape, example[1]))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "mbgpD3E6gM2P"
      },
      "source": [
        "# TODO: Reformat Images and Create Batches\n",
        "\n",
        "In the cell below create a function that reformats all images to the resolution expected by MobileNet v2 (224, 224) and normalizes them. The function should take in an `image` and a `label` as arguments and should return the new `image` and corresponding `label`. Then create training and validation batches of size `32`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "we_ftzQxNf7e",
        "colab": {}
      },
      "source": [
        "IMAGE_RES = 224\n",
        "\n",
        "def format_image(image, label):\n",
        "  image = tf.image.resize(image, (IMAGE_RES, IMAGE_RES))/255.0\n",
        "  return image, label\n",
        "\n",
        "BATCH_SIZE = 32\n",
        "\n",
        "train_batches = training_set.shuffle(num_training_examples//4).map(format_image).batch(BATCH_SIZE).prefetch(1)\n",
        "\n",
        "validation_batches = validation_set.map(format_image).batch(BATCH_SIZE).prefetch(1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "JzV457OXreQP"
      },
      "source": [
        "# Do Simple Transfer Learning with TensorFlow Hub\n",
        "\n",
        "Let's now use TensorFlow Hub to do Transfer Learning. Remember, in transfer learning we reuse parts of an already trained model and change the final layer, or several layers, of the model, and then retrain those layers on our own dataset.\n",
        "\n",
        "### TODO: Create a Feature Extractor\n",
        "In the cell below create a `feature_extractor` using MobileNet v2. Remember that the partial model from TensorFlow Hub (without the final classification layer) is called a feature vector. Go to the [TensorFlow Hub documentation](https://tfhub.dev/s?module-type=image-feature-vector&q=tf2) to see a list of available feature vectors. Click on the `tf2-preview/mobilenet_v2/feature_vector`. Read the documentation and get the corresponding `URL` to get the MobileNet v2 feature vector. Finally, create a `feature_extractor` by using `hub.KerasLayer` with the correct `input_shape` parameter."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "5wB030nezBwI",
        "colab": {}
      },
      "source": [
        "URL = \"https://tfhub.dev/google/tf2-preview/mobilenet_v2/feature_vector/4\"\n",
        "feature_extractor = hub.KerasLayer(URL,\n",
        "                                   input_shape=(IMAGE_RES, IMAGE_RES, 3))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "CtFmF7A5E4tk"
      },
      "source": [
        "### TODO: Freeze the Pre-Trained Model\n",
        "\n",
        "In the cell below freeze the variables in the feature extractor layer, so that the training only modifies the final classifier layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Jg5ar6rcE4H-",
        "colab": {}
      },
      "source": [
        "feature_extractor.trainable = False"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "RPVeouTksO9q"
      },
      "source": [
        "### TODO: Attach a classification head\n",
        "\n",
        "In the cell below create a `tf.keras.Sequential` model, and add the pre-trained model and the new classification layer. Remember that the classification layer must have the same number of classes as our Flowers dataset. Finally print a summary of the Sequential model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "mGcY27fY1q3Q",
        "colab": {}
      },
      "source": [
        "model = tf.keras.Sequential([\n",
        "  feature_extractor,\n",
        "  layers.Dense(num_classes)\n",
        "])\n",
        "\n",
        "model.summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "OHbXQqIquFxQ"
      },
      "source": [
        "### TODO: Train the model\n",
        "\n",
        "In the cell bellow train this model like any other, by first calling `compile` and then followed by `fit`. Make sure you use the proper parameters when applying both methods. Train the model for only 6 epochs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "3n0Wb9ylKd8R",
        "colab": {}
      },
      "source": [
        "model.compile(\n",
        "  optimizer='adam',\n",
        "  loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
        "  metrics=['accuracy'])\n",
        "\n",
        "EPOCHS = 6\n",
        "\n",
        "history = model.fit(train_batches,\n",
        "                    epochs=EPOCHS,\n",
        "                    validation_data=validation_batches)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "76as-K8-vFQJ"
      },
      "source": [
        "You can see we get ~88% validation accuracy with only 6 epochs of training, which is absolutely awesome. This is a huge improvement over the model we created in the previous lesson, where we were able to get ~76% accuracy with 80 epochs of training. The reason for this difference is that MobileNet v2 was carefully designed over a long time by experts, then trained on a massive dataset (ImageNet)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "SLxTcprUqJaq"
      },
      "source": [
        "# TODO: Plot Training and Validation Graphs\n",
        "\n",
        "In the cell below, plot the training and validation accuracy/loss graphs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "d28dhbFpr98b",
        "colab": {}
      },
      "source": [
        "acc = history.history['accuracy']\n",
        "val_acc = history.history['val_accuracy']\n",
        "\n",
        "loss = history.history['loss']\n",
        "val_loss = history.history['val_loss']\n",
        "\n",
        "epochs_range = range(EPOCHS)\n",
        "\n",
        "plt.figure(figsize=(8, 8))\n",
        "plt.subplot(1, 2, 1)\n",
        "plt.plot(epochs_range, acc, label='Training Accuracy')\n",
        "plt.plot(epochs_range, val_acc, label='Validation Accuracy')\n",
        "plt.legend(loc='lower right')\n",
        "plt.title('Training and Validation Accuracy')\n",
        "\n",
        "plt.subplot(1, 2, 2)\n",
        "plt.plot(epochs_range, loss, label='Training Loss')\n",
        "plt.plot(epochs_range, val_loss, label='Validation Loss')\n",
        "plt.legend(loc='upper right')\n",
        "plt.title('Training and Validation Loss')\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "5zmoDisGvNye"
      },
      "source": [
        "What is a bit curious here is that validation performance is better than training performance, right from the start to the end of execution.\n",
        "\n",
        "One reason for this is that validation performance is measured at the end of the epoch, but training performance is the average values across the epoch.\n",
        "\n",
        "The bigger reason though is that we're reusing a large part of MobileNet which is already trained on Flower images. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "kb__ZN8uFn-D"
      },
      "source": [
        "# TODO: Check Predictions\n",
        "\n",
        "In the cell below get the label names from the dataset info and convert them into a NumPy array. Print the array to make sure you have the correct label names."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "W_Zvg2i0fzJu",
        "colab": {}
      },
      "source": [
        "class_names = np.array(dataset_info.features['label'].names)\n",
        "\n",
        "print(class_names)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "4Olg6MsNGJTL"
      },
      "source": [
        "### TODO: Create an Image Batch and Make Predictions\n",
        "\n",
        "In the cell below, use the `next()` function to create an `image_batch` and its corresponding `label_batch`. Convert both the `image_batch` and `label_batch` to numpy arrays using the `.numpy()` method. Then use the `.predict()` method to run the image batch through your model and make predictions. Then use the `np.argmax()` function to get the indices of the best prediction for each image. Finally convert the indices of the best predictions to class names."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "fCLVCpEjJ_VP",
        "colab": {}
      },
      "source": [
        "image_batch, label_batch = next(iter(train_batches))\n",
        "\n",
        "\n",
        "image_batch = image_batch.numpy()\n",
        "label_batch = label_batch.numpy()\n",
        "\n",
        "predicted_batch = model.predict(image_batch)\n",
        "predicted_batch = tf.squeeze(predicted_batch).numpy()\n",
        "\n",
        "predicted_ids = np.argmax(predicted_batch, axis=-1)\n",
        "predicted_class_names = class_names[predicted_ids]\n",
        "\n",
        "print(predicted_class_names)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "CkGbZxl9GZs-"
      },
      "source": [
        "### TODO: Print True Labels and Predicted Indices\n",
        "\n",
        "In the cell below, print the true labels and the indices of predicted labels."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "nL9IhOmGI5dJ",
        "colab": {}
      },
      "source": [
        "print(\"Labels:           \", label_batch)\n",
        "print(\"Predicted labels: \", predicted_ids)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "gJDyzEfYuFcW"
      },
      "source": [
        "# Plot Model Predictions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "wC_AYRJU9NQe",
        "colab": {}
      },
      "source": [
        "plt.figure(figsize=(10,9))\n",
        "for n in range(30):\n",
        "  plt.subplot(6,5,n+1)\n",
        "  plt.subplots_adjust(hspace = 0.3)\n",
        "  plt.imshow(image_batch[n])\n",
        "  color = \"blue\" if predicted_ids[n] == label_batch[n] else \"red\"\n",
        "  plt.title(predicted_class_names[n].title(), color=color)\n",
        "  plt.axis('off')\n",
        "_ = plt.suptitle(\"Model predictions (blue: correct, red: incorrect)\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "7QBKxS5CuKhc"
      },
      "source": [
        "# TODO: Perform Transfer Learning with the Inception Model\n",
        "\n",
        "Go to the [TensorFlow Hub documentation](https://tfhub.dev/s?module-type=image-feature-vector&q=tf2) and click on `tf2-preview/inception_v3/feature_vector`. This feature vector corresponds to the Inception v3 model. In the cells below, use transfer learning to create a CNN that uses Inception v3 as the pretrained model to classify the images from the Flowers dataset. Note that Inception, takes as input, images that are 299 x 299 pixels. Compare the accuracy you get with Inception v3 to the accuracy you got with MobileNet v2."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "wVII2H9ZNNQf",
        "colab": {}
      },
      "source": [
        "IMAGE_RES = 299\n",
        "\n",
        "(training_set, validation_set), dataset_info = tfds.load(\n",
        "    'tf_flowers', \n",
        "    with_info=True, \n",
        "    as_supervised=True, \n",
        "    split=['train[:70%]', 'train[70%:]'],\n",
        ")\n",
        "train_batches = training_set.shuffle(num_training_examples//4).map(format_image).batch(BATCH_SIZE).prefetch(1)\n",
        "validation_batches = validation_set.map(format_image).batch(BATCH_SIZE).prefetch(1)\n",
        "\n",
        "URL = \"https://tfhub.dev/google/tf2-preview/inception_v3/feature_vector/4\"\n",
        "feature_extractor = hub.KerasLayer(URL,\n",
        "  input_shape=(IMAGE_RES, IMAGE_RES, 3),\n",
        "  trainable=False)\n",
        "\n",
        "model_inception = tf.keras.Sequential([\n",
        "  feature_extractor,\n",
        "  tf.keras.layers.Dense(num_classes)\n",
        "])\n",
        "\n",
        "model_inception.summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "idcaQKWAPgL0",
        "colab": {}
      },
      "source": [
        "model_inception.compile(\n",
        "  optimizer='adam', \n",
        "  loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
        "  metrics=['accuracy'])\n",
        "\n",
        "EPOCHS = 6\n",
        "\n",
        "history = model_inception.fit(train_batches,\n",
        "                    epochs=EPOCHS,\n",
        "                    validation_data=validation_batches)"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}