pa5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PA5: ConvNets\n",
    "\n",
    "In this project, we will be visualizing and manipulating AlexNet [1]:\n",
    "<img src=\"figures/alexnet.png\"/>\n",
    "\n",
    "For this project, we are using <a href=\"http://caffe.berkeleyvision.org/\">Caffe</a>, an open-source deep learning library that has an efficient implementation of AlexNet.  Other similar libraries include <a href=\"http://torch.ch/\">Torch</a>, <a href=\"http://deeplearning.net/software/theano/\">Theano</a>, and <a href=\"https://www.tensorflow.org/\">TensorFlow</a>.  While the original AlexNet implementation by Alex Krizhevsky is <a href=\"https://code.google.com/p/cuda-convnet/\">cuda-convnet</a>, caffe has a nicer API.\n",
    "\n",
    "Some parts of this assignment were adapted/inspired from a <a href=\"http://cs231n.stanford.edu/\">Stanford cs231</a> assignment.  The parts that are similar have been modified heavily and ported to caffe.\n",
    "\n",
    "[1] <a href=\"https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf\">Krizhevsky et al, \"ImageNet Classification with Deep Convolutional Neural Networks\", NIPS 2012</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 0. Getting Started\n",
    "\n",
    "## Installing Caffe\n",
    "\n",
    "To get you started, we have an installation script:\n",
    "```bash\n",
    "./install-caffe.sh\n",
    "```\n",
    "which will clone Caffe to `~/caffe`, build it, test it, and install the Python interface (`pycaffe`).  After running the script, you will need to restart your iPython Notebook Kernel (menu: Kernel --> Restart).  While this notebook should be self-sufficient, you can find more information on the <a href=\"http://tutorial.caffe.berkeleyvision.org/\">Caffe tutorial page</a> or the <a href=\"http://caffe.berkeleyvision.org/\">Caffe home page</a>.\n",
    "\n",
    "**Note: It will take about 30-60 minutes to install caffe, depending on your machine.**\n",
    "\n",
    "## Download / load AlexNet\n",
    "\n",
    "The below cell downloads AlexNet: both the structure of the network (a `.prototxt` file) and the weights for each layer (a `.caffemodel` file).  The original caffe network is defined in the directory `~/caffe/models/bvlc_alexnet/`.  The below cell will patch the original file (`deploy.prototxt`) to allow you to backprop all the way to the input image, saving it to `patched_alexnet.prototxt` (in the current directory).\n",
    "\n",
    "Before images can be passed into AlexNet, they first must be \"preprocessed\" meaning that they are scaled to the correct range (0 to 255), a pre-computed mean is subtracted, the color channel order is flipped (RGB to BGR), and the channels are transposed.  In more detail, Python loads images in \"RGB\" `(red, green, blue)` channel order, and this must be flipped to \"BGR\" channel order.  Further, images usually have the shape `(height, width, color channel)`, but in caffe they have shape `(color channel, height, width)`.  When there are multiple images saved in the same matrix, they have shape `(batch index, color channel, height, width)` and are abbreviated as `(N, C, H, W)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Auto reload modules marked with aimport\n",
    "%load_ext autoreload\n",
    "%autoreload 1\n",
    "\n",
    "# Fix the PYTHONPATH before importing caffe\n",
    "import os, sys\n",
    "caffe_root = os.path.expanduser('~/caffe/')\n",
    "sys.path.insert(0, caffe_root + 'python')\n",
    "import caffe\n",
    "\n",
    "# Helper function to load AlexNet (take a look in load_alexnet.py if you're curious)\n",
    "%aimport load_alexnet\n",
    "net, transformer = load_alexnet.load_alexnet(caffe_root, gpu=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Ignore warnings\n",
    "\n",
    "If you get a warning that looks like this, you can safely ignore it:\n",
    "\n",
    "`caffe/python/caffe/pycaffe.py:13: RuntimeWarning: to-Python converter for boost::shared_ptr<caffe::Net<float> > already registered; second conversion method ignored.\n",
    "  from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, `\n",
    "\n",
    "BVLC is aware of the issue, and there are open GitHub Issues to propose fixes (https://github.com/BVLC/caffe/pull/3960, if you're curious)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Helper functions\n",
    "\n",
    "These functions help convert between caffe format (`(color channel, height, width)` in `BGR` order) and the more common Python image format needed for saving/display (`height, width, color channel)` in `RGB` order)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os, sys,  math, random, subprocess\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from scipy.ndimage.filters import gaussian_filter\n",
    "from IPython.display import clear_output, Image, display, HTML\n",
    "from google.protobuf import text_format\n",
    "from cStringIO import StringIO\n",
    "import PIL.Image\n",
    "\n",
    "# Configure matplotlib to render inline\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (10, 10)        # large images\n",
    "plt.rcParams['image.interpolation'] = 'nearest'  # don't interpolate: show square pixels\n",
    "plt.rcParams['image.cmap'] = 'gray'  # use grayscale output rather than a (potentially misleading) color heatmap\n",
    "\n",
    "def preprocess(img):\n",
    "    \"\"\" Process an image by subtracting the mean and converting RGB to BGR \"\"\"\n",
    "    return transformer.preprocess('data', img)\n",
    "\n",
    "def deprocess(img):\n",
    "    \"\"\" Undo image processing (add back the mean and convert BGR to RGB) \"\"\"\n",
    "    return transformer.deprocess('data', img)\n",
    "\n",
    "def load_and_preprocess_image(fname):\n",
    "    \"\"\" Load an image, and process it \"\"\"\n",
    "    image = caffe.io.load_image(fname)\n",
    "    return preprocess(image)\n",
    "\n",
    "def save_image(img, fname):\n",
    "    \"\"\" Save a floating point image, assumed to be in the range [0, 1] \"\"\"\n",
    "    assert img.ndim == 2 or img.ndim == 3  # must be grayscale or RGB\n",
    "    parent_dir = os.path.dirname(fname)\n",
    "    if not os.path.exists(parent_dir):\n",
    "        os.makedirs(parent_dir)\n",
    "    pil_image = PIL.Image.fromarray(np.clip(img * 255.0, 0, 255).astype(np.uint8))\n",
    "    pil_image.save(fname)\n",
    "    \n",
    "def showarray(img, fmt='jpeg'):\n",
    "    \"\"\" Display an image inline in the notebook, assumed to be in the range [0, 1] \"\"\"\n",
    "    assert img.ndim == 2 or img.ndim == 3  # must be grayscale or RGB\n",
    "    img = np.uint8(np.clip(img * 255.0, 0, 255))\n",
    "    f = StringIO()\n",
    "    PIL.Image.fromarray(img).save(f, fmt)\n",
    "    display(Image(data=f.getvalue()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Caffe API: Accessing Data\n",
    "\n",
    "### Layer, top, bottom\n",
    "\n",
    "In caffe, each layer in the neural net is called a `Layer` and has a name, such as \"conv1\".  These are defined in the file `patched_alexnet.prototxt`.\n",
    "\n",
    "For example, this is the specification for `conv1` (taken from `patched_alexnet.prototxt`):\n",
    "```\n",
    "layer {\n",
    "  name: \"conv1\"\n",
    "  type: \"Convolution\"\n",
    "  bottom: \"data\"\n",
    "  top: \"conv1\"\n",
    "  param {\n",
    "    lr_mult: 1\n",
    "    decay_mult: 1\n",
    "  }\n",
    "  param {\n",
    "    lr_mult: 2\n",
    "    decay_mult: 0\n",
    "  }\n",
    "  convolution_param {\n",
    "    num_output: 96\n",
    "    kernel_size: 11\n",
    "    stride: 4\n",
    "  }\n",
    "```\n",
    "\n",
    "The \"bottom\" is the _input_ to a layer and the \"top\" is the _output_ of a layer.  Here we can see that we have a layer called `conv1`, it is a convolution, it takes input from a blob named `data`, and produces an output blob named `conv1` (which happens to share the same name as the layer).\n",
    "\n",
    "### Blob: data and diff\n",
    "\n",
    "In caffe, intermediate values between layers are saved as \"blobs\".  The blobs are like a centralized key-value store where variables can be saved by name.  For example, when `conv1` runs, it produces an output (\"top\") that gets saved into the `conv1` blob.\n",
    "\n",
    "Each `Blob` stores both `data` and a `diff`.  The `data` is the activations (computed in the forwards pass), and the `diff` is the gradient (computed in the backwards pass).  By convention, when \"gradient\" or \"diff\" is used by itself, it is always the gradient of the loss with respect to that blob.  Note that the forwards pass may use the `diff` as a temporary place to save values, so `diff` only contains valid values after performing a backwards pass.\n",
    "\n",
    "For example, to access the data for the `conv1` layer, which saves its output to a blob named `conv1`, you would write `net.blobs['conv1'].data`, and to access its diff, you would write `net.blobs['conv1'].diff`. By convention, the output of each layer (called its \"top\") is saved in a blob with the same name as that layer, but this doesn't always have to be the case.\n",
    "\n",
    "**Reading from a blob**: \n",
    "`Blob` objects have a numpy-like interface, so you can access data as `net.blobs['conv1'].data[0, ...]` (to get the activations of the first image in the batch), and you can measure shape with `net.blobs['conv1'].shape`.\n",
    "Note that when grabbing data from a blob, make a copy of it (using `np.copy`).  Otherwise, the next forwards/backwards pass may destroy your saved data because it was a reference and not a copy.\n",
    "\n",
    "**Writing to a blob**:\n",
    "Assign data using `net.blobs['conv1'].data[0, ...] = X`.  The same applies for `diff`.\n",
    "\n",
    "### In-place layers\n",
    "\n",
    "The next layer after `conv1` is `relu1`:\n",
    "```\n",
    "layer {\n",
    "  name: \"relu1\"\n",
    "  type: \"ReLU\"\n",
    "  bottom: \"conv1\"\n",
    "  top: \"conv1\"\n",
    "}\n",
    "```\n",
    "Notice that the top and bottom are the same.  This means that the blob `conv1` is _overwritten_ with the result of the ReLU operation.  This is important: this means that if you access `conv1`, it will be _after_ applying ReLU.  There is no blob named `relu1`; `relu1` is the name of the layer, not the top blob.\n",
    "\n",
    "### Params\n",
    "\n",
    "Each `Layer` contains a number of parameters, saved as blobs in the dictionary `net.params`.  By convention, the weights are index 0 and the biases are index 1.  For example, the weights for `conv1` are `net.params['conv1'][0].data`, and the gradient of those weights is `net.params['conv1'][0].diff`.\n",
    "\n",
    "### Forward and backward\n",
    "\n",
    "You can compute forwards and backwards passes using these functions:\n",
    "\n",
    "* `net.forward()`: complete forwards pass from `data` all the way to the last layer `prob`.\n",
    "\n",
    "* `net.forward(start=A, end=B)`: only compute from A to B.  Note that ReLU is a separate layer, and thus if you want to include ReLU in the pass, you need to specify the appropriate ReLU layer and not the layer just before it.\n",
    "\n",
    "* `net.backward()`: complete backwards pass from the last layer (`prob`) back to the input layer (`data`).  Note that by default, the backwards pass stops at `conv1`, but we have patched the network (specified `force_backward: true`) to make it go back to `data`.\n",
    "\n",
    "* `net.backward(start=A, end=B)`: backwards pass starting from A, and working back towards B.\n",
    "\n",
    "### Viewing layer and parameter shapes AlexNet\n",
    "\n",
    "Let's visualize the shape of each layer and parameter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. AlexNet: Visualizing its structure\n",
    "\n",
    "First we will visualize the structure of AlexNet by inspecting its filters, looking at feature map sizes, and visualizing the activations inside."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# for each layer, show the output shape\n",
    "print \"Layer output shapes:\\n\" + ('-' * 30)\n",
    "for layer_name, blob in net.blobs.iteritems():\n",
    "    print \"%s\\t %s\" % (layer_name, blob.data.shape)\n",
    "\n",
    "print \"\\nLayer parameter shapes:\\n\" + ('-' * 30)\n",
    "for layer_name, param in net.params.iteritems():\n",
    "    print \"%s\\t weights: %s  \\tbias: %s\" % (layer_name, param[0].data.shape, param[1].data.shape)\n",
    "    \n",
    "print \"\\nLayer parameter counts:\\n\" + ('-' * 30)\n",
    "total = 0\n",
    "for layer_name, param in net.params.iteritems():\n",
    "    c = np.prod(param[0].data.shape) + np.prod(param[1].data.shape)\n",
    "    print \"%s\\t %s\" % (layer_name, c, )\n",
    "    total += c\n",
    "print \"\\nTotal parameters: %s\" % total"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explanation of layer sizes\n",
    "\n",
    "_(Run the above cell)_\n",
    "\n",
    "Remember that activations have _structure_, so for the input data layer, we have 3 color channels, and images are 227 pixels across and 227 pixels tall.  For the first  layer (conv1), the output has shape 55x55, because that convolutional filter has a stride > 1; this causes subsampling.\n",
    "\n",
    "Notice that for the activations, the first dimension in every layer is 1.  This is because we are only running one image at a time; the batchsize is 1.\n",
    "\n",
    "The layers fc6, fc7, fc8 are fully connected layers.  Thus they don't have 3D structures and are vectors (with 4096, 4096, and 1000 dimensions respectively).\n",
    "\n",
    "The weights for convolutions are 4D (as discussed in class) and the weights for fully connected layers are 2D."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing conv1 filters\n",
    "\n",
    "Filters in `conv1` are unique in that they take RGB images as input.  This means that we can visualize them as RGB images.  For all other layers, we cannot view them as nice little colored squares because they are much higher dimensional."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def vis_square(data, title=None):\n",
    "    \"\"\"Take an array of shape (n, height, width) or (n, height, width, 3)\n",
    "       and visualize each (height, width) thing in a grid of size approx. sqrt(n) by sqrt(n)\"\"\"\n",
    "    \n",
    "    # normalize data for display\n",
    "    data = (data - data.min()) / (data.max() - data.min())\n",
    "    \n",
    "    # force the number of filters to be square\n",
    "    n = int(np.ceil(np.sqrt(data.shape[0])))\n",
    "    padding = (((0, n ** 2 - data.shape[0]),\n",
    "               (0, 2), (0, 2))                 # add some space between filters\n",
    "               + ((0, 0),) * (data.ndim - 3))  # don't pad the last dimension (if there is one)\n",
    "    data = np.pad(data, padding, mode='constant', constant_values=1)  # pad with ones (white)\n",
    "    \n",
    "    # tile the filters into an image\n",
    "    data = data.reshape((n, n) + data.shape[1:]).transpose((0, 2, 1, 3) + tuple(range(4, data.ndim + 1)))\n",
    "    data = data.reshape((n * data.shape[1], n * data.shape[3]) + data.shape[4:])\n",
    "    \n",
    "    # plot it\n",
    "    plt.figure(figsize=(8, 8))\n",
    "    plt.imshow(data)\n",
    "    plt.axis('off')\n",
    "    if title: plt.title(title)\n",
    "\n",
    "filters = net.params['conv1'][0].data\n",
    "vis_square(filters.transpose(0, 2, 3, 1), title=\"Visualizing filters in conv1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing AlexNet activations.\n",
    "\n",
    "Let's take an example image (of a Pug) and pass it through AlexNet.\n",
    "\n",
    "Visualize the activations inside all of these layers, using the code below.  Lighter values have higher magnitude, and darker values have smaller magnitude."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "example_fname = 'dataset/test-dog/HOND1.jpg'\n",
    "display(Image(filename=example_fname))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "net.blobs['data'].reshape(1, 3, 227, 227)\n",
    "net.blobs['data'].data[0, ...] = load_and_preprocess_image(example_fname)\n",
    "net.forward()\n",
    "\n",
    "for layer_name, param in (net.blobs.iteritems()):\n",
    "    shape_str = 'x'.join(str(s) for s in net.blobs[layer_name].data.shape[1:])\n",
    "    if layer_name != 'prob' and not layer_name.startswith('fc'):\n",
    "        feat = net.blobs[layer_name].data[0, ...]\n",
    "        vis_square(feat, title=\"Visualizing layer: %s (%s channels)\" % (layer_name, shape_str))\n",
    "    else:\n",
    "        feat = net.blobs[layer_name].data[0]\n",
    "        plt.figure(figsize=(15, 3))\n",
    "        plt.plot(feat.flat)\n",
    "        plt.title(\"Visualizing layer: %s (%s channels)\" % (layer_name, shape_str))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question\n",
    "\n",
    "Consider the visualizations produced by the above cell.  Why does fc8 have negative values, but fc6 and fc7 are only positive?\n",
    "\n",
    "HINT: look at the structure of the network in the file `patched_alexnet.prototxt` (current directory).\n",
    "\n",
    "**TODO 1**: Put your answer in `student.py` (as a comment)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Looking up the class names\n",
    "\n",
    "The output probability vectors from our network will be numerical scores for each of 1000 ImageNet categories. We'll download the English names corresponding to each category so we can get a sense of how well AlexNet is doing, and then plot the scores for the top 10 classes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "output_prob = net.blobs['prob'].data[0]\n",
    "ilsvrc_labels_file = caffe_root + 'data/ilsvrc12/synset_words.txt'\n",
    "if not os.path.exists(ilsvrc_labels_file):\n",
    "    print 'downloading synset words...'\n",
    "    subprocess.call([caffe_root + 'data/ilsvrc12/get_ilsvrc_aux.sh'])\n",
    "ilsvrc_labels = np.loadtxt(ilsvrc_labels_file, str, delimiter='\\t')\n",
    "\n",
    "top_k = 10\n",
    "top_indices = np.argsort(output_prob)[-top_k:]\n",
    "plt.figure(figsize=(12, 8))\n",
    "plt.barh(np.arange(top_k) + 0.5, output_prob[top_indices], align='center')\n",
    "plt.xlabel('prob')\n",
    "plt.yticks(np.arange(top_k) + 0.5, ilsvrc_labels[top_indices])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What to expect\n",
    "\n",
    "We can see that this example was correctly classified, and with high confidence, despite the dog wearing a misleading costume!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Dog vs Food: Classification\n",
    "\n",
    "Let's classify dog vs food.  We have prepared a test set of dogs dressed up like hotdogs, and hotdogs cut to look like animals.\n",
    "\n",
    "The below cell loads the dataset and visualizes all the images in the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import glob, uuid\n",
    "\n",
    "def show_images(fnames, correct=None, size=128):\n",
    "    # print all the images inline, optionally color-coding which are correct\n",
    "    # (the uuid is to avoid caching)\n",
    "    inline_html = ''.join([\n",
    "        \"<img src='%s?%s' style='width: %spx; margin: 2px; display: inline; border: %s' />\" % (\n",
    "            fname, uuid.uuid4(), size,\n",
    "            (\"4px solid green\" if correct[i] else \"8px solid red\")\n",
    "            if correct else \"1px solid black\"\n",
    "        )\n",
    "        for i, fname in enumerate(fnames)\n",
    "    ])\n",
    "    display(HTML(inline_html))\n",
    "\n",
    "# Class names\n",
    "classes = ['dog', 'food']\n",
    "\n",
    "# Search the local filesystem for files\n",
    "test_fnames = [sorted(glob.glob('dataset/test-%s/*.jpg' % cname)) for cname in classes]\n",
    "for cidx, cname in enumerate(classes):\n",
    "    print \"Found %s %s test images\" % (len(test_fnames[cidx]), cname)\n",
    "    show_images(test_fnames[cidx], size=64)\n",
    "    \n",
    "# Make sure we found the images\n",
    "assert len(test_fnames[0]) == 33\n",
    "assert len(test_fnames[1]) == 25"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Repurposing the ILSVRC2012 Classifier\n",
    "\n",
    "AlexNet was trained to recognize one of 1000 classes.  We can repurpose it for our \"food vs dog\" task by remapping the categories.\n",
    "\n",
    "Below is a helper function to process a batch of images (you need to run the cell once)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Helper function to process a batch of images\n",
    "def batch_forward_pass(fnames, blob_name):\n",
    "    \"\"\" Process a batch of images.  Each image is loaded one-by-one, its output is extracted\n",
    "    \n",
    "    :param fnames: list of filenames to process\n",
    "    :param blob_name: which blob to extract from the network each forwards pass\n",
    "    \"\"\"\n",
    "    outputs = []\n",
    "    for i, fname in enumerate(fnames):\n",
    "        print '\\rProcessing: %s/%s (%r)' % (i+1, len(fnames), fname),\n",
    "        # load an image and pre-process it\n",
    "        # (subtract mean, convert to BGR order, scale up to 255)\n",
    "        data = load_and_preprocess_image(fname)\n",
    "        # put the data at the beginning of the network\n",
    "        net.blobs['data'].reshape(1, 3, 227, 227)\n",
    "        net.blobs['data'].data[0, ...] = data\n",
    "        # forward pass: compute every layer\n",
    "        net.forward()\n",
    "        # extract and store the output\n",
    "        net_output = np.copy(net.blobs[blob_name].data[0, :])\n",
    "        outputs.append(net_output)\n",
    "    print \"\"\n",
    "    return np.array(outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To estimate the probability of \"food\", we take the sum of all food-related ILSVRC2012 classes, and to estimate the probability of \"dog\", we do the same for dogs.  Dog is class 0, and food is class 1.\n",
    "\n",
    "Write a method to compute the probability of our two classes, based on the AlexNet output.  Normalize your answer so that it sums to 1.\n",
    "\n",
    "**TODO 2**: Implement `convert_ilsvrc2012_probs_to_dog_vs_food_probs` in `student.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%aimport student\n",
    "\n",
    "# batch forward\n",
    "all_probs = []\n",
    "for cidx, cname in enumerate(classes):\n",
    "    batch_fnames = test_fnames[cidx] # current batch of images\n",
    "    N = len(batch_fnames) # number of items in the batch\n",
    "    probs_ilsvrc = batch_forward_pass(batch_fnames, blob_name='prob')\n",
    "    \n",
    "    probs = student.convert_ilsvrc2012_probs_to_dog_vs_food_probs(probs_ilsvrc)\n",
    "    \n",
    "    # Check shape\n",
    "    assert probs.shape == (N, 2)\n",
    "    # Check that probs are normalized\n",
    "    np.testing.assert_almost_equal(np.sum(probs, axis=1), np.ones(N), decimal=5)\n",
    "    \n",
    "    all_probs.append(probs)\n",
    "all_probs = np.array(all_probs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Measure the accuracy\n",
    "\n",
    "You should expect to get around 85% and 96% (+/- 1%) for dog and food respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# compute the accuracy of our method\n",
    "for cidx, cname in enumerate(classes):\n",
    "    num_images =  len(test_fnames[cidx])\n",
    "    predicted = np.argmax(all_probs[cidx], axis=1)\n",
    "    num_correct = np.sum(predicted == cidx)\n",
    "    accuracy = 100.0 * float(num_correct) / num_images\n",
    "    print \"Accuracy: %s %s/%s (%s%%)\" % (cname, num_correct, num_images, accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Dog vs Food: Visualization\n",
    "\n",
    "We can sort the predictions by the \"dog\" score and \"food\" score.  The images are sorted according to how much AlexNet thinks the image belongs to that category.\n",
    "\n",
    "Images are colored green/red depending on whether the prediction was correct.\n",
    "\n",
    "For this assignment, you should expect that the incorrect predictions are all near the bottom (with the lowest score).  This is a property that is very desirable -- mistakes only happen with lower scores.  Note that in general real-world tasks, this does not always happen for free.  If you want to be able to estimate the confidence of being correct, you need to separately predict that, and it is not easy to predict.\n",
    "\n",
    "**TODO 3:** Implement `get_prediction_descending_order_indices` in `student.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "%aimport student\n",
    "\n",
    "for cidx, cname in enumerate(classes):\n",
    "    print \"\"\n",
    "    print \"Test images for: %s\" % cname\n",
    "    \n",
    "    # which items are correct?\n",
    "    predicted = np.argmax(all_probs[cidx], axis=1)\n",
    "    correct = [p == cidx for p in predicted]\n",
    "    \n",
    "    # Sort the classes in reverse order\n",
    "    order = student.get_prediction_descending_order_indices(all_probs[cidx], cidx)\n",
    "    assert len(order) == len(test_fnames[cidx]) # check shape\n",
    "    \n",
    "    show_images(\n",
    "        fnames=[test_fnames[cidx][i] for i in order],\n",
    "        correct=[correct[i] for i in order],\n",
    "        size=128\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Visualizing saliency\n",
    "\n",
    "Using our pre-trained AlexNet, we will compute class saliency maps as described in Section 3.1 of [2].  As mentioned in Section 2 of the paper, you should compute the gradient of the image with respect to the unnormalized class score (`fc8`), not with respect to the normalized class probability (`prob`).  You will need to use the `forward` and `backward` methods of the `net` to compute gradients with respect to the image.\n",
    "\n",
    "We want to compute: $${\\partial s_y \\over \\partial I}$$\n",
    "\n",
    "where $s_y$ is the score for class $y$ before the final Softmax layer (i.e. layer `fc8`).\n",
    "\n",
    "We will then visualize the squared magnitude of this (max across color channels), to estimate the saliency of the class across the input image.  See [1] for more details and intuition.\n",
    "\n",
    "**TODO 4**: Implement `compute_dscore_dimage` in `student.py`.\n",
    "\n",
    "**HINT**: To compute the desired gradient, imagine that you have created an extra layer that takes $s$ as input (i.e. layer `fc8`), and outputs $s_y$.  For the purposes of this derivation, imagine that the \"loss\" is $L = s_y$.  If you compute the derivative of this layer, then you will have ${\\partial L \\over \\partial s}$.  Note that this is exactly the definition of the `diff` for the $s$ layer (i.e. the derivative of the loss $L$ with respect to the activations in $s$).  Thus, you can set the `diff` for the $s$ layer (i.e. layer `fc8`) equal to ${\\partial L \\over \\partial s} = {\\partial s_y \\over \\partial s}$.  Then, continue the backprop process starting from layer `fc8` and continue back to the image (`data` layer).\n",
    "\n",
    "**NOTE**: You don't need to call `net.forward()` in your function; this has already been run for you.  Same for all gradient functions you implement below.\n",
    "\n",
    "[2] <a href=\"http://arxiv.org/pdf/1312.6034.pdf\">Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. \"Deep Inside Convolutional Networks: Visualising\n",
    "Image Classification Models and Saliency Maps\", ICLR Workshop 2014.</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "%aimport student\n",
    "\n",
    "def visualize_saliency(data):\n",
    "    net.blobs['data'].reshape(1, 3, 227, 227)\n",
    "    net.blobs['data'].data[0, ...] = data\n",
    "    net.forward()\n",
    "    argmax_idx = np.argmax(net.blobs['prob'].data[0])\n",
    "    grad = student.compute_dscore_dimage(net, data, argmax_idx)\n",
    "    vis = grad * grad\n",
    "    vis /= np.percentile(vis, q=99.9)\n",
    "    vis = np.max(vis, axis=0)\n",
    "    return vis\n",
    "\n",
    "num_images = 6\n",
    "for cidx, cname in enumerate(classes):\n",
    "    fnames = []\n",
    "    for i, fname in enumerate(test_fnames[cidx][:num_images]):\n",
    "        print '\\rProcessing: %s/%s (%r)' % (i+1, num_images, fname),\n",
    "        \n",
    "        data = load_and_preprocess_image(fname)\n",
    "        vis_image = visualize_saliency(data)\n",
    "        \n",
    "        # Check the shape and save\n",
    "        assert vis_image.shape == (227, 227)\n",
    "        vis_fname = 'vis/%s-%s.png' % (cidx, i)\n",
    "        save_image(vis_image, vis_fname)\n",
    "        \n",
    "        fnames.append(fname)\n",
    "        fnames.append(vis_fname)\n",
    "    \n",
    "    print \"\\nSaliency maps for %s\" % cname\n",
    "    show_images(fnames, size=200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Fooling AlexNet\n",
    "\n",
    "For many machine learning models, it is possible to \"fool\" them by tweaking the image slightly so that the image is predicted to become any category [1].  Given any image, and any target class, you can perform gradient ascent to maximize the score of that target class (equivalently, gradient descent on the negative score of that class), stopping when the network confidently predicts it as the target class.\n",
    "\n",
    "Again, maximize the score with respect to the unnormalized class score (`fc8`) and not the normalized class probability (`prob`).  You can reuse the `compute_dscore_dimage` you defined in the previous cell.\n",
    "\n",
    "In addition to maximizing the score (minimizing the negative score), also add a regularizer that computes the L2 norm between the original image, and the fooling image.  The final gradient will be the sum of the gradient from the regularizer and the gradient from maximizing the class score.\n",
    "\n",
    "We can write this as a loss $L$:\n",
    "$$\n",
    "L = -s_y(I) + R(I)\n",
    "$$\n",
    "\n",
    "where $$\n",
    "R(I) = 0.5 \\lambda \\|I - I_\\text{orig}\\|_2^2\n",
    "$$\n",
    "$y$ is the target class, and $\\lambda$ is the regularization.\n",
    "\n",
    "#### Momentum\n",
    "When optimizing functions with ConvNets, typically you use use gradient descent with momentum, which has the update rule:\n",
    "$$V_t = \\mu V_{t-1} - \\alpha G$$\n",
    "$$I_t = I_{t-1} + V_{t}$$\n",
    "\n",
    "where $V$ is the velocity, $\\alpha$ is the learning rate, $\\mu$ is the momentum parameter, $t$ is the iteration number, and $G = \\frac{\\partial L}{\\partial I_{t-1}}$ is the gradient.\n",
    "\n",
    "To improve stability, we will use a slightly different update, which normalizes the gradient $G$ to have unit norm:\n",
    "$$V_t = \\mu V_{t-1} - \\alpha \\frac{G}{\\|G\\|}$$\n",
    "$$I_t = I_{t-1} + V_{t}$$\n",
    "\n",
    "**TODO 5a**: Implement the above update rule, `normalized_sgd_with_momentum_update`, in `student.py`.\n",
    "\n",
    "** TODO 5b**: Implement `fooling_image_gradient` which returns the gradient of the loss with respect to the image $({\\partial L \\over \\partial I})$.  You can reuse the function you implemented above (`compute_dscore_dimage`) to compute ${\\partial s_y \\over \\partial I}$.\n",
    "\n",
    "[1] Szegedy et al, \"Intriguing properties of neural networks\", ICLR 2014"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%aimport student\n",
    "\n",
    "def make_fooling_image(fname, target_class, learning_rate, regularization,\n",
    "                       num_iter, momentum, threshold=0.9):\n",
    "    \"\"\"\n",
    "    Fool AlexNet into thinking that any image has a particular class, by perturbing it just a little bit\n",
    "    \n",
    "    :param fname: starting image filename\n",
    "    :param target_class: the class that this will become after optimization\n",
    "    :param learning_rate: either a constant, or a function that returns the learning rate at each iteration\n",
    "    :param regularization: lambda parameter to multiply the regularizer\n",
    "    :param num_iter: maximum number of iterations\n",
    "    :param momentum: amount of momentum to use in the SGD \n",
    "    :param threshold: the target score for target_class\n",
    "    \"\"\"\n",
    "    data = load_and_preprocess_image(fname)\n",
    "    net.blobs['data'].reshape(1, 3, 227, 227)\n",
    "    velocity = np.zeros_like(data)\n",
    "    \n",
    "    # This is the original image (used by the regularizer)\n",
    "    orig_data = np.copy(data)\n",
    "    for i in xrange(num_iter):\n",
    "        net.blobs['data'].data[0, ...] = data\n",
    "        net.forward()\n",
    "        cur_prob = net.blobs['prob'].data[0, target_class]\n",
    "        \n",
    "        # compute the gradient\n",
    "        grad = student.fooling_image_gradient(\n",
    "            net, orig_data, data, target_class, regularization)\n",
    "        \n",
    "        # update the image with the SGD rule\n",
    "        data, velocity = student.normalized_sgd_with_momentum_update(\n",
    "            data, grad, velocity, momentum, learning_rate)\n",
    "        \n",
    "        # visualize the current state\n",
    "        showarray(deprocess(data))\n",
    "        print fname\n",
    "        print \"Fooling AlexNet into thinking this is a: %r...\" % ilsvrc_labels[target_class]\n",
    "        print '(%s/%s), %s%% confidence' % (i+1, num_iter, cur_prob * 100)\n",
    "        clear_output(wait=True)\n",
    "        \n",
    "        if cur_prob > threshold:\n",
    "            break\n",
    "    \n",
    "    delta = (data - orig_data)\n",
    "    return data, delta\n",
    "\n",
    "num_images = 2\n",
    "all_fnames = []\n",
    "target_class=113\n",
    "for cidx, cname in enumerate(classes):\n",
    "    fnames = []\n",
    "    for i, fname in enumerate(test_fnames[cidx][:num_images]):\n",
    "        data, delta = make_fooling_image(\n",
    "            fname,\n",
    "            target_class=target_class,\n",
    "            learning_rate=100,\n",
    "            regularization=5e-5,\n",
    "            num_iter=100,\n",
    "            momentum=0.9)\n",
    "\n",
    "        fooling_fname = 'fooling/%s-%s.png' % (cidx, i)\n",
    "        save_image(deprocess(data), fooling_fname)\n",
    "        \n",
    "        # Save 5x magnified delta image\n",
    "        delta = 0.5 + (5.0/255.0) * delta\n",
    "        delta = delta.transpose(1, 2, 0)  # BGR --> RGB\n",
    "        delta_fname = 'fooling/%s-%s-delta.png' % (cidx, i)\n",
    "        save_image(delta, delta_fname)\n",
    "        \n",
    "        fnames.append([fname, fooling_fname, delta_fname])\n",
    "    all_fnames.append(fnames)\n",
    "    \n",
    "for cidx, cname in enumerate(classes):\n",
    "    print (\"\\nLeft: original, middle: fooling image, right: difference magnified by 5x (gray is 0).\\n\"\n",
    "        \"AlexNet will classify the middle image in each row as %r with high confidence\" % (\n",
    "        ilsvrc_labels[target_class]))\n",
    "    for group in all_fnames[cidx]:\n",
    "        show_images(group, size=200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Class visualization\n",
    "\n",
    "We can visualize the knowledge inside AlexNet by starting with random noise and maximizing the probability of a certain class [2, 4].  This is similar to the idea of making a \"fooling\" image, except that the regularizer is a little different.\n",
    "\n",
    "Again, we can write this as a loss $L$ depending on the image $I$:\n",
    "$$\n",
    "L = -s_y(I) + R(I)\n",
    "$$\n",
    "\n",
    "where $$\n",
    "R(I) = 0.5 \\lambda \\|I\\|_2^2\n",
    "$$\n",
    "$y$ is the target class, and $\\lambda$ is the regularization.\n",
    "\n",
    "**TODO 6**: Implement `class_visualization_gradient` (in `student.py`): compute the gradient of the loss $({\\partial L \\over \\partial I})$, which is a combination of the gradient from $-{\\partial s_y \\over \\partial I}$ and the regularization ${\\partial R(I) \\over \\partial I}$.  As before, call `compute_dscore_dimage` to compute ${\\partial s_y \\over \\partial I}$.\n",
    "\n",
    "[2] <a href=\"http://arxiv.org/pdf/1312.6034.pdf\">Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. \"Deep Inside Convolutional Networks: Visualising\n",
    "Image Classification Models and Saliency Maps\", ICLR Workshop 2014.</a>\n",
    "\n",
    "[4] <a href=\"http://yosinski.com/media/papers/Yosinski__2015__ICML_DL__Understanding_Neural_Networks_Through_Deep_Visualization__.pdf\">Yosinski et al, \"Understanding Neural Networks Through Deep Visualization\", ICML 2015 Deep Learning Workshop</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%aimport student\n",
    "\n",
    "def make_class_visualization(target_class, learning_rate, regularization, num_iter,\n",
    "                             max_jitter, blur_sigma, momentum):\n",
    "    \"\"\"\n",
    "    Visualize an ILSVRC2012 class by maximizing the probability of that class, starting from random.\n",
    "    \n",
    "    :param target_class: what ILSVRC2012 class to visualize\n",
    "    :param learning_rate: either a constant, or a function that returns the learning rate at each iteration\n",
    "    :param regularization: lambda parameter to multiply the regularizer\n",
    "    :param num_iter: number of iterations\n",
    "    :param max_jitter: amount of jitter to add for regularization\n",
    "    :param blur_sigma: blur each iteration by this amount.\n",
    "    :param momentum: amount of momentum to use in the SGD update\n",
    "    \"\"\"\n",
    "    data = 100 * np.random.randn(3, 227, 227)\n",
    "    net.blobs['data'].reshape(1, 3, 227, 227)\n",
    "    velocity = np.zeros_like(data)\n",
    "    \n",
    "    for i in xrange(num_iter):\n",
    "        # Random jitter to regularize\n",
    "        ox, oy = np.random.randint(-max_jitter, max_jitter+1, 2)\n",
    "        data = np.roll(np.roll(data, ox, -1), oy, -2)\n",
    "        velocity = np.roll(np.roll(velocity, ox, -1), oy, -2)\n",
    "        \n",
    "        # Compute the current class score\n",
    "        net.blobs['data'].data[0, ...] = data\n",
    "        net.forward()\n",
    "        cur_prob = net.blobs['prob'].data[0, target_class]\n",
    "        \n",
    "        # Compute the gradient\n",
    "        grad = student.class_visualization_gradient(\n",
    "            net, data, target_class, regularization)\n",
    "        \n",
    "        # Normalized SGD+Momentum update\n",
    "        data, velocity = student.normalized_sgd_with_momentum_update(\n",
    "            data, grad, velocity, momentum, learning_rate)\n",
    "        \n",
    "        # Undo jitter\n",
    "        data = np.roll(np.roll(data, -ox, -1), -oy, -2)\n",
    "        velocity = np.roll(np.roll(velocity, -ox, -1), -oy, -2)\n",
    "        \n",
    "        # blur the image every iteration\n",
    "        for c in xrange(3):\n",
    "            data[c, ...] = gaussian_filter(data[c, ...], sigma=blur_sigma)\n",
    "        \n",
    "        # Visualize our current result\n",
    "        showarray(deprocess(data))\n",
    "        print \"Generating visualization of: %r...\" % ilsvrc_labels[target_class]\n",
    "        print '(%s/%s), %s%% confidence' % (i+1, num_iter, cur_prob * 100)\n",
    "        clear_output(wait=True)\n",
    "    \n",
    "    return deprocess(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate images!\n",
    "\n",
    "Use `make_class_visualization` to visualize an example Flamingo or Tarantula.  You can play around with the hyperparameters or generate new classes, but you must at least generate a Flamingo or Tarantula.\n",
    "\n",
    "The list of classes is in `~/caffe/data/ilsvrc12/synset_words.txt` (this is downloaded in a cell above)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data = make_class_visualization(\n",
    "    target_class=130, # flamingo\n",
    "    learning_rate=200,\n",
    "    regularization=1e-7,\n",
    "    num_iter=250,\n",
    "    max_jitter=5,\n",
    "    blur_sigma=0.4,\n",
    "    momentum=0.9,\n",
    ")\n",
    "save_image(data, 'classvis/class130-flamingo.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data = make_class_visualization(\n",
    "    target_class=76, # tarantula\n",
    "    learning_rate=200,\n",
    "    regularization=1e-7,\n",
    "    num_iter=250,\n",
    "    max_jitter=5,\n",
    "    blur_sigma=0.4,\n",
    "    momentum=0.9,\n",
    ")\n",
    "save_image(data, 'classvis/class76-tarantula.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Try other classes here!\n",
    "# (make sure to call save_image with a new filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7. Feature Inversion\n",
    "\n",
    "Instead of generating new examples, we can try and _reconstruct_ examples starting from the hidden representation of a single layer [5].  While [4] and [5] use better regularizers, we are going to use a simple L2 regularizer.\n",
    "\n",
    "Concretely, given a image $I$, let $\\phi_\\ell(I)$ be the activations at layer $\\ell$ of the convolutional network $\\phi$. We wish to find an image $I$ with a similar feature representation as $I'$ at layer $\\ell$ of the network $\\phi$ by solving the optimization problem\n",
    "\n",
    "$$\n",
    "L = (0.5/M) \\|\\phi_\\ell(I) - \\phi_\\ell(I')\\|_2^2 + R(I)\n",
    "$$\n",
    "\n",
    "where $\\|\\cdot\\|_2^2$ is the squared Euclidean norm, and $M$ is the number of entries in $\\phi_\\ell(I)$.  $M$ was added to make it easier to set $\\lambda$, and $R(I)$ is a regularizer.  We can solve this optimization problem using gradient descent (with momentum), computing gradients with respect to the generated image.  We will use L2 regularization of the form:\n",
    "\n",
    "$$\n",
    "R(I) = 0.5 \\lambda \\|I\\|_2^2\n",
    "$$\n",
    "\n",
    "To improve the reconstruction quality, we will blur the image every few iterations, which is like adding an implicit regularizer to the optimization.\n",
    "\n",
    "**TODO 7a**: Implement `feature_inversion_gradient`, which computes the gradient of the above loss function with respect to the image, ${\\partial L \\over \\partial I}$.\n",
    "\n",
    "**HINT:** Imagine that you have added an extra Euclidean loss layer at some layer in the network.  Backprop through that one layer manually (in code that you write), insert the resulting gradient into the network at the right place, and backprop to get the gradient with respect to the image.  Finally, add the gradient from the regularizer $R$.  You should only have to call `net.backward` exactly once.\n",
    "\n",
    "[5] <a href=\"http://arxiv.org/pdf/1412.0035.pdf\">Aravindh Mahendran, Andrea Vedaldi, \"Understanding Deep Image Representations by Inverting them\", CVPR 2015</a>\n",
    "\n",
    "[4] <a href=\"http://yosinski.com/media/papers/Yosinski__2015__ICML_DL__Understanding_Neural_Networks_Through_Deep_Visualization__.pdf\">Yosinski et al, \"Understanding Neural Networks Through Deep Visualization\", ICML 2015 Deep Learning Workshop</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%aimport student\n",
    "\n",
    "def make_feature_inversion(fname, blob_name, variable_learning_rate, regularization,\n",
    "                           num_iter, blur_every, momentum):\n",
    "    \"\"\"\n",
    "    Solve for a feature inversion for a given filename, by reconstructing from a certain layer (blob_name).\n",
    "    \n",
    "    :param variable_learning_rate: a function that returns the learning rate at each iteration\n",
    "    :param regularization: lambda parameter to multiply the regularizer\n",
    "    :param num_iter: number of iterations\n",
    "    :param blur_every: how many iterations to wait between blurring the image\n",
    "    :param momentum: amount of momentum to use in the SGD update\n",
    "    \"\"\"\n",
    "    orig_data = load_and_preprocess_image(fname)\n",
    "    net.blobs['data'].data[0, ...] = orig_data\n",
    "    net.forward(end=blob_name)\n",
    "    \n",
    "    # target feature value\n",
    "    target_feat = np.copy(net.blobs[blob_name].data[0])\n",
    "    \n",
    "    # start out with random gaussian noise\n",
    "    data = 20 * np.random.randn(3, 227, 227)\n",
    "    net.blobs['data'].reshape(1, 3, 227, 227)\n",
    "    \n",
    "    # momentum velocity\n",
    "    velocity = np.zeros_like(data)\n",
    "    \n",
    "    for i in xrange(num_iter):\n",
    "        net.blobs['data'].data[0, ...] = data\n",
    "        net.forward(end=blob_name)\n",
    "        \n",
    "        # learning rate: allow it to vary\n",
    "        learning_rate = variable_learning_rate(i)\n",
    "        \n",
    "        # current feature value\n",
    "        cur_feat = net.blobs[blob_name].data[0, ...]\n",
    "        \n",
    "        # loss function\n",
    "        loss_target = 0.5 * np.sum(np.square(cur_feat - target_feat)) / target_feat.size\n",
    "        loss_regulariztion = 0.5 * regularization * np.sum(np.square(data))\n",
    "        loss = loss_target + loss_regulariztion\n",
    "        \n",
    "        # loss gradient\n",
    "        grad = student.feature_inversion_gradient(\n",
    "            net, data, blob_name, target_feat, regularization)\n",
    "        \n",
    "        # Normalized SGD+Momentum update\n",
    "        data, velocity = student.normalized_sgd_with_momentum_update(\n",
    "            data, grad, velocity, momentum, learning_rate)\n",
    "        \n",
    "        # Periodically blur the image\n",
    "        if i > 0 and i < num_iter - 1 and (i % blur_every) == 0:\n",
    "            # Blur each color channel separately\n",
    "            for c in xrange(3):\n",
    "                data[c, ...] = gaussian_filter(data[c, ...], sigma=0.5)\n",
    "\n",
    "        showarray(deprocess(data))\n",
    "        print \"(%s/%s) Inverting features for %r\" % (i+1, num_iter, fname)\n",
    "        print \"Blob name: %r, learning_rate: %.3f, loss: %.3f (%.3f target + %.3f reg)\" % (\n",
    "            blob_name, learning_rate, loss, loss_target, loss_regulariztion)\n",
    "        clear_output(wait=True)\n",
    "    \n",
    "    return deprocess(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Original image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "invert_fname = test_fnames[0][10]\n",
    "display(Image(filename=invert_fname))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Reconstructed images\n",
    "\n",
    "HINT: If you read the paper by Mahendran and Vedaldi, you'll see that reconstructions from deep features tend not to look much like the original image, so you shouldn't expect the results to look like the reconstruction above. You should be able to get an image that shows some discernable structure within 100-1000 iterations.\n",
    "\n",
    "`conv1` should look pretty close to the original image.  `conv1` should be pretty fast to compute (several iterations per second), but the later layers will be slower (1-5 seconds per iteration).\n",
    "\n",
    "When debugging, we recommend making sure that conv1 works before trying the other layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data = make_feature_inversion(invert_fname,\n",
    "           blob_name='conv1',\n",
    "           variable_learning_rate=lambda i: 1000.0 * np.exp(-i/100.0),\n",
    "           regularization=1e-8,\n",
    "           num_iter=200,\n",
    "           blur_every=5,\n",
    "           momentum=0.9)\n",
    "save_image(data, 'inversion/conv1.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data = make_feature_inversion(invert_fname,\n",
    "           blob_name='conv2',\n",
    "           variable_learning_rate=lambda i: 1000.0 * np.exp(-i/100.0),\n",
    "           regularization=1e-8,\n",
    "           num_iter=200,\n",
    "           blur_every=5,\n",
    "           momentum=0.9)\n",
    "save_image(data, 'inversion/conv2.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data = make_feature_inversion(invert_fname,\n",
    "           blob_name='conv3',\n",
    "           variable_learning_rate=lambda i: 1000.0 * np.exp(-i/100.0),\n",
    "           regularization=1e-7,\n",
    "           num_iter=200,\n",
    "           blur_every=5,\n",
    "           momentum=0.9)\n",
    "save_image(data, 'inversion/conv3.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data = make_feature_inversion(invert_fname,\n",
    "           blob_name='conv4',\n",
    "           variable_learning_rate=lambda i: 1000.0 * np.exp(-i/100.0),\n",
    "           regularization=5e-8,\n",
    "           num_iter=200,\n",
    "           blur_every=5,\n",
    "           momentum=0.9)\n",
    "save_image(data, 'inversion/conv4.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data = make_feature_inversion(invert_fname,\n",
    "           blob_name='conv5',\n",
    "           variable_learning_rate=lambda i: 1000.0 * np.exp(-i/100.0),\n",
    "           regularization=1e-8,\n",
    "           num_iter=200,\n",
    "           blur_every=5,\n",
    "           momentum=0.9)\n",
    "save_image(data, 'inversion/conv5.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data = make_feature_inversion(invert_fname,\n",
    "           blob_name='fc6',\n",
    "           variable_learning_rate=lambda i: 1000.0 * np.exp(-i/100.0),\n",
    "           regularization=1e-8,\n",
    "           num_iter=200,\n",
    "           blur_every=2,\n",
    "           momentum=0.9)\n",
    "save_image(data, 'inversion/fc6.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data = make_feature_inversion(invert_fname,\n",
    "           blob_name='fc8',\n",
    "           variable_learning_rate=lambda i: 1000.0 * np.exp(-i/100.0),\n",
    "           regularization=1e-10,\n",
    "           num_iter=200,\n",
    "           blur_every=2,\n",
    "           momentum=0.9)\n",
    "save_image(data, 'inversion/fc8.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question\n",
    "\n",
    "(a) What happens to the quality of the reconstruction as you reconstruct from higher layers?  Why does this happen?  What does this suggest about the representation at each layer?\n",
    "\n",
    "(b) Why did we adjust the regularization parameter to a different value for some layers?  How might the objective function be changed so that a different value isn't needed for each layer?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO 7b**: Answer in `student.py` as a comment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}