05_Representations.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Important concepts of lecture 4:\n",
    "\n",
    "* from n-hot vectors (sparse) to embeddings (dense inputs)\n",
    "* baselines for your projects (classification, sequence prediction)\n",
    "* short introduction to RNNs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Neural Networks - Representations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Recap: Feed-forward Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\n",
    "$$NN_{MLP1}(\\mathbf{x})=g(\\mathbf{xW^1+b^1})\\mathbf{W^2}+\\mathbf{b^2}$$\n",
    "\n",
    "<img src=\"pics/yg-compgraph1.png\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "However, what is the input $\\textbf{x}$?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Recap: Features so far\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Before we go further, let's make a detour and recap: how did we represent a training instance in a traditional classifier so far?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "For instance, recall our example from week 1: training a Logistic Regression classifier for sentiment classification. \n",
    "\n",
    "* Describe in words: what were the features we used? I.e., how did we represent a training instance $\\textbf{x}$?\n",
    "* How can you now describe the entire sentiment training data set as a matrix $X$, i.e.,  what are the rows and columns of $X$? $$ X = \\{\\mathbf{x_1}, ... , \\mathbf{x_n}\\} $$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So far we used **sparse** inputs (n-hot encodings)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "After this lecture you should:\n",
    "* know about **distributional similarity** (embeddings: --traditional:LSA--, --neural:word2vec--)\n",
    "* understand the difference between **discrete** (one-hot) and **dense** feature representations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Representations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**discrete representation**\n",
    "\n",
    "$$\\mathbf{x}_{cat} = [0,0,0,0,0,0,1] $$\n",
    "$$\\mathbf{x}_{dog} = [0,0,0,0,1,0,0] $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**similarity** on discrete representations? For any pair of words: $$\\mathbf{x}_{cat} \\wedge \\mathbf{x}_{dog} = 0$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Probably the biggest jump when moving from traditional linear models with sparse inputs to deep neural networks is to stop representing each feature as a unique dimension, but instead represent them as **dense vectors** (Goldberg, 2015)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Instead of using discrete representations, we will **embed** words into a high-dimensional feature space and represent each word by a lower-dimensional dense *vector* (aka. *embedding*):\n",
    "\n",
    "$$\\mathbf{x}_{cat} = [0.9, 0.1, 0.3, 0.8]$$\n",
    "$$\\mathbf{x}_{dog} = [0.8, 0.2, 0.7, 0.6]$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Representing words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**\"You shall know a word by the company it keeps\"** (Firth, J. R. 1957:11)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<img src=\"pics/flødebolle.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### \"The company it keeps\": word co-occurence matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We can represent the \"company\" of a word in terms of a word co-occurence matrix. On the rows we have the words, on the columns their context.\n",
    "\n",
    "**Contexts** can be of different types, for example:\n",
    "* entire documents\n",
    "* paragraphs\n",
    "* a window around the word"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Example corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "corpus = [\"She enjoys Groningen .\", \n",
    "          \"I like Cockatoos .\", \n",
    "          \"She likes good food .\", \n",
    "          \"I like Groningen .\",\n",
    "          \"good times\"\n",
    "         ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['.', 'Cockatoos', 'Groningen', 'I', 'She', 'enjoys', 'food', 'good', 'like', 'likes', 'times']\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "vocab = set(np.concatenate([s.split() for s in corpus],0))\n",
    "print(sorted(vocab))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Co-occurence matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vocab size:  11\n",
      "[[1. 1. 1. 1. 0.]\n",
      " [0. 1. 0. 0. 0.]\n",
      " [1. 0. 0. 1. 0.]\n",
      " [0. 1. 0. 1. 0.]\n",
      " [1. 0. 1. 0. 0.]\n",
      " [1. 0. 0. 0. 0.]\n",
      " [0. 0. 1. 0. 0.]\n",
      " [0. 0. 1. 0. 1.]\n",
      " [0. 1. 0. 1. 0.]\n",
      " [0. 0. 1. 0. 0.]\n",
      " [0. 0. 0. 0. 1.]]\n"
     ]
    }
   ],
   "source": [
    "# let's build a co-occurence matrix \n",
    "# rows: indices of words\n",
    "# columns: each column is a document, register whether the word appeared in the context\n",
    "## (in practice: many more contexts, different weighting schemes etc..)\n",
    "w2i = {w: i for i,w in enumerate(sorted(vocab))}\n",
    "i2w = {i: w for i,w in enumerate(w2i)}\n",
    "\n",
    "coocurrence_matrix = np.zeros((len(vocab),len(corpus)))\n",
    "for col_idx, sentence in enumerate(corpus):\n",
    "    sentence = sentence.split()\n",
    "    for word in sentence:\n",
    "        word_idx = w2i[word]\n",
    "        coocurrence_matrix[(word_idx,col_idx)] +=1\n",
    "\n",
    "print(\"vocab size: \", len(w2i))\n",
    "print(coocurrence_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "with row info:\n",
      "0\t.\t[ 1.  1.  1.  1.  0.]\n",
      "1\tCockatoos\t[ 0.  1.  0.  0.  0.]\n",
      "2\tGroningen\t[ 1.  0.  0.  1.  0.]\n",
      "3\tI\t[ 0.  1.  0.  1.  0.]\n",
      "4\tShe\t[ 1.  0.  1.  0.  0.]\n",
      "5\tenjoys\t[ 1.  0.  0.  0.  0.]\n",
      "6\tfood\t[ 0.  0.  1.  0.  0.]\n",
      "7\tgood\t[ 0.  0.  1.  0.  1.]\n",
      "8\tlike\t[ 0.  1.  0.  1.  0.]\n",
      "9\tlikes\t[ 0.  0.  1.  0.  0.]\n",
      "10\ttimes\t[ 0.  0.  0.  0.  1.]\n"
     ]
    }
   ],
   "source": [
    "print(\"with row info:\")\n",
    "for i, row in enumerate(coocurrence_matrix):\n",
    "    print(\"{}\\t{}\\t{}\".format(i, i2w[i], row))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Co-occurence matrix\n",
    "\n",
    "* **dimensionality**: number of words $|V|$ (size of vocabulary) times number of documents (typically number of documents is huge)\n",
    "* we want to **reduce** its dimensionality"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### LSA - Latent Semantic Analysis (Singular Value Decomposition - SVD)\n",
    "\n",
    "Approximate a matrix $\\mathbf{C}$ through a decomposition into three submatrices (**of smaller dimensionality**):\n",
    "\n",
    "$$\\mathbf{C} \\approx \\mathbf{U \\sum V^T}$$\n",
    "\n",
    "<img src=\"https://simonpaarlberg.com/posts/2012-06-28-latent-semantic-analyses/box2.png\">\n",
    "\n",
    "NB. $=$ should be $\\approx$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(11, 2)\n",
      "(2,)\n",
      "(2, 5)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.utils.extmath import randomized_svd\n",
    "\n",
    "# reduce space to, say, 2 dimensions (for simplicity here)\n",
    "U, Sigma, VT = randomized_svd(coocurrence_matrix, \n",
    "                              n_components=2)\n",
    "print(U.shape)\n",
    "print(Sigma.shape)\n",
    "print(VT.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Visualizing the vector space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'times', '.', 'I', 'Cockatoos', 'Groningen', 'likes', 'enjoys', 'like', 'food', 'She', 'good'}\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD8CAYAAACfF6SlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl4VOX99/H3lyQEhJCAoILyEOCxlJAFyGJQJGEpIC2gAoJajQtFrNpSK1exbrhUcXmEqlir/kB+FgVFEKyoiIAIBSSRQMtmBFEhCEEMEDYTcj9/JExZApmQSSbkfF7XlStz7rnn3N9zIJ85c5+ZOeacQ0REvKVOsAsQEZHqp/AXEfEghb+IiAcp/EVEPEjhLyLiQQp/EREPUviLiHiQwl9ExIMU/iIiHhQa7AJOpWnTpi46OjrYZYiInFWysrJ2OeealdevxoZ/dHQ0mZmZwS5DROSsYmbf+NNP0z4iIh6k8Jdqs2XLFmJjY4Ndhoig8BcR8aQaO+cvwffoo48ydepUWrZsSdOmTUlMTKRXr16MHDmSAwcO0LZtWyZNmkTjxo3Jzs4usz0rK4tbbrmFc845h65duwZ7k0SklI78pUyZmZm88847rFq1ipkzZ/pOvt944408+eSTrFmzhri4OB5++OHTtt98880899xzLFu2LGjbIiInU/h72LurtnHZuAW0HvM+l41bwLurtvnuW7JkCQMHDqR+/fpERETQv39/9u/fT35+PmlpaQBkZGSwePFi9uzZ41f7DTfcUP0bKSJl0rSPR727ahv3zvw3BwuPALAt/yD3zvw3AFd2upBAXOHNOYeZVXo9IhJ4OvL3qKc/2ugL/qMOFh7h6Y82AtC1a1fee+89Dh06REFBAe+//z4NGjSgcePGfPbZZwC8/vrrpKWlERkZWWZ7VFQUkZGRLFmyBICpU6dW4xaKyOnoyN+jcvMPnrY9OTmZAQMGkJCQQKtWrUhKSiIyMpIpU6b4Tuy2adOGyZMnA5yyffLkyb4Tvn369KmejQuwv/zlL7zxxhuEhIRQp04d/v73vzN06FAyMzNp2rRpsMsTOSNWUy/gnpSU5PQJ36pz2bgFbCvjCeDCqPosHdMDgIKCAho2bMiBAwfo1q0bL7/8Mp07d67uUoNq2bJl3H333SxatIjw8HB27drFTz/9xKWXXqrwlxrJzLKcc0nl9dO0j0eN7tOO+mEhx7XVDwthdJ92vuURI0bQsWNHOnfuzKBBgzwX/ADbt2+nadOmhIeHA9C0aVNatGgBwPPPP0/nzp2Ji4tjw4YNAOzfv59bbrmF5ORkOnXqxOzZs4NWu8jp6Mjfw95dtY2nP9pIbv5BWkTVZ3SfdlzZ6cJgl1WjFBQU0LVrVw4cOECvXr0YOnQoaWlpREdH88c//pG77rqLF198kS+++IJXX32VP//5z8TExPDrX/+a/Px8UlJSWLVqFQ0aNAj2pohH+Hvkrzl/D7uy04WeD/t3vt/NE5u3s+1wIReGh3Fvm+YMuqCJ7/6GDRuSlZXFZ599xsKFCxk6dCjjxo0D4OqrrwYgMTGRmTNnAjBv3jzmzJnDM888A8ChQ4f49ttvad++fTVvmcjpKfzFs975fjf3bPyOg8Ulr363Hi7kno3fARz3BBASEkJ6ejrp6enExcUxZcoUAN9UUEhICEVFRUDJ21vfeecd2rVrh0hNpjl/8awnNm/3Bf9RB4sdT2ze7lveuHEjOTk5vuXs7GxatWp1ynX26dOH559/3vc5iVWrVgW4apHAUPiLZ207XFhue0FBARkZGcTExBAfH8+6desYO3bsKdf5wAMPUFhYSHx8PLGxsTzwwAOBLlskIAJywtfM+gJ/BUKAV51z407RbzDwNpDsnDvt2Vyd8JWqlvSvtWwt4wngovAwMi/tEISKRCqv2t7qaWYhwETgCiAGuNbMYsroFwH8DlhR2TFFAuHeNs2pX+f4r5+oX8e4t03zIFUkUn0CMe2TAnzlnNvsnPsJmAYMLKPfo8BTwKEAjClSaYMuaMIz7VpyUXgYRskR/zPtWh53slektgrEu30uBL47ZnkrcMmxHcysE9DSOfdPM7snAGOKBMSgC5oo7MWTAnHkX9bXNvpOJJhZHWA88MdyV2Q2wswyzSwzLy8vAKWJiEhZAhH+W4GWxyxfBOQesxwBxAKLzGwLkArMMbOTTkg45152ziU555KaNWsWgNJERKQsgQj/lcDFZtbazOoCw4A5R+90zu1xzjV1zkU756KB5cCA8t7tIyIiVafS4e+cKwLuBD4C1gNvOefWmtkjZjagsusXEZHAC8jXOzjn5gJzT2h78BR90wMxpoiInDl9wldExIMU/iIiHqTwFxHxIIW/iIgHKfw9KDw8nPbt23P11VczePBgAF577TXuvPPOCq0nOjqaXbt2VUWJIlLFdDEXDyosLGTu3Lm0bt062KWISJAo/D1m5MiROOcYMGAAAwYM4Pnnnyc6Opr9+/eTnJwMwLRp0/jtb39L8+bNqV+/PlFRUezZs4eioiLCwsIoLCykZcuW5Obmkp6eTlhYGIsXLyYiIiLIWyci/tK0j8e89NJLmBkLFy5k69at1KtXjzVr1jBo0CDmz5/PrFmzuPvuuxkxYgRr166lUaNGfPPNN6xcuZKOHTvy9ddfs2rVKr7//nsKCwtZtGgRn332GfXr1w/2polIBSj8a6GsRU/w4XsxzP+kLR++F0PWoifK7JeZmUlUVBQAMTEx7Nmzh7/85S80bdqUESNGAPCf//yHb775hri4OKZNm0adOnXYt28fV199NSEhIbz88svk5+cTGqoXkSJnE4V/LZO16Al2//Q/hDU4jBmENTjM7p/+p8wngBOv4hYaGkpBQQGHDv33kgvFxcWcd955LF26lHbt2rF8+XIiIiIYM2YMDRo04NChQ6SmprJhw4Yq3zYRCRyFfy2Tt+916oQeH+p1Qh15+14/qW9KSgr5+fkAbNiwgYYNGzJ79mx27drF+PHjAUhISMA5R6NGjejWrRvPPPMMAJMmTWLv3r387ne/IykpSeEvcpbRa/VaJvScw363jxo1ipkzZxIfH8/+/fvp2bMn7dq1Y+bMmfTv35+PP/6Yhg0bEhsbS3x8PIcOHeLgwYMsW7aM4uJiwsLCSEtLIz4+niuuuKKqN01EAkjhX8sUHQgnrMHJQV90INx3u7i4GICmTZuyd+/ek/qmp6ezb9++qitSRIJO0z61TLOIGyguOv7iasVFRrOIG4JUkYjURAr/WiYx/V6a1L2Vwv3hOAeF+8NpUvdWEtPvDXZpIlKDaNqnFioJeoW9iJyajvxFRDxI4S8i4kEKfxERD1L4i4h4kMJfRMSDFP4iIh6k8BcR8SCFv4iIByn8RUQ8SOEvIuJBCn8REQ9S+IuIeJDCX0TEgxT+IiIepPAXEfEghb+IiAcp/EVEPEjhLyLiQQp/EREPUviLiHiQwl9ExIMU/iIiHhSQ8Dezvma20cy+MrMxZdx/t5mtM7M1ZvaJmbUKxLgiInJmKh3+ZhYCTASuAGKAa80s5oRuq4Ak51w8MAN4qrLjiojImQvEkX8K8JVzbrNz7idgGjDw2A7OuYXOuQOli8uBiwIwroiInKFAhP+FwHfHLG8tbTuVW4EPyrrDzEaYWaaZZebl5QWgNBERKUsgwt/KaHNldjT7NZAEPF3W/c65l51zSc65pGbNmgWgNBERKUtoANaxFWh5zPJFQO6JncysF3AfkOacOxyAcUVE5AwF4sh/JXCxmbU2s7rAMGDOsR3MrBPwd2CAc25nAMYUEZFKqHT4O+eKgDuBj4D1wFvOubVm9oiZDSjt9jTQEHjbzLLNbM4pViciItUgENM+OOfmAnNPaHvwmNu9AjGOiIgEhj7hKyLiQQp/EREPUviLiHiQwl9ExIMU/iIiHuSp8M/Pz+fFF18EIDc3l8GDBwe5IhGR4PBs+Ldo0YIZM2YEuSIRkeDwVPiPGTOGTZs20bFjR4YMGUJsbCwAr732GldeeSX9+/endevWvPDCCzz77LN06tSJ1NRUdu/eDcCmTZvo27cviYmJXH755WzYsAGAt99+m9jYWBISEujWrVvQtk9ExG/OuRr5k5iY6ALt66+/dh06dDjp9uTJk13btm3d3r173c6dO12jRo3c3/72N+ecc6NGjXLjx493zjnXo0cP9+WXXzrnnFu+fLnr3r27c8652NhYt3XrVueccz/++GPA6xYR8ReQ6fzI2IB8wrcmeXfVNp7+aCO5+QdpEVWf0X3acWWn033DdInu3bsTERFBREQEkZGR9O/fH4C4uDjWrFlDQUEB//rXvxgyZIjvMYcPl3w/3WWXXcZNN93ENddcw9VXX101GyYiEkC1KvzfXbWNe2f+m4OFRwDYln+Qe2f+G6DcJ4Dw8HDf7Tp16viW69SpQ1FREcXFxURFRZGdnX3SY1966SVWrFjB+++/T8eOHcnOzubcc88N1GaJiARcrZrzf/qjjb7gP+pg4RGe/mgjABEREezbt++M1t2oUSNat27N22+/DZRMl61evRooORdwySWX8Mgjj9C0aVO+++67061KRCToatWRf27+wdO2n3vuuVx22WXExsbSvn37Cq9/6tSp3H777Tz22GMUFhYybNgwEhISGD16NDk5OTjn6NmzJwkJCZXajprkpZde4pxzzuHGG28MdikiEkBWcn6g5klKSnKZmZkVesxl4xawrYwngAuj6rN0TI9AlSYiUmOZWZZzLqm8frVq2md0n3bUDws5rq1+WAij+7QLUkU10z/+8Q9SUlLo2LEjt912G0eOHKFhw4bcd999JCQkkJqayo4dOwAYO3YszzzzDADZ2dmkpqYSHx/PVVddxY8//simTZvo3Lmzb905OTkkJiYCJW+tjYmJIT4+nnvuuaf6N1RETqlWhf+VnS7kiavjuDCqPkbJEf8TV8f59W4fr1i/fj3Tp09n6dKlZGdnExISwtSpU9m/fz+pqamsXr2abt268corr5z02BtvvJEnn3ySNWvWEBcXx8MPP0zbtm2JjIz0nQifPHkyN910E7t372bWrFmsXbuWNWvWcP/991f3porIadSqOX8oeQLwetjvX7WTvR9t4Uj+YUKiwmnUJ5oGnc4D4JNPPiErK4vk5GQADh48yHnnnUfdunX51a9+BUBiYiIff/zxcevcs2cP+fn5pKWlAZCRkeF72+vw4cOZPHkyzz77LNOnT+fzzz+nUaNG1KtXj+HDh/PLX/7St24RqRlq1ZG/lAR//swcjuSXfAbhSP5h8mfmsH9VyaWTnXNkZGSQnZ1NdnY2GzduZOzYsYSFhWFmAISEhFBUVOT3mIMGDeKDDz7gn//8J4mJiZx77rmEhoby+eefM2jQIN5991369u0b+I0VkTOm8K9l9n60BVdYfFybKyxm70dbAOjZsyczZsxg586SJ4Pdu3fzzTfflLveyMhIGjduzGeffQbA66+/7nsVUK9ePfr06cPtt9/OzTffDEBBQQF79uyhX79+TJgwoczPR4hI8NS6aR+vO3rEf6r2mJgYHnvsMXr37k1xcTFhYWFMnDjxtOs8+opgypQpjBw5kgMHDtCmTRsmT57s63P99dczc+ZMevfuDcC+ffsYOHAghw4dwjnH+PHjA7F5IhIgCv9aJiQqvMwngJCo/36CeejQoQwdOvS4+wsKCny3Bw8e7Pu66x9++IFWrVoB0LFjR5YvX17muEuWLOGWW24hJKTk3VbNmzfn888/r9zGiEiVUfjXMo36RJM/M+e4qR8Lq0OjPtEVXtcDDzzAihUrGDt27Gn7XXXVVWzatIkFCxZUeAwRCY5a9SEvKXG6d/uISO3m74e8dORfCzXodJ7CXkROS+/2ERHxIIW/iIgHKfxFRDxI4S8i4kEKfxERD1L4i4h4kMJfRMSDFP4iIh6k8BcR8SCFv4iIByn8RURqijVvwfhYGBtV8nvNW1U2lL7bR0SkJljzFrz3Oyg8WLK857uSZYD4awI+nI78RURqgk8e+W/wH1V4sKS9VL9+/cjNzQ3IcAEJfzPra2YbzewrMxtTxv3hZja99P4VZhYdiHFFRGqNPVvLbZ87dy4tWrQIyHCVDn8zCwEmAlcAMcC1ZhZzQrdbgR+dc/8XGA88WdlxRURqlciLKtZeSYE48k8BvnLObXbO/QRMAwae0GcgMKX09gygpx29MKyIiEDPByGs/vFtYfVL2qtAIML/QuC7Y5a3lraV2cc5VwTsAc49cUVmNsLMMs0sMy8vLwCliYicJeKvgf7PQWRLwEp+93+uSk72QmDe7VPWEfyJ14b0pw/OuZeBl6HkMo6VL01E5CwSf02Vhf2JAnHkvxVoeczyRcCJp6N9fcwsFIgEdgdgbBEROQOBCP+VwMVm1trM6gLDgDkn9JkDZJTeHgwscDX1yvEiIh5Q6Wkf51yRmd0JfASEAJOcc2vN7BEg0zk3B/gf4HUz+4qSI/5hlR1XRETOXEA+4eucmwvMPaHtwWNuHwKGBGIsERGpPH3CV0TEgxT+IiIepPAXEfEghb+IiAcp/EVEPEjhLyLiQQp/EREPUviLiHiQwl9ExIMU/iIiHqTwFwmgHTt2cN1119GmTRsSExPp0qULs2bNCsi6hw8fzrp16wKyLpGAfLePiIBzjiuvvJKMjAzeeOMNAL755hvmzDn+S26LiooIDa34n96rr74akDpFQEf+IgGzYMEC6taty8iRI31trVq14q677uK1115jyJAh9O/fn969e+OcY/To0cTGxhIXF8f06dMBWLRoEenp6QwePJif//znXH/99Rz99vP09HQyMzMBaNiwIffddx8JCQmkpqayY8cOADZt2kRqairJyck8+OCDNGzY0FfL008/TXJyMvHx8Tz00EMAbNmyhfbt2/Ob3/yGDh060Lt3bw4ePFgt+0uCS+EvEiBr166lc+fOp7x/2bJlTJkyhQULFjBz5kyys7NZvXo18+fPZ/To0Wzfvh2AVatWMWHCBNatW8fmzZtZunTpSevav38/qamprF69mm7duvHKK68A8Pvf/57f//73rFy5khYtWvj6z5s3j5ycHD7//HOys7PJyspi8eLFAOTk5HDHHXewdu1aoqKieOeddwK5W6SGUviL+OnLFd8z5c9LmThyAVP+vJQvV3x/2v533HEHCQkJJCcnA/CLX/yCJk2aALBkyRKuvfZaQkJCOP/880lLS2PlypUApKSkcNFFF1GnTh06duzIli1bTlp33bp1+dWvfgVAYmKir8+yZcsYMqTk29Ovu+46X/958+Yxb948OnXqROfOndmwYQM5OTkAtG7dmo4dO560LqndNOcv4ocvV3zPwqkbKPqpGICC3YdZOHUDAD+75AIAOnTocNxR88SJE9m1axdJSUkANGjQwHff6S5kFx4e7rsdEhJCUVHRSX3CwsIws9P2OZZzjnvvvZfbbrvtuPYtW7acNJ6mfbxBR/4iflg2e5Mv+I8q+qmYZbM3+ZZ79OjBoUOH+Nvf/uZrO3DgQJnr69atG9OnT+fIkSPk5eWxePFiUlJSKl1namqq7wlo2rRpvvY+ffowadIkCgoKANi2bRs7d+6s9Hhy9lL4i/ihYPfhctvNjHfffZdPP/2U1q1bk5KSQkZGBk8++eRJj7vqqquIj48nISGBHj168NRTT3HBBRdUus4JEybw7LPPkpKSwvbt24mMjASgd+/eXHfddXTp0oW4uDgGDx7Mvn37Kj2enL2spl5HPSkpyR19Z4NIsE3589IynwAaNgkn4/HLglBR2Q4cOED9+vUxM6ZNm8abb77J7Nmzg12WVCMzy3LOJZXXT3P+In7oMrDtcXP+AKF169BlYNsgVnWyrKws7rzzTpxzREVFMWnSpGCXJDWUwl/ED0dP6i6bvYmC3Ydp2CScLgPb+tprissvv5zVq1cHuww5Cyj8Rfz0s0suqHFhL3KmdMJXRMSDFP4iIh6k8BcR8SCFv4iIByn8RUQ8SOEvIuJBCn8REQ9S+IuIeJDCX0TEgxT+IiIepPAXEfEghb+IiAcp/EVEPEjhLyLiQQp/D/v+++8ZNmwYbdu2JSYmhn79+vHll19WaB3p6elU5IprEyZMOOV1bUWk+lQq/M2siZl9bGY5pb8bl9Gno5ktM7O1ZrbGzIZWZkwJDOccV111Fenp6WzatIl169bx+OOPs2PHjiodV+EvUjNU9sh/DPCJc+5i4JPS5RMdAG50znUA+gITzCyqkuNKJS1cuJCwsDBGjhzpa+vYsSNdu3Zl9OjRxMbGEhcXx/Tp0333P/XUU8TFxZGQkMCYMcf/UxcXF5ORkcH9998PwO23305SUhIdOnTgoYceAuC5554jNzeX7t270717dwDefPNN4uLiiI2N5U9/+pNvfWW1HzlyhJtuuslX2/jx46tm54h4gXPujH+AjUDz0tvNgY1+PGY1cHF5/RITE51Unb/+9a9u1KhRJ7XPmDHD9erVyxUVFbnvv//etWzZ0uXm5rq5c+e6Ll26uP379zvnnPvhhx+cc86lpaW5ZcuWuWHDhrnHHnvMt56j9xcVFbm0tDS3evVq55xzrVq1cnl5ec4557Zt2+Zatmzpdu7c6QoLC1337t3drFmzTtmemZnpevXq5Rvjxx9/rJqdI3IWAzKdH/ld2SP/851z20ufRLYD552us5mlAHWBTZUcV6rIkiVLuPbaawkJCeH8888nLS2NlStXMn/+fG6++WbOOeccAJo0aeJ7zG233UZsbCz33Xefr+2tt96ic+fOdOrUibVr17Ju3bqTxlq5ciXp6ek0a9aM0NBQrr/+ehYvXnzK9jZt2rB582buuusuPvzwQxo1alT1O0Sklio3/M1svpn9p4yfgRUZyMyaA68DNzvnik/RZ4SZZZpZZl5eXkVWL2XY89575PToyfr2MeT06Mme997z3dehQweysrJOekzJgcPJnHOYWZn3XXrppSxcuJBDhw4B8PXXX/PMM8/wySefsGbNGn75y1/67vN3rLI0btyY1atXk56ezsSJExk+fHiZ/USkfOWGv3Oul3Mutoyf2cCO0lA/Gu47y1qHmTUC3gfud84tP81YLzvnkpxzSc2aNTuzLRKgJPi3P/AgRbm54BxFublsf+BB3xNAjx49OHz4MK+88orvMStXrqRx48ZMnz6dI0eOkJeXx+LFi0lJSaF3795MmjTJd7J29+7dvsfdeuut9OvXjyFDhlBUVMTevXtp0KABkZGR7Nixgw8++MDXNyIign379gFwySWX8Omnn7Jr1y6OHDnCm2++SVpa2inbd+3aRXFxMYMGDeLRRx/liy++qI5dKVIrhVby8XOADGBc6e/ZJ3Yws7rALOB/nXNvV3I88dPO8RNwJxxtu0OH2Dl+ApH9+2NmzJo1i1GjRjFu3Djq1atHdHQ0EyZMoKCggISEBMyMp556igsuuIC+ffuSnZ1NUlISdevWpV+/fjz++OO+dd99993s2bOHG264galTp9KpUyc6dOhAmzZtuOyyy3z9RowYwRVXXEHz5s1ZuHAhTzzxBN27d8c5R79+/Rg4sOQFZVntq1ev5uabb6a4uNjXR0TOjJ3qJbZfDzY7F3gL+D/At8AQ59xuM0sCRjrnhpvZr4HJwNpjHnqTcy77dOtOSkpyFXn/uBxvffsYKOvf1oz260+efxeR2sHMspxzSeX1q9SRv3PuB6BnGe2ZwPDS2/8A/lGZcaTiQps3L5nyKaNdRESf8K2lzvvDKKxeveParF49zvvDqCBVJCI1SWXn/KWGiuzfHyiZ+y/avp3Q5s057w+jfO0i4m0K/1ossn9/hb2IlEnTPiIiHqTwFxHxIIW/iIgHKfxFRDxI4S8i4kEKfxERD1L4i4h4kMJfRMSDFP4iIh6k8BcR8SCFv4iIByn8RUQ8SOEvIuJBCn8REQ9S+IuIeJDCX0TEgxT+IiIepPAXEfEghb+IiAcp/EVEPEjhLyLiQQp/EREPUviLiHiQwl9ExIMU/iIiHqTwFxHxIIW/iIgHKfxFRDxI4S8i4kEKfxERD1L4i4h4kMJfJMDMDIDc3FwGDx4MwGuvvcadd94ZzLJEjqPwF6kiLVq0YMaMGcEuQ6RMCn+RKrJlyxZiY2NPan///ffp0qULu3btIi8vj0GDBpGcnExycjJLly4NQqXiRaGVebCZNQGmA9HAFuAa59yPp+jbCFgPzHLO6fWveNKsWbN49tlnmTt3Lo0bN+a6667jD3/4A127duXbb7+lT58+rF+/PthligdUKvyBMcAnzrlxZjamdPlPp+j7KPBpJccTOWstXLiQzMxM5s2bR6NGjQCYP38+69at8/XZu3cv+/btIyIiIlhlikdUNvwHAumlt6cAiygj/M0sETgf+BBIquSYIkE1Zf4U1q9YT3hhOIfDDtP+kvZk9Moo93Ft2rRh8+bNfPnllyQllfwZFBcXs2zZMurXr1/VZYscp7Jz/uc757YDlP4+78QOZlYH+H/A6EqOJRJ0U+ZPIWdpDvUK62EY9QrrkbM0hynzp5T72FatWjFz5kxuvPFG1q5dC0Dv3r154YUXfH2ys7OrrHaRY5Ub/mY238z+U8bPQD/H+C0w1zn3nR9jjTCzTDPLzMvL83P1ItVn/Yr1hLrjXzCHulDWr/Bvnr5du3ZMnTqVIUOGsGnTJp577jkyMzOJj48nJiaGl156qSrKFjmJOefO/MFmG4F059x2M2sOLHLOtTuhz1TgcqAYaAjUBV50zo053bqTkpJcZmbmGdcmUhUeGvsQhp3U7nA8PPbhIFQkcjwzy3LOlTu9XtlpnznA0cnODGD2iR2cc9c75/6Pcy4auAf43/KCX6SmOhx2uELtIjVVZcN/HPALM8sBflG6jJklmdmrlS1OpKZpf0l7iqzouLYiK6L9Je2DVJHImanUtE9V0rSP1FRn+m4fkerg77RPZd/qKeI5Gb0yoFewqxCpHH29g4iIByn8RUQ8SOEvIuJBCn8REQ9S+IuIeFCNfaunmeUB35zQ3BTYFYRyAuls3wbVH1yqP7jOhvpbOeealdepxoZ/Wcws05/3r9ZkZ/s2qP7gUv3BdbbXfyxN+4iIeJDCX0TEg8628H852AUEwNkF6MvRAAADtklEQVS+Dao/uFR/cJ3t9fucVXP+IiISGGfbkb+IiARAjQx/M+trZhvN7KvSC8OfeH+4mU0vvX+FmUVXf5Wn5kf93czsCzMrMrPBwajxdPyo/24zW2dma8zsEzNrFYw6T8ePbRhpZv82s2wzW2JmMcGo81TKq/+YfoPNzJlZjXoHih/7/yYzyyvd/9lmNjwYdZ6KP/vfzK4p/TtYa2ZvVHeNleacq1E/QAiwCWhDyVW/VgMxJ/T5LfBS6e1hwPRg113B+qOBeOB/gcHBrvkM6u8OnFN6+/aatP8rsA2Njrk9APgw2HVXpP7SfhHAYmA5kBTsuiu4/28CXgh2rZWo/2JgFdC4dPm8YNdd0Z+aeOSfAnzlnNvsnPsJmAaceL3ggcDRK2bPAHqa2cnX1guOcut3zm1xzq2h5NKWNY0/9S90zh0oXVwOXFTNNZbHn23Ye8xiA6Amnfzy528A4FHgKeBQdRbnB3/rr6n8qf83wETn3I8Azrmd1VxjpdXE8L8QOPZi71tL28rs45wrAvYA51ZLdeXzp/6arKL13wp8UKUVVZxf22Bmd5jZJkoC9HfVVJs/yq3fzDoBLZ1z/6zOwvzk7/+hQaVThzPMrGX1lOYXf+r/GfAzM1tqZsvNrG+1VRcgNTH8yzqCP/GozJ8+wVKTa/OH3/Wb2a+BJODpKq2o4vzaBufcROdcW+BPwP1VXpX/Tlu/mdUBxgN/rLaKKsaf/f8eEO2ciwfm899X8jWBP/WHUjL1kw5cC7xqZlFVXFdA1cTw3wocexRwEZB7qj5mFgpEArurpbry+VN/TeZX/WbWC7gPGOCcq2lXL6/ov8E04Moqrahiyqs/AogFFpnZFiAVmFODTvqWu/+dcz8c8//mFSCxmmrzh78ZNNs5V+ic+xrYSMmTwdkj2CcdyjjZEgpsBlrz35MtHU7ocwfHn/B9K9h1V6T+Y/q+Rs074evP/u9EyQmxi4NdbyW24eJjbvcHMoNd95n8Hyrtv4iadcLXn/3f/JjbVwHLg113BevvC0wpvd2Ukmmic4Nde4W2M9gFnGLn9wO+LA2Y+0rbHqHkKBOgHvA28BXwOdAm2DVXsP5kSo4c9gM/AGuDXXMF658P7ACyS3/mBLvmM9iGvwJrS+tfeLpwrYn1n9C3RoW/n/v/idL9v7p0//882DVXsH4DngXWAf8GhgW75or+6BO+IiIeVBPn/EVEpIop/EVEPEjhLyLiQQp/EREPUviLiHiQwl9ExIMU/iIiHqTwFxHxoP8P3h3W3eu7MjgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "size of U: (11, 2)\n",
      "vector for 'likes': [0.15663919 0.32849558]\n",
      "vector for 'enjoys': [0.15680191 0.09115486]\n",
      "vector for 'good': [0.17807516 0.45032677]\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "print(vocab)\n",
    "for word in vocab:\n",
    "    i=w2i[word]\n",
    "    plt.text(U[i,0]+0.01,U[i,1], word)\n",
    "    plt.plot(U[i,0],U[i,1], 'o')\n",
    "plt.show()\n",
    "print(\"size of U:\", U.shape)\n",
    "print(\"vector for 'likes':\", U[w2i[\"likes\"]])\n",
    "print(\"vector for 'enjoys':\", U[w2i[\"enjoys\"]])\n",
    "print(\"vector for 'good':\", U[w2i[\"good\"]])\n",
    "#for w in vocab:\n",
    "#    print(\"vector for '{}': {}\".format(w, U[w2i[w]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Similarity\n",
    "\n",
    "**cosine** similarity \n",
    "\n",
    "<img src=\"https://simonpaarlberg.com/posts/2012-06-28-latent-semantic-analyses/eq1.png\">\n",
    "<img src=\"https://simonpaarlberg.com/posts/2012-06-28-latent-semantic-analyses/vector_example2.png\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**Exercise**: Calculate the cosine distance between the words *good* and *enjoys* as well as *enjoys* and *likes*. (Hint: you can use the *cosine* **distance** function from *scipy.spatial.distance*, notice it is 1 minus cosine similarity). What is the distance between a word and itself?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cosine distances:\n",
      "good<>enjoys:     0.21\n",
      "enjoys<>likes: 0.17\n",
      "good<>good: 0.00\n"
     ]
    }
   ],
   "source": [
    "## solution:\n",
    "from scipy.spatial.distance import cosine\n",
    "\n",
    "v_likes = U[w2i[\"likes\"]]\n",
    "v_enjoys = U[w2i[\"enjoys\"]]\n",
    "v_good = U[w2i[\"good\"]]\n",
    "\n",
    "print(\"cosine distances:\")\n",
    "print(\"good<>enjoys:     {0:.2f}\".format(cosine(v_good, v_enjoys)))\n",
    "print(\"enjoys<>likes: {0:.2f}\".format(cosine(v_enjoys, v_likes)))\n",
    "print(\"good<>good: {0:.2f}\".format(cosine(v_good, v_good)))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Deep learning approach: Directly learning word vectors (embeddings)\n",
    "\n",
    "* SVD: computation cost scales quadratically with size of co-occurence matrix; difficult to integrate new words\n",
    "* **Idea**: directly learn word vectors (e.g. word2vec)\n",
    "    * NLP (almost) from Scratch (Collobert & Weston, 2008)\n",
    "    * word2vec (Mikolov et al, 2013)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Main idea of word2vec\n",
    "\n",
    "* instead of capturing co-occurence statistics of words\n",
    "* **predict context** (surrounding words of every word); in particular, predict words in a window of length $m$ around the current word"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$o$ is the outside word (context), $c$ is the current center word; \n",
    "\n",
    "Maximize the probability of a word in the context ($o$) given the current word $c$:\n",
    "\n",
    "$$p(o|c) = \\frac{exp(u_o^T v_c)}{\\sum_{w=1}^W exp(u_w^T v_c)}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<img src=\"http://www.gabormelli.com/RKB/images/a/a6/skip-gram_NNLM_architecture.150216.jpg\">\n",
    "\n",
    "At the end you can read off the embedding vector from the Embedding layer! Voilà!\n",
    "\n",
    "NB. denominator $\\sum$ over all words! In practice, *negative sampling* is used (randomly choose a word which is not in context as a negative sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In deep learning we represent words as vectors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**a) sparse representation vs b) dense representation**  (Figure 1 in Yoav Goldberg's primer)\n",
    "<img src=\"pics/sparsevsdense.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Traditional vs deep learning approach to feature extraction (representations)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The common pipeline of extracting features **for an NLP model with a Neural Network** then becomes:\n",
    "\n",
    "* extract a set of core linguistic features $f_1,..f_n$\n",
    "* define a **vector** for **each feature** (lookup Embedding table)\n",
    "* **combine** vectors of features to get the vector representation for the **instance** $\\mathbf{x}$ (**dense representation**)\n",
    "* use $\\mathbf{x}$ as representation for an instance, train the model\n",
    "\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Lets compare this to our traditional approach - the common pipeline of extracting features for an NLP model is:\n",
    "\n",
    "* extract a set of core linguistic features $f_1,..f_n$\n",
    "* define a vector whose length is the total number of features with a 1 at position k if the k-th feature is active; this feature vector represents the **instance** $\\mathbf{x}$  (**sparse representation**, n-hot encoding)\n",
    "* use $\\mathbf{x}$ as representation for an instance, train the model\n",
    "\n",
    "Now it should be clear why it is called sparse vs dense feature representation.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### How do you combine different feature vector representations?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "In an NLP application, $\\mathbf{x}$ is usually composed of various embedding vectors.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Following the notation in Goldberg (2015), chapter 4, let's use the function $c(\\cdot)$ as **feature combiner** that creates our input embeddings layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "A common choice for $c$ is **concatenation**:\n",
    "\n",
    "$\\mathbf{x} = c(f_1, f_2, f_3) = [v(f_1); v(f_2); v(f_3)]$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Alternatively, $c$ could be the **sum of the embedding vectors**:\n",
    "\n",
    "$\\mathbf{x} = c(f_1, f_2, f3) = [v(f_1)+v(f_2)+v(f_3)] $\n",
    "\n",
    "or its **mean**:\n",
    "\n",
    "$\\mathbf{x} = c(f_1, f_2, f3) = [mean(v(f_1),v(f_2),v(f_3))] $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "In many papers $v$ is often referred to as the embeddings layer or lookup layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Our example from before with explicit input representation\n",
    "\n",
    "For instance, let us explicitly state the input representation. Suppose we use the concatentation operator, then our network above becomes:\n",
    "\n",
    "<img src=\"pics/nn.png\" width=300> \n",
    "\n",
    "since: \n",
    "$\\mathbf{x} = c(f_1, f_2, f_3) = [v(f_1); v(f_2); v(f_3)] $\n",
    "\n",
    "then: \n",
    "\n",
    "$NN_{MLP1}(\\mathbf{x})=g(\\mathbf{[v(f_1); v(f_2); v(f_3)]W^1+b^1})\\mathbf{W^2}+\\mathbf{b^2}$\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "As a computational graph:\n",
    "<img src=\"pics/yg-compgraph2.png\">\n",
    "Note, as in the previous notebook, that the dimensions in the computational graph do not correspond to those in the network depicted above"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The values of the *embedding vectors* are treated as model parameters and trained together with the other parameters of the model (weights)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Unrolled graph with concrete input, expected output, and loss node, (Goldberg Figure 3c):\n",
    "<img src=\"pics/yg-compgraph3.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Example: animacy classification\n",
    "\n",
    "Exercise: \n",
    "\n",
    "* add an embedding layer to the animacy classification example. For now use a simple concatenation as representation. What performance do you get?\n",
    "\n",
    "* add code that reads off the embedding layer from the network and stores it in a file \"vectors.txt\". Once you have this embedding vector you can inspect it (e.g. find nearest neighbors) as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Florida', 0.9196957945823669),\n",
       " ('Arlington', 0.9070820808410645),\n",
       " ('California', 0.8937997817993164),\n",
       " ('Buffalo', 0.8546850085258484),\n",
       " ('Canada', 0.8518165349960327),\n",
       " ('Japan', 0.8483419418334961),\n",
       " ('Europe', 0.8347345590591431),\n",
       " ('Houston', 0.8322141170501709),\n",
       " ('Virginia', 0.8314700126647949),\n",
       " ('France', 0.8283012509346008)]"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# once we have read off the embeddings after training the animacy classifier, and \n",
    "# stored them in file 'vectors.txt' we load it for inspection\n",
    "from gensim.models.keyedvectors import KeyedVectors\n",
    "w2v = KeyedVectors.load_word2vec_format('./vectors.txt', binary=False)\n",
    "w2v.most_similar(positive=['Texas'])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('elect', 0.851630449295044),\n",
       " ('word', 0.8355662822723389),\n",
       " ('letting', 0.8245967030525208),\n",
       " ('teach', 0.8199971914291382),\n",
       " ('swaying', 0.81275475025177),\n",
       " ('parks', 0.811647891998291),\n",
       " ('finally', 0.7990972995758057),\n",
       " ('wishes', 0.793840765953064),\n",
       " ('repainted', 0.7927937507629395),\n",
       " ('jus-', 0.787428617477417)]"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w2v.most_similar(positive=['send'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "These vectors are not traditional word vectors learned with word2vec (i.e. skip-gram or CBOW model). Instead we read them off from our animacy classifier (they are not trained with the word2vec objective, but are a by-product of the classifier). Nevertheless this shows us that we can also get embeddings from a neural network with dense (embedding) inputs!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Word Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "So, in deep learning approaches to NLP words are represented as dense vectors. Where do these word vectors (embeddings) come from?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* **off-the-shelf embeddings**: you can also use trained, available embeddings (estimated with e.g. *word2vec*, *GLoVe* or *FastText*) and *initialize* the embedding layer of the network with your pretrained (unsupervised) word embeddings\n",
    "* **task-specific embeddings**: you could also train your embeddings from scratch with the data for your task. In this case, the vectors are typically **randomly initialized** (small numbers around 0) and *trained with the network*. At the end you can read them off the network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Remember, today we have seen tree ways to get embeddings:\n",
    "\n",
    "1. Traditional methods (also called 'count' methods): SVD on a co-occurence matrix (=LSA)\n",
    "2. Neural method #1 (also called 'predict' methods): e.g. word2vec (train on a large unlabeled corpus)\n",
    "3. Neural method #2 (also a 'predict' method, but task-specific): train your embeddings on your supervised task, read them off at the end (typically less used as you will have less supervised training data, it's easier to get loads of unlabeled text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Inputs of different lengths\n",
    "\n",
    "In our animacy classification example we have made one simplification: the input is always of the same size (namely, 5 words). \n",
    "\n",
    "However, in NLP we seldom have fixed size inputs; sentences are of different length. The neural network however needs inputs of fixed size. So how to deal with this?\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "* create an input of fixed size, e.g. using the mean embedding vector\n",
    "* use a model that can deal with variable size inputs, e.g. a recurrent neural network (depending on the deep learning toolkit you use, you might still need to *pad* sequences to a fixed length)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### References\n",
    "\n",
    "* Yoav Goldberg's primer (chapters 2 and 5): [A Primer on Neural Network Models for Natural Language Processing](http://arxiv.org/abs/1510.00726)\n",
    "* D. Rao & B. McMahan's NLP with PyTorch (chapter 5).\n",
    "* Simon Paarlberg's [blog on LSA](https://simonpaarlberg.com/post/latent-semantic-analyses/)\n",
    "* Richard Socher's [lecture on word vectors](https://www.youtube.com/watch?v=xhHOL3TNyJs)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}