RNN.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# reveal.js presentation configuration\n",
    "from notebook.services.config import ConfigManager\n",
    "\n",
    "cm = ConfigManager()\n",
    "cm.update('livereveal', {\n",
    "              'theme': 'league',\n",
    "              'transition': 'fade',\n",
    "              'center': 'false',\n",
    "              'overview' : 'true',\n",
    "              'start_slideshow_at': 'selected'\n",
    "})\n",
    "\n",
    "# imports\n",
    "import theano\n",
    "from theano import tensor\n",
    "import codecs\n",
    "import numpy\n",
    "import sys\n",
    "from blocks import initialization\n",
    "from blocks import roles\n",
    "from blocks.model import Model\n",
    "from blocks.bricks import Linear, NDimensionalSoftmax\n",
    "from blocks.bricks.parallel import Fork\n",
    "from blocks.bricks.recurrent import GatedRecurrent\n",
    "from blocks.bricks.lookup import LookupTable\n",
    "from blocks.filter import VariableFilter\n",
    "from blocks.extensions.saveload import load_parameter_values\n",
    "from blocks.bricks import NDimensionalSoftmax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Language modeling with RNN\n",
    "\n",
    "[Fabio A. González](http://dis.unal.edu.co/~fgonza/), Universidad Nacional de Colombia"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Setup\n",
    "* Training data: Biblia Reina Valera 1960\n",
    "* Software:\n",
    "  * [Blocks](https://github.com/mila-udem/blocks): \"Blocks is a framework that helps you build neural network models on top of Theano\"\n",
    "  * [Theano](http://deeplearning.net/software/theano/): \"Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of chars: 978848\n",
      "Vocabulary size: 85\n"
     ]
    }
   ],
   "source": [
    "# Load training file to get vocabulary\n",
    "text_file = 'biblia.txt' # input file\n",
    "with codecs.open(text_file, 'r', 'utf-8') as f:\n",
    "    data = f.read()\n",
    "    \n",
    "chars = list(set(data))\n",
    "vocab_size = len(chars)\n",
    "char_to_ix = {ch: i for i, ch in enumerate(chars)}\n",
    "ix_to_char = {i: ch for i, ch in enumerate(chars)}\n",
    "print \"Total number of chars:\", len(data)\n",
    "print \"Vocabulary size:\", vocab_size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s será medido. 3 ¿Y por qué miras la paja que está en el ojo de tu hermano, y no echas de ver la viga que está en tu propio ojo? 4 ¿O cómo dirás a tu hermano: Déjame sacar la paja de tu ojo, y he aquí la viga en el ojo tuyo? 5 ¡Hipócrita! saca primero la viga de tu propio ojo, y entonces verás bien para sacar la paja del ojo de tu hermano.\r\n",
      "\r\n",
      "6 No deis lo santo a los perros, ni echéis vuestras perlas delante de los cerdos, no sea que las pisoteen, y se vuelvan y os despedacen.\r\n",
      "\r\n",
      "La oración, y la regla de oro\r\n",
      "\r\n",
      "(Lc. 11.9-13; 6.31)\r\n",
      "\r\n",
      "7 Pedid, y se os dará; buscad, y hallaréis; llamad, y se os abrirá. 8 Porque todo aquel que pide, recibe; y el que busca, halla; y al que llama, se le abrirá. 9 ¿Qué hombre hay de vosotros, que si su hijo le pide pan, le dará una piedra? 10 ¿O si le pide un pescado, le dará una serpiente? 11 Pues si vosotros, siendo malos, sabéis dar buenas dádivas a vuestros hijos, ¿cuánto más vuestro Padre que está en los cielos dará buenas cosas a los que le pidan? 12 \n"
     ]
    }
   ],
   "source": [
    "print data[21000:22000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Network architecture\n",
    "\n",
    "<img src=\"rnn_architecture.jpg\" width= 400>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Define the layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# Define the model structure\n",
    "embedding_size = 256 # number of hidden units per layer\n",
    "\n",
    "# Input\n",
    "lookup = LookupTable(length=vocab_size, dim=embedding_size)\n",
    "\n",
    "# Layer 1\n",
    "fork1 = Fork(output_names=['linear1', 'gates1'],\n",
    "             input_dim=embedding_size, output_dims=[embedding_size, embedding_size * 2])\n",
    "fork1.name = 'fork1'\n",
    "grnn1 = GatedRecurrent(dim=embedding_size)\n",
    "grnn1.name = 'grnn1'\n",
    "\n",
    "# Layer 2\n",
    "fork2 = Fork(output_names=['linear2', 'gates2'],\n",
    "             input_dim=embedding_size, output_dims=[embedding_size, embedding_size * 2])\n",
    "fork2.name = 'fork2'\n",
    "grnn2 = GatedRecurrent(dim=embedding_size)\n",
    "grnn2.name = 'grnn2'\n",
    "\n",
    "# Softmax layer\n",
    "hidden_to_output = Linear(name='hidden_to_output', input_dim=embedding_size,\n",
    "                          output_dim=vocab_size)\n",
    "softmax = NDimensionalSoftmax()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Connect the layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# Propagate x until top brick to get y_hat predictions\n",
    "x = tensor.imatrix('features')  # input\n",
    "y = tensor.imatrix('targets')   # output\n",
    "embedding = lookup.apply(x)\n",
    "linear1, gates1 = fork1.apply(embedding)\n",
    "h1 = grnn1.apply(linear1, gates1)\n",
    "h1.name = 'h1'\n",
    "linear2, gates2 = fork2.apply(h1)\n",
    "h2 = grnn2.apply(linear2, gates2)\n",
    "h2.name = 'h2'\n",
    "linear3 = hidden_to_output.apply(h2)\n",
    "linear3.name = 'linear3'\n",
    "y_hat = softmax.apply(linear3, extra_ndim=1)\n",
    "y_hat.name = 'y_hat'\n",
    "\n",
    "# COST\n",
    "cost = softmax.categorical_cross_entropy(y, linear3, extra_ndim=1).mean()\n",
    "cost.name = 'cost'\n",
    "\n",
    "model = Model(cost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Load parameters and build Theano graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# Load model parameters from a file\n",
    "model_file_name = 'model2.pkl'\n",
    "model_params = model.get_parameter_dict().keys()\n",
    "param_vals = {k:v for k,v in load_parameter_values(model_file_name).iteritems() if k in model_params}\n",
    "model.set_parameter_values(param_vals)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The output file is available at theano_graph.svg\n"
     ]
    }
   ],
   "source": [
    "# Define Theano graph\n",
    "y, x = model.inputs\n",
    "softmax = NDimensionalSoftmax()\n",
    "linear_output = [v for v in model.variables if v.name == 'linear3'][0]\n",
    "y_hat = softmax.apply(linear_output, extra_ndim=1)\n",
    "predict = theano.function([x], y_hat)\n",
    "theano.printing.pydotprint(predict, outfile=\"theano_graph.svg\", format = 'svg', var_with_name_simple=True)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"theano_graph.svg\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# Sampling function\n",
    "def sample(nchars, text, temperature=1.0, seed=None):\n",
    "    # represent the entry text as an array of indices \n",
    "    text = ''.join([ch for ch in text if ch in char_to_ix])\n",
    "    x_curr = numpy.expand_dims(numpy.array([char_to_ix[ch] for ch in text], dtype='uint8'), axis=1)\n",
    "    if seed:\n",
    "        numpy.random.seed(seed)\n",
    "    sample_string = '' + text\n",
    "    for _ in range(nchars):\n",
    "        probs = predict(x_curr).squeeze()[-1].astype('float64') # predicts the probability of the next char\n",
    "        # based on the temperature decides whether to sample from the predicted probability distribution\n",
    "        # or choose the char with the maximum probability\n",
    "        if numpy.random.binomial(1, temperature) == 1: \n",
    "            probs = probs / probs.sum()\n",
    "            sample = numpy.random.multinomial(1, probs).nonzero()[0][0]\n",
    "        else:\n",
    "            sample = probs.argmax()\n",
    "        #sys.stdout.write('.')\n",
    "        sample_string += ix_to_char[sample]\n",
    "        if x_curr.shape[0] < 50:\n",
    "            x_curr = numpy.roll(x_curr, -1)\n",
    "            x_curr[-1] = sample\n",
    "        else:\n",
    "            x_curr = numpy.vstack((x_curr, [sample])).astype('int32')\n",
    "    sys.stdout.write('\\n')\n",
    "    return sample_string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Sampling from the model\n",
    "\n",
    "* The model calculates the probability of the next word given the previous words:  \n",
    "$$P(w_t | w_{t-1}, w_{t-2},\\dots, w_{1})$$\n",
    "* We sample from the model using this conditional probability\n",
    "  ```python\n",
    "  for i in [1..n]:\n",
    "      P = predict_next() \n",
    "      bin_var = sample_binomial(temperature)\n",
    "      if bin_var:\n",
    "          w_i = sample_multinomial(P) \n",
    "      else:\n",
    "          w_i = P.argmax() \n",
    "  ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Sampling from the model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Y Jesús dijo: ¿Qué harán los principales sacerdotes y de los que estaban en el camino a todos los que estaban en el camino a todos los que estaban el camino, y los que con él estaban lanzado en el templo de Dios, y de la tierra de la circuncisión es su carne, la cual es mi corazón estar con ellos, y los de los tributos para presentar a los que estaban en el camino a todos los que estaban en el camino a todos los misterios de la tierra de los cielos se apartan a los que estaban en el corazón de los hombres q\n"
     ]
    }
   ],
   "source": [
    "print sample(500, u\"Y Jesús dijo\", temperature = 0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Probability of a text\n",
    "* The probability of a text is:  \n",
    "$$P(w_1, \\dots, w_n) = P(w_1)\\prod_{i=2}^{n}\\ P(w_i | w_{i-1},\\dots, w_{1})$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# Function to calculate the probability of a text\n",
    "def log_likelihood(text):\n",
    "    text = ''.join([ch for ch in text if ch in char_to_ix])\n",
    "    x_curr = numpy.expand_dims(numpy.array([char_to_ix[ch] for ch in text], dtype='uint8'), axis=1)\n",
    "    probs = predict(x_curr).squeeze()\n",
    "    return sum([numpy.log(probs[i,c]) for i,c in enumerate(x_curr[1:].flatten())])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Most likely phrases from a bag of words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "27.0533799628 hombre hombre a la ama\n",
      "27.0533799628 hombre hombre a la ama\n",
      "28.0204834143 hombre hombre ama a la\n",
      "28.0204834143 hombre hombre ama a la\n",
      "28.9577404847 a hombre hombre la ama\n",
      "28.9577404847 a hombre hombre la ama\n",
      "29.6325383139 hombre a hombre la ama\n",
      "29.6325383139 hombre a hombre la ama\n",
      "30.1323473039 ama a hombre hombre la\n",
      "30.1323473039 ama a hombre hombre la\n",
      "30.8650613899 hombre hombre la ama a\n",
      "30.8650613899 hombre hombre la ama a\n",
      "31.1142256059 a hombre la hombre ama\n",
      "31.1142256059 a hombre la hombre ama\n",
      "31.2628376472 ama a hombre la hombre\n",
      "31.2628376472 ama a hombre la hombre\n",
      "31.4612043043 hombre ama a hombre la\n",
      "31.4612043043 hombre ama a hombre la\n",
      "31.7704667995 hombre hombre ama la a\n",
      "31.7704667995 hombre hombre ama la a\n"
     ]
    }
   ],
   "source": [
    "from itertools import permutations\n",
    "bow =  [u' ', 'hombre', 'ama', 'la', 'a']\n",
    "perms = [' '.join(perm) for perm in permutations(bow)]\n",
    "for p, t in sorted([(-log_likelihood(text),text) for text in perms])[:20]:\n",
    "    print p, t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Least likely phrases "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "41.9514358657 mujer a la ama hombre\n",
      "42.1607716451 mujer la hombre ama a\n",
      "42.2761766982 la a hombre ama mujer\n",
      "42.4480295589 hombre mujer la a ama\n",
      "42.5642225358 ama hombre la a mujer\n",
      "42.6927156776 mujer hombre la a ama\n",
      "42.7488565983 la mujer a ama hombre\n",
      "42.9256959793 ama hombre a mujer la\n",
      "42.9949665482 ama la a hombre mujer\n",
      "43.1326335912 mujer a ama la hombre\n",
      "43.1956629937 mujer la a hombre ama\n",
      "43.757676189 la ama hombre a mujer\n",
      "44.0174657961 la a mujer ama hombre\n",
      "44.6651412403 hombre la a ama mujer\n",
      "46.1478429234 mujer ama la a hombre\n",
      "46.1694953692 la a ama mujer hombre\n",
      "46.5567532544 mujer la ama hombre a\n",
      "47.7684307758 mujer a ama hombre la\n",
      "53.1923222735 la a ama hombre mujer\n",
      "57.9175873573 mujer la a ama hombre\n"
     ]
    }
   ],
   "source": [
    "perms = [' '.join(perm) for perm in permutations(bow)]\n",
    "for p, t in sorted([(-log_likelihood(text),text) for text in perms])[-20:]:\n",
    "    print p, t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Morphology"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from itertools import permutations\n",
    "from random import shuffle\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11.6193473339 mpn\n",
      "15.2685871124 nmp\n",
      "20.0848698616 mnp\n",
      "21.4446115494 npm\n",
      "24.6974582672 pmn\n",
      "------------------\n",
      "15.2685871124 nmp\n",
      "20.0848698616 mnp\n",
      "21.4446115494 npm\n",
      "24.6974582672 pmn\n",
      "25.9515581131 pnm\n"
     ]
    }
   ],
   "source": [
    "text = list(u'mnp')\n",
    "perms = [''.join(perm) for perm in permutations(text)]\n",
    "for p, t in sorted([(-log_likelihood(text),text) for text in perms])[:5]:\n",
    "    print p, t\n",
    "print \"------------------\"\n",
    "for p, t in sorted([(-log_likelihood(text),text) for text in perms])[-5:]:\n",
    "    print p, t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Structure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "(Lc. 12.23-39)\r\n",
      "\r\n",
      "Muerte de los de Cataleza de los que estaba el campo para que estable decir, y no soy la palabra de los que estaba el señores de los que estaba en el fin de Simón habla entre vosotros falsos de los que estaba en el calor a los que éste hablas, como cardaba a tu carne, sino lo que estaba en el caminable demonio con todos los que estaban en la carne. \r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print sample(400, u\"(Lc. \", temperature = 0.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## A model trained from a different dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of chars: 11120595\n",
      "Vocabulary size: 152\n"
     ]
    }
   ],
   "source": [
    "# Load training file to get vocabulary\n",
    "text_file = 'reg1.txt' # input file\n",
    "with codecs.open(text_file, 'r', 'utf-8') as f:\n",
    "    data = f.read()\n",
    "chars = [u'\\u0432', u'\\u1293', u'\\u2014', u'\\u2018', u'\\u201c', u'\\U00061ba1', u' ', u'$', u'(', u',', u'\\xaf', u'\\\\', u'\\u03b1', u'0', u'\\xb3', u'4', u'\\xb7', u'8', u'\\xbb', u'\\u203c', u'\\xbf', u'@', u'\\u9839', u'\\xd7', u'\\u263b', u'\\u05e1', u'`', u'\\xe3', u'd', u'\\xe7', u'h', u'l', u'\\xef', u'p', u'\\xf3', u't', u'x', u'\\ufffd', u'|', u'\\ufeff', u'\\u2003', u'j', u'\\u200b', u'\\U000ee28c', u'\\u2013', u'\\xa0', u'#', u\"'\", u'\\xa8', u'+', u'/', u'\\xb0', u'3', u'\\xb4', u'7', u';', u'?', u'\\U00065ca1', u'\\u044b', u'\\u2022', u'[', u'_', u'\\u0161', u'\\xe0', u'c', u'\\U00061b65', u'\\xe4', u'g', u'\\xe8', u'k', u'\\u2026', u'\\xec', u'o', u's', u'\\u0434', u'\\xf4', u'w', u'{', u'\\xfc', u'\\U00061285', u'\\u2030', u'\\n', u'\\ud299', u'\\xa1', u'\"', u'&', u'\\xa9', u'*', u'\\xad', u'.', u'\\xb1', u'2', u'6', u':', u'\\xbd', u'>', u'\\u0436', u'\\u2663', u'\\u2665', u'^', u'\\xe1', u'b', u'f', u'\\xe9', u'\\u266a', u'\\xed', u'n', u'\\xf1', u'r', u'v', u'\\xf9', u'z', u'\\xfd', u'\\u266b', u'~', u'\\t', u'\\u2113', u'\\x92', u'\\u2019', u'\\u201d', u'!', u'\\u9ba0', u'\\xa2', u'%', u'\\u1ba4', u')', u'\\xaa', u'-', u'\\U0006192f', u'\\xae', u'1', u'5', u'9', u'\\xba', u'=', u'\\u0441', u'\\u0430', u'\\U000ee825', u']', u'a', u'\\xe2', u'e', u'i', u'\\xea', u'm', u'q', u'\\xf2', u'u', u'\\u92a8', u'y', u'\\xfa', u'}']\n",
    "vocab_size = len(chars)\n",
    "char_to_ix = {ch: i for i, ch in enumerate(chars)}\n",
    "ix_to_char = {i: ch for i, ch in enumerate(chars)}\n",
    "print \"Total number of chars:\", len(data)\n",
    "print \"Vocabulary size:\", vocab_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# Input\n",
    "lookup = LookupTable(length=vocab_size, dim=embedding_size)\n",
    "# Layer 1\n",
    "fork1 = Fork(output_names=['linear1', 'gates1'],\n",
    "             input_dim=embedding_size, output_dims=[embedding_size, embedding_size * 2])\n",
    "fork1.name = 'fork1'\n",
    "grnn1 = GatedRecurrent(dim=embedding_size)\n",
    "grnn1.name = 'grnn1'\n",
    "# Layer 2\n",
    "fork2 = Fork(output_names=['linear2', 'gates2'],\n",
    "             input_dim=embedding_size, output_dims=[embedding_size, embedding_size * 2])\n",
    "fork2.name = 'fork2'\n",
    "grnn2 = GatedRecurrent(dim=embedding_size)\n",
    "grnn2.name = 'grnn2'\n",
    "# Softmax layer\n",
    "hidden_to_output = Linear(name='hidden_to_output', input_dim=embedding_size,\n",
    "                          output_dim=vocab_size)\n",
    "softmax = NDimensionalSoftmax()\n",
    "\n",
    "# Propagate x until top brick to get y_hat predictions\n",
    "x = tensor.imatrix('features')  # input\n",
    "y = tensor.imatrix('targets')   # output\n",
    "embedding = lookup.apply(x)\n",
    "linear1, gates1 = fork1.apply(embedding)\n",
    "h1 = grnn1.apply(linear1, gates1)\n",
    "h1.name = 'h1'\n",
    "linear2, gates2 = fork2.apply(h1)\n",
    "h2 = grnn2.apply(linear2, gates2)\n",
    "h2.name = 'h2'\n",
    "linear3 = hidden_to_output.apply(h2)\n",
    "linear3.name = 'linear3'\n",
    "y_hat = softmax.apply(linear3, extra_ndim=1)\n",
    "y_hat.name = 'y_hat'\n",
    "\n",
    "# COST\n",
    "cost = softmax.categorical_cross_entropy(y, linear3, extra_ndim=1).mean()\n",
    "cost.name = 'cost'\n",
    "\n",
    "model = Model(cost)\n",
    "\n",
    "# Load model parameters from a file\n",
    "model_file_name = 'model4.pkl'\n",
    "model_params = model.get_parameter_dict().keys()\n",
    "param_vals = {k:v for k,v in load_parameter_values(model_file_name).iteritems() if k in model_params}\n",
    "model.set_parameter_values(param_vals)\n",
    "\n",
    "# Define Theano graph\n",
    "y, x = model.inputs\n",
    "softmax = NDimensionalSoftmax()\n",
    "linear_output = [v for v in model.variables if v.name == 'linear3'][0]\n",
    "y_hat = softmax.apply(linear_output, extra_ndim=1)\n",
    "predict = theano.function([x], y_hat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "bienaventurado con la mano me la paso pensando en ti, no me hagas esperar\n",
      "y a mi compañera\n",
      "\n",
      "y siempre serás mejor que tu... receten y se mueren\n",
      "\n",
      "y solo por un beso\n",
      "y tu me miras y no te tengo\n",
      "es mi bomba pa que se acabe la mano\n",
      "y si tu quieres mami tu sabes que yo soy un tipo que no se acabe la mano\n",
      "y si tu quieres mami tu sabes que yo soy tu juguete\n",
      "y si tu te me estas tentando\n",
      "\n",
      "y si tu quieres mami tu sabes que yo soy tu nene\n",
      "\n",
      "y si tu te vas a encontrar la pared\n",
      "tu y yo en ti me mata el dembow\n",
      "es que tu me \n"
     ]
    }
   ],
   "source": [
    "print sample(500, u\"bienaventurado\", temperature = 0.1, seed=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "recuerda si tu quieres mami\n",
      "recuerda si tu quieres mami\n",
      "recuerda si tu quieres mami\n"
     ]
    }
   ],
   "source": [
    "!grep -i \"si tu quieres mami\" reg.txt \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}