examples/fields.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Use of fields associated to sequence records\n",
    "\n",
    "In is most simple form of use, `seqp` stores data sequences. However, sometimes it is useful to be able to add extra pieces of data associated to the sequence.\n",
    "\n",
    "An example might be: in a Neural Machine Translation scenario, we might want to store extra information about each a sentence along with its token IDs, like its dependency parse, the words POS tags, etc.\n",
    "\n",
    "In this notebook we will illustrate such a setup: apart from tokenizing the sentence storing its token IDs, we will use [spacy](https://spacy.io/) to get the sentence dependency parse and we will store it along with the token IDs.\n",
    "\n",
    "This notebook overlaps a bit with the [basic read/write notebook](https://github.com/noe/seqp/blob/master/examples/basic_read_write.ipynb) and might be helpful to review it first if you are not familiar with `seqp`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The idea in this notebook is to:\n",
    "1. Retrieve a text file from the internet.\n",
    "2. Extract a word-level vocabulary from the text.\n",
    "3. Segment the text into sentences.\n",
    "4. For each sentence:\n",
    "    - encode it as token IDs\n",
    "    - extract its dependency parse.\n",
    "    - store everything with `seqp`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text file download\n",
    "\n",
    "First, lets' download a text file to play with. It will be the Universal Declaration of Human Rights (UDHR)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "!wget -q http://research.ics.aalto.fi/cog/data/udhr/txt/eng.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vocabulary extraction\n",
    "\n",
    "We read all the file contents..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "import re\n",
    "from seqp.vocab import Vocabulary, VocabularyCollector\n",
    "\n",
    "file_name = 'eng.txt'\n",
    "\n",
    "with open(file_name) as f:\n",
    "    lines = [line.strip() for line in f]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...then we segment each line into sentences with spacy..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "import spacy\n",
    "from itertools import chain\n",
    "\n",
    "nlp = spacy.load('en')\n",
    "\n",
    "sents_in_text = sum((list(nlp(line).sents) for line in lines), [])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and now we extract the vocabulary from the sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "tokens_in_text = [str(t) for sent in sents_in_text for t in sent]\n",
    "\n",
    "collector = VocabularyCollector()\n",
    "for token in tokens_in_text:\n",
    "    collector.add_symbol(token)\n",
    "\n",
    "vocab = collector.consolidate(max_num_symbols=5000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Store records with fields with `seqp`\n",
    "\n",
    "We now will for each sentence, encode it in token IDs and store them with `seqp` along with the dependencies (i.e. the index of the head of each token in the sentence):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from seqp.hdf5 import Hdf5RecordWriter\n",
    "\n",
    "SEQ_FIELD = 'seq'\n",
    "DEPS_FIELD = 'deps'\n",
    "FIELDS = [SEQ_FIELD, DEPS_FIELD]\n",
    "\n",
    "output_file = 'udhr_eng.hdf5'\n",
    "\n",
    "with Hdf5RecordWriter(output_file, FIELDS, SEQ_FIELD) as writer:\n",
    "\n",
    "    # save vocabulary along with the records\n",
    "    writer.add_metadata({'vocab': vocab.to_json()})\n",
    "\n",
    "    for sent in sents_in_text:\n",
    "        tokens = [str(w) for w in sent]\n",
    "        token_ids = vocab.encode(tokens, add_eos=False, use_unk=True)\n",
    "        head_indexes = [w.head.i for w in sent]\n",
    "        record = {SEQ_FIELD: np.array(token_ids),\n",
    "                  DEPS_FIELD: np.array(head_indexes)}\n",
    "        writer.write(record)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read records back\n",
    "\n",
    "Now we will back a few records from the file we just wrote, to ensure everything works properly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "text": [
      "Sentence: Universal Declaration of Human Rights\n",
      "Deps: [1 1 1 4 2]\n",
      "\n",
      "Sentence: Preamble\n",
      "Deps: [0]\n",
      "\n"
     ],
     "output_type": "stream"
    }
   ],
   "source": [
    "from seqp.hdf5 import Hdf5RecordReader\n",
    "\n",
    "MAX_LINES_TO_PRINT = 3\n",
    "\n",
    "with Hdf5RecordReader(output_file) as reader:\n",
    "\n",
    "    loaded_vocab = Vocabulary.from_json(reader.metadata('vocab'))\n",
    "\n",
    "    for idx, length in reader.indexes_and_lengths():\n",
    "        if idx >= MAX_LINES_TO_PRINT:\n",
    "            break\n",
    "        record = reader.retrieve(idx)\n",
    "        tokens = loaded_vocab.decode(record[SEQ_FIELD])\n",
    "        print(\"Sentence: \" + \" \".join(tokens))\n",
    "        deps = record[DEPS_FIELD]\n",
    "        print(\"Deps: \" + str(deps) + '\\n')\n",
    "        "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "source": [],
    "metadata": {
     "collapsed": false
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}