notebooks/7_WorkingTexts.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with texts\n",
    "\n",
    "Text Normalization and Pre-processing Pipelines. \n",
    "\n",
    "Let's go back to our brief introduction to working with texts and expand upon it some more (see NOTEBOOK 5)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## A Standard Natural Language Processing (NLP) Pipeline\n",
    "\n",
    "![alt text](images/pipeline.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### What we hope to get as the final result..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Go on the Stanford CoreNLP [online demo](http://corenlp.run/) and parse the following sentence:\n",
    "\n",
    "> I think there is a world market for maybe five computers.\n",
    "\n",
    "(allegedly said by Thomas J. Watson, CEO of IBM, in 1943)\n",
    "\n",
    "This might be useful: [list of Penn Treebank POS tags.](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A Few Recommendations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [NLTK](http://www.nltk.org/) (Python)\n",
    "\n",
    "\n",
    "- has a variety of ready/easy-to-use implementations in several languages (some tools available for Dutch as well)\n",
    "\n",
    "\n",
    "- has a variety of APIs for other packages (e.g. Stanford CoreNLP)\n",
    "\n",
    "\n",
    "- lacks Information Extraction features (e.g. coreference resolution, relation extraction)\n",
    "\n",
    "\n",
    "- several implementations are not state-of-the-art and/or are rather slow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [Stanford CoreNLP](https://stanfordnlp.github.io/CoreNLP/) (Java)\n",
    "\n",
    "\n",
    "- includes virtually any NLP tool\n",
    "\n",
    "\n",
    "- basic distribution for English, but models for the following languages are available: Arabic, Chinese, French, German, and Spanish\n",
    "\n",
    "\n",
    "- implementations are often State-of-the-Art (though some component may not be the most efficient)\n",
    "\n",
    "\n",
    "- has Python wrappers\n",
    "\n",
    "\n",
    "- needs a bit more technical skills than using NLTK"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [spaCy](https://spacy.io) (Python)\n",
    "\n",
    "\n",
    "- industry-ready: scale, performance, getting things done, lacks some flexibility\n",
    "\n",
    "\n",
    "- many models for many languages: English German French Spanish Portuguese Italian Dutch Greek\n",
    "\n",
    "\n",
    "- implementations are often State-of-the-Art\n",
    "\n",
    "\n",
    "- has Python wrappers and the very clean \"spaCy pipeline\" programming interface \n",
    "\n",
    "\n",
    "- comes with many machine learning pre-trained models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [pattern](https://www.clips.uantwerpen.be/pages/pattern-nl) (NL-EN/Python)\n",
    "\n",
    "\n",
    "- toolbox for Dutch\n",
    "\n",
    "\n",
    "- easy to use\n",
    "\n",
    "\n",
    "- contains modules for PoS tagging, sentiment analysis and a WordNet interface\n",
    "\n",
    "\n",
    "- many interesting datasets available\n",
    "\n",
    "\n",
    "- available for [English](https://www.clips.uantwerpen.be/pages/pattern-en) as well"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When to use what? Chose by yourself: https://spacy.io/usage/facts-figures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task-dependent choces"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### A note on terminology:\n",
    "\n",
    "* **Type** = abstract ideal form of a sequence of characters (not necessarily a word in a language!)\n",
    "* **Token** = instantiation of a type in our corpus or document\n",
    "\n",
    "For example: *to be or not to be.*\n",
    "\n",
    "* 6 tokens (to, be, or, not, to, be)\n",
    "* 4 types (to, be, or, not)\n",
    "\n",
    "The types coincide with the vocabulary.\n",
    "\n",
    "Note that not all types might be proper words nor have a related lemma in an external lexicon."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Cleaning\n",
    "\n",
    "- remove lists, tables, figure captions, page headers, etc.\n",
    "\n",
    "\n",
    "\n",
    "- text from XML/HTML: remove unwanted code (e.g. head elements, boilerplates, comments, javascript code).\n",
    "\n",
    "\n",
    "\n",
    "- text from PDF: remove noisy characters\n",
    "\n",
    "\n",
    "- OCR errors are systematic: e.g. an \"`m`\" recognized as a \"`rn`\", an \"`e`\" as a \"`c`\", an \"`l`\" as a \"`1`\", an \"`f`\" recognized as a \"`s`\".\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Normalization and filtering\n",
    "\n",
    "Reducing the valid vocabulary space or even the text itself by filtering or standardizing. Standardizing might be considered as the task of creating equivalence classes of types. These decisions are application-dependent, might come from trial and error but should motivated in principle.\n",
    "\n",
    "- lowercasing and abbreviations. If you want \"`Train`\" and \"`train`\" to be counted as token of the same type, you should \"`lower()`\" the text, however case-folding may remove disciminative information, e.g., \"`U.S.`\" \"`US`\" vs. \"`us`\".\n",
    "\n",
    "\n",
    "- Hyphenation, accents, ... (\"`text-minig`\", \"`textmining`\", \"`text mining`\"; \"`cliche`\", \"`cliché`\")\n",
    "\n",
    "\n",
    "- Punctuation, digits, misspells, ...\n",
    "\n",
    "\n",
    "- Very frequent or very rare words.\n",
    "\n",
    "\n",
    "- Stopwords (usually, these are very frequent words!).\n",
    "\n",
    "\n",
    "- Specialized portions of the text (e.g., footnotes).\n",
    "\n",
    "\n",
    "- Normalization can also be done via relations, e.g. considering \"`car`\" and \"`automobile`\" as synonyms. We can use WordNet for that. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ces, par, étée, mais, ne, nos, d, êtes, eussions, avais, de, notre, étant, serez, qu, suis, le, eue, aurais, seriez, fus, fussent, les, étions, aura, n, l, seront, même, sa, se, y, un, fusses, pour, avec, ton, étais, on, seraient, à, ayez, ayons, je, fussiez, tes, au, avions, as, fut, aux, eurent, lui, mes, moi, serions, aurait, aurions, ait, fussions, en, serais, aient, aurez, soyons, fusse, eussent, il, te, eûmes, fûtes, avez, vous, ou, ce, serons, ils, leur, ont, avaient, me, votre, son, une, aviez, auras, dans, eux, toi, fûmes, eûtes, auriez, c, que, pas, ai, étante, sommes, aies, eût, avait, et, furent, qui, sur, étées, eussiez, aurai, aurons, serai, avons, aie, eues, ma, était, la, été, eu, seras, soient, serait, mon, es, ayants, ayant, ayantes, tu, soyez, eusse, nous, eut, du, s, eusses, auront, eus, auraient, étiez, ta, j, étés, t, est, sont, vos, ayante, étantes, soit, étants, m, sera, étaient, elle, ses, sois, fût, des'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "\", \".join(set(stopwords.words('french')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentence Segmentation\n",
    "\n",
    "In its base form, a text is a stream of characters. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[The Man Who Was Thursday by G. K. Chesterton 1908]\n",
      "\n",
      "To Edmund Clerihew Bentley\n",
      "\n",
      "A cloud was on the mind of men, and wailing went the weather,\n",
      "Yea, a sick cloud upon the soul when we were boys together.\n",
      "Science announced nonentity and art admired decay;\n",
      "The world was old and ended: but you and I were gay;\n",
      "Round us in antic order their crippled vices came--\n",
      "Lust that had lost its laughter, fear that had lost its shame.\n",
      "Like the white lock of Whistler, that lit our aimless gloom,\n",
      "Men showed their own white feather as proudly as a plume.\n",
      "Life was a fly that faded, and death a drone that stung;\n",
      "The world was very old indeed when you and I were young.\n",
      "They twisted even decent sin to shapes not to be named:\n",
      "Men were ashamed of honour; but we were not ashamed.\n",
      "Weak if we were and foolish, not thus we failed, not thus;\n",
      "When that black Baal blocked the heavens he had no hymns from us\n",
      "Children we were--our forts of sand were even as weak as eve,\n",
      "High as they went we piled them up to break that bitter sea.\n"
     ]
    }
   ],
   "source": [
    "text = nltk.corpus.gutenberg.raw('chesterton-thursday.txt')\n",
    "print(text[:1010])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['austen-emma.txt',\n",
       " 'austen-persuasion.txt',\n",
       " 'austen-sense.txt',\n",
       " 'bible-kjv.txt',\n",
       " 'blake-poems.txt',\n",
       " 'bryant-stories.txt',\n",
       " 'burgess-busterbrown.txt',\n",
       " 'carroll-alice.txt',\n",
       " 'chesterton-ball.txt',\n",
       " 'chesterton-brown.txt',\n",
       " 'chesterton-thursday.txt',\n",
       " 'edgeworth-parents.txt',\n",
       " 'melville-moby_dick.txt',\n",
       " 'milton-paradise.txt',\n",
       " 'shakespeare-caesar.txt',\n",
       " 'shakespeare-hamlet.txt',\n",
       " 'shakespeare-macbeth.txt',\n",
       " 'whitman-leaves.txt']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.corpus.gutenberg.fileids()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#nltk.download('stopwords')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first NLP processing step is **the identification of the sentence boundaries**:\n",
    "\n",
    "- some pipelines perform the sentence splitting phase before the tokenization (e.g., NLTK)\n",
    "\n",
    "\n",
    "- while other uses information about the tokens (e.g., CoreNLP) and/or about the parse tree (e.g., spaCy) to identify sentence boundaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ The Man Who Was Thursday by G . K . Chesterton 1908 ]\n",
      "To Edmund Clerihew Bentley\n",
      "A cloud was on the mind of men , and wailing went the weather , Yea , a sick cloud upon the soul when we were boys together .\n",
      "Science announced nonentity and art admired decay ; The world was old and ended : but you and I were gay ; Round us in antic order their crippled vices came -- Lust that had lost its laughter , fear that had lost its shame .\n",
      "Like the white lock of Whistler , that lit our aimless gloom , Men showed their own white feather as proudly as a plume .\n",
      "Life was a fly that faded , and death a drone that stung ; The world was very old indeed when you and I were young .\n",
      "They twisted even decent sin to shapes not to be named : Men were ashamed of honour ; but we were not ashamed .\n",
      "Weak if we were and foolish , not thus we failed , not thus ; When that black Baal blocked the heavens he had no hymns from us Children we were -- our forts of sand were even as weak as eve , High as they went we piled them up to break that bitter sea .\n"
     ]
    }
   ],
   "source": [
    "# getting rid of newlines\n",
    "text = nltk.corpus.gutenberg.raw('chesterton-thursday.txt').replace(\"\\n\", \" \") \n",
    "\n",
    "# the NLTK corpus sents() returns the tokenized words as well \n",
    "for sentence in nltk.corpus.gutenberg.sents('chesterton-thursday.txt')[:8]:\n",
    "    print(\" \".join(sentence))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ... but **what is a sentence?**\n",
    " \n",
    " - something ending with \"`.`\", \"`!`\" or \"`?`\", maybe?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['[The Man Who Was Thursday by G',\n",
       " ' K',\n",
       " ' Chesterton 1908]  To Edmund Clerihew Bentley  A cloud was on the mind of men, and wailing went the weather, Yea, a sick cloud upon the soul when we were boys togethe']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re\n",
    "re.split(r\"[.?!]\", text[:200])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[The Man Who Was Thursday by G. K. Chesterton 1908]  To Edmund Clerihew Bentley  A cloud was on the mind of men, and wailing went the weather, Yea, a sick cloud upon the soul when we were boys togethe\n"
     ]
    }
   ],
   "source": [
    "print(text[:200])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not really, a period can be used also to denote:\n",
    "\n",
    "- an abbreviation (47% of the periods in the Wall Street Journal denote abbreviations)\n",
    "\n",
    "- decimal point\n",
    "\n",
    "- ellipsis\n",
    "\n",
    "- url or e-mail address..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Question marks and exclamation marks can be used in:\n",
    "\n",
    "- emoticons and slang\n",
    "\n",
    "- computer code \n",
    "\n",
    "- embedded quotations..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even if heuristic solutions are conceivable (e.g. CoreNLP)...\n",
    "\n",
    "- e.g. *a period is not a sentence boundary if the preceding token is title-cased*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...the best option is to treat the sentence boundary identification as a **binary classification problem** (e.g. NLTK), in which possible features can be:\n",
    "\n",
    "- case of the surrounding tokens\n",
    "\n",
    "\n",
    "- length of the surrounding tokens\n",
    "\n",
    "\n",
    "- lemma/type/form of the surrounding tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[The Man Who Was Thursday by G. K. Chesterton 1908]  To Edmund Clerihew Bentley  A cloud was on the mind of men, and wailing went the weather, Yea, a sick cloud upon the soul when we were boys together.\n",
      "\n",
      "Science announced nonentity and art admired decay; The world was old and ended: but you and I were gay; Round us in antic order their crippled vices came-- Lust that had lost its laughter, fear that had lost its shame.\n",
      "\n",
      "Like the white lock of Whistler, that lit our aimless gloom, Men showed their own white feather as proudly as a plume.\n",
      "\n",
      "Life was a fly that faded, and death a drone that stung; The world was very old indeed when you and I were young.\n",
      "\n",
      "They twisted even decent sin to shapes not to be named: Men were ashamed of honour; but we were not ashamed.\n",
      "\n",
      "Weak if we were and foolish, not thus we failed, not thus; When that black Baal blocked the heavens he had no hymns from us Children we were--our forts of sand were even as weak as eve, High as they went we piled them up to break that bitter sea.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i in nltk.tokenize.sent_tokenize(text[:1010]):\n",
    "    print(i+\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "The accuracy of these classifiers can be as high as over 99%"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "NLTK come packed with classifiers trained on 17 languages, among which Dutch:\n",
    "\n",
    "- there are methods that allows you to train your own models as well. This can be useful if you want to process special texts like tweets or facebook messages..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "a_dutch_email = \"\"\"Uw abonnement op uw OV-chipkaart activeren.\n",
    "Het is belangrijk dat u het abonnement op uw OV-chipkaart activeert.\n",
    "Dit kunt u doen bij de NS-kaartautomaat of bij één van de overige ophaalpunten voor de OV-chipkaart. Kijk voor een overzicht van de ophaalpunten op www.ov-chipkaart.nl/adresvinder.\n",
    "Hoe activeert u uw abonnement bij de NS-kaartautomaat?\n",
    "Houd uw OV-chipkaart voor de kaartlezer rechts onder het scherm.\n",
    "Kies 'Ophalen bestelling' en volg de aanwijzingen op het scherm.\n",
    "Voordat u aan uw reis begint dient er minimaal €10,- saldo op uw OV-chipkaart te staan.\n",
    "Vergeet niet in en uit te checken tijdens uw reis.\n",
    "Heeft u nog vragen?\n",
    "We helpen u graag verder. Kijk op www.ns.nl of stel uw vraag op het NS Serviceforum (www.ns.nl/forum). U kunt ook contact opnemen met NS Klantenservice via telefoonnummer 030 - 751 51 55 of via Twitter @NS_online. We staan dag en nacht voor u klaar.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Uw abonnement op uw OV-chipkaart activeren.',\n",
       " 'Het is belangrijk dat u het abonnement op uw OV-chipkaart activeert.',\n",
       " 'Dit kunt u doen bij de NS-kaartautomaat of bij één van de overige ophaalpunten voor de OV-chipkaart.',\n",
       " 'Kijk voor een overzicht van de ophaalpunten op www.ov-chipkaart.nl/adresvinder.',\n",
       " 'Hoe activeert u uw abonnement bij de NS-kaartautomaat?',\n",
       " 'Houd uw OV-chipkaart voor de kaartlezer rechts onder het scherm.',\n",
       " \"Kies 'Ophalen bestelling' en volg de aanwijzingen op het scherm.\",\n",
       " 'Voordat u aan uw reis begint dient er minimaal €10,- saldo op uw OV-chipkaart te staan.',\n",
       " 'Vergeet niet in en uit te checken tijdens uw reis.',\n",
       " 'Heeft u nog vragen?',\n",
       " 'We helpen u graag verder.',\n",
       " 'Kijk op www.ns.nl of stel uw vraag op het NS Serviceforum (www.ns.nl/forum).',\n",
       " 'U kunt ook contact opnemen met NS Klantenservice via telefoonnummer 030 - 751 51 55 of via Twitter @NS_online.',\n",
       " 'We staan dag en nacht voor u klaar.']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.tokenize.sent_tokenize(a_dutch_email, language='dutch')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenization\n",
    "\n",
    "Tokenization is the task of splitting a string of characters into chunks of characters.\n",
    "\n",
    "It is a task that is usually tackled with rule-based or regex-based algorithms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['[', 'The', 'Man', 'Who', 'Was', 'Thursday', 'by', 'G', '.', 'K', '.', 'Chesterton', '1908', ']', 'To', 'Edmund', 'Clerihew', 'Bentley', 'A', 'cloud', 'was', 'on', 'the', 'mind', 'of', 'men', ',', 'and', 'wailing', 'went', 'the', 'weather', ',', 'Yea', ',', 'a', 'sick', 'cloud', 'upon', 'the', 'soul', 'when', 'we', 'were', 'boys', 'together', '.', 'Science', 'announced', 'nonentity', 'and', 'art', 'admired', 'decay', ';', 'The', 'world', 'was', 'old', 'and', 'ended', ':', 'but', 'you', 'and', 'I', 'were', 'gay', ';', 'Round', 'us', 'in', 'antic', 'order', 'their', 'crippled', 'vices', 'came', '--', 'Lust', 'that', 'had', 'lost', 'its', 'laughter', ',', 'fear', 'that', 'had', 'lost', 'its', 'shame', '.', 'Like', 'the', 'white', 'lock', 'of', 'Whistler', ',', 'that', 'lit', 'our', 'aimless', 'gloom', ',', 'Men', 'showed', 'their', 'own', 'white', 'feather', 'as', 'proudly', 'as', 'a', 'plume', '.', 'Life', 'was', 'a', 'fly', 'that', 'faded', ',', 'and', 'death', 'a', 'drone', 'that', 'stung', ';', 'The', 'world', 'was', 'very', 'old', 'indeed', 'when', 'you', 'and', 'I', 'were', 'young', '.', 'They', 'twisted', 'even', 'decent', 'sin', 'to', 'shapes', 'not', 'to', 'be', 'named', ':', 'Men', 'were', 'ashamed', 'of', 'honour', ';', 'but', 'we', 'were', 'not', 'ashamed', '.', 'Weak', 'if', 'we', 'were', 'and', 'foolish', ',', 'not', 'thus', 'we', 'failed', ',', 'not', 'thus', ';', 'When', 'that', 'black', 'Baal', 'blocked', 'the', 'heavens', 'he', 'had', 'no', 'hymns', 'from', 'us', 'Children', 'we', 'were', '--', 'our', 'forts', 'of', 'sand', 'were', 'even', 'as', 'weak', 'as', 'eve', ',', 'High', 'as', 'they', 'went', 'we', 'piled', 'them', 'up', 'to', 'break', 'that', 'bitter', 'sea', '.']\n"
     ]
    }
   ],
   "source": [
    "# it can be performed BEFORE sentence splitting\n",
    "print(nltk.corpus.gutenberg.words('chesterton-thursday.txt')[:226])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['[', 'The', 'Man', 'Who', 'Was', 'Thursday', 'by', 'G', '.', 'K', '.', 'Chesterton', '1908', ']']\n",
      "['To', 'Edmund', 'Clerihew', 'Bentley']\n",
      "['A', 'cloud', 'was', 'on', 'the', 'mind', 'of', 'men', ',', 'and', 'wailing', 'went', 'the', 'weather', ',', 'Yea', ',', 'a', 'sick', 'cloud', 'upon', 'the', 'soul', 'when', 'we', 'were', 'boys', 'together', '.']\n",
      "['Science', 'announced', 'nonentity', 'and', 'art', 'admired', 'decay', ';', 'The', 'world', 'was', 'old', 'and', 'ended', ':', 'but', 'you', 'and', 'I', 'were', 'gay', ';', 'Round', 'us', 'in', 'antic', 'order', 'their', 'crippled', 'vices', 'came', '--', 'Lust', 'that', 'had', 'lost', 'its', 'laughter', ',', 'fear', 'that', 'had', 'lost', 'its', 'shame', '.']\n",
      "['Like', 'the', 'white', 'lock', 'of', 'Whistler', ',', 'that', 'lit', 'our', 'aimless', 'gloom', ',', 'Men', 'showed', 'their', 'own', 'white', 'feather', 'as', 'proudly', 'as', 'a', 'plume', '.']\n",
      "['Life', 'was', 'a', 'fly', 'that', 'faded', ',', 'and', 'death', 'a', 'drone', 'that', 'stung', ';', 'The', 'world', 'was', 'very', 'old', 'indeed', 'when', 'you', 'and', 'I', 'were', 'young', '.']\n",
      "['They', 'twisted', 'even', 'decent', 'sin', 'to', 'shapes', 'not', 'to', 'be', 'named', ':', 'Men', 'were', 'ashamed', 'of', 'honour', ';', 'but', 'we', 'were', 'not', 'ashamed', '.']\n",
      "['Weak', 'if', 'we', 'were', 'and', 'foolish', ',', 'not', 'thus', 'we', 'failed', ',', 'not', 'thus', ';', 'When', 'that', 'black', 'Baal', 'blocked', 'the', 'heavens', 'he', 'had', 'no', 'hymns', 'from', 'us', 'Children', 'we', 'were', '--', 'our', 'forts', 'of', 'sand', 'were', 'even', 'as', 'weak', 'as', 'eve', ',', 'High', 'as', 'they', 'went', 'we', 'piled', 'them', 'up', 'to', 'break', 'that', 'bitter', 'sea', '.']\n"
     ]
    }
   ],
   "source": [
    "# it can be performed AFTER sentence splitting\n",
    "for s in nltk.corpus.gutenberg.sents('chesterton-thursday.txt')[:8]:\n",
    "    print(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some of the most recurrent issues with **Tokenization in English**:\n",
    "\n",
    "- contractions: *I'm*, *isn't*, *Gianluca's*\n",
    "\n",
    "\n",
    "- emoticons: *:)*, *;-)*\n",
    "\n",
    "\n",
    "- proper names: *J.K. Rowling*, *F.B.I.*, *O'Neill*\n",
    "\n",
    "\n",
    "- hyphenation can be used in many ways:\n",
    "\n",
    "    - single word: *e-mail*, *co-operation*\n",
    "    - how many words? *so-called*, *pro-gun*\n",
    "    - grouping of words for non-strictly-linguistic purposes: *take-it-or-leave*\n",
    "\n",
    "\n",
    "\n",
    "- multi word expressions and collocations:\n",
    "\n",
    "    - phrasal verbs: *give up*, *work out*, *move in*\n",
    "    - multi-token proper names: *New York*, *Las Vegas*\n",
    "    - compound nominals: *car park*, *attorney general*, *part of speech*\n",
    "    - ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The complexity of this task is also **language-dependent**:\n",
    "\n",
    "- e.g. compounds in German: *Donaudampfschifffahrtselektrizitätenhauptbetriebswerkbauunterbeamtengesellschaft* (i.e. \"Association for subordinate officials of the head office management of the Danube steamboat electrical services\")\n",
    "\n",
    "\n",
    "- Chinese is written without spaces between words: 天花 (\"small pot\") vs. 天 (\"sky\") and 花 (\"flower\")."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Simplest (but suboptimal) solution: **segment around space characters**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['[The', 'Man', 'Who', 'Was', 'Thursday', 'by', 'G.', 'K.', 'Chesterton', '1908]', 'To', 'Edmund', 'Clerihew', 'Bentley', 'A', 'cloud', 'was', 'on', 'the', 'mind', 'of', 'men,', 'and', 'wailing', 'went', 'the', 'weather,', 'Yea,', 'a', 'sick', 'cloud', 'upon', 'the', 'soul', 'when', 'we', 'were', 'boys', 'together.', 'Science', 'announced', 'nonentity', 'and', 'art', 'admired', 'decay;', 'The', 'world', 'was', 'old', 'and', 'ended:', 'but', 'you', 'and', 'I', 'were', 'gay;', 'Round', 'us', 'in', 'antic', 'order', 'their', 'crippled', 'vices', 'came--', 'Lust', 'that', 'had', 'lost', 'its', 'laughter,', 'fear', 'that', 'had', 'lost', 'its', 'shame.', 'Like', 'the', 'white', 'lock', 'of', 'Whistler,', 'that', 'lit', 'our', 'aimless', 'gloom,', 'Men', 'showed', 'their', 'own', 'white', 'feather', 'as', 'proudly', 'as', 'a', 'plume.', 'Life', 'was', 'a', 'fly', 'that', 'faded,', 'and', 'death', 'a', 'drone', 'that', 'stung;', 'The', 'world', 'was', 'very', 'old', 'indeed', 'when', 'you', 'and', 'I', 'were', 'young.', 'They', 'twisted', 'even', 'decent', 'sin', 'to', 'shapes', 'not', 'to', 'be', 'named:', 'Men', 'were', 'ashamed', 'of', 'honour;', 'but', 'we', 'were', 'not', 'ashamed.', 'Weak', 'if', 'we', 'were', 'and', 'foolish,', 'not', 'thus', 'we', 'failed,', 'not', 'thus;', 'When', 'that', 'black', 'Baal', 'blocked', 'the', 'heavens', 'he', 'had', 'no', 'hymns', 'from', 'us', 'Children', 'we', 'were--our', 'forts', 'of', 'sand', 'were', 'even', 'as', 'weak', 'as', 'eve,', 'High', 'as', 'they', 'went', 'we', 'piled', 'them', 'up', 'to', 'break', 'that', 'b']\n"
     ]
    }
   ],
   "source": [
    "print(text[:1000].split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many issues with this solution, among which the production of tokens such as `'plume.'`, `'thus;'` and `'were--our'`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NLTK's recommended word tokenizer is the Penn `nltk.tokenize.TreebankWordTokenizer().tokenize()`, that can be called in many ways, among which the easier is by using the `nltk.tokenize.word_tokenize()` wrapper:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['[', 'The', 'Man', 'Who', 'Was', 'Thursday', 'by', 'G.', 'K.', 'Chesterton', '1908', ']', 'To', 'Edmund', 'Clerihew', 'Bentley', 'A', 'cloud', 'was', 'on', 'the', 'mind', 'of', 'men', ',', 'and', 'wailing', 'went', 'the', 'weather', ',', 'Yea', ',', 'a', 'sick', 'cloud', 'upon', 'the', 'soul', 'when', 'we', 'were', 'boys', 'together', '.', 'Science', 'announced', 'nonentity', 'and', 'art', 'admired', 'decay', ';', 'The', 'world', 'was', 'old', 'and', 'ended', ':', 'but', 'you', 'and', 'I', 'were', 'gay', ';', 'Round', 'us', 'in', 'antic', 'order', 'their', 'crippled', 'vices', 'came', '--', 'Lust', 'that', 'had', 'lost', 'its', 'laughter', ',', 'fear', 'that', 'had', 'lost', 'its', 'shame', '.', 'Like', 'the', 'white', 'lock', 'of', 'Whistler', ',', 'that', 'lit', 'our', 'aimless', 'gloom', ',', 'Men', 'showed', 'their', 'own', 'white', 'feather', 'as', 'proudly', 'as', 'a', 'plume', '.', 'Life', 'was', 'a', 'fly', 'that', 'faded', ',', 'and', 'death', 'a', 'drone', 'that', 'stung', ';', 'The', 'world', 'was', 'very', 'old', 'indeed', 'when', 'you', 'and', 'I', 'were', 'young', '.', 'They', 'twisted', 'even', 'decent', 'sin', 'to', 'shapes', 'not', 'to', 'be', 'named', ':', 'Men', 'were', 'ashamed', 'of', 'honour', ';', 'but', 'we', 'were', 'not', 'ashamed', '.', 'Weak', 'if', 'we', 'were', 'and', 'foolish', ',', 'not', 'thus', 'we', 'failed', ',', 'not', 'thus', ';', 'When', 'that', 'black', 'Baal', 'blocked', 'the', 'heavens', 'he', 'had', 'no', 'hymns', 'from', 'us', 'Children', 'we', 'were', '--', 'our', 'forts', 'of', 'sand', 'were', 'even', 'as', 'weak', 'as', 'eve', ',', 'High', 'as', 'they', 'went', 'we', 'piled', 'them', 'up', 'to', 'break', 'that', 'b']\n"
     ]
    }
   ],
   "source": [
    "# this is the same as nltk.tokenize.TreebankWordTokenizer().tokenize(text[:1010])\n",
    "print(nltk.tokenize.word_tokenize(text[:1000]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NLTK offers alternative word tokenizer, that may be more appropriate according to the applications you are developing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['That', 'U.S.A.', 'poster-print', \"doesn't\", 'cost', '$12.40...']\n"
     ]
    }
   ],
   "source": [
    "nltk_text = 'That U.S.A. poster-print doesn\\'t cost $12.40...'\n",
    "print(nltk_text.split())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['That', 'U.S.A.', 'poster-print', 'does', \"n't\", 'cost', '$', '12.40', '...']\n"
     ]
    }
   ],
   "source": [
    "nltk_text = 'That U.S.A. poster-print doesn\\'t cost $12.40...'\n",
    "print(nltk.tokenize.word_tokenize(nltk_text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['That', 'U', '.', 'S', '.', 'A', '.', 'poster', '-', 'print', 'doesn', \"'\", 't', 'cost', '$', '12', '.', '40', '...']\n"
     ]
    }
   ],
   "source": [
    "# nltk.tokenize.wordpunct_tokenize splits all punctuations into separate tokens\n",
    "print(nltk.tokenize.wordpunct_tokenize(nltk_text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['That', 'U.S.A.', 'poster-print', 'doesn', \"'\", 't', 'cost', '$12.40', '...']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# nltk.regexp_tokenize() allows you to build a customized tokenizer by specifying a regex pattern\n",
    "pattern = r'''(?x)    # set flag to allow verbose regexps\n",
    "(?:[A-Z]\\.)+          # abbreviations, e.g. U.S.A.\n",
    "|\\w+(?:-\\w+)*         # words with optional internal hyphens\n",
    "|\\$?\\d+(?:\\.\\d+)?%?   # currency and percentages, e.g. $12.40, 82%\n",
    "|\\.\\.\\.               # ellipsis\n",
    "|[][.,;\"\\'?():-_`]    # these are separate tokens\n",
    "'''\n",
    "\n",
    "nltk.regexp_tokenize(nltk_text, pattern)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stemming and Lemmatization\n",
    "\n",
    "\n",
    "The goal of both stemming and lemmatization is to reduce inflectional forms (e.g. *dreaming*, *dreamed*) and sometimes derivationally related forms of a word to a common \"base form\" (e.g. *eat*).\n",
    "\n",
    "*Stemming targets the stem, or the part of a word which remains stable to inflectional manipulations. E.g., \"eats\" -> \"eat\".*\n",
    "\n",
    "*Lemmatization targets the lemma, or the canonical dictionary key associated with the word. E.g., \"is\" -> \"be\".*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stemming\n",
    "\n",
    "- Stemming in practice usually refers to a **crude heuristic** process that (usually) removes the inflectional **suffixes** of a word\n",
    "\n",
    "\n",
    "- The result of stemming **might not be an actual word**\n",
    "\n",
    "\n",
    "- Stemmers are very fast and do not require PoS information"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NLTK offers several algorithms, among which the most used are: \n",
    "\n",
    "- the [Porter](https://tartarus.org/martin/PorterStemmer/) Stemmer: the oldest algorithm (1979), it is regex-based, it is still the most used even if more computationally intensive\n",
    "\n",
    "\n",
    "- the algorithm consists of 5 phases of word reductions, applied sequentially as a cascade.\n",
    "\n",
    "\n",
    "- Example of rules in phase 1:\n",
    "\n",
    "    * SSES -> SS (*caresses -> caress*)\n",
    "    * IES -> I (*ponies -> poni*)\n",
    "    * S ->  (*cats -> cat*)\n",
    "\n",
    "\n",
    "- Example of rules in phase 5 (measuring if a token could stand by itself):\n",
    "\n",
    "    * EMENT ->  (would reduce *replacement* to *replac*, but leave *cement* as cement because \"`c`\" alone is too short)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'unfriendli'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.PorterStemmer().stem(\"unfriendliness\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'cement'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.PorterStemmer().stem(\"cements\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'replac'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.PorterStemmer().stem(\"replace\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- the [Lancaster](http://www.comp.lancs.ac.uk/computing/research/stemming/) Stemmers: the most aggressive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'unfriend'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# aggressive\n",
    "nltk.LancasterStemmer().stem(\"unfriendliness\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'cem'"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.LancasterStemmer().stem(\"cements\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "happi\n",
      "happy\n",
      "happi\n"
     ]
    }
   ],
   "source": [
    "# sometimes works better than the Porter Stemmer\n",
    "print(nltk.PorterStemmer().stem(\"happiness\"))\n",
    "print(nltk.LancasterStemmer().stem(\"happiness\"))\n",
    "print(nltk.stem.snowball.SnowballStemmer(\"english\").stem(\"happiness\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [Snowball](http://snowballstem.org/) is a language for creating stemmers, and is available in many languages among which Dutch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('arabic', 'danish', 'dutch', 'english', 'finnish', 'french', 'german', 'hungarian', 'italian', 'norwegian', 'porter', 'portuguese', 'romanian', 'russian', 'spanish', 'swedish')\n"
     ]
    }
   ],
   "source": [
    "print(nltk.stem.snowball.SnowballStemmer.languages)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Wanna see a demo on how it works in dutch? Try `nltk.stem.snowball.demo()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a general rule, the SnowballStemmer for English (a.k.a. Porter2), works better than the original Porter stemmer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gener\n",
      "generous\n"
     ]
    }
   ],
   "source": [
    "print(nltk.PorterStemmer().stem(\"generously\"))\n",
    "print(nltk.stem.snowball.SnowballStemmer(\"english\").stem(\"generously\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lemmatization\n",
    "\n",
    "Lemmatizers use a **vocabulary** and **morphological information** to apply a more articulated set of rules in order to return **actual lemmas** (a.k.a. dictionary entries)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The WordNet lemmatizer removes the affixes of a token only if the resulting lemma is in WordNet\n",
    "\n",
    "- It is based on the WordNet `morphy` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'woman'"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# irregularities are easier to handle with a lemmatizer (if the word is in the vocabulary)\n",
    "nltk.WordNetLemmatizer().lemmatize(\"women\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'flying'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# but it doesn't really handle some inflected forms ...\n",
    "nltk.WordNetLemmatizer().lemmatize(\"flying\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'fly'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# ... but it is a PoS tagging issue, really. Let's tell it that we have the verb\n",
    "nltk.WordNetLemmatizer().lemmatize(\"flying\", pos = \"v\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A quick-and-dirty comparison of Stemmers and the WN Lemmatizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT \t\t\t Porter \t Snowball \t Lancaster \t WN Lemmatizer\n",
      "-----------------------------------------------------------------------------------------\n",
      "agreed:  \t\t agre\t\t agre\t\t agree\t\t agreed\n",
      "caresses:  \t\t caress\t\t caress\t\t caress\t\t caress\n",
      "cement:  \t\t cement\t\t cement\t\t cem\t\t cement\n",
      "colonizer:  \t\t colon\t\t colon\t\t colon\t\t colonizer\n",
      "denied:  \t\t deni\t\t deni\t\t deny\t\t denied\n",
      "flies:  \t\t fli\t\t fli\t\t fli\t\t fly\n",
      "flying:  \t\t fli\t\t fli\t\t fly\t\t flying\n",
      "humbled:  \t\t humbl\t\t humbl\t\t humbl\t\t humbled\n",
      "itemization:  \t\t item\t\t item\t\t item\t\t itemization\n",
      "meeting:  \t\t meet\t\t meet\t\t meet\t\t meeting\n",
      "mules:  \t\t mule\t\t mule\t\t mul\t\t mule\n",
      "provision:  \t\t provis\t\t provis\t\t provid\t\t provision\n",
      "reference:  \t\t refer\t\t refer\t\t ref\t\t reference\n",
      "sensational:  \t\t sensat\t\t sensat\t\t sens\t\t sensational\n",
      "sized:  \t\t size\t\t size\t\t siz\t\t sized\n",
      "stating:  \t\t state\t\t state\t\t stat\t\t stating\n",
      "traditional:  \t\t tradit\t\t tradit\t\t tradit\t\t traditional\n",
      "women:  \t\t women\t\t women\t\t wom\t\t woman\n"
     ]
    }
   ],
   "source": [
    "words = ['caresses', 'flies', 'flying', 'mules', 'denied', 'agreed', 'humbled', 'sized',  'meeting', 'stating', \n",
    "         'itemization', 'sensational', 'traditional', 'reference', 'colonizer', 'provision', 'cement', 'women']\n",
    "\n",
    "print(\"INPUT\", \"\\t\\t\\t\", \"Porter\", \"\\t\", \"Snowball\", \"\\t\", \"Lancaster\", \"\\t\", \"WN Lemmatizer\")\n",
    "print(\"-\"*89)\n",
    "for w in sorted(words):\n",
    "    print(w+\":  \", end='')\n",
    "    for stemmer_function in [nltk.PorterStemmer().stem, nltk.stem.snowball.SnowballStemmer(\"english\").stem, \\\n",
    "                             nltk.LancasterStemmer().stem, nltk.WordNetLemmatizer().lemmatize]:\n",
    "        print(\"\\t\\t\", stemmer_function(w), end='')\n",
    "    print(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How to choose?\n",
    "\n",
    "As a rule of thumb:\n",
    "\n",
    "- if you need to work with **real words**, use a lemmatizer (PoS information helps a lot)\n",
    "\n",
    "\n",
    "- if you need **speed**, choose a stemmer\n",
    "\n",
    "\n",
    "- if want to link morphological derivations (e.g. \"dreamable\") to their bases, a stemmer can be a quick-and-dirty solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## PoS Tagging\n",
    "\n",
    "The proccess of automatically classifying words into their **lexical categories** \n",
    "\n",
    "- lexical categories are also called words classes, parts-of-speech, lexical tags..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lexical categories are divided into two main groups:\n",
    "\n",
    "- **open word classes** which constantly acquire new members:\n",
    "    - nouns, verbs, adjectives and adverbs\n",
    "\n",
    "\n",
    "- **closed class** when their inventory does not change over time (or much less so):\n",
    "    - e.g., pronouns, determiners, prepositions, conjunctions, auxiliary and modal verbs, numerals..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The collection of PoS tags used for a particular task is known as a **tagset**\n",
    "\n",
    "- The most common English tagset is the [**Penn Treebank**](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html) one, containing the following tags:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you might need to do this to download the related resource\n",
    "#nltk.download('tagsets')\n",
    "#nltk.download('universal_tagset')\n",
    "#nltk.download('brown')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "$: dollar\n",
      "    $ -$ --$ A$ C$ HK$ M$ NZ$ S$ U.S.$ US$\n",
      "'': closing quotation mark\n",
      "    ' ''\n",
      "(: opening parenthesis\n",
      "    ( [ {\n",
      "): closing parenthesis\n",
      "    ) ] }\n",
      ",: comma\n",
      "    ,\n",
      "--: dash\n",
      "    --\n",
      ".: sentence terminator\n",
      "    . ! ?\n",
      ":: colon or ellipsis\n",
      "    : ; ...\n",
      "CC: conjunction, coordinating\n",
      "    & 'n and both but either et for less minus neither nor or plus so\n",
      "    therefore times v. versus vs. whether yet\n",
      "CD: numeral, cardinal\n",
      "    mid-1890 nine-thirty forty-two one-tenth ten million 0.5 one forty-\n",
      "    seven 1987 twenty '79 zero two 78-degrees eighty-four IX '60s .025\n",
      "    fifteen 271,124 dozen quintillion DM2,000 ...\n",
      "DT: determiner\n",
      "    all an another any both del each either every half la many much nary\n",
      "    neither no some such that the them these this those\n",
      "EX: existential there\n",
      "    there\n",
      "FW: foreign word\n",
      "    gemeinschaft hund ich jeux habeas Haementeria Herr K'ang-si vous\n",
      "    lutihaw alai je jour objets salutaris fille quibusdam pas trop Monte\n",
      "    terram fiche oui corporis ...\n",
      "IN: preposition or conjunction, subordinating\n",
      "    astride among uppon whether out inside pro despite on by throughout\n",
      "    below within for towards near behind atop around if like until below\n",
      "    next into if beside ...\n",
      "JJ: adjective or numeral, ordinal\n",
      "    third ill-mannered pre-war regrettable oiled calamitous first separable\n",
      "    ectoplasmic battery-powered participatory fourth still-to-be-named\n",
      "    multilingual multi-disciplinary ...\n",
      "JJR: adjective, comparative\n",
      "    bleaker braver breezier briefer brighter brisker broader bumper busier\n",
      "    calmer cheaper choosier cleaner clearer closer colder commoner costlier\n",
      "    cozier creamier crunchier cuter ...\n",
      "JJS: adjective, superlative\n",
      "    calmest cheapest choicest classiest cleanest clearest closest commonest\n",
      "    corniest costliest crassest creepiest crudest cutest darkest deadliest\n",
      "    dearest deepest densest dinkiest ...\n",
      "LS: list item marker\n",
      "    A A. B B. C C. D E F First G H I J K One SP-44001 SP-44002 SP-44005\n",
      "    SP-44007 Second Third Three Two * a b c d first five four one six three\n",
      "    two\n",
      "MD: modal auxiliary\n",
      "    can cannot could couldn't dare may might must need ought shall should\n",
      "    shouldn't will would\n",
      "NN: noun, common, singular or mass\n",
      "    common-carrier cabbage knuckle-duster Casino afghan shed thermostat\n",
      "    investment slide humour falloff slick wind hyena override subhumanity\n",
      "    machinist ...\n",
      "NNP: noun, proper, singular\n",
      "    Motown Venneboerger Czestochwa Ranzer Conchita Trumplane Christos\n",
      "    Oceanside Escobar Kreisler Sawyer Cougar Yvette Ervin ODI Darryl CTCA\n",
      "    Shannon A.K.C. Meltex Liverpool ...\n",
      "NNPS: noun, proper, plural\n",
      "    Americans Americas Amharas Amityvilles Amusements Anarcho-Syndicalists\n",
      "    Andalusians Andes Andruses Angels Animals Anthony Antilles Antiques\n",
      "    Apache Apaches Apocrypha ...\n",
      "NNS: noun, common, plural\n",
      "    undergraduates scotches bric-a-brac products bodyguards facets coasts\n",
      "    divestitures storehouses designs clubs fragrances averages\n",
      "    subjectivists apprehensions muses factory-jobs ...\n",
      "PDT: pre-determiner\n",
      "    all both half many quite such sure this\n",
      "POS: genitive marker\n",
      "    ' 's\n",
      "PRP: pronoun, personal\n",
      "    hers herself him himself hisself it itself me myself one oneself ours\n",
      "    ourselves ownself self she thee theirs them themselves they thou thy us\n",
      "PRP$: pronoun, possessive\n",
      "    her his mine my our ours their thy your\n",
      "RB: adverb\n",
      "    occasionally unabatingly maddeningly adventurously professedly\n",
      "    stirringly prominently technologically magisterially predominately\n",
      "    swiftly fiscally pitilessly ...\n",
      "RBR: adverb, comparative\n",
      "    further gloomier grander graver greater grimmer harder harsher\n",
      "    healthier heavier higher however larger later leaner lengthier less-\n",
      "    perfectly lesser lonelier longer louder lower more ...\n",
      "RBS: adverb, superlative\n",
      "    best biggest bluntest earliest farthest first furthest hardest\n",
      "    heartiest highest largest least less most nearest second tightest worst\n",
      "RP: particle\n",
      "    aboard about across along apart around aside at away back before behind\n",
      "    by crop down ever fast for forth from go high i.e. in into just later\n",
      "    low more off on open out over per pie raising start teeth that through\n",
      "    under unto up up-pp upon whole with you\n",
      "SYM: symbol\n",
      "    % & ' '' ''. ) ). * + ,. < = > @ A[fj] U.S U.S.S.R * ** ***\n",
      "TO: \"to\" as preposition or infinitive marker\n",
      "    to\n",
      "UH: interjection\n",
      "    Goodbye Goody Gosh Wow Jeepers Jee-sus Hubba Hey Kee-reist Oops amen\n",
      "    huh howdy uh dammit whammo shucks heck anyways whodunnit honey golly\n",
      "    man baby diddle hush sonuvabitch ...\n",
      "VB: verb, base form\n",
      "    ask assemble assess assign assume atone attention avoid bake balkanize\n",
      "    bank begin behold believe bend benefit bevel beware bless boil bomb\n",
      "    boost brace break bring broil brush build ...\n",
      "VBD: verb, past tense\n",
      "    dipped pleaded swiped regummed soaked tidied convened halted registered\n",
      "    cushioned exacted snubbed strode aimed adopted belied figgered\n",
      "    speculated wore appreciated contemplated ...\n",
      "VBG: verb, present participle or gerund\n",
      "    telegraphing stirring focusing angering judging stalling lactating\n",
      "    hankerin' alleging veering capping approaching traveling besieging\n",
      "    encrypting interrupting erasing wincing ...\n",
      "VBN: verb, past participle\n",
      "    multihulled dilapidated aerosolized chaired languished panelized used\n",
      "    experimented flourished imitated reunifed factored condensed sheared\n",
      "    unsettled primed dubbed desired ...\n",
      "VBP: verb, present tense, not 3rd person singular\n",
      "    predominate wrap resort sue twist spill cure lengthen brush terminate\n",
      "    appear tend stray glisten obtain comprise detest tease attract\n",
      "    emphasize mold postpone sever return wag ...\n",
      "VBZ: verb, present tense, 3rd person singular\n",
      "    bases reconstructs marks mixes displeases seals carps weaves snatches\n",
      "    slumps stretches authorizes smolders pictures emerges stockpiles\n",
      "    seduces fizzes uses bolsters slaps speaks pleads ...\n",
      "WDT: WH-determiner\n",
      "    that what whatever which whichever\n",
      "WP: WH-pronoun\n",
      "    that what whatever whatsoever which who whom whosoever\n",
      "WP$: WH-pronoun, possessive\n",
      "    whose\n",
      "WRB: Wh-adverb\n",
      "    how however whence whenever where whereby whereever wherein whereof why\n",
      "``: opening quotation mark\n",
      "    ` ``\n"
     ]
    }
   ],
   "source": [
    "nltk.help.upenn_tagset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some distinctions are quite **hard for both humans and machines** :\n",
    "\n",
    "- preposition (IN), particle (RP) or adverb (RB)?\n",
    "    - You'll never get *around*/RP to watching all those movies\n",
    "    - Let's go *around*/IN the corner\n",
    "    - It costs *around*/RB 500 bucks\n",
    "\n",
    "\n",
    "- past participle (VBN) or adjective (JJ)?\n",
    "    - What the teacher said, has *interested*/VBN the student\n",
    "    - The student was *interested*/?? in what the teacher was saying\n",
    "    - The *interested*/JJ student was listening to the teacher\n",
    "\n",
    "\n",
    "- sometimes nouns can act like adjectives (i.e. they modify nouns):\n",
    "    - *science* teacher\n",
    "    - *government road accident research* centre"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Insecure about a **specific tag**? Ask NLTK!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "JJ: adjective or numeral, ordinal\n",
      "    third ill-mannered pre-war regrettable oiled calamitous first separable\n",
      "    ectoplasmic battery-powered participatory fourth still-to-be-named\n",
      "    multilingual multi-disciplinary ...\n"
     ]
    }
   ],
   "source": [
    "nltk.help.upenn_tagset(\"JJ\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NN: noun, common, singular or mass\n",
      "    common-carrier cabbage knuckle-duster Casino afghan shed thermostat\n",
      "    investment slide humour falloff slick wind hyena override subhumanity\n",
      "    machinist ...\n",
      "NNP: noun, proper, singular\n",
      "    Motown Venneboerger Czestochwa Ranzer Conchita Trumplane Christos\n",
      "    Oceanside Escobar Kreisler Sawyer Cougar Yvette Ervin ODI Darryl CTCA\n",
      "    Shannon A.K.C. Meltex Liverpool ...\n",
      "NNPS: noun, proper, plural\n",
      "    Americans Americas Amharas Amityvilles Amusements Anarcho-Syndicalists\n",
      "    Andalusians Andes Andruses Angels Animals Anthony Antilles Antiques\n",
      "    Apache Apaches Apocrypha ...\n",
      "NNS: noun, common, plural\n",
      "    undergraduates scotches bric-a-brac products bodyguards facets coasts\n",
      "    divestitures storehouses designs clubs fragrances averages\n",
      "    subjectivists apprehensions muses factory-jobs ...\n"
     ]
    }
   ],
   "source": [
    "# a group of tags\n",
    "nltk.help.upenn_tagset(\"NN*\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Other tagsets** are available in NLTK, e.g., the Brown corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NN: noun, singular, common\n",
      "    failure burden court fire appointment awarding compensation Mayor\n",
      "    interim committee fact effect airport management surveillance jail\n",
      "    doctor intern extern night weekend duty legislation Tax Office ...\n",
      "NN$: noun, singular, common, genitive\n",
      "    season's world's player's night's chapter's golf's football's\n",
      "    baseball's club's U.'s coach's bride's bridegroom's board's county's\n",
      "    firm's company's superintendent's mob's Navy's ...\n",
      "NN+BEZ: noun, singular, common + verb 'to be', present tense, 3rd person singular\n",
      "    water's camera's sky's kid's Pa's heat's throat's father's money's\n",
      "    undersecretary's granite's level's wife's fat's Knife's fire's name's\n",
      "    hell's leg's sun's roulette's cane's guy's kind's baseball's ...\n",
      "NN+HVD: noun, singular, common + verb 'to have', past tense\n",
      "    Pa'd\n",
      "NN+HVZ: noun, singular, common + verb 'to have', present tense, 3rd person singular\n",
      "    guy's Knife's boat's summer's rain's company's\n",
      "NN+IN: noun, singular, common + preposition\n",
      "    buncha\n",
      "NN+MD: noun, singular, common + modal auxillary\n",
      "    cowhand'd sun'll\n",
      "NN+NN: noun, singular, common, hyphenated pair\n",
      "    stomach-belly\n",
      "NNS: noun, plural, common\n",
      "    irregularities presentments thanks reports voters laws legislators\n",
      "    years areas adjustments chambers $100 bonds courts sales details raises\n",
      "    sessions members congressmen votes polls calls ...\n",
      "NNS$: noun, plural, common, genitive\n",
      "    taxpayers' children's members' States' women's cutters' motorists'\n",
      "    steelmakers' hours' Nations' lawyers' prisoners' architects' tourists'\n",
      "    Employers' secretaries' Rogues' ...\n",
      "NNS+MD: noun, plural, common + modal auxillary\n",
      "    duds'd oystchers'll\n",
      "NP: noun, singular, proper\n",
      "    Fulton Atlanta September-October Durwood Pye Ivan Allen Jr. Jan.\n",
      "    Alpharetta Grady William B. Hartsfield Pearl Williams Aug. Berry J. M.\n",
      "    Cheshire Griffin Opelika Ala. E. Pelham Snodgrass ...\n",
      "NP$: noun, singular, proper, genitive\n",
      "    Green's Landis' Smith's Carreon's Allison's Boston's Spahn's Willie's\n",
      "    Mickey's Milwaukee's Mays' Howsam's Mantle's Shaw's Wagner's Rickey's\n",
      "    Shea's Palmer's Arnold's Broglio's ...\n",
      "NP+BEZ: noun, singular, proper + verb 'to be', present tense, 3rd person singular\n",
      "    W.'s Ike's Mack's Jack's Kate's Katharine's Black's Arthur's Seaton's\n",
      "    Buckhorn's Breed's Penny's Rob's Kitty's Blackwell's Myra's Wally's\n",
      "    Lucille's Springfield's Arlene's\n",
      "NP+HVZ: noun, singular, proper + verb 'to have', present tense, 3rd person singular\n",
      "    Bill's Guardino's Celie's Skolman's Crosson's Tim's Wally's\n",
      "NP+MD: noun, singular, proper + modal auxillary\n",
      "    Gyp'll John'll\n",
      "NPS: noun, plural, proper\n",
      "    Chases Aderholds Chapelles Armisteads Lockies Carbones French Marskmen\n",
      "    Toppers Franciscans Romans Cadillacs Masons Blacks Catholics British\n",
      "    Dixiecrats Mississippians Congresses ...\n",
      "NPS$: noun, plural, proper, genitive\n",
      "    Republicans' Orioles' Birds' Yanks' Redbirds' Bucs' Yankees' Stevenses'\n",
      "    Geraghtys' Burkes' Wackers' Achaeans' Dresbachs' Russians' Democrats'\n",
      "    Gershwins' Adventists' Negroes' Catholics' ...\n",
      "NR: noun, singular, adverbial\n",
      "    Friday home Wednesday Tuesday Monday Sunday Thursday yesterday tomorrow\n",
      "    tonight West East Saturday west left east downtown north northeast\n",
      "    southeast northwest North South right ...\n",
      "NR$: noun, singular, adverbial, genitive\n",
      "    Saturday's Monday's yesterday's tonight's tomorrow's Sunday's\n",
      "    Wednesday's Friday's today's Tuesday's West's Today's South's\n",
      "NR+MD: noun, singular, adverbial + modal auxillary\n",
      "    today'll\n",
      "NRS: noun, plural, adverbial\n",
      "    Sundays Mondays Saturdays Wednesdays Souths Fridays\n"
     ]
    }
   ],
   "source": [
    "# Brown corpus tagset\n",
    "\n",
    "nltk.help.brown_tagset(\"NN*\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using a Tagger"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `nltk.pos_tag()` method allows you to launch the default NLTK pos tagger model on a **tokenized text** (i.e. on a list of words):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('And', 'CC'), ('now', 'RB'), ('for', 'IN'), ('something', 'NN'), ('completely', 'RB'), ('different', 'JJ')]\n"
     ]
    }
   ],
   "source": [
    "text = nltk.word_tokenize(\"And now for something completely different\")\n",
    "# text = nltk.word_tokenize(\"I'm going to my place.\")\n",
    "print(nltk.pos_tag(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PoS-tagging requires a certain amount of disambiguation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('They', 'PRP'), ('refuse', 'VBP'), ('to', 'TO'), ('permit', 'VB'), ('us', 'PRP'), ('to', 'TO'), ('obtain', 'VB'), ('the', 'DT'), ('refuse', 'NN'), ('permit', 'NN')]\n"
     ]
    }
   ],
   "source": [
    "# compare the two classifications of the type \"refuse\"\n",
    "text = nltk.word_tokenize(\"They refuse to permit us to obtain the refuse permit\")\n",
    "print(nltk.pos_tag(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `nltk.pos_tag_sents()` method is recommened for efficient tagging of more than one sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[('And', 'CC'), ('now', 'RB'), ('for', 'IN'), ('something', 'NN'), ('completely', 'RB'), ('different', 'JJ')], [('They', 'PRP'), ('refuse', 'VBP'), ('to', 'TO'), ('permit', 'VB'), ('us', 'PRP'), ('to', 'TO'), ('obtain', 'VB'), ('the', 'DT'), ('refuse', 'NN'), ('permit', 'NN')]]\n"
     ]
    }
   ],
   "source": [
    "print(nltk.pos_tag_sents([nltk.word_tokenize(\"And now for something completely different\"),\\\n",
    "                          nltk.word_tokenize(\"They refuse to permit us to obtain the refuse permit\"), ]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other tagsets can be selected by using the `tagset` parameter. \n",
    "\n",
    "- For instance, to use the following **Universal tagset** proposed by [Petrov et al (2011)](https://arxiv.org/abs/1104.2086):\n",
    "\n",
    "|  tag  | meaning |\n",
    "|:-----:|:---------|\n",
    "| VERB | verbs (all tenses and modes) |\n",
    "| NOUN | nouns (common and proper) |\n",
    "| PRON | pronouns |\n",
    "| ADJ | adjectives |\n",
    "| ADV | adverbs |\n",
    "| ADP | adpositions (prepositions and postpositions) |\n",
    "| CONJ | conjunctions |\n",
    "| DET | determiners |\n",
    "| NUM | cardinal numbers |\n",
    "| PRT | particles or other function words |\n",
    "| X | other: foreign words, typos, abbreviations |\n",
    "| \\. | punctuation |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('They', 'PRON'), ('refuse', 'VERB'), ('to', 'PRT'), ('permit', 'VERB'), ('us', 'PRON'), ('to', 'PRT'), ('obtain', 'VERB'), ('the', 'DET'), ('refuse', 'NOUN'), ('permit', 'NOUN')]\n"
     ]
    }
   ],
   "source": [
    "text = nltk.word_tokenize(\"They refuse to permit us to obtain the refuse permit\")\n",
    "print(nltk.pos_tag(text, tagset=\"universal\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Available Taggers\n",
    "\n",
    "The default pos tagger model used by NLTK is the [Perceptron](https://github.com/sloria/textblob-aptagger) tagger. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NLTK provides other pos taggers among which:\n",
    "\n",
    "- the [TnT](http://www.coli.uni-saarland.de/~thorsten/tnt/) tagger: `nltk.tag.tnt.TnT()`\n",
    "\n",
    "\n",
    "- the [CRF](http://www.nltk.org/_modules/nltk/tag/crf.html) tagger: `nltk.tag.CRFTagger()`\n",
    "\n",
    "\n",
    "- various simple [N-gram](http://www.nltk.org/book/ch05.html#general-n-gram-tagging) taggers:  `nltk.tag.UnigramTagger()`, `nltk.tag.BigramTagger()`, `nltk.tag.TrigramTagger()` \n",
    "\n",
    "\n",
    "- a [regex-based](http://www.nltk.org/book/ch05.html#the-regular-expression-tagger) tagger `nltk.tag.sequential.RegexpTagger()`\n",
    "\n",
    "\n",
    "- a tagger that assign always the same tag ot each token: `nltk.tag.DefaultTagger()`, that can be used as a [backoff](http://www.nltk.org/book/ch05.html#combining-taggers) strategy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Using the Stanford PoS Tagger"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "NLTK provides interfaces to three tools of the Stanford CoreNLP suite:\n",
    "\n",
    "- the Stanford Log-linear Part-Of-Speech Tagger\n",
    "\n",
    "\n",
    "- the Stanford Parser \n",
    "\n",
    "\n",
    "- the Stanford Named Entity Recognizer (NER)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "The Stanford POS Tagger can be download from the [download section](https://nlp.stanford.edu/software/tagger.html#Download) of the official site. Two versions are available:\n",
    "\n",
    "\n",
    "- the basic English Stanford Tagger [25 MB]\n",
    "\n",
    "\n",
    "- the full Stanford Tagger [129 MB]\n",
    "\n",
    "Both versions contain a lot of pos tagger models, you can find more infos in `models/README-Models.txt "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the Stanford POS Tagger with the nltk.tag.stanford module"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download one of the two Stanford packages above (the basic 25 MB one will do for now). Extract to your notebook working directory, and create an instance of `StanfordPOSTagger` with path arguments to a language model (here: 'english-bidirectional-distsim.tagger') and the Stanford tagger jar file (here: 'stanford-postagger-3.9.1.jar'). Using the `tag` method, we can now apply POS tags to a (split) string."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "scrolled": true
   },
   "source": [
    "```Python\n",
    "from nltk.tag import StanfordPOSTagger\n",
    "\n",
    "st = StanfordPOSTagger('./stanford-postagger-2018-02-27/models/english-bidirectional-distsim.tagger',\\\n",
    "                       path_to_jar='./stanford-postagger-2018-02-27/stanford-postagger-3.9.1.jar')\n",
    "\n",
    "st.tag('What is the airspeed of an unladen swallow ?'.split())\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Suggested Readings**\n",
    ">\n",
    ">A nice tutorial on how NLTK taggers can be evaluated is available on [Natemccoy's Blog](https://natemccoy.github.io/2016/10/27/evaluatingnltktaggerstutorial.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tagged Corpora\n",
    "\n",
    "Several of the corpora included with NLTK have been tagged for their part-of-speech. \n",
    "\n",
    "Whenever a corpus contains tagged text, the NLTK corpus interface will have a `tagged_words()` method that will print `(token, tag)` tuples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('Pierre', 'NNP'), ('Vinken', 'NNP'), (',', ','), ('61', 'CD'), ('years', 'NNS'), ('old', 'JJ'), (',', ','), ('will', 'MD'), ('join', 'VB'), ('the', 'DT')]\n"
     ]
    }
   ],
   "source": [
    "print(nltk.corpus.treebank.tagged_words()[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A related method is `tagged_sents()`, that will retain the sentence structure of the text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[('Pierre', 'NNP'), ('Vinken', 'NNP'), (',', ','), ('61', 'CD'), ('years', 'NNS'), ('old', 'JJ'), (',', ','), ('will', 'MD'), ('join', 'VB'), ('the', 'DT'), ('board', 'NN'), ('as', 'IN'), ('a', 'DT'), ('nonexecutive', 'JJ'), ('director', 'NN'), ('Nov.', 'NNP'), ('29', 'CD'), ('.', '.')], [('Mr.', 'NNP'), ('Vinken', 'NNP'), ('is', 'VBZ'), ('chairman', 'NN'), ('of', 'IN'), ('Elsevier', 'NNP'), ('N.V.', 'NNP'), (',', ','), ('the', 'DT'), ('Dutch', 'NNP'), ('publishing', 'VBG'), ('group', 'NN'), ('.', '.')], [('Rudolph', 'NNP'), ('Agnew', 'NNP'), (',', ','), ('55', 'CD'), ('years', 'NNS'), ('old', 'JJ'), ('and', 'CC'), ('former', 'JJ'), ('chairman', 'NN'), ('of', 'IN'), ('Consolidated', 'NNP'), ('Gold', 'NNP'), ('Fields', 'NNP'), ('PLC', 'NNP'), (',', ','), ('was', 'VBD'), ('named', 'VBN'), ('*-1', '-NONE-'), ('a', 'DT'), ('nonexecutive', 'JJ'), ('director', 'NN'), ('of', 'IN'), ('this', 'DT'), ('British', 'JJ'), ('industrial', 'JJ'), ('conglomerate', 'NN'), ('.', '.')]]\n"
     ]
    }
   ],
   "source": [
    "print(nltk.corpus.treebank.tagged_sents()[:3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not all corpora employ the same tagset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('The', 'AT'), ('Fulton', 'NP-TL'), ('County', 'NN-TL'), ('Grand', 'JJ-TL'), ('Jury', 'NN-TL'), ('said', 'VBD'), ('Friday', 'NR'), ('an', 'AT'), ('investigation', 'NN'), ('of', 'IN'), (\"Atlanta's\", 'NP$'), ('recent', 'JJ'), ('primary', 'NN'), ('election', 'NN'), ('produced', 'VBD'), ('``', '``'), ('no', 'AT'), ('evidence', 'NN'), (\"''\", \"''\"), ('that', 'CS'), ('any', 'DTI'), ('irregularities', 'NNS'), ('took', 'VBD'), ('place', 'NN'), ('.', '.'), ('The', 'AT'), ('jury', 'NN'), ('further', 'RBR'), ('said', 'VBD'), ('in', 'IN'), ('term-end', 'NN'), ('presentments', 'NNS'), ('that', 'CS'), ('the', 'AT'), ('City', 'NN-TL'), ('Executive', 'JJ-TL'), ('Committee', 'NN-TL'), (',', ','), ('which', 'WDT'), ('had', 'HVD'), ('over-all', 'JJ'), ('charge', 'NN'), ('of', 'IN'), ('the', 'AT'), ('election', 'NN'), (',', ','), ('``', '``'), ('deserves', 'VBZ'), ('the', 'AT'), ('praise', 'NN'), ('and', 'CC'), ('thanks', 'NNS'), ('of', 'IN'), ('the', 'AT'), ('City', 'NN-TL'), ('of', 'IN-TL'), ('Atlanta', 'NP-TL'), (\"''\", \"''\"), ('for', 'IN'), ('the', 'AT'), ('manner', 'NN'), ('in', 'IN'), ('which', 'WDT'), ('the', 'AT'), ('election', 'NN'), ('was', 'BEDZ'), ('conducted', 'VBN'), ('.', '.')]\n"
     ]
    }
   ],
   "source": [
    "print(nltk.corpus.brown.tagged_words()[:68])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Things can be made comparable by using the built-in [mapping](https://github.com/slavpetrov/universal-pos-tags) to the Universal Tagset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('The', 'DET'), ('Fulton', 'NOUN'), ('County', 'NOUN'), ('Grand', 'ADJ'), ('Jury', 'NOUN'), ('said', 'VERB'), ('Friday', 'NOUN'), ('an', 'DET'), ('investigation', 'NOUN'), ('of', 'ADP'), (\"Atlanta's\", 'NOUN'), ('recent', 'ADJ'), ('primary', 'NOUN'), ('election', 'NOUN'), ('produced', 'VERB'), ('``', '.'), ('no', 'DET'), ('evidence', 'NOUN'), (\"''\", '.'), ('that', 'ADP'), ('any', 'DET'), ('irregularities', 'NOUN'), ('took', 'VERB'), ('place', 'NOUN'), ('.', '.'), ('The', 'DET'), ('jury', 'NOUN'), ('further', 'ADV'), ('said', 'VERB'), ('in', 'ADP'), ('term-end', 'NOUN'), ('presentments', 'NOUN'), ('that', 'ADP'), ('the', 'DET'), ('City', 'NOUN'), ('Executive', 'ADJ'), ('Committee', 'NOUN'), (',', '.'), ('which', 'DET'), ('had', 'VERB'), ('over-all', 'ADJ'), ('charge', 'NOUN'), ('of', 'ADP'), ('the', 'DET'), ('election', 'NOUN'), (',', '.'), ('``', '.'), ('deserves', 'VERB'), ('the', 'DET'), ('praise', 'NOUN'), ('and', 'CONJ'), ('thanks', 'NOUN'), ('of', 'ADP'), ('the', 'DET'), ('City', 'NOUN'), ('of', 'ADP'), ('Atlanta', 'NOUN'), (\"''\", '.'), ('for', 'ADP'), ('the', 'DET'), ('manner', 'NOUN'), ('in', 'ADP'), ('which', 'DET'), ('the', 'DET'), ('election', 'NOUN'), ('was', 'VERB'), ('conducted', 'VERB'), ('.', '.')]\n"
     ]
    }
   ],
   "source": [
    "print(nltk.corpus.brown.tagged_words(tagset='universal')[:68])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('Pierre', 'NOUN'), ('Vinken', 'NOUN'), (',', '.'), ('61', 'NUM'), ('years', 'NOUN'), ('old', 'ADJ'), (',', '.'), ('will', 'VERB'), ('join', 'VERB'), ('the', 'DET'), ('board', 'NOUN'), ('as', 'ADP'), ('a', 'DET'), ('nonexecutive', 'ADJ'), ('director', 'NOUN'), ('Nov.', 'NOUN'), ('29', 'NUM'), ('.', '.'), ('Mr.', 'NOUN'), ('Vinken', 'NOUN'), ('is', 'VERB'), ('chairman', 'NOUN'), ('of', 'ADP'), ('Elsevier', 'NOUN'), ('N.V.', 'NOUN'), (',', '.'), ('the', 'DET'), ('Dutch', 'NOUN'), ('publishing', 'VERB'), ('group', 'NOUN'), ('.', '.'), ('Rudolph', 'NOUN'), ('Agnew', 'NOUN'), (',', '.'), ('55', 'NUM'), ('years', 'NOUN'), ('old', 'ADJ'), ('and', 'CONJ'), ('former', 'ADJ'), ('chairman', 'NOUN'), ('of', 'ADP'), ('Consolidated', 'NOUN'), ('Gold', 'NOUN'), ('Fields', 'NOUN'), ('PLC', 'NOUN'), (',', '.'), ('was', 'VERB'), ('named', 'VERB'), ('*-1', 'X'), ('a', 'DET'), ('nonexecutive', 'ADJ'), ('director', 'NOUN'), ('of', 'ADP'), ('this', 'DET'), ('British', 'ADJ'), ('industrial', 'ADJ'), ('conglomerate', 'NOUN'), ('.', '.')]\n"
     ]
    }
   ],
   "source": [
    "print(nltk.corpus.treebank.tagged_words(tagset='universal')[:58])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exploiring Tagged  Corpora"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1. Let's find the most frequent nouns of each noun part-of-speech type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "?nltk.ConditionalFreqDist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NN [('year', 137), ('time', 97), ('state', 88), ('week', 85), ('man', 72)]\n",
      "NN$ [(\"year's\", 13), (\"world's\", 8), (\"state's\", 7), (\"nation's\", 6), (\"city's\", 6)]\n",
      "NN$-HL [(\"Golf's\", 1), (\"Navy's\", 1)]\n",
      "NN$-TL [(\"President's\", 11), (\"Administration's\", 3), (\"Army's\", 3), (\"League's\", 3), (\"University's\", 3)]\n",
      "NN-HL [('sp.', 2), ('problem', 2), ('Question', 2), ('cut', 2), ('party', 2)]\n",
      "NN-NC [('ova', 1), ('eva', 1), ('aya', 1)]\n",
      "NN-TL [('President', 88), ('House', 68), ('State', 59), ('University', 42), ('City', 41)]\n",
      "NN-TL-HL [('Fort', 2), ('Mayor', 1), ('Commissioner', 1), ('City', 1), ('Oak', 1)]\n",
      "NNS [('years', 101), ('members', 69), ('people', 52), ('sales', 51), ('men', 46)]\n",
      "NNS$ [(\"children's\", 7), (\"women's\", 5), (\"men's\", 3), (\"janitors'\", 3), (\"taxpayers'\", 2)]\n",
      "NNS$-HL [(\"Dealers'\", 1), (\"Idols'\", 1)]\n",
      "NNS$-TL [(\"Women's\", 4), (\"States'\", 3), (\"Giants'\", 2), (\"Princes'\", 1), (\"Bombers'\", 1)]\n",
      "NNS-HL [('Wards', 1), ('deputies', 1), ('bonds', 1), ('aspects', 1), ('Decisions', 1)]\n",
      "NNS-TL [('States', 38), ('Nations', 11), ('Masters', 10), ('Communists', 9), ('Rules', 9)]\n",
      "NNS-TL-HL [('Nations', 1)]\n"
     ]
    }
   ],
   "source": [
    "def findtags(tag_prefix, tagged_text):\n",
    "    cfd = nltk.ConditionalFreqDist((tag, word) for (word, tag) in tagged_text\n",
    "                                  if tag.startswith(tag_prefix))\n",
    "    return dict((tag, cfd[tag].most_common(5)) for tag in cfd.conditions())\n",
    "\n",
    "tagdict = findtags('NN', nltk.corpus.brown.tagged_words(categories='news'))\n",
    "\n",
    "for tag in sorted(tagdict):\n",
    "    print(tag, tagdict[tag])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 2. Suppose we're studying the word \"`often`\" and want to look at the part-of-speech tag of the following words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "VERB  ADV  ADP  ADJ    .  PRT \n",
      "  37    8    7    6    4    2 \n"
     ]
    }
   ],
   "source": [
    "brown_lrnd_tagged = nltk.corpus.brown.tagged_words(categories='learned', tagset='universal')\n",
    "\n",
    "tags = [b[1] for (a, b) in nltk.bigrams(brown_lrnd_tagged) if a[0] == 'often']\n",
    "\n",
    "nltk.FreqDist(tags).tabulate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">### Exercises\n",
    "> \n",
    "> \n",
    "> \n",
    "> \n",
    "> \n",
    ">\n",
    "> - Use the word frequencies in the Brown corpus to:\n",
    ">\n",
    ">   - print the top 50 most frequent unigrams and bigrams (together with their frequencies). A bigram is a group of two subsequent words, e.g., \"this is good\": \"this is\", \"is good\". Check the `nltk.bigrams()` function\n",
    ">   - plot them too\n",
    ">\n",
    ">\n",
    "> - Lemmatize the Brown corpus and use the lemma frequencies to plot the 50 most frequent unigrams and bigrams. So, the same analysis as in the previous exercise, but with lemma frequencies this time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}