BasicSentenceMarkup.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Demonstration of Basic Sentence Markup with pyConTextNLP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "pyConTextNLP uses NetworkX directional graphs to represent the markup: nodes in the graph will be the concepts that are identified in the sentence and edges in the graph will be the relationships between those concepts. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pyConTextNLP.pyConTextGraph as pyConText\n",
    "import pyConTextNLP.itemData as itemData\n",
    "import networkx as nx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ``pyConTextGraph`` contains the bulk of the pyConTextNLP functionality, including basic class definitions such as the ``ConTextMarkup`` class that represents the markup of a sentence.\n",
    "* ``itemData`` contains a class definition for an itemData and functions for reading itemData definitions which are assumed to be in a tab seperated file that is specified as either a local file or a remote resource. In this example we will read definitions straight from the GitHub repository.\n",
    "    * An ``itemData`` in its most basic form is a four-tuple consisting of \n",
    "        1. A **literal** (e.g. \"pulmonary embolism\", \"no definite evidence of\")\n",
    "        1. A **category** (e.g. \"CRITICAL_FINDING\", \"PROBABLE_EXISTENCE\")\n",
    "        1. A **regular expression** that defines how to identify the literal concept. If no regular expression is specified, a regular expression will be built directly from the literal by wrapping it with word boundaries (e.g. r\"\"\"\\bpulmonary embolism\\b\"\"\")\n",
    "        1. A **rule** that defines how the concept works in the sentence (e.g. a negation term that looks **forward** in the sentence)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sentences\n",
    "\n",
    "These example reports are taken from (with modification) the [MIMIC2 demo data set](https://physionet.org/mimic2/) that is a publically available database of de-identified medical records for deceased individuals. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reports = [\n",
    "    \"\"\"IMPRESSION: Evaluation limited by lack of IV contrast; however, no evidence of\n",
    "      bowel obstruction or mass identified within the abdomen or pelvis. Non-specific interstitial opacities and bronchiectasis seen at the right\n",
    "     base, suggestive of post-inflammatory changes.\"\"\",\n",
    "    \"\"\"IMPRESSION: Evidence of early pulmonary vascular congestion and interstitial edema. Probable scarring at the medial aspect of the right lung base, with no\n",
    "     definite consolidation.\"\"\"\n",
    "    ,\n",
    "    \"\"\"IMPRESSION:\n",
    "     \n",
    "     1.  2.0 cm cyst of the right renal lower pole.  Otherwise, normal appearance\n",
    "     of the right kidney with patent vasculature and no sonographic evidence of\n",
    "     renal artery stenosis.\n",
    "     2.  Surgically absent left kidney.\"\"\",\n",
    "    \"\"\"IMPRESSION:  No pneumothorax.\"\"\",\n",
    "    \"\"\"IMPRESSION: No definite pneumothorax\"\"\"\n",
    "    \"\"\"IMPRESSION:  New opacity at the left lower lobe consistent with pneumonia.\"\"\"\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read the ``itemData`` definitions\n",
    "\n",
    "We're reading directly from GitHub. You could read from a local file using a `file://` URL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "modifiers = itemData.instantiateFromCSVtoitemData(\n",
    "    \"https://raw.githubusercontent.com/chapmanbe/pyConTextNLP/master/KB/lexical_kb_05042016.tsv\")\n",
    "targets = itemData.instantiateFromCSVtoitemData(\n",
    "    \"https://raw.githubusercontent.com/chapmanbe/pyConTextNLP/master/KB/utah_crit.tsv\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example function to analyze each sentence\n",
    "\n",
    "This the function we'll use for each report. The following section of this document steps through each line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def markup_sentence(s, modifiers, targets, prune_inactive=True):\n",
    "    \"\"\"\n",
    "    \"\"\"\n",
    "    markup = pyConText.ConTextMarkup()\n",
    "    markup.setRawText(s)\n",
    "    markup.cleanText()\n",
    "    markup.markItems(modifiers, mode=\"modifier\")\n",
    "    markup.markItems(targets, mode=\"target\")\n",
    "    markup.pruneMarks()\n",
    "    markup.dropMarks('Exclusion')\n",
    "    # apply modifiers to any targets within the modifiers scope\n",
    "    markup.applyModifiers()\n",
    "    markup.pruneSelfModifyingRelationships()\n",
    "    if prune_inactive:\n",
    "        markup.dropInactiveModifiers()\n",
    "    return markup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We're going to start with our simplest of sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'IMPRESSION:  No pneumothorax.'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reports[3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### marking up a sentence\n",
    "\n",
    "We start by creating an instance of the ``ConTextMarkup`` class. This is a subclass of a NetworkX DiGraph. Information will be stored in the nodes and edges. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "markup = pyConText.ConTextMarkup()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "isinstance(markup,nx.DiGraph)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#### Set the text to be processed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________\n",
      "rawText: impression:  no pneumothorax.\n",
      "cleanedText: None\n",
      "__________________________________________\n",
      "\n",
      "29\n"
     ]
    }
   ],
   "source": [
    "markup.setRawText(reports[3].lower())\n",
    "print(markup)\n",
    "print(len(markup.getRawText()))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Clean the text\n",
    "\n",
    "Prior to processing we do some basic cleaning of the text, sucha s replacing multiple white spaces with a single space. You'll notice this in the spacing between the colon and \"no\" in the ``raw`` and ``clean`` versions of the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________\n",
      "rawText: impression:  no pneumothorax.\n",
      "cleanedText: impression: no pneumothorax.\n",
      "__________________________________________\n",
      "\n",
      "28\n"
     ]
    }
   ],
   "source": [
    "markup.cleanText()\n",
    "print(markup)\n",
    "print(len(markup.getText()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Identify concepts in the sentence\n",
    "\n",
    "The ``markItems`` method takes a list of itemData and uses the regular expressions to identify any instances of the itemData in the sentence. With the ``mode`` keyword we specify whether these ``itemData`` are targets or modifiers. This value will be stored as a data attribute of the node that is created in the graph for any identified concepts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(<id> 215455046329351668380259283016206441424 </id> <phrase> no </phrase> <category> ['definite_negated_existence'] </category> , {'category': 'modifier'})]\n",
      "<class 'pyConTextNLP.pyConTextGraph.tagObject'>\n"
     ]
    }
   ],
   "source": [
    "markup.markItems(modifiers, mode=\"modifier\")\n",
    "print(markup.nodes(data=True))\n",
    "print(type(markup.nodes()[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What does our initial markup look like?\n",
    "\n",
    "* We've identified one concept in the sentence: ``no``\n",
    "* We've created a ``tagObject`` for this concept which keeps track of the actual phrase identified by the regular expression, what the category of the itemData was (``definite_negated_existence``), this is a list because there can be multiple categories. There is also an absurdly long identifier for the node. Note that our mode ``modifier`` has been stored as a data element of the node. In NetworkX each node (or edge) has a dictionary for data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Now let's markup the targets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "markup.markItems(targets, mode=\"target\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(<id> 215496472358685501776852452868303018960 </id> <phrase> pneumothorax </phrase> <category> ['pneumothorax'] </category> , {'category': 'target'}), (<id> 215455046329351668380259283016206441424 </id> <phrase> no </phrase> <category> ['definite_negated_existence'] </category> , {'category': 'modifier'})]\n"
     ]
    }
   ],
   "source": [
    "print(markup.nodes(data=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What does our markup look like now?\n",
    "\n",
    "We've added another node to the graph. This time the ``target`` ``pneumothorax``."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Prune Marks\n",
    "\n",
    "After identifying concepts, we prune concepts that are a subset of another identified concept. This results in no changes here, but the importance will be shown later with a different sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<id> 215496472358685501776852452868303018960 </id> <phrase> pneumothorax </phrase> <category> ['pneumothorax'] </category> , <id> 215455046329351668380259283016206441424 </id> <phrase> no </phrase> <category> ['definite_negated_existence'] </category> ]\n"
     ]
    }
   ],
   "source": [
    "markup.pruneMarks()\n",
    "print(markup.nodes())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Are there any relationships in our markup?\n",
    "\n",
    "We do not yet have any relationships (edges) between our concepts (target and modifier edges)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[]\n"
     ]
    }
   ],
   "source": [
    "print(markup.edges())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Apply modifiers\n",
    "\n",
    "We now call the ``applyModifiers`` method of the ConTextMarkup object to identify any relationships between the nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(<id> 215455046329351668380259283016206441424 </id> <phrase> no </phrase> <category> ['definite_negated_existence'] </category> , <id> 215496472358685501776852452868303018960 </id> <phrase> pneumothorax </phrase> <category> ['pneumothorax'] </category> )]\n"
     ]
    }
   ],
   "source": [
    "markup.applyModifiers()\n",
    "print(markup.edges())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We now have a relationship!\n",
    "\n",
    "We now have a directed edge between our ``no`` node and our ``pneumothorax`` node. This will be interepreted as ``pneumothorax`` being a definitely negated concept in the sentence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What's next?\n",
    "\n",
    "The value of pruning is shown in [this](./BasicSentencemarkupPart2.ipynb) notebook."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}