IntroductionToPyConTextNLP.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simple Clinical Natural Language Processing with pyConTextNLP\n",
    "\n",
    "In this notebook we introduce the basics of pyConTextNLP, a simple Python tool that we have used extensively for processing clinical text, including radiology, psychiatry, etc.\n",
    "\n",
    "pyConTextNLP is built around the concept of **targets** and **modifiers**: the target is the concept we are interested in identifying (like a cough or a pulmonary embolism); a modifier is a concept that changes the target in some sense (e.g. historical, severity, certainty, negation).\n",
    "\n",
    "pyConTextNLP relies on [regular expressions](RegularExpressions.ipynb) to identify concepts (both targets and modifiers) within a sentence and then uses simple lexical rules to assign relationships between the identified targets and modifiers. Internally, pyConTextNLP uses graphs. Targets and modifiers are nodes in the graph and relationships between modifiers and targets are edges in the graph.\n",
    "\n",
    "## Specifying targets, modifiers, and rules\n",
    "\n",
    "pyConTextNLP uses a four-tuple to represent concepts. Within the program we create an instance of an ``itemData`` class. Each ``itemData`` consists of the following four attributres:\n",
    "\n",
    "1. A **literal** (e.g. \"pulmonary embolism\", \"no definite evidence of\"): This is a lingustic representation of the target or modifier we want to identify\n",
    "1. A **category** (e.g. \"CRITICAL_FINDING\", \"PROBABLE_EXISTENCE\"): This is the label we want applied to the literal when we see it in text\n",
    "1. A **regular expression** that defines how to identify the literal concept. If no regular expression is specified, a regular expression will be built directly from the literal by wrapping it with word boundaries (e.g. r\"\"\"\\bpulmonary embolism\\b\"\"\")\n",
    "1. A **rule** that defines how the concept works in the sentence (e.g. a negation term that looks **forward** in the sentence). this only applies to modifiers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!pip install -U pycontextnlp==0.6.1.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pyConTextNLP.pyConTextGraph as pyConText\n",
    "import pyConTextNLP.itemData as itemData"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The task: Identify patients with pulmonary embolism from radiology reports\n",
    "## Step 1: how is the concept of pulmonary embolism represented in the reports - fill in the list below with literals you want to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "mytargets = itemData.itemData()\n",
    "mytargets.extend([[\"pulmonary embolism\", \"CRITICAL_FINDING\", \"\", \"\"],\n",
    "                   [\"pneumonia\", \"CRITICAL_FINDING\", \"\", \"\"]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(mytargets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!pip install -U radnlp==0.2.0.8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentence Splitting\n",
    "\n",
    "pyConTextNLP operates on a *sentence* level and so the first step we need to take is to split our document into individual sentences. pyConTextNLP comes with a simple sentence splitter class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pyConTextNLP.helpers as helpers\n",
    "spliter = helpers.sentenceSplitter()\n",
    "spliter.splitSentences(\"This is Dr. Chapman's first sentence. This is the 2.0 sentence.\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, sentence splitting is a common NLP task and so most full-fledged NLP applications provide sentence splitters. We usually rely on the sentence splitter that is part of the [TextBlob](https://textblob.readthedocs.io/en/dev/) package, which in turn relies on the Natural Language Toolkit ([NLTK](http://www.nltk.org/)). So before proceeding we need to download some NLTK resources with the following command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!python -m textblob.download_corpora"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [6950]",
   "language": "python",
   "name": "Python [6950]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}