CS_1_Codes_Desc_Embeddings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "naPWqtUVigTd"
   },
   "source": [
    "---\n",
    "---\n",
    "# **`CASE STUDY - 1` :: Healthcare Provider Fraudulent Detection**\n",
    "\n",
    "<img src=\"Display_Pic.png\" alt=\"drawing\" width=\"950\"/>\n",
    "\n",
    "---\n",
    "---\n",
    "\n",
    "\n",
    "**This notebook contains the sentence embeddings generated on the ICD-9 Universal Diagnosis and Procedure Codes from the pre-trained Bio-BERT model performed on the publicly available dataset at [Kaggle](https://www.kaggle.com/datasets/rohitrox/healthcare-provider-fraud-detection-analysis) with the intent.**\n",
    "\n",
    "\n",
    "**Kindly checkout below link for gaining BUSINESS related insights related to this problem ::**\n",
    "\n",
    "- [Deck : Detailed Explanation](https://docs.google.com/presentation/d/1Thuw_eZskafkl9W3xYuEVsjgTJKFeEkzEfAmczH96Uw/)\n",
    "\n",
    "\n",
    "**Kindly checkout below link for TECHNICAL design document ::**\n",
    "\n",
    "- [Technical Document](https://docs.google.com/document/d/10z9xbn4dZWkforlAszCDaa1M0roAW-MOf2pjJHSnEXk/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PHkkU2ExpgPR"
   },
   "source": [
    "# **`Notebook Contents`**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "toc",
    "id": "ZsqH5e9kpIzv"
   },
   "source": [
    ">[CASE STUDY - 1 :: Healthcare Provider Fraudulent Detection](#scrollTo=naPWqtUVigTd)\n",
    "\n",
    ">[Notebook Contents](#scrollTo=PHkkU2ExpgPR)\n",
    "\n",
    ">>[Downloading_Train_Data_Files](#scrollTo=Ry-TJ1anmAdO)\n",
    "\n",
    ">>[Importing_Libraries](#scrollTo=ThPD8j4PnuVH)\n",
    "\n",
    ">>>[VECTOR_Embeddings](#scrollTo=xhJqQq7R5XSz)\n",
    "\n",
    ">>>>[Pre-processing the CODES](#scrollTo=Er0qcZNk5XS8)\n",
    "\n",
    ">>[SUMMARY](#scrollTo=tbWnhQ0qgiCm)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ry-TJ1anmAdO"
   },
   "source": [
    "## **Downloading_Train_Data_Files**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "swatiZXYnuOv",
    "outputId": "411069db-33ae-44fe-b538-ba0bbc0b7f82"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading...\n",
      "From: https://drive.google.com/uc?id=12zSQN2FOxmuXFhz2xzPNussPisEfVP5w\n",
      "To: /content/Train_Outpatientdata-1542865627584.csv\n",
      "100% 77.4M/77.4M [00:00<00:00, 113MB/s] \n",
      "Downloading...\n",
      "From: https://drive.google.com/uc?id=13XyBakfHiG-BNQPrYFXAHlsOcfICOTpx\n",
      "To: /content/Train_Beneficiarydata-1542865627584.csv\n",
      "100% 11.4M/11.4M [00:00<00:00, 40.2MB/s]\n",
      "Downloading...\n",
      "From: https://drive.google.com/uc?id=1dLxl4vkykPcm4Zj0abYR0Ohr7STQHg-1\n",
      "To: /content/Train-1542865627584.csv\n",
      "100% 87.1k/87.1k [00:00<00:00, 78.4MB/s]\n",
      "Downloading...\n",
      "From: https://drive.google.com/uc?id=1rFER-7VuYb7GfCYeJrfxPidgK0lwqw3R\n",
      "To: /content/Train_Inpatientdata-1542865627584.csv\n",
      "100% 8.57M/8.57M [00:00<00:00, 62.2MB/s]\n",
      "Downloading...\n",
      "From: https://drive.google.com/uc?id=1092Kqiqo1VFhVOgy2ttuW4Vu1pRTEOKp\n",
      "To: /content/CMS32_DESC_LONG_SHORT_SG.xlsx\n",
      "100% 194k/194k [00:00<00:00, 65.3MB/s]\n",
      "Downloading...\n",
      "From: https://drive.google.com/uc?id=1mmzARIlfF3TkZQljk74bZ6xkl7azyy3K\n",
      "To: /content/CMS32_DESC_LONG_SHORT_DX.xlsx\n",
      "100% 664k/664k [00:00<00:00, 112MB/s]\n",
      "Downloading...\n",
      "From: https://drive.google.com/uc?id=11TPcDjNF5Ptk1bf3c7qM5RLaSfZIbXV2\n",
      "To: /content/Test-1542969243754.csv\n",
      "100% 14.9k/14.9k [00:00<00:00, 22.7MB/s]\n",
      "Downloading...\n",
      "From: https://drive.google.com/uc?id=1PvYHPMxQNNj-E1VvNhGF7nHOiCe9J-xm\n",
      "To: /content/Test_Outpatientdata-1542969243754.csv\n",
      "100% 18.8M/18.8M [00:00<00:00, 83.1MB/s]\n",
      "Downloading...\n",
      "From: https://drive.google.com/uc?id=1RxMPDYXRoRQxUzw9ly1D68Oe2ONIiQbm\n",
      "To: /content/Test_Beneficiarydata-1542969243754.csv\n",
      "100% 5.33M/5.33M [00:00<00:00, 31.8MB/s]\n",
      "Downloading...\n",
      "From: https://drive.google.com/uc?id=1naCnYPra543q1omchA_Y0Xir5twojaeA\n",
      "To: /content/Test_Inpatientdata-1542969243754.csv\n",
      "100% 2.02M/2.02M [00:00<00:00, 126MB/s]\n"
     ]
    }
   ],
   "source": [
    "## TRAIN set files\n",
    "!gdown 12zSQN2FOxmuXFhz2xzPNussPisEfVP5w\n",
    "!gdown 13XyBakfHiG-BNQPrYFXAHlsOcfICOTpx\n",
    "!gdown 1dLxl4vkykPcm4Zj0abYR0Ohr7STQHg-1\n",
    "!gdown 1rFER-7VuYb7GfCYeJrfxPidgK0lwqw3R\n",
    "!gdown 1092Kqiqo1VFhVOgy2ttuW4Vu1pRTEOKp\n",
    "!gdown 1mmzARIlfF3TkZQljk74bZ6xkl7azyy3K\n",
    "\n",
    "## TEST set files\n",
    "!gdown 11TPcDjNF5Ptk1bf3c7qM5RLaSfZIbXV2\n",
    "!gdown 1PvYHPMxQNNj-E1VvNhGF7nHOiCe9J-xm\n",
    "!gdown 1RxMPDYXRoRQxUzw9ly1D68Oe2ONIiQbm\n",
    "!gdown 1naCnYPra543q1omchA_Y0Xir5twojaeA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ThPD8j4PnuVH"
   },
   "source": [
    "## **Importing_Libraries**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "KPorbuy15XR7",
    "outputId": "d43fcf3c-b897-4824-f354-54d41e91e411"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
      "Collecting pyspark==3.0.1\n",
      "  Downloading pyspark-3.0.1.tar.gz (204.2 MB)\n",
      "\u001b[K     |████████████████████████████████| 204.2 MB 29 kB/s \n",
      "\u001b[?25hCollecting py4j==0.10.9\n",
      "  Downloading py4j-0.10.9-py2.py3-none-any.whl (198 kB)\n",
      "\u001b[K     |████████████████████████████████| 198 kB 65.9 MB/s \n",
      "\u001b[?25hBuilding wheels for collected packages: pyspark\n",
      "  Building wheel for pyspark (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for pyspark: filename=pyspark-3.0.1-py2.py3-none-any.whl size=204612243 sha256=b7ce1c4e9f5505babc0708bf5a4cdb4d77cf04b42193250c8a2958c402439fe6\n",
      "  Stored in directory: /root/.cache/pip/wheels/5e/34/fa/b37b5cef503fc5148b478b2495043ba61b079120b7ff379f9b\n",
      "Successfully built pyspark\n",
      "Installing collected packages: py4j, pyspark\n",
      "Successfully installed py4j-0.10.9 pyspark-3.0.1\n"
     ]
    }
   ],
   "source": [
    "pip install pyspark==3.0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "bWEWi2la5XR9",
    "outputId": "83cdcde0-1adf-40b4-94a8-928bd09f2141"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
      "Collecting nlu\n",
      "  Downloading nlu-4.0.0-py3-none-any.whl (570 kB)\n",
      "\u001b[K     |████████████████████████████████| 570 kB 12.7 MB/s \n",
      "\u001b[?25hCollecting spark-nlp<4.1.0,>=4.0.0\n",
      "  Downloading spark_nlp-4.0.2-py2.py3-none-any.whl (532 kB)\n",
      "\u001b[K     |████████████████████████████████| 532 kB 77.1 MB/s \n",
      "\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from nlu) (1.21.6)\n",
      "Requirement already satisfied: pandas>=1.3.5 in /usr/local/lib/python3.7/dist-packages (from nlu) (1.3.5)\n",
      "Requirement already satisfied: pyarrow>=0.16.0 in /usr/local/lib/python3.7/dist-packages (from nlu) (6.0.1)\n",
      "Collecting dataclasses\n",
      "  Downloading dataclasses-0.6-py3-none-any.whl (14 kB)\n",
      "Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=1.3.5->nlu) (2022.2.1)\n",
      "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=1.3.5->nlu) (2.8.2)\n",
      "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.7.3->pandas>=1.3.5->nlu) (1.15.0)\n",
      "Installing collected packages: spark-nlp, dataclasses, nlu\n",
      "Successfully installed dataclasses-0.6 nlu-4.0.0 spark-nlp-4.0.2\n"
     ]
    }
   ],
   "source": [
    "pip install nlu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "l6qT6r78nuZC"
   },
   "outputs": [],
   "source": [
    "import nlu\n",
    "import os\n",
    "import sys\n",
    "import math\n",
    "import scipy as scipy\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xhJqQq7R5XSz"
   },
   "source": [
    "### **`VECTOR_Embeddings`**\n",
    "\n",
    "- Generating the vector embeddings of ::\n",
    "    - `CLAIM Admit Diagnosis Codes`\n",
    "    - `Diagnosis Codes`\n",
    "    - `Procedure Codes`\n",
    "\n",
    "\n",
    "- For now, I'm not including the `Dx Related Group Code` for generating the similarity score b/w these features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "IngalsnE5XSz"
   },
   "outputs": [],
   "source": [
    "# Importing the ICD-9 Dx and Proc Codes Descriptions\n",
    "CAD_Diag_codes_df = pd.read_excel(\"CMS32_DESC_LONG_SHORT_DX.xlsx\", dtype=\"str\")\n",
    "PROC_codes_df = pd.read_excel(\"CMS32_DESC_LONG_SHORT_SG.xlsx\", dtype=\"str\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "I-r3xi4d5XS0"
   },
   "outputs": [],
   "source": [
    "# Will use the Long Descriptions\n",
    "CAD_Diag_codes_df.drop(['SHORT DESCRIPTION'], axis=1, inplace=True)\n",
    "PROC_codes_df.drop(['SHORT DESCRIPTION'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-uK8mbUA5XS0"
   },
   "outputs": [],
   "source": [
    "# Renaming the columns\n",
    "CAD_Diag_codes_df.columns = ['Dx_Code', 'Dx_Desc']\n",
    "PROC_codes_df.columns = ['Proc_Code', 'Proc_Desc']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "SahbxpoW5XS1",
    "outputId": "21522b2c-d257-40c5-b1e4-4f9e7eac2ba3"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-4bc9f813-ff82-4489-ad62-b80c5a27a59c\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Dx_Code</th>\n",
       "      <th>Dx_Desc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0010</td>\n",
       "      <td>Cholera due to vibrio cholerae</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0011</td>\n",
       "      <td>Cholera due to vibrio cholerae el tor</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0019</td>\n",
       "      <td>Cholera, unspecified</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0020</td>\n",
       "      <td>Typhoid fever</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0021</td>\n",
       "      <td>Paratyphoid fever A</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-4bc9f813-ff82-4489-ad62-b80c5a27a59c')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-4bc9f813-ff82-4489-ad62-b80c5a27a59c button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-4bc9f813-ff82-4489-ad62-b80c5a27a59c');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "  Dx_Code                                Dx_Desc\n",
       "0    0010         Cholera due to vibrio cholerae\n",
       "1    0011  Cholera due to vibrio cholerae el tor\n",
       "2    0019                   Cholera, unspecified\n",
       "3    0020                          Typhoid fever\n",
       "4    0021                    Paratyphoid fever A"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Few CAD and Dx codes and their descriptions\n",
    "CAD_Diag_codes_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "a1LKzR5n5XS1",
    "outputId": "a3c78e95-5586-49cc-b877-120c1b9f4f6a"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-319b3bed-64ad-45b1-bf18-7524d2fa139e\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Proc_Code</th>\n",
       "      <th>Proc_Desc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0001</td>\n",
       "      <td>Therapeutic ultrasound of vessels of head and ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0002</td>\n",
       "      <td>Therapeutic ultrasound of heart</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0003</td>\n",
       "      <td>Therapeutic ultrasound of peripheral vascular ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0009</td>\n",
       "      <td>Other therapeutic ultrasound</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0010</td>\n",
       "      <td>Implantation of chemotherapeutic agent</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-319b3bed-64ad-45b1-bf18-7524d2fa139e')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-319b3bed-64ad-45b1-bf18-7524d2fa139e button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-319b3bed-64ad-45b1-bf18-7524d2fa139e');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "  Proc_Code                                          Proc_Desc\n",
       "0      0001  Therapeutic ultrasound of vessels of head and ...\n",
       "1      0002                    Therapeutic ultrasound of heart\n",
       "2      0003  Therapeutic ultrasound of peripheral vascular ...\n",
       "3      0009                       Other therapeutic ultrasound\n",
       "4      0010             Implantation of chemotherapeutic agent"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Few PROC codes and their descriptions\n",
    "PROC_codes_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "2tD5fMXy5XS2",
    "outputId": "41b8673a-7c7b-45ac-90c5-cd172c6a5e1e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((14567, 2), 14567)"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CAD_Diag_codes_df.shape, CAD_Diag_codes_df[\"Dx_Code\"].nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HpIszXBG5XS3"
   },
   "source": [
    "- There are 14,567 unique Diagnosis Codes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "PwSL2hBK5XS4",
    "outputId": "f4bef950-2e95-4cb9-9ecb-d1e09e91bb25"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((3882, 2), 3882)"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "PROC_codes_df.shape, PROC_codes_df[\"Proc_Code\"].nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tafLf4iV5XS6"
   },
   "source": [
    "- There are 3882 unique Procedures Codes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "1A-fziQT5XS7",
    "outputId": "b75979f3-baa5-4a97-b59c-35d016426954"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dx_Code    object\n",
      "Dx_Desc    object\n",
      "dtype: object \n",
      "\n",
      " Proc_Code    object\n",
      "Proc_Desc    object\n",
      "dtype: object\n"
     ]
    }
   ],
   "source": [
    "print(CAD_Diag_codes_df.dtypes, \"\\n\"*2, PROC_codes_df.dtypes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Er0qcZNk5XS8"
   },
   "source": [
    "#### **Pre-processing the CODES**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ZMw15U1Z5XS8",
    "outputId": "9bdc2761-62a4-4358-9763-7c6c56c6bb53"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dx_Code --> 6079 \n",
      "Dx_Code Description --> Unspecified disorder of penis\n",
      "===============================================================\n",
      "Dx_Code --> 29699 \n",
      "Dx_Code Description --> Other specified episodic mood disorder\n",
      "===============================================================\n",
      "Dx_Code --> 71652 \n",
      "Dx_Code Description --> Unspecified polyarthropathy or polyarthritis, upper arm\n",
      "===============================================================\n",
      "Dx_Code --> 1542 \n",
      "Dx_Code Description --> Malignant neoplasm of anal canal\n",
      "===============================================================\n",
      "Dx_Code --> 2409 \n",
      "Dx_Code Description --> Goiter, unspecified\n",
      "===============================================================\n",
      "Dx_Code --> 5373 \n",
      "Dx_Code Description --> Other obstruction of duodenum\n",
      "===============================================================\n",
      "Dx_Code --> 9895 \n",
      "Dx_Code Description --> Toxic effect of venom\n",
      "===============================================================\n",
      "Dx_Code --> 37412 \n",
      "Dx_Code Description --> Mechanical ectropion\n",
      "===============================================================\n"
     ]
    }
   ],
   "source": [
    "# Printing some of the random CAD and Dx codes and their descriptions\n",
    "random_cad_dx_desc = np.random.randint(low=0, high=CAD_Diag_codes_df.shape[0]-1, size=8)\n",
    "\n",
    "for idx in random_cad_dx_desc:\n",
    "    print(\"Dx_Code --> {} \\nDx_Code Description --> {}\".format(CAD_Diag_codes_df['Dx_Code'].iloc[idx], \n",
    "                                                                 CAD_Diag_codes_df['Dx_Desc'].iloc[idx]))\n",
    "    print(\"===============================================================\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "FEEqVLM-5XS9",
    "outputId": "45b5be51-f4d2-41be-93b2-d03a74a41b31"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Proc_Code --> 8023 \n",
      "Proc_code Description --> Arthroscopy, wrist\n",
      "===============================================================\n",
      "Proc_Code --> 5849 \n",
      "Proc_code Description --> Other repair of urethra\n",
      "===============================================================\n",
      "Proc_Code --> 8341 \n",
      "Proc_code Description --> Excision of tendon for graft\n",
      "===============================================================\n",
      "Proc_Code --> 5503 \n",
      "Proc_code Description --> Percutaneous nephrostomy without fragmentation\n",
      "===============================================================\n",
      "Proc_Code --> 8734 \n",
      "Proc_code Description --> Intrathoracic lymphangiogram\n",
      "===============================================================\n",
      "Proc_Code --> 3862 \n",
      "Proc_code Description --> Other excision of vessels, other vessels of head and neck\n",
      "===============================================================\n",
      "Proc_Code --> 0972 \n",
      "Proc_code Description --> Other repair of punctum\n",
      "===============================================================\n",
      "Proc_Code --> 8779 \n",
      "Proc_code Description --> Other x-ray of the urinary system\n",
      "===============================================================\n"
     ]
    }
   ],
   "source": [
    "# Printing some of the random Proc codes and their descriptions\n",
    "random_proc_codes_desc = np.random.randint(low=0, high=PROC_codes_df.shape[0]-1, size=8)\n",
    "\n",
    "for idx in random_proc_codes_desc:\n",
    "    print(\"Proc_Code --> {} \\nProc_code Description --> {}\".format(PROC_codes_df['Proc_Code'].iloc[idx], \n",
    "                                                                   PROC_codes_df['Proc_Desc'].iloc[idx]))\n",
    "    print(\"===============================================================\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "CzoQTLVq5XS-"
   },
   "outputs": [],
   "source": [
    "# Importing the libraries\n",
    "import nltk\n",
    "from nltk import word_tokenize\n",
    "from nltk.corpus import stopwords\n",
    "from string import punctuation\n",
    "from scipy.spatial import distance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "E3nZmKE35XS_",
    "outputId": "6760d6d9-b6fb-4ad5-f5cd-b61d55f1f389"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/stopwords.zip.\n",
      "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
      "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Downloading the required components\n",
    "nltk.download('stopwords')\n",
    "nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "y5ITPB3J5XTA",
    "outputId": "45c6a7ce-bc31-4a23-fc7d-d14d27c9052f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', \"you're\", \"you've\", \"you'll\", \"you'd\", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', \"she's\", 'her', 'hers', 'herself', 'it', \"it's\", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', \"that'll\", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', \"don't\", 'should', \"should've\", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', \"aren't\", 'couldn', \"couldn't\", 'didn', \"didn't\", 'doesn', \"doesn't\", 'hadn', \"hadn't\", 'hasn', \"hasn't\", 'haven', \"haven't\", 'isn', \"isn't\", 'ma', 'mightn', \"mightn't\", 'mustn', \"mustn't\", 'needn', \"needn't\", 'shan', \"shan't\", 'shouldn', \"shouldn't\", 'wasn', \"wasn't\", 'weren', \"weren't\", 'won', \"won't\", 'wouldn', \"wouldn't\"]\n"
     ]
    }
   ],
   "source": [
    "# Stopwords\n",
    "eng_stopwords = stopwords.words('english')\n",
    "eng_stopwords.remove('no')\n",
    "eng_stopwords.remove('nor')\n",
    "eng_stopwords.remove('not')\n",
    "\n",
    "print(eng_stopwords)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "VPt392kv5XTB"
   },
   "outputs": [],
   "source": [
    "# Pre-processing the sentences\n",
    "stp_words = set(eng_stopwords)\n",
    "def preprocess_sentence(text, stop_words=stp_words):\n",
    "  \"\"\"\n",
    "  Description : This function is created for pre-processing the sentences of ICD given codes description\n",
    "  \"\"\"\n",
    "    text = text.replace('/', ' / ')\n",
    "    text = text.replace('.-', ' .- ')\n",
    "    text = text.replace('.', ' . ')\n",
    "    text = text.replace('\\'', ' \\' ')\n",
    "    text = text.lower()\n",
    "    \n",
    "    tokens = [token for token in word_tokenize(text) if token not in punctuation and token not in stop_words]\n",
    "\n",
    "    return ' '.join(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "mfMyKd3q5XTC"
   },
   "outputs": [],
   "source": [
    "# Pre-process every description sentence\n",
    "CAD_Diag_codes_df['Dx_Desc_pp'] = CAD_Diag_codes_df['Dx_Desc'].apply(lambda row: preprocess_sentence(row))\n",
    "PROC_codes_df['Proc_Desc_pp'] = PROC_codes_df['Proc_Desc'].apply(lambda row: preprocess_sentence(row))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "VZarQCIZ5XTD",
    "outputId": "620c6256-e458-42bf-ccec-a28eb5e0dbb9"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-da3f38a0-5f75-42e4-afdf-c655d9467713\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Dx_Code</th>\n",
       "      <th>Dx_Desc</th>\n",
       "      <th>Dx_Desc_pp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0010</td>\n",
       "      <td>Cholera due to vibrio cholerae</td>\n",
       "      <td>cholera due vibrio cholerae</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0011</td>\n",
       "      <td>Cholera due to vibrio cholerae el tor</td>\n",
       "      <td>cholera due vibrio cholerae el tor</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0019</td>\n",
       "      <td>Cholera, unspecified</td>\n",
       "      <td>cholera unspecified</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0020</td>\n",
       "      <td>Typhoid fever</td>\n",
       "      <td>typhoid fever</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0021</td>\n",
       "      <td>Paratyphoid fever A</td>\n",
       "      <td>paratyphoid fever</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-da3f38a0-5f75-42e4-afdf-c655d9467713')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-da3f38a0-5f75-42e4-afdf-c655d9467713 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-da3f38a0-5f75-42e4-afdf-c655d9467713');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "  Dx_Code                                Dx_Desc  \\\n",
       "0    0010         Cholera due to vibrio cholerae   \n",
       "1    0011  Cholera due to vibrio cholerae el tor   \n",
       "2    0019                   Cholera, unspecified   \n",
       "3    0020                          Typhoid fever   \n",
       "4    0021                    Paratyphoid fever A   \n",
       "\n",
       "                           Dx_Desc_pp  \n",
       "0         cholera due vibrio cholerae  \n",
       "1  cholera due vibrio cholerae el tor  \n",
       "2                 cholera unspecified  \n",
       "3                       typhoid fever  \n",
       "4                   paratyphoid fever  "
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CAD_Diag_codes_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "D7ZOkVC95XTE",
    "outputId": "970bbc22-eb6b-41eb-e52c-c26bd90c14d6"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-4de4aedb-ba72-414e-80f6-03401a9b2273\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Proc_Code</th>\n",
       "      <th>Proc_Desc</th>\n",
       "      <th>Proc_Desc_pp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0001</td>\n",
       "      <td>Therapeutic ultrasound of vessels of head and ...</td>\n",
       "      <td>therapeutic ultrasound vessels head neck</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0002</td>\n",
       "      <td>Therapeutic ultrasound of heart</td>\n",
       "      <td>therapeutic ultrasound heart</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0003</td>\n",
       "      <td>Therapeutic ultrasound of peripheral vascular ...</td>\n",
       "      <td>therapeutic ultrasound peripheral vascular ves...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0009</td>\n",
       "      <td>Other therapeutic ultrasound</td>\n",
       "      <td>therapeutic ultrasound</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0010</td>\n",
       "      <td>Implantation of chemotherapeutic agent</td>\n",
       "      <td>implantation chemotherapeutic agent</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-4de4aedb-ba72-414e-80f6-03401a9b2273')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-4de4aedb-ba72-414e-80f6-03401a9b2273 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-4de4aedb-ba72-414e-80f6-03401a9b2273');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "  Proc_Code                                          Proc_Desc  \\\n",
       "0      0001  Therapeutic ultrasound of vessels of head and ...   \n",
       "1      0002                    Therapeutic ultrasound of heart   \n",
       "2      0003  Therapeutic ultrasound of peripheral vascular ...   \n",
       "3      0009                       Other therapeutic ultrasound   \n",
       "4      0010             Implantation of chemotherapeutic agent   \n",
       "\n",
       "                                        Proc_Desc_pp  \n",
       "0           therapeutic ultrasound vessels head neck  \n",
       "1                       therapeutic ultrasound heart  \n",
       "2  therapeutic ultrasound peripheral vascular ves...  \n",
       "3                             therapeutic ultrasound  \n",
       "4                implantation chemotherapeutic agent  "
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "PROC_codes_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "O6JJD_cA5XTE",
    "outputId": "11950f43-4430-4264-f217-738aa38ccbf6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sent_biobert_pubmed_pmc_base_cased download started this may take some time.\n",
      "Approximate size to download 386.7 MB\n",
      "[OK!]\n"
     ]
    }
   ],
   "source": [
    "# Loading the pre-trained Bio-BERT model\n",
    "biobert_pt_model = nlu.load('en.embed_sentence.biobert.pubmed_pmc_base_cased')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "m_ezQWpC5XTF",
    "outputId": "c1893e69-8fb9-4808-ce8f-a409704e13b3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sentence_detector_dl download started this may take some time.\n",
      "Approximate size to download 354.6 KB\n",
      "[OK!]\n"
     ]
    }
   ],
   "source": [
    "# Getting the sentence embeddings from the pre-trained Bio-BERT Model\n",
    "CAD_Dx_codes_embeddings_df = biobert_pt_model.predict(CAD_Diag_codes_df['Dx_Desc_pp'], output_level='sentence')\n",
    "Proc_codes_embeddings_df = biobert_pt_model.predict(PROC_codes_df['Proc_Desc_pp'], output_level='sentence')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5N-O-6NR5XTH",
    "outputId": "cf48ed36-ba33-448d-f5e6-8b28ace7d88d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((14567, 2), (3882, 2))"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CAD_Dx_codes_embeddings_df.shape, Proc_codes_embeddings_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "JImkuwIm5XTK",
    "outputId": "2c9a507c-e190-4951-f679-41acf623b3d3"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-6f314ec5-705c-4e88-8b10-a6fd1863be90\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence</th>\n",
       "      <th>sentence_embedding_biobert</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>cholera due vibrio cholerae</td>\n",
       "      <td>[0.2308211326599121, 0.012130413204431534, -0....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>cholera due vibrio cholerae el tor</td>\n",
       "      <td>[0.27864786982536316, 0.07002799957990646, -0....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>cholera unspecified</td>\n",
       "      <td>[0.4682339131832123, 0.07824743539094925, -0.1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>typhoid fever</td>\n",
       "      <td>[0.18718130886554718, 0.1722823977470398, -0.1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>paratyphoid fever</td>\n",
       "      <td>[0.23444253206253052, 0.3054424822330475, -0.1...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-6f314ec5-705c-4e88-8b10-a6fd1863be90')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-6f314ec5-705c-4e88-8b10-a6fd1863be90 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-6f314ec5-705c-4e88-8b10-a6fd1863be90');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "                             sentence  \\\n",
       "0         cholera due vibrio cholerae   \n",
       "1  cholera due vibrio cholerae el tor   \n",
       "2                 cholera unspecified   \n",
       "3                       typhoid fever   \n",
       "4                   paratyphoid fever   \n",
       "\n",
       "                          sentence_embedding_biobert  \n",
       "0  [0.2308211326599121, 0.012130413204431534, -0....  \n",
       "1  [0.27864786982536316, 0.07002799957990646, -0....  \n",
       "2  [0.4682339131832123, 0.07824743539094925, -0.1...  \n",
       "3  [0.18718130886554718, 0.1722823977470398, -0.1...  \n",
       "4  [0.23444253206253052, 0.3054424822330475, -0.1...  "
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CAD_Dx_codes_embeddings_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "k--7DtHk5XTK",
    "outputId": "7a6b00a5-020c-48b3-99ed-07b7d00e2012"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-1391bd6f-ced6-4a0b-baa0-18b08f282b27\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence</th>\n",
       "      <th>sentence_embedding_biobert</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>therapeutic ultrasound vessels head neck</td>\n",
       "      <td>[-0.07812043279409409, 0.12884767353534698, -0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>therapeutic ultrasound heart</td>\n",
       "      <td>[-0.013654773123562336, 0.13431645929813385, -...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>therapeutic ultrasound peripheral vascular ves...</td>\n",
       "      <td>[0.008313322439789772, 0.14970511198043823, 0....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>therapeutic ultrasound</td>\n",
       "      <td>[0.052392199635505676, 0.180985689163208, -0.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>implantation chemotherapeutic agent</td>\n",
       "      <td>[-0.04786144942045212, 0.47679537534713745, -0...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-1391bd6f-ced6-4a0b-baa0-18b08f282b27')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-1391bd6f-ced6-4a0b-baa0-18b08f282b27 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-1391bd6f-ced6-4a0b-baa0-18b08f282b27');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "                                            sentence  \\\n",
       "0           therapeutic ultrasound vessels head neck   \n",
       "1                       therapeutic ultrasound heart   \n",
       "2  therapeutic ultrasound peripheral vascular ves...   \n",
       "3                             therapeutic ultrasound   \n",
       "4                implantation chemotherapeutic agent   \n",
       "\n",
       "                          sentence_embedding_biobert  \n",
       "0  [-0.07812043279409409, 0.12884767353534698, -0...  \n",
       "1  [-0.013654773123562336, 0.13431645929813385, -...  \n",
       "2  [0.008313322439789772, 0.14970511198043823, 0....  \n",
       "3  [0.052392199635505676, 0.180985689163208, -0.0...  \n",
       "4  [-0.04786144942045212, 0.47679537534713745, -0...  "
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Proc_codes_embeddings_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "WxLjzPex5XTL"
   },
   "outputs": [],
   "source": [
    "CAD_Diag_codes_df['Dx_embeddings'] = CAD_Dx_codes_embeddings_df['sentence_embedding_biobert']\n",
    "PROC_codes_df['Proc_embeddings'] = Proc_codes_embeddings_df['sentence_embedding_biobert']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "IhRQxYO-5XTM",
    "outputId": "90892f4d-6cbf-4187-8c57-dfbba643994a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((768,), numpy.ndarray)"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Checking the length and type of embeddings vector\n",
    "CAD_Diag_codes_df['Dx_embeddings'].iloc[0].shape, type(CAD_Diag_codes_df['Dx_embeddings'].iloc[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jUi2ZSEKsYLa"
   },
   "outputs": [],
   "source": [
    "# Preparing the dictionary of CAD and Dx CODE and its EMBEDDING\n",
    "cad_diag_codes_embeddings_dict = dict(CAD_Diag_codes_df[['Dx_Code','Dx_embeddings']].values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "_c0pKsm_uG5X",
    "outputId": "8d1fc5dd-d258-46d6-9923-f389f8badbf4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "14567"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Checking the length of dict keys\n",
    "len(cad_diag_codes_embeddings_dict.keys()), len(cad_diag_codes_embeddings_dict.values())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "DTluyTaKt5Xa"
   },
   "outputs": [],
   "source": [
    "# Preparing the dictionary of PROC CODE and its EMBEDDING\n",
    "proc_codes_embeddings_dict = dict(PROC_codes_df[['Proc_Code','Proc_embeddings']].values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "T1JPyg1IuVNb",
    "outputId": "3fa3f8d5-93fd-4a1a-f899-70c79af64cd2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3882, 3882)"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Checking the length of dict keys\n",
    "len(proc_codes_embeddings_dict.keys()), len(proc_codes_embeddings_dict.values())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ftqFdguvxLKU"
   },
   "outputs": [],
   "source": [
    "# Storing the CAD,Dx and PROC CODES DICTIONARIES physically\n",
    "import pickle\n",
    "with open('cad_diag_codes_embeddings.pkl', 'wb') as f:\n",
    "    pickle.dump(cad_diag_codes_embeddings_dict, f)\n",
    "\n",
    "with open('proc_codes_embeddings.pkl', 'wb') as f:\n",
    "    pickle.dump(proc_codes_embeddings_dict, f)\n",
    "      \n",
    "with open('cad_diag_codes_embeddings.pkl', 'rb') as f:\n",
    "    loaded_cad_dict = pickle.load(f)\n",
    "\n",
    "with open('proc_codes_embeddings.pkl', 'rb') as f:\n",
    "    loaded_proc_dict = pickle.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "4AYNow6framo",
    "outputId": "87b749f8-3aaa-43e9-ba3a-4f407f739974"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mounted at /content/drive\n"
     ]
    }
   ],
   "source": [
    "# mount the drive\n",
    "from google.colab import drive\n",
    "drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "nacEJJkmz9br"
   },
   "outputs": [],
   "source": [
    "# copy it there\n",
    "!cp cad_diag_codes_embeddings.pkl /content/drive/MyDrive/AAIC_Case_Study_1/Dataset\n",
    "!cp proc_codes_embeddings.pkl /content/drive/MyDrive/AAIC_Case_Study_1/Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tbWnhQ0qgiCm"
   },
   "source": [
    "## **`SUMMARY`**\n",
    "\n",
    "- I also tried using SENT2VEC model pre-tarined on Biomedical PubMed and PMC dataset that internaly uses FastText as its framework.\n",
    "  - But, the model was so huge that it was crashing my system RAM.\n",
    "\n",
    "\n",
    "- I also tried using Bio-BERT pre-trained models trained on PubMed and PMC datasets available on HuggingFaces.\n",
    "  - And, these were giving some configuration related errors. Also, I checked on internet that people have raised some out-of-memory concerns related to these as well.\n",
    "\n",
    "- Thus, I ended up using the NLU library from John Snow Labs NLU that provides SOTA for NLP&NLU with 5000+ of pretrained models in 200+ languages. \n",
    "  - It enables swift and simple development and research with its powerful Pythonic and Keras inspired API. It is powerd by John Snow Labs powerful Spark NLP library."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "machine_shape": "hm",
   "name": "CS_1_Codes_Desc_Embeddings.ipynb",
   "provenance": []
  },
  "gpuClass": "standard",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}