Custom Medical VQA Test.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "14abc76c-a9f1-4bc7-b88a-760bf8096a3f",
   "metadata": {},
   "source": [
    "# Custom Medical VQA Test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99b4bfe2-fce0-4702-a803-01db9dcab7eb",
   "metadata": {},
   "source": [
    "### Goal: \n",
    "Using pre-trained LLM and scraped caption : image dataset, train a VQA model to accurately answer questions based on medical textbook. Explore whether an encoder or decoder model is more appropriate.\n",
    "\n",
    "### Textbook Source:\n",
    "https://drive.google.com/drive/u/2/folders/12mL45XMDRSxhkgMH_PIeQAAsAtbv-X2W\n",
    "\n",
    "### TODO:\n",
    "- question forming: {question : answering : image pairings} from scraped dictionary using coordinate classifier (potentially ask SRI experts)\n",
    "- vision + text modalities: use LLM for text and ??? (resnet CV) for image modality (potentially ask SRI experts; maybe ask for online resources if they have any)\n",
    "- Once modalities in place, reference MedBLIP / MedPalm papers + architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d84b07d7-2402-4b93-92a3-fc3f6c50d419",
   "metadata": {},
   "source": [
    "### Pairing + Question Forming"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62ed9e36-fd74-4c42-82f7-aa1297e554ed",
   "metadata": {},
   "source": [
    "##### Data Loading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0b5a8e1b-d0d4-4318-8ff0-bbcca35c684e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "import regex as re\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9614c477-ade0-4896-9aba-ac23357981b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# IMPORTANT: Run scrape notebook first\n",
    "PDF_URL = \"General - Mandell - Core Radiology (1e).pdf\"\n",
    "\n",
    "assert os.path.exists(f\"book-scrape/scrape_out/{PDF_URL.split('.pdf')[0]}\")\n",
    "TEXT_DATA_FOLDER_URL = f\"book-scrape/scrape_out/{PDF_URL.split('.pdf')[0]}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2f064610-6b61-436b-8b56-759beed48f72",
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_data = pd.DataFrame()\n",
    "\n",
    "for ch_num, fjson in enumerate(os.listdir(TEXT_DATA_FOLDER_URL)):\n",
    "    ftype = fjson.split('.')[-1]\n",
    "    if ftype != 'json':\n",
    "        continue\n",
    "    fpath = TEXT_DATA_FOLDER_URL + f'/{fjson}'\n",
    "\n",
    "    raw_file_data = pd.read_json(fpath)\n",
    "    raw_file_data['ch'] = [ch_num] * len(raw_file_data)\n",
    "    raw_data = pd.concat([raw_data, raw_file_data])\n",
    "\n",
    "raw_data = raw_data.set_index(['ch'])\n",
    "raw_data = raw_data.sort_index()[['header', 'body', 'images',\n",
    "                                  'label_range', 'pg_range']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ec11c048-7e0f-4d0a-94b0-528172538587",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>header</th>\n",
       "      <th>body</th>\n",
       "      <th>images</th>\n",
       "      <th>label_range</th>\n",
       "      <th>pg_range</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ch</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[STARTUP]</td>\n",
       "      <td>Approach to interpreting an angiogram Normal ...</td>\n",
       "      <td>[scrape_out/General - Mandell - Core Radiology...</td>\n",
       "      <td>[0, 5]</td>\n",
       "      <td>[707, 708]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Air embolism</td>\n",
       "      <td>Percutaneous transluminal angioplasty (PTA) Em...</td>\n",
       "      <td>[scrape_out/General - Mandell - Core Radiology...</td>\n",
       "      <td>[0, 0]</td>\n",
       "      <td>[709, 712]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>superior vena cava (s Vc)</td>\n",
       "      <td>Congenital anomalies of the superior vena cava...</td>\n",
       "      <td>[scrape_out/General - Mandell - Core Radiology...</td>\n",
       "      <td>[1, 0]</td>\n",
       "      <td>[712, 721]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Hepatic artery aneurysmMesenteric ischemiaAcut...</td>\n",
       "      <td>Role of interventional radiology in gastrointe...</td>\n",
       "      <td>[scrape_out/General - Mandell - Core Radiology...</td>\n",
       "      <td>[0, 2]</td>\n",
       "      <td>[722, 724]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Atherosclerotic renal artery stenosisFibromusc...</td>\n",
       "      <td>Fibromuscular dysplasia (medial fibroplasia su...</td>\n",
       "      <td>[scrape_out/General - Mandell - Core Radiology...</td>\n",
       "      <td>[0, 3]</td>\n",
       "      <td>[725, 725]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>OncocytomaAngiomyolipoma (AML)</td>\n",
       "      <td>Axial T 1-weighted mRI shows a mass in the low...</td>\n",
       "      <td>[scrape_out/General - Mandell - Core Radiology...</td>\n",
       "      <td>[0, 6]</td>\n",
       "      <td>[726, 726]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Renal traumaRenal arteriovenous fistulas and m...</td>\n",
       "      <td>grade Iv renal injury: Selective d SA angiogra...</td>\n",
       "      <td>[scrape_out/General - Mandell - Core Radiology...</td>\n",
       "      <td>[0, 3]</td>\n",
       "      <td>[727, 728]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Nutcracker syndromeMay–Thürner</td>\n",
       "      <td>may–Thürner: Axial contrast-enhanced CT (left ...</td>\n",
       "      <td>[scrape_out/General - Mandell - Core Radiology...</td>\n",
       "      <td>[0, 1]</td>\n",
       "      <td>[729, 732]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Varicocele</td>\n",
       "      <td>Retroaortic left renal vein: Initial digital s...</td>\n",
       "      <td>[scrape_out/General - Mandell - Core Radiology...</td>\n",
       "      <td>[0, 2]</td>\n",
       "      <td>[733, 733]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Percutaneous transhepatic cholangiography (Ptc)</td>\n",
       "      <td>Biliary intervention overview and technique Th...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[0, 2]</td>\n",
       "      <td>[734, 734]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Sclerosing cholangitisMalignant biliary obstru...</td>\n",
       "      <td>klatskin tumor: Cholangiogram\\ndemonstrates bi...</td>\n",
       "      <td>[scrape_out/General - Mandell - Core Radiology...</td>\n",
       "      <td>[0, 3]</td>\n",
       "      <td>[735, 735]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Acute cholecystitis</td>\n",
       "      <td>Contrast-enhanced CT shows diffuse gallbladder...</td>\n",
       "      <td>[scrape_out/General - Mandell - Core Radiology...</td>\n",
       "      <td>[0, 5]</td>\n",
       "      <td>[736, 736]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Percutaneous nephrostomy indications and techn...</td>\n",
       "      <td>Percutaneous nephrostomy (PCN) indications Per...</td>\n",
       "      <td>[scrape_out/General - Mandell - Core Radiology...</td>\n",
       "      <td>[0, 0]</td>\n",
       "      <td>[737, 739]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Iliac atherosclerotic disease</td>\n",
       "      <td>Leriche syndrome: Axial CT angiogram (left ima...</td>\n",
       "      <td>[scrape_out/General - Mandell - Core Radiology...</td>\n",
       "      <td>[1, 2]</td>\n",
       "      <td>[739, 739]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Persistent sciatic artery</td>\n",
       "      <td>Case courtesy Michael Hanley, MD, University o...</td>\n",
       "      <td>[scrape_out/General - Mandell - Core Radiology...</td>\n",
       "      <td>[0, 4]</td>\n",
       "      <td>[740, 742]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Chronic arterial occlusive diseaseFemoropoplit...</td>\n",
       "      <td>Type d: Chronic total occlusion: Surgery is tr...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[0, 2]</td>\n",
       "      <td>[743, 743]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Acute thromboembolic diseasePopliteal aneurysm...</td>\n",
       "      <td>Overview of thoracic outlet syndromes Coronal ...</td>\n",
       "      <td>[scrape_out/General - Mandell - Core Radiology...</td>\n",
       "      <td>[3, 4]</td>\n",
       "      <td>[743, 747]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>surgical dialysis access</td>\n",
       "      <td>Arteriovenous fistula Polytetrafluorethylene (...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[0, 6]</td>\n",
       "      <td>[748, 748]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Hypothenar hammer</td>\n",
       "      <td>Hypothenar hammer: digital subtraction angiogr...</td>\n",
       "      <td>[scrape_out/General - Mandell - Core Radiology...</td>\n",
       "      <td>[0, 2]</td>\n",
       "      <td>[749, 749]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Buerger diseaseRaynaud diseaseThromboembolic d...</td>\n",
       "      <td>kaufman, J. &amp; Lee, m. vascular and Interventio...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[0, 1]</td>\n",
       "      <td>[750, 751]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               header  \\\n",
       "ch                                                      \n",
       "0                                           [STARTUP]   \n",
       "0                                        Air embolism   \n",
       "0                           superior vena cava (s Vc)   \n",
       "0   Hepatic artery aneurysmMesenteric ischemiaAcut...   \n",
       "0   Atherosclerotic renal artery stenosisFibromusc...   \n",
       "0                      OncocytomaAngiomyolipoma (AML)   \n",
       "0   Renal traumaRenal arteriovenous fistulas and m...   \n",
       "0                      Nutcracker syndromeMay–Thürner   \n",
       "0                                          Varicocele   \n",
       "0     Percutaneous transhepatic cholangiography (Ptc)   \n",
       "0   Sclerosing cholangitisMalignant biliary obstru...   \n",
       "0                                 Acute cholecystitis   \n",
       "0   Percutaneous nephrostomy indications and techn...   \n",
       "0                       Iliac atherosclerotic disease   \n",
       "0                           Persistent sciatic artery   \n",
       "0   Chronic arterial occlusive diseaseFemoropoplit...   \n",
       "0   Acute thromboembolic diseasePopliteal aneurysm...   \n",
       "0                            surgical dialysis access   \n",
       "0                                   Hypothenar hammer   \n",
       "0   Buerger diseaseRaynaud diseaseThromboembolic d...   \n",
       "\n",
       "                                                 body  \\\n",
       "ch                                                      \n",
       "0    Approach to interpreting an angiogram Normal ...   \n",
       "0   Percutaneous transluminal angioplasty (PTA) Em...   \n",
       "0   Congenital anomalies of the superior vena cava...   \n",
       "0   Role of interventional radiology in gastrointe...   \n",
       "0   Fibromuscular dysplasia (medial fibroplasia su...   \n",
       "0   Axial T 1-weighted mRI shows a mass in the low...   \n",
       "0   grade Iv renal injury: Selective d SA angiogra...   \n",
       "0   may–Thürner: Axial contrast-enhanced CT (left ...   \n",
       "0   Retroaortic left renal vein: Initial digital s...   \n",
       "0   Biliary intervention overview and technique Th...   \n",
       "0   klatskin tumor: Cholangiogram\\ndemonstrates bi...   \n",
       "0   Contrast-enhanced CT shows diffuse gallbladder...   \n",
       "0   Percutaneous nephrostomy (PCN) indications Per...   \n",
       "0   Leriche syndrome: Axial CT angiogram (left ima...   \n",
       "0   Case courtesy Michael Hanley, MD, University o...   \n",
       "0   Type d: Chronic total occlusion: Surgery is tr...   \n",
       "0   Overview of thoracic outlet syndromes Coronal ...   \n",
       "0   Arteriovenous fistula Polytetrafluorethylene (...   \n",
       "0   Hypothenar hammer: digital subtraction angiogr...   \n",
       "0   kaufman, J. & Lee, m. vascular and Interventio...   \n",
       "\n",
       "                                               images label_range    pg_range  \n",
       "ch                                                                             \n",
       "0   [scrape_out/General - Mandell - Core Radiology...      [0, 5]  [707, 708]  \n",
       "0   [scrape_out/General - Mandell - Core Radiology...      [0, 0]  [709, 712]  \n",
       "0   [scrape_out/General - Mandell - Core Radiology...      [1, 0]  [712, 721]  \n",
       "0   [scrape_out/General - Mandell - Core Radiology...      [0, 2]  [722, 724]  \n",
       "0   [scrape_out/General - Mandell - Core Radiology...      [0, 3]  [725, 725]  \n",
       "0   [scrape_out/General - Mandell - Core Radiology...      [0, 6]  [726, 726]  \n",
       "0   [scrape_out/General - Mandell - Core Radiology...      [0, 3]  [727, 728]  \n",
       "0   [scrape_out/General - Mandell - Core Radiology...      [0, 1]  [729, 732]  \n",
       "0   [scrape_out/General - Mandell - Core Radiology...      [0, 2]  [733, 733]  \n",
       "0                                                  []      [0, 2]  [734, 734]  \n",
       "0   [scrape_out/General - Mandell - Core Radiology...      [0, 3]  [735, 735]  \n",
       "0   [scrape_out/General - Mandell - Core Radiology...      [0, 5]  [736, 736]  \n",
       "0   [scrape_out/General - Mandell - Core Radiology...      [0, 0]  [737, 739]  \n",
       "0   [scrape_out/General - Mandell - Core Radiology...      [1, 2]  [739, 739]  \n",
       "0   [scrape_out/General - Mandell - Core Radiology...      [0, 4]  [740, 742]  \n",
       "0                                                  []      [0, 2]  [743, 743]  \n",
       "0   [scrape_out/General - Mandell - Core Radiology...      [3, 4]  [743, 747]  \n",
       "0                                                  []      [0, 6]  [748, 748]  \n",
       "0   [scrape_out/General - Mandell - Core Radiology...      [0, 2]  [749, 749]  \n",
       "0                                                  []      [0, 1]  [750, 751]  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_data.head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18e93ae8-f99d-4670-a788-9fd141e41c37",
   "metadata": {},
   "source": [
    "##### Question Generation (QG)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "8d649681-8c73-482b-b749-f5c0e6924db1",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/developer/anaconda3/envs/andy/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "# Reference: https://huggingface.co/mrm8488/t5-base-finetuned-question-generation-ap\n",
    "from transformers import AutoModelWithLMHead, AutoTokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "cbe22978",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading (…)okenizer_config.json: 100%|████| 25.0/25.0 [00:00<00:00, 131kB/s]\n",
      "Downloading (…)lve/main/config.json: 100%|█| 1.23k/1.23k [00:00<00:00, 7.91MB/s]\n",
      "The `xla_device` argument has been deprecated in v4.4.0 of Transformers. It is ignored and you can safely remove it from your `config.json` file.\n",
      "Downloading (…)ve/main/spiece.model: 100%|███| 792k/792k [00:00<00:00, 3.21MB/s]\n",
      "Downloading (…)cial_tokens_map.json: 100%|█| 1.79k/1.79k [00:00<00:00, 10.1MB/s]\n",
      "The `xla_device` argument has been deprecated in v4.4.0 of Transformers. It is ignored and you can safely remove it from your `config.json` file.\n",
      "You are using the default legacy behaviour of the <class 'transformers.models.t5.tokenization_t5.T5Tokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565\n",
      "The `xla_device` argument has been deprecated in v4.4.0 of Transformers. It is ignored and you can safely remove it from your `config.json` file.\n",
      "/home/developer/anaconda3/envs/andy/lib/python3.11/site-packages/transformers/models/auto/modeling_auto.py:1509: FutureWarning: The class `AutoModelWithLMHead` is deprecated and will be removed in a future version. Please use `AutoModelForCausalLM` for causal language models, `AutoModelForMaskedLM` for masked language models and `AutoModelForSeq2SeqLM` for encoder-decoder models.\n",
      "  warnings.warn(\n",
      "The `xla_device` argument has been deprecated in v4.4.0 of Transformers. It is ignored and you can safely remove it from your `config.json` file.\n",
      "/home/developer/anaconda3/envs/andy/lib/python3.11/site-packages/torch/cuda/__init__.py:546: UserWarning: Can't initialize NVML\n",
      "  warnings.warn(\"Can't initialize NVML\")\n",
      "/home/developer/anaconda3/envs/andy/lib/python3.11/site-packages/torch/cuda/__init__.py:651: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:109.)\n",
      "  return torch._C._cuda_getDeviceCount() if nvml_count < 0 else nvml_count\n",
      "Downloading model.safetensors: 100%|███████| 1.19G/1.19G [00:25<00:00, 45.7MB/s]\n"
     ]
    }
   ],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(\"mrm8488/t5-base-finetuned-question-generation-ap\")\n",
    "model = AutoModelWithLMHead.from_pretrained(\"mrm8488/t5-base-finetuned-question-generation-ap\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "94647f3e-c407-48ce-b950-79fe3ee9138e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_question(answer, context, max_length=64):\n",
    "    input_text = \"answer: %s  context: %s </s>\" % (answer, context)\n",
    "    features = tokenizer([input_text], return_tensors='pt')\n",
    "\n",
    "    output = model.generate(input_ids=features['input_ids'], \n",
    "               attention_mask=features['attention_mask'],\n",
    "               max_length=max_length)\n",
    "\n",
    "    return tokenizer.decode(output[0])\n",
    "\n",
    "\n",
    "def form_question_set(answers, context):\n",
    "    \"\"\"\n",
    "    returns list of (question, answer) pairs\n",
    "    \"\"\"\n",
    "    qa_pairs = []\n",
    "    \n",
    "    # for single answer string\n",
    "    if type(answers) == str: return [(get_question(answers, context), answers)]\n",
    "    \n",
    "    # for list of answers\n",
    "    for answer in answers:\n",
    "        qa_pairs.append((get_question(answer, context), answer))\n",
    "    return qa_pairs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "85c459a1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('<pad> question: What is a common complication of embolization?</s>', 'Air embolism'), ('<pad> question: Who kindly provided the catheters for this procedure?</s>', 'Timothy P. Killoran, MD, Brigham and Women’s Hospital')]\n"
     ]
    }
   ],
   "source": [
    "context = \"Percutaneous transluminal angioplasty (PTA) Embolic materials Complications of embolization Catheter sizing High-flow catheters Selective and superselective catheters Standard floppy tip wires Case courtesy Timothy P. Killoran, MD, Brigham and Women’s Hospital.\\ngiant cell arteritis: Selective angiogram of the axillary artery (catheter is not visible) shows mild narrowing\\nof the axillary artery, complete occlusion of the axillary/brachial artery at the origin of the brachial artery\\n(yellow arrow), and an irregular appearance of the posterior circumflex humeral artery (red arrow).\\n• Aortic disease is discussed in the cardiovascular imaging section.\"\n",
    "answers = [\"Air embolism\", \"Timothy P. Killoran, MD, Brigham and Women’s Hospital\"]\n",
    "\n",
    "print(form_question_set(answers, context))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95330df1",
   "metadata": {},
   "source": [
    "### Text & Vision Modalities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "e965fd2a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# qa dataset formation\n",
    "\n",
    "qa_dataset = pd.DataFrame()\n",
    "\n",
    "for n in range(len(raw_data)):\n",
    "    context, answer = raw_data.iloc[n]['body'], raw_data.iloc[n]['header']\n",
    "    qa = form_question_set(answer, context)\n",
    "    for q, a in qa:\n",
    "        new_row = pd.DataFrame({\"question\":[q], \"answer\":[a]})\n",
    "        qa_dataset = pd.concat([qa_dataset, new_row], ignore_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "78e4ae98",
   "metadata": {},
   "outputs": [],
   "source": [
    "# qa_dataset.to_csv('qa_dataset.csv')\n",
    "# qa_dataset = pd.read_csv('qa_dataset.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "1fa3dc57",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "472\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>answer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>&lt;pad&gt; question: What is the name of the angiog...</td>\n",
       "      <td>[STARTUP]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>&lt;pad&gt; question: What is a common complication ...</td>\n",
       "      <td>Air embolism</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>&lt;pad&gt; question: What is SVC?&lt;/s&gt;</td>\n",
       "      <td>superior vena cava (s Vc)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>&lt;pad&gt; question: What is the role of interventi...</td>\n",
       "      <td>Hepatic artery aneurysmMesenteric ischemiaAcut...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>&lt;pad&gt; question: What is the name of the diseas...</td>\n",
       "      <td>Atherosclerotic renal artery stenosisFibromusc...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>&lt;pad&gt; question: What is AML?&lt;/s&gt;</td>\n",
       "      <td>OncocytomaAngiomyolipoma (AML)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>&lt;pad&gt; question: What was the grade Iv injury?&lt;/s&gt;</td>\n",
       "      <td>Renal traumaRenal arteriovenous fistulas and m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>&lt;pad&gt; question: What is the name of the syndro...</td>\n",
       "      <td>Nutcracker syndromeMay–Thürner</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>&lt;pad&gt; question: What is the name of the vein t...</td>\n",
       "      <td>Varicocele</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>&lt;pad&gt; question: What is the name of the proced...</td>\n",
       "      <td>Percutaneous transhepatic cholangiography (Ptc)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>&lt;pad&gt; question: What is the name of the diseas...</td>\n",
       "      <td>Sclerosing cholangitisMalignant biliary obstru...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>&lt;pad&gt; question: What is the name of the condit...</td>\n",
       "      <td>Acute cholecystitis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>&lt;pad&gt; question: What is the PCN?&lt;/s&gt;</td>\n",
       "      <td>Percutaneous nephrostomy indications and techn...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>&lt;pad&gt; question: What is the name of the diseas...</td>\n",
       "      <td>Iliac atherosclerotic disease</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>&lt;pad&gt; question: What is the name of the enlarg...</td>\n",
       "      <td>Persistent sciatic artery</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>&lt;pad&gt; question: What is the type of occlusion?...</td>\n",
       "      <td>Chronic arterial occlusive diseaseFemoropoplit...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>&lt;pad&gt; question: What is the name of the thorac...</td>\n",
       "      <td>Acute thromboembolic diseasePopliteal aneurysm...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>&lt;pad&gt; question: What is the clinical evaluatio...</td>\n",
       "      <td>surgical dialysis access</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>&lt;pad&gt; question: What is used to determine the ...</td>\n",
       "      <td>Hypothenar hammer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>&lt;pad&gt; question: What is the name of the diseas...</td>\n",
       "      <td>Buerger diseaseRaynaud diseaseThromboembolic d...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             question  \\\n",
       "0   <pad> question: What is the name of the angiog...   \n",
       "1   <pad> question: What is a common complication ...   \n",
       "2                    <pad> question: What is SVC?</s>   \n",
       "3   <pad> question: What is the role of interventi...   \n",
       "4   <pad> question: What is the name of the diseas...   \n",
       "5                    <pad> question: What is AML?</s>   \n",
       "6   <pad> question: What was the grade Iv injury?</s>   \n",
       "7   <pad> question: What is the name of the syndro...   \n",
       "8   <pad> question: What is the name of the vein t...   \n",
       "9   <pad> question: What is the name of the proced...   \n",
       "10  <pad> question: What is the name of the diseas...   \n",
       "11  <pad> question: What is the name of the condit...   \n",
       "12               <pad> question: What is the PCN?</s>   \n",
       "13  <pad> question: What is the name of the diseas...   \n",
       "14  <pad> question: What is the name of the enlarg...   \n",
       "15  <pad> question: What is the type of occlusion?...   \n",
       "16  <pad> question: What is the name of the thorac...   \n",
       "17  <pad> question: What is the clinical evaluatio...   \n",
       "18  <pad> question: What is used to determine the ...   \n",
       "19  <pad> question: What is the name of the diseas...   \n",
       "\n",
       "                                               answer  \n",
       "0                                           [STARTUP]  \n",
       "1                                        Air embolism  \n",
       "2                           superior vena cava (s Vc)  \n",
       "3   Hepatic artery aneurysmMesenteric ischemiaAcut...  \n",
       "4   Atherosclerotic renal artery stenosisFibromusc...  \n",
       "5                      OncocytomaAngiomyolipoma (AML)  \n",
       "6   Renal traumaRenal arteriovenous fistulas and m...  \n",
       "7                      Nutcracker syndromeMay–Thürner  \n",
       "8                                          Varicocele  \n",
       "9     Percutaneous transhepatic cholangiography (Ptc)  \n",
       "10  Sclerosing cholangitisMalignant biliary obstru...  \n",
       "11                                Acute cholecystitis  \n",
       "12  Percutaneous nephrostomy indications and techn...  \n",
       "13                      Iliac atherosclerotic disease  \n",
       "14                          Persistent sciatic artery  \n",
       "15  Chronic arterial occlusive diseaseFemoropoplit...  \n",
       "16  Acute thromboembolic diseasePopliteal aneurysm...  \n",
       "17                           surgical dialysis access  \n",
       "18                                  Hypothenar hammer  \n",
       "19  Buerger diseaseRaynaud diseaseThromboembolic d...  "
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(len(qa_dataset))\n",
    "qa_dataset.head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8115ad58",
   "metadata": {},
   "source": [
    "### Appendix / Old Code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6c3be94",
   "metadata": {},
   "outputs": [],
   "source": [
    "# x-y classification\n",
    "\n",
    "def classify(row):\n",
    "    # basic approach: find center of image, and get closest relevant text or N/A\n",
    "    # returns new df indexed by ???? with text and relevant image\n",
    "    # possibly add radius check; if text within certain radius, it gets added to df\n",
    "    image_coords = np.array(row['image_coords'])\n",
    "    text_coords = np.array(row['text_coords'])\n",
    "    return_df = pd.DataFrame(['image_ref', 'header', 'text'])\n",
    "\n",
    "    # potentially use a kd tree, but n*m should be good enough for 2-dimensions\n",
    "    for img_ref, image_coord_ls in enumerate(image_coords):\n",
    "        img_center = (image_coord_ls[0] + image_coord_ls[1])/2\n",
    "        min_dist = 10000\n",
    "        min_text_ref = -1\n",
    "        for text_ref, text_coord_ls in enumerate(text_coords):\n",
    "            text_center = text_coord_ls[0]\n",
    "            dist = np.sqrt(np.sum((img_center - text_center) ** 2))\n",
    "            if dist < min_dist:\n",
    "                min_dist = dist\n",
    "                min_text_ref = text_ref\n",
    "        entry = pd.DataFrame({'image_ref': row['images'][image_ref], \n",
    "                              'header': row['header'], \n",
    "                              'text': row['body']})\n",
    "        return_df = pd.concat(return_df, entry)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afb5a5c6",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}