Custom Medical LLM Test.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4ac80916-0e00-47aa-b296-989986af8195",
   "metadata": {},
   "source": [
    "# Custom Medical LLM Test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0176709-f481-4fb6-9bc7-f73070c7e78a",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### Goal: \n",
    "Train a LLM using data from a selected medical textbook.\n",
    "\n",
    "### More to Explore:\n",
    "Could potentially use RAG as an alternative instead, like SRI researchers may pursue.\n",
    "\n",
    "### TODOs:\n",
    "- evaluate on Brent & Helms Radiology Textbook for more text and less images\n",
    "- train LLM and evaluate training loss using SotA, not Bigram Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45ac4a54-cd8c-4adf-957a-8f3f66f071b6",
   "metadata": {},
   "source": [
    "### Process:\n",
    "\n",
    "##### Data Intake - Train text data page by page, including images\n",
    "1. clean text data\n",
    "2. image extractor\n",
    "3. Need to train classifier that associates text with respective image - if we can get coordinates, that'd be huge\n",
    "5. For now, probably just storing images and text in dict\n",
    "\n",
    "##### Joint Heads\n",
    "4. Create a LLM - this will be used as a BERT-esque LLM in the text encoder\n",
    "5. (Hard) CNN for images\n",
    "\n",
    "##### Encoder-Decoder Model\n",
    "6. Train Encoder\n",
    "7. Train Decoder"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d0222ea-56bf-4749-80d1-474ffc9bb97e",
   "metadata": {},
   "source": [
    "# Reading Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "516d9b3d-b73d-4a0d-b5ab-794c48c2c250",
   "metadata": {},
   "source": [
    "**Reference Max's textbook scraping notebook first!** Courtesy of Max Vogel :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "13c4d2a1-c6cb-4b95-8028-01d2fc3ce9e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "import regex as re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a66d52ff-9a5f-4ddd-a2b2-cb6662be457c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# IMPORTANT: Run scrape notebook first\n",
    "PDF_URL = \"General - Mandell - Core Radiology (1e).pdf\"\n",
    "\n",
    "assert os.path.exists(f\"book-scrape/scrape_out/{PDF_URL.split('.pdf')[0]}\")\n",
    "TEXT_DATA_FOLDER_URL = f\"book-scrape/scrape_out/{PDF_URL.split('.pdf')[0]}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c0fffea6-0ad4-47e1-b892-826dc604a715",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'label_range': [0, 0],\n",
       "  'pg_range': [14, 16],\n",
       "  'header_font': [['Calibri', 14.0]],\n",
       "  'header': 'Anatomy',\n",
       "  'body_font': [['TwCenMT', 11.0]],\n",
       "  'body': 'Interlobar fi ssures Mechanisms of atelectasis Each of the five lobes tends to collapse in a predictable direction, as shown above.',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/0.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/1.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/2.png']},\n",
       " {'label_range': [0, 0],\n",
       "  'pg_range': [17, 18],\n",
       "  'header_font': [['TwCenMT', 11.0]],\n",
       "  'header': 'Right upper lobe atelectasis',\n",
       "  'body_font': [['Calibri', 10.0]],\n",
       "  'body': 'Case courtesy Ritu R. Gill, MD, MPH, Brigham and Women’s Hospital.',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/3.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/4.png']},\n",
       " {'label_range': [1, 0],\n",
       "  'pg_range': [18, 19],\n",
       "  'header_font': [['TwCenMT', 11.0]],\n",
       "  'header': 'Right lower lobe atelectasis',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'Left lower lobe collapse: Frontal and lateral radiographs demonstrate a triangular retrocardiac opacity\\nrepresenting the collapsed left lower lobe (red arrows). There is loss of concavity of the left heart\\nborder (the flat waist sign; yellow arrow).\\nCase courtesy Ritu R. Gill, MD, MPH, Brigham and Women’s Hospital.',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/5.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/6.png']},\n",
       " {'label_range': [1, 2],\n",
       "  'pg_range': [19, 27],\n",
       "  'header_font': [['Calibri', 14.0]],\n",
       "  'header': 'round atelectasis',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'Right middle lobe atelectasis: Frontal chest radiograph shows an indistinct opacity in the right lung with\\nfocal silhouetting of the right heart border (arrow). There is elevation of the right hemidiaphragm due\\nto volume loss. The lateral radiograph shows a wedge-shaped opacity (arrow) projecting over the midheart representing the collapsed right middle lobe.\\nRound atelectasis: Noncontrast CT shows a rounded\\nopacity in the medial right lower lobe (red arrows). This\\nexample meets all five criteria for round atelectasis\\nincluding adjacent pleural abnormality (effusion), opacity\\nin contact with the pleura, round shape, volume loss in\\nthe affected lobe, and the comet tail sign (yellow arrows)\\nrepresenting curved vessels and bronchi leading to the\\nfocus of round atelectasis.\\npulmonary veins (and lymphatics, not pictured)\\nrun in the interlobular septa Consolidation and ground glass Schematic demonstrates complete\\nhazy filling of the alveolus. The\\npulmonary vessels are still visible.\\nground glass opacification: Noncontrast CT shows diffuse\\nground glass opacification (ggo). The pulmonary architecture,\\nincluding vasculature and bronchi, can be still seen, which\\nis characteristic for ggo. Although these imaging findings\\nare nonspecific, this was a case of acute respiratory distress\\nsyndrome (ARd S).\\nAlveolar proteinosis, an idiopathic disease characterized by alveolar filling by a proteinaceous\\nsubstance. The distribution is typically central, with sparing of the periphery.\\nCoronal schemati c shows centralpredominant\\nground glass att enuati on.\\nPeripheral ground glass or consolidation Coronal CT demonstrates centralpredominant\\nground glass and associated septal thickening\\n(the crazy paving patt ern) in a case of alveolar\\nproteinosis.\\nAxial CT shows peripheral and subpleural ground glass\\natt enuati on. This was a case of organizing pneumonia.\\nSchematic demonstrates smooth\\ninterlobular septal thickening.\\nSchematic demonstrates smooth Smooth interlobular septal thickening: CT demonstrates smooth\\nthickening of the interlobular septa (arrows) in pulmonary\\nedema.\\nNodular septal thickening: CT shows a dominant mass in the\\nright lung (yellow arrows) with peripheral nodularity and septal\\nthickening. This was a case of lymphangitic carcinomatosis.\\nSchematic demonstrates interlobular\\nseptal thickening and ground glass\\nopacification.\\nAxial CT shows interlobular septal thickening in regions of\\nground glass opacification, representing crazy paving. This\\nwas a case of alveolar proteinosis, the entity in which crazy\\npaving was first described.\\nCentrilobular nodules Schematic demonstrates a centrilobular\\nnodule, located at the center of the\\npulmonary lobule.\\nAxial CT demonstrates innumerable subcentimeter\\ncentrilobular nodules of ground glass attenuation (arrows).\\nNone of the nodules extends to the pleural surface, which\\nis typical of a centrilobular distribution. This was a case of\\nrespiratory bronchiolitis interstitial lung disease (RBILd).\\nDiffuse panbronchiolitis is a chronic inflammatory disorder characterized by lymphoid\\nhyperplasia in the walls of the respiratory bronchioles resulting in bronchiolectasis. It typically\\naffects patients of Asian descent.\\nSilicosis, an inhalation lung disease that develops in response to inhaled silica particles, is\\ncharacterized by upper lobe predominant centrilobular and perilymphatic nodules.\\nPerilymphatic nodules: Schematic of the secondary pulmonary lobule (top left\\nimage) demonstrates two of the three distributions of\\nperilymphatic nodules. The gray nodules are located along\\nthe bronchovascular bundle and the white nodules are\\nlocated along the interlobular septa.\\nSchematic of the lungs (left image) demonstrates the\\nperibronchovascular and subpleural/fissural distribution.',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/7.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/8.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/9.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/10.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/11.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/12.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/13.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/14.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/15.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/16.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/17.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/18.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/19.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/20.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/21.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/22.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/23.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/24.png']},\n",
       " {'label_range': [0, 2],\n",
       "  'pg_range': [28, 32],\n",
       "  'header_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'header': 'Random nodules:',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'Schematic shows several nodules\\ncentered on an opacified small airway Treein-bud nodularity: Axial CT show numerous small nodules\\n(arrows) “budding” off of linear branching structures in\\nthe right middle lobe. This case was secondary to atypical\\nmycobacteria.\\nMultiple cavitary nodules Coronal schematic shows numerous cavitary\\nlesions bilaterally.\\nAxial CT shows a single spiculated cavitary lesion\\nin the left upper lobe (arrow). This was a case of\\nsquamous cell carcinoma.\\nAxial CT shows numerous cavitary and noncavitary\\nlesions bilaterally, in a random distribution. This was\\na case of tricuspid endocarditis and septic emboli.\\nCase courtesy Michael Hanley, MD, University of\\nVirginia Health System.\\n• Multiple cavitary lesions are typically vascular or spread through the vascular system: Coronal schematic shows numerous thin-walled\\ncystic lesions bilaterally.\\nEmphysema, which tends to be upperlobe predominant in a smoker.\\nPneumocystis jiroveci pneumonia, which features cysts in latestage disease.\\nAxial CT shows bilateral thin-walled cysts that are\\nof varying sizes but are predominantly regular in\\nshape. There is a small left pleural effusion. This was\\na case of lymphangioleiomyomatosis.\\nPneumatocele, which is an air-filled space caused by prior lung trauma or infection.\\nUpper lobe fibrotic changes Coronal CT shows bibasilar fibrosis and honeycombing\\n(arrows), with relative sparing of the upper lobes. This\\nwas a case of idiopathic pulmonary fibrosis.\\nCoronal CT shows upperlobe predominant\\nsubpleural fibrosis and traction bronchiectasis. A\\npathologic diagnosis was not established in this cas',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/25.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/26.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/27.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/28.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/29.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/30.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/31.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/32.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/33.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/34.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/35.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/36.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/37.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/38.png']},\n",
       " {'label_range': [0, 6],\n",
       "  'pg_range': [33, 34],\n",
       "  'header_font': [['TwCenMT', 11.0], ['Calibri', 14.0]],\n",
       "  'header': 'clinical classification of pneumoniaHospital acquired pneumonia (HAP)Health care associated pneumonia (HCAP)Ventilator associated pneumonia (VAP)Pneumonia in the immunocompromised patientradiographic patterns of infectionLobular pneumonia (bronchopneumonia)Round pneumoniaPulmonary abscessPulmonary gangreneEmpyemaPneumatoceleBronchopleural fistula (BPF)',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'Split pleura sign: Contrast-enhanced CT\\nshows enhancement of the thickened\\nvisceral and parietal pleural layers (arrows),\\nwhich encase a pleural fluid collection.\\nThe split pleura sign is seen in the majority\\nof exudative effusions, although it is not\\nspecific. Similar findings can be seen\\nin malignant effusion, mesothelioma,\\nfibrothorax, and after talc pleurodesis.',\n",
       "  'images': []},\n",
       " {'label_range': [0, 1],\n",
       "  'pg_range': [35, 36],\n",
       "  'header_font': [['TwCenMT', 11.0]],\n",
       "  'header': 'Primary tuberculosis',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': \"Primary tuberculosis: Chest radiograph (left image) shows a vague right upper lung opacity (arrow). CT\\nshows a patchy opacification (arrow) in the lower portion of the right upper lobe with adjacent treein-bud nodularity. The patient's sputum grew Mycobacterium tuberculosis.\\nReactivation (postprimary) tuberculosis Case courtesy Michael Hanley, MD, University of Virginia Health System.\",\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/39.png']},\n",
       " {'label_range': [2, 0],\n",
       "  'pg_range': [36, 37],\n",
       "  'header_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'header': 'Tuberculous adenopathy: Contrast-enhanced\\nneck CT shows marked rightsided adenopathy\\n(arrows) with peripheral enhancement and\\ncentral necrosis.',\n",
       "  'body_font': [['Calibri', 10.0]],\n",
       "  'body': 'Case courtesy Ritu R. Gill, MD, MPH, Brigham\\nand Women’s Hospital.\\nReactivation TB: Frontal chest radiograph (left image) shows a cavitary lesion in the left upper lobe\\n(arrow), confirmed by CT (arrow). There was no significant mediastinal adenopathy. The differential\\ndiagnosis of this appearance would include cavitary primary lung cancer, which would be expected to\\nfeature a thicker wall.\\nCase courtesy Ritu R. Gill, MD, MPH, Brigham and Women’s Hospital.',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/40.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/41.png']},\n",
       " {'label_range': [1, 4],\n",
       "  'pg_range': [37, 37],\n",
       "  'header_font': [['TwCenMT', 11.0]],\n",
       "  'header': 'Miliary tuberculosis',\n",
       "  'body_font': [['Calibri', 10.0]],\n",
       "  'body': 'Case courtesy Ritu R. Gill, MD, MPH, Brigham and Women’s Hospital.\\nHealed tuberculosis: Radiograph shows scarring, volume loss, and superior hilar retraction (arrows). CT\\nshows apical scarring.\\nmiliary tuberculosis: Radiograph and CT show innumerable tiny nodules in a random pattern, reflecting\\nhematogenous seeding of tuberculosis.',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/42.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/43.png']},\n",
       " {'label_range': [0, 3],\n",
       "  'pg_range': [38, 38],\n",
       "  'header_font': [['TwCenMT', 11.0]],\n",
       "  'header': 'Atypical mycobacteria infectionHistoplasma capsulatumCoccidioides immitis and Blastomyces dermatitidis',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'Mycobacterium avium intracellulare infection: Coronal (left) and axial CT show right upper lobe and\\nlingular treein-bud opacities and bronchiectasis, with more focal consolidation in the lingula (arrow).',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/44.png']},\n",
       " {'label_range': [0, 2],\n",
       "  'pg_range': [39, 39],\n",
       "  'header_font': [['TwCenMT', 11.0]],\n",
       "  'header': 'Pneumocystis jiroveci pneumonia',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'Pneumocystis pneumonia with cystic change:\\nRadiograph (top left image) shows multiple\\nupperlobe predominant small nodular\\nopacities. CT (top image) better defines\\nthe nodular opacities as being primarily\\ncentrilobular in distribution, with confluent\\nground glass attenuation and multiple small\\ncysts/pneumatoceles (arrows).\\nCases courtesy Ritu R. Gill, MD, MPH, Brigham\\nand Women’s Hospital.',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/45.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/46.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/47.png']},\n",
       " {'label_range': [0, 3],\n",
       "  'pg_range': [40, 41],\n",
       "  'header_font': [['TwCenMT', 11.0], ['Calibri', 14.0]],\n",
       "  'header': 'A����������Saprophytic aspergillosis (aspergilloma)',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'Fingerin-glove sign of ABPA: Chest radiograph shows upperlobe predominant bronchiectasis with\\nbranching perihilar opacities most prominent on the right (arrow). CT shows dense bronchial mucoid\\nimpaction representing the fingerin-glove sign (arrow).\\nAspergilloma and Monod sign: CT shows an\\nirregular opacity (arrows) resting dependent\\nwithin a left upper lobe cavity, representing\\nan aspergilloma (arrows). The Monod\\nsign is the curvilinear air surrounding the\\naspergilloma. This patient has prior TB\\nwith biapical scarring and prior left apical\\ncavitation.\\nCase courtesy Ritu R. Gill, MD, MPH, Brigham\\nand Women’s Hospital.',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/48.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/49.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/50.png']},\n",
       " {'label_range': [0, 4],\n",
       "  'pg_range': [42, 42],\n",
       "  'header_font': [['TwCenMT', 11.0]],\n",
       "  'header': 'Airwayinvasive aspergillosisAngioinvasive aspergillosis',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'Halo sign: CT shows a left upper lobe\\nmass with peripheral ground glass\\n(yellow arrow), representing pulmonary\\nhemorrhage.\\nAir crescent sign: Noncontrast CT shows\\na cavitary lesion in the right upper lung\\nwith a crescent-shaped air collection\\nanteriorly (arrows).\\nCase courtesy Ritu R. Gill, MD, MPH,\\nBrigham and Women’s Hospital.',\n",
       "  'images': []},\n",
       " {'label_range': [0, 0],\n",
       "  'pg_range': [43, 44],\n",
       "  'header_font': [['Calibri', 14.0]],\n",
       "  'header': 'Pulmonary edema',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'mild pulmonary edema: Radiograph demonstrates moderate enlargement of the cardiac silhouette\\nand sternotomy wires. There are increased interstitial markings with kerley B lines (arrows). Chest\\nCT shows smooth interlobular septal thickening (arrows), corresponding to the kerley B lines seen on\\nradiography. mild geographic ground glass attenuation likely corresponds to early alveolar filling.\\nRight mainstem bronchus intubation: Chest\\nradiograph shows the endotracheal tube terminating\\nin the right mainstem bronchus (yellow arrow), a few\\ncentimeters distal to the carina (red arrow).',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/51.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/52.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/53.png']},\n",
       " {'label_range': [0, 6],\n",
       "  'pg_range': [45, 46],\n",
       "  'header_font': [['TwCenMT', 11.0]],\n",
       "  'header': 'Pulmonary artery catheter',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'Peripherally inserted central catheter (PICC) in the azygos vein: Frontal radiograph shows a left-sided\\nPICC coursing medially at the confluence of the brachiocephalic vein and the Sv C (arrow). Lateral\\nradiograph shows that the PICC curves anteriorly before heading inferiorly and posteriorly into the\\nazygos vein (arrow).\\nclinical overview of lung cancer Epidemiology Overview of the solitary pulmonary nodule Nodule morphology essentially diagnostic for a benign etiology Nodule morphology suggesting, but not diagnostic for, a benign etiology Nodule morphology suggesting malignancy',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/54.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/55.png']},\n",
       " {'label_range': [0, 2],\n",
       "  'pg_range': [47, 47],\n",
       "  'header_font': [['TwCenMT', 11.0], ['Calibri', 14.0]],\n",
       "  'header': 'Histologic subtypes of lung cancerAdenocarcinoma',\n",
       "  'body_font': [['UHEZNZ+Calibri', 9.0]],\n",
       "  'body': 'Regardless of risk, either PET, biopsy, or at least three followups at 3, 9, and 24 months.',\n",
       "  'images': []},\n",
       " {'label_range': [0, 2],\n",
       "  'pg_range': [48, 48],\n",
       "  'header_font': [['TwCenMT', 11.0]],\n",
       "  'header': 'Bronchioloalveolar carcinoma (BAC)',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'SCC: Axial CT shows a spiculated cavitary lesion\\n(arrow) shown at pathology to be squamous\\ncell carcinoma.\\nNonmucinous BAC (multifocal):\\nAxial CT shows a solid nodule\\nwith air bronchograms (yellow\\narrow) and a more peripheral\\nground glass nodule (red arrow).',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/56.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/57.png']},\n",
       " {'label_range': [0, 0],\n",
       "  'pg_range': [49, 51],\n",
       "  'header_font': [['TwCenMT', 11.0]],\n",
       "  'header': 'Small cell carcinomaLarge cell carcinomaCarcinoid tumor',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'mucinous BAC is an important differential consideration for chronic ground glass or consolidation, ofte\\nwith air bronchograms.\\nThe CT angiogram sign (not imaged above) describes the especially prominent appearance of\\nenhancing pulmonary vessels seen in a low attenuation, mucinrich consolidation of mucinous BAC.\\nmucinous BAC is an important differential consideration for chronic ground glass or consolidation, ofte\\nwith air bronchograms.\\nSolitary pulmonary nodule or lung mass Case courtesy Ritu R. Gill, MD, MPH, Brigham and Women’s Hospital.',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/58.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/59.png']},\n",
       " {'label_range': [1, 3],\n",
       "  'pg_range': [51, 51],\n",
       "  'header_font': [['TwCenMT', 11.0]],\n",
       "  'header': 'Lymphangitic carcinomatosis\\n_J:_______:',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'Superior sulcus tumor: Chest radiograph shows nearly imperceptible increased opacity in the right apex\\n(arrow). A superior sulcus mass is clearly evident on CT (arrows), with encasement of the vasculature\\nand effacement of the superior mediastinal fat.\\nLymphangitic carcinomatosis: Axial (left image) and coronal CT shows asymmetric, nodular septal\\nthickening of the right lung in a patient with metastatic uterine sarcoma. Aside from a trace effusion,\\nthe left lung is clear.',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/60.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/61.png']},\n",
       " {'label_range': [0, 6],\n",
       "  'pg_range': [52, 54],\n",
       "  'header_font': [['TwCenMT', 11.0], ['Calibri', 14.0]],\n",
       "  'header': 'Pneumothoraxstaging of lung cancer',\n",
       "  'body_font': [['TwCenMT', 11.0]],\n",
       "  'body': 'Overview of lung cancer staging Treatment based on staging Case courtesy Ritu R. Gill, MD, MPH, Brigham and Women’s Hospital.\\n: Stage IV Case courtesy Ritu R. Gill, MD, MPH, Brigham and Women’s Hospital.\\nExample staging: Stage IV Case courtesy Ritu R. Gill, MD, MPH, Brigham and Women’s Hospital.\\nStage IIIA lung cancer: Axial CT in lung windows (left image) shows a right upper lobe peripheral mass\\n(arrow) that abuts the pleura. There is diff use centrilobular emphysema. Softti ssue window CT shows\\nipsilateral right paratracheal adenopathy (arrow). The T staging is T 2 (both for size >3 cm and invasion\\nof pleura) and the N staging is N 2 for ipsilateral mediasti nal nodes. The overall stage is therefore IIIA\\nand this tumor is resectable.\\nStage Iv lung cancer: Axial CT (left image) shows a dominant, spiculated right upper lobe T 2 mass. more\\ninferiorly (right image), a small contralateral nodule (arrow) is present. This nodule was determined to\\nbe malignant on follow-up, for an m staging of m 1a and stage Iv disease.\\nStage Iv lung cancer: Axial CT (left image) shows a left upper lobe mass with ipsilateral hilar adenopathy\\n(arrows). Coronal CT through the upper abdomen shows bilateral adrenal masses (arrows), which were\\nconfi rmed to be metastases (m 1b), representi ng stage Iv disease.',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/62.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/63.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/64.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/65.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/66.png']},\n",
       " {'label_range': [0, 2],\n",
       "  'pg_range': [55, 55],\n",
       "  'header_font': [['Calibri', 14.0]],\n",
       "  'header': 'Pulmonary hypertension',\n",
       "  'body_font': [['TwCenMT', 11.0]],\n",
       "  'body': 'Definition of pulmonary hypertension Overview of pulmonary hypertension classification',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/67.png']},\n",
       " {'label_range': [0, 4],\n",
       "  'pg_range': [57, 57],\n",
       "  'header_font': [['TwCenMT', 11.0]],\n",
       "  'header': 'Pulmonary hypertension due to leftto-right cardiac shunts – WHO group 1, precapillary',\n",
       "  'body_font': [['TwCenMT', 11.0]],\n",
       "  'body': 'ulmonary hypertension due to leftto-right cardiac shunts – WHO group 1, precapillary Pulmonary venous hypertension – WHO group 2, postcapillary Chronic thromboembolic pulmonary hypertension (CTEPH) – WHO group 4, precapillary',\n",
       "  'images': []},\n",
       " {'label_range': [0, 1],\n",
       "  'pg_range': [58, 59],\n",
       "  'header_font': [['Calibri', 14.0]],\n",
       "  'header': 'Pulmonary embolism (PE)',\n",
       "  'body_font': [['TwCenMT', 11.0]],\n",
       "  'body': 'Clinical diagnosis Imaging of pulmonary embolism massive pulmonary embolism: CT pulmonary angiogram shows a large, nearlyocclusive filling defect in\\nthe right main pulmonary artery (arrows) extending proximally to the bifurcation.\\nCardiac evaluation Pitfalls of CT pulmonary angiogram',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/68.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/69.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/70.png']},\n",
       " {'label_range': [0, 1],\n",
       "  'pg_range': [60, 64],\n",
       "  'header_font': [['Calibri', 14.0]],\n",
       "  'header': 'Idiopathic interstitial pneumonias',\n",
       "  'body_font': [['TwCenMT', 11.0]],\n",
       "  'body': 'Overview of idiopathic interstitial pneumonias Type I pneumocytes form alveolar wall and participate in gas exchange.\\nType II pneumocytes produce surfactant, which prevents atelectasis.\\nAlveolar macrophages ingest and process inhaled particulate materials.\\nIPF: Noncontrast CT shows\\nbibasilar subpleural honeycombing\\n(yellow arrow) and tracti on\\nbronchiectasis with cysti c change\\n(red arrow).\\nCo P: Axial CT shows patchy\\nconsolidati ve and ground glass\\nopaciti es in a peribronchovascular\\ndistributi on. There is a reverse halo\\nor atoll sign in the right lower lobe\\n(arrow), where a central lucency is\\nsurrounded by a peripheral opacity.\\nCase courtesy Seth Kligerman, MD,\\nUniversity of Maryland.\\nContrast-enhanced CT in a diff erent\\npati ent with Co P shows patchy\\nconsolidati on in a peribronchial and\\nsubpleural locati on.\\nCase courtesy Ritu R. Gill, MD, MPH,\\nBrigham and Women’s Hospital.\\nRBILd: Axial CT shows\\ninnumerable subtle ground\\nglass centrilobular nodules\\n(arrows). This case does\\nnot feature ground glass\\nopacifi cati on. Centrilobular\\nnodules are a nonspecifi c\\nfi nding, and the other common\\ndisease featuring centrilobular\\nnodules is hypersensiti vity\\npneumoniti s.\\nCase courtesy Ritu R. Gill, MD,\\nMPH, Brigham and Women’s\\nHospital.',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/71.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/72.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/73.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/74.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/75.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/76.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/77.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/78.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/79.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/80.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/81.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/82.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/83.png']},\n",
       " {'label_range': [0, 1],\n",
       "  'pg_range': [68, 68],\n",
       "  'header_font': [['TwCenMT', 11.0]],\n",
       "  'header': 'Hypersensitivity pneumonitis (HSP)',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'If there is relative sparing of the bases, chronic HSP is much more likely than IPF.',\n",
       "  'images': []},\n",
       " {'label_range': [2, 0],\n",
       "  'pg_range': [68, 70],\n",
       "  'header_font': [['TwCenMT', 11.0], ['Calibri', 14.0]],\n",
       "  'header': 'PneumoconiosesSimple pulmonary eosinophilia (Löffler syndrome)Chronic eosinophilic pneumoniaPulmonary vasculitisChurg–Strauss',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'Chronic eosinophilic pneumonia: Chest radiograph demonstrates upperlobe and peripheral\\nconsolidation. Axial CT shows bilateral patchy peripheral consolidative opacities. Although this is a\\nnonspecific appearance, the appearance was unchanged over multiple priors, which suggests against\\nbacterial pneumonia.\\nAlveolar hemorrhage due to microscopic polyangiitis: Axial CT shows basal predominant central ground\\nglass with associated mild interlobular septal thickening. This patient had renal insufficiency and had a\\nPANCA positive vasculitis with alveolar hemorrhage.',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/84.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/85.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/86.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/87.png']},\n",
       " {'label_range': [0, 3],\n",
       "  'pg_range': [71, 71],\n",
       "  'header_font': [['TwCenMT', 11.0]],\n",
       "  'header': 'Drug toxicityRadiation lung injury',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'Acute radiation pneumonitis: Coronal and axial CT demonstrate geographic left upper lung ground\\nglass (arrows) with nonanatomic superior and lateral linear margins.\\nRadiation fibrosis: Chest radiograph and enhanced CT shows paramediastinal fibrotic changes\\nwithin the radiation port. despite radiation, the patient has persistent left mediastinal adenopathy\\n(arrow).',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/88.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/89.png']},\n",
       " {'label_range': [0, 2],\n",
       "  'pg_range': [72, 76],\n",
       "  'header_font': [['TwCenMT', 11.0]],\n",
       "  'header': 'Sarcoidosis',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'Stage 1 sarcoidosis: Frontal (left image) and lateral radiographs show symmetric adenopathy\\ndemonstrating the 1-2-3 sign:\\nRight paratracheal adenopathy (yellow arrows, “1”)\\nRight hilar adenopathy (red arrow, “2”)\\nLeft hilar adenopathy (red arrow, “3”) The lateral radiograph demonstrates the donut sign with adenopathy circumferentially encircling\\nthe trachea (arrows).\\nThe lung parenchyma is normal. Case courtesy Ritu R. Gill, MD, MPH, Brigham and Women’s Hospital.\\nSarcoidosis: Axial CT in a different\\npatient shows peripherallycalcified\\n(\"eggshell\") prevascular and\\nparatracheal lymph nodes (arrows).\\nCase courtesy Ritu R. Gill, MD, MPH,\\nBrigham and Women’s Hospital.\\nCT in a patient with sarcoidosis shows tiny\\nperibronchovascular and fissural (subpleural)\\nnodules in the bilateral upper lobes.\\nCase courtesy Darryl Sneag, MD, Brigham and\\nWomen’s Hospital.\\nCT in a different patient with sarcoidosis shows\\nslightly larger nodules in a peribronchovascular\\ndistribution.\\nCase courtesy Ritu R. Gill, MD, MPH, Brigham\\nand Women’s Hospital.\\nPerilymphatic nodules are found along the course of the pulmonary lymphatics, which are\\nperibronchovascular, subpleural, and within the interlobular septa.\\nAxial contrast-enhanced CT\\nthrough the liver and spleen\\nin a patient with sarcoidosis\\nshows innumerable small\\nhypoattenuating splenic nodul Case courtesy Michael Hanley,\\nMD, University of Virginia Health\\nSystem.\\nPulmonary Langerhans cell histiocytosis:\\nChest radiograph shows multiple upperlobe\\npredominant cysts with the suggestion of\\nmultiple small nodules. There is a moderate l ase courtesy Ritu R. Gill, MD,\\nPH, Brigham and Women’s\\nospital.\\nPulmonary alveolar\\nproteinosis with\\nsuperimposed Nocardia: CT shows bilateral crazy\\npaving predominantly in\\nthe right lung, with a dense\\nconsolidative opacity in\\nthe left upper lobe (arrow)\\nrepresenting superimposed\\nNocardia pneumonia.\\nCase courtesy Ritu R. Gill, MD,\\nMPH, Brigham and Women’s\\nHospital.\\nPatient 1 (top radiograph and top CT) has end-stage LAm with severe cystic change and moderate\\nright apical and posterior pneumothoraces.\\nPatient 2 (bottom left CT) shows a less severe\\nstage of disease, with multiple smoothwalled\\ncysts and bilateral pleural effusions. Although the\\npleural fluid was not sampled in this case, there is\\nan association of LAm with chylous effusion.\\nCases courtesy Ritu R. Gill, MD, MPH, Brigham\\nand Women’s Hospital.',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/90.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/91.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/92.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/93.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/94.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/95.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/96.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/97.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/98.png']},\n",
       " {'label_range': [0, 1],\n",
       "  'pg_range': [77, 81],\n",
       "  'header_font': [['Calibri', 14.0]],\n",
       "  'header': 'Anatomy of the mediastinum',\n",
       "  'body_font': [['TwCenMT', 11.0]],\n",
       "  'body': 'Anterior mediastinum Middle mediastinum Posterior mediastinum An “interface” is formed by the contact of two soft tissue structures of different densities.\\nAortopulmonary (AP) window Retrosternal clear space Left superior intercostal vein (LSIV) Detection of an anterior mediastinal mass Detection of a middle mediastinal mass Detection of a posterior mediastinal mass displacement of the left paraspinal line: Frontal chest radiograph (left) shows lateral displacement\\nand bowing of the left paraspinal line (arrows), suggestive of a posterior mediastinal mass. Axial\\ncontrast-enhanced CT shows a nonenhancing homogeneous mass (arrow) effacing the esophagus.\\nAlthough extremely rare, this was a liposarcoma.\\nOverview of prevascular anterior mediastinal masses Invasive thymoma with drop metastasis: Contrast-enhanced CT shows a slightly heterogeneous mass\\n(yellow arrow) anterior to the aorta and Sv C, with two pleuralbased metastases (red arrows).',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/99.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/100.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/101.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/102.png']},\n",
       " {'label_range': [0, 1],\n",
       "  'pg_range': [82, 82],\n",
       "  'header_font': [['TwCenMT', 11.0]],\n",
       "  'header': 'Germ cell tumor (GCT)',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'Teratoma: Chest radiograph shows an extra mediastinal contour (yellow arrow) lateral to the aortic\\narch (red arrow). Because the mass does not silhouette the aorta it cannot be in the posterior\\nmediastinum. on CT there is an anterior mediastinal mass containing a large focus of fat (yellow\\narrow). The differential in this case would include a thymolipoma.',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/103.png']},\n",
       " {'label_range': [0, 4],\n",
       "  'pg_range': [83, 83],\n",
       "  'header_font': [['TwCenMT', 11.0]],\n",
       "  'header': 'LymphomaNonlymphomatous adenopathy',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'Inferior extension of a goiter into the mediastinum: Chest radiograph demonstrates a right superior\\nmediastinal mass (arrow) with obscuration of the\\nsuperior border of the mass, representing the\\ncervicothoracic sign, which implies that the mass is\\nin continuity with the soft tissues of the neck.\\nNoncontrast CT shows a large heterogeneous mass\\n(arrows) with coarse calcifications replacing the\\nthyroid gland.',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/104.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/105.png']},\n",
       " {'label_range': [0, 1],\n",
       "  'pg_range': [84, 84],\n",
       "  'header_font': [['TwCenMT', 11.0]],\n",
       "  'header': 'Castleman disease',\n",
       "  'body_font': [['TwCenMT', 11.0]],\n",
       "  'body': 'Overview of precardiac anterior mediastinal masses',\n",
       "  'images': []},\n",
       " {'label_range': [0, 0],\n",
       "  'pg_range': [85, 88],\n",
       "  'header_font': [['TwCenMT', 11.0], ['Calibri', 14.0]],\n",
       "  'header': 'Foregut duplication cystPosterior mediastinal massNeurogenic tumorDescending thoracic aortic aneurysmLateral meningoceleEsophageal neoplasmForegut duplication cystParaspinal abscess',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'Foregut duplication cyst: Frontal radiograph shows a subtle opacity overlying the right heart (arrow),\\nshown to be a fluid attenuation lesion adjacent to the esophagus on CT.\\nosteomyelitis and paraspinal phlegmon: Radiograph shows displacement of the bilateral paraspinal\\nlines (arrows). Sagittal T 1 (middle image) and T 2-weighted mRI (right image) shows loss of disk space\\nheight and marrow edema in two adjacent lower thoracic vertebral bodies. There is associated\\nparaspinal phlegmonous change (arrows), which caused the paraspinal line displacement on\\nradiography.\\nOverview of non-neoplastic diffuse tracheal disease Case courtesy Ritu R. Gill, MD, MPH, Brigham and Women’s Hospital.',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/106.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/107.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/108.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/109.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/110.png']},\n",
       " {'label_range': [1, 4],\n",
       "  'pg_range': [88, 88],\n",
       "  'header_font': [['TwCenMT', 11.0]],\n",
       "  'header': 'Amyloidosis',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'Tracheal tuberculosis: Axial CT in soft-tissue window through the upper trachea (left image) shows\\ncircumferential tracheal thickening (arrows). Coronal curvedmultiplanar reformation (right image)\\nshows multifocal tracheal stenosis (arrows) and left mainstem bronchus stenosis. These images were\\nobtained approximately one month after treatment for active postprimary tuberculosis.\\nTracheal amyloid: Axial CT images show\\nnodular and irregular thickening of the\\ntrachea (arrows). This pattern is not specific,\\nand the differential would include also\\ninclude sarcoidosis, multifocal adenoid cystic\\ncarcinoma, and tracheal metastases.\\nCase courtesy Ritu R. Gill, MD, MPH, Brigham\\nand Women’s Hospital.',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/111.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/112.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/113.png']},\n",
       " {'label_range': [0, 5],\n",
       "  'pg_range': [89, 89],\n",
       "  'header_font': [['TwCenMT', 11.0], ['Calibri', 14.0]],\n",
       "  'header': 'Rare causes of focal tracheal stenosisLarge airwaysBronchiectasis',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'Exaggerated immune response, as seen in Ab PA and vasculitis.\\n• Extremely uncommon causes of focal tracheal stenosis include Behçet and Crohn disease.\\nImpaired immunity: congenital immunodeficiency, transplant recipients, and agammaglobulinemia.',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/114.png']},\n",
       " {'label_range': [0, 1],\n",
       "  'pg_range': [90, 90],\n",
       "  'header_font': [['TwCenMT', 11.0]],\n",
       "  'header': 'Broncholithiasis',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'Congenital connective tissue disorders – Mounier–Kuhn (a connective tissue disorder causing\\ntracheobronchomegaly leading to recurrent pneumonia), or Williams–campbell (a rare disorder of\\nthe distal bronchial cartilage, which may be congenital or acquired as a sequela of viral infection).',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/115.png']},\n",
       " {'label_range': [0, 3],\n",
       "  'pg_range': [91, 91],\n",
       "  'header_font': [['TwCenMT', 11.0]],\n",
       "  'header': 'Centrilobular emphysemaParaseptal emphysema',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'Centrilobular and paraseptal emphysema: Coronal (left image) and axial CT demonstrate both\\ncentrilobular and paraseptal emphysema. Centrilobular emphysema is predominant in the upper lobes\\n(yellow arrows) and paraseptal emphysema is seen anteromedially (blue arrows).\\nAll smokingrelated lung disease (RB, RBILd, d IP, PLCH, and emphysema) may be within the same\\nspectrum of disease caused by macrophagemediated inflammation in reaction to inhaled particles\\nand toxins.',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/116.png']},\n",
       " {'label_range': [0, 0],\n",
       "  'pg_range': [92, 93],\n",
       "  'header_font': [['TwCenMT', 11.0], ['Calibri', 14.0]],\n",
       "  'header': 'Airway tumorsSquamous cell carcinoma (SCC)',\n",
       "  'body_font': [['Calibri', 10.0]],\n",
       "  'body': 'Case courtesy Ritu R. Gill, MD, MPH, Brigham and Women’s Hospital.\\nEndotracheal squamous cell carcinoma: Radiograph shows a subtle tracheal filling defect (arrows). on\\nCT there is a heterogeneously enhancing, wellcircumscribed mass markedly narrowing the tracheal\\nlumen (arrow).\\nCase courtesy Ritu R. Gill, MD, MPH, Brigham and Women’s Hospital.',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/117.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/118.png']},\n",
       " {'label_range': [1, 2],\n",
       "  'pg_range': [93, 93],\n",
       "  'header_font': [['TwCenMT', 11.0]],\n",
       "  'header': 'Carcinoid',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'Tracheal adenoid cystic carcinoma: CT (left image) shows irregular circumferential tracheal thickening\\n(arrow). Postcontrast coronal T 1-weighted mRI (right image) shows an enhancing nodular mass\\nextending into the tracheal lumen (arrows).',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/119.png',\n",
       "   'scrape_out/General - Mandell - Core Radiology (1e)/images/120.png']},\n",
       " {'label_range': [0, 2],\n",
       "  'pg_range': [94, 94],\n",
       "  'header_font': [['TwCenMT', 11.0]],\n",
       "  'header': 'Tracheal lymphomaEndobronchial metastasis',\n",
       "  'body_font': [['TwCenMT', 11.0]],\n",
       "  'body': 'irect invasion of the central airways by adjacent malignancy',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/121.png']},\n",
       " {'label_range': [0, 3],\n",
       "  'pg_range': [95, 95],\n",
       "  'header_font': [['TwCenMT', 11.0]],\n",
       "  'header': 'MesotheliomaMetastasesMultiple myeloma/plasmacytoma',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'mesothelioma: Contrast-enhanced CT through the thorax (left image) shows extensive nodular pleural\\nthickening of the left hemithorax (arrows). Images through the upper abdomen show extensive soft\\ntissue abnormality with invasion of the chest wall (arrows).',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/122.png']},\n",
       " {'label_range': [0, 2],\n",
       "  'pg_range': [96, 97],\n",
       "  'header_font': [['TwCenMT', 11.0], ['Calibri', 14.0]],\n",
       "  'header': 'Pleural effusionTransudateExudate',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'Fibrous tumor of the pleura: CT topogram (left image) shows a round opacity (arrow) with a\\ncircumscribed medial margin and indisti nct lateral margin, suggesti ng a pleuralbased mass. CT\\nconfi rms that the mass (arrow) is pleuralbased, with a broad att achment to the pleura.\\nEisenhuber, E. Signs in imaging - the tree in bud sign. Radiology 222, 771-2 (2002).\\nEngoren, m. Lack of association between atelectasis and fever. Chest 107, 81-84 (1995).\\nmarshall, g.B. et al. Signs in thoracic imaging. Journal of Thoracic Imaging 21, 76-90 (2006).',\n",
       "  'images': ['scrape_out/General - Mandell - Core Radiology (1e)/images/123.png']},\n",
       " {'label_range': [3, 8],\n",
       "  'pg_range': [97, 97],\n",
       "  'header_font': [['Calibri', 11.0]],\n",
       "  'header': 'Pulmonary infection:',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'Franquet, T. et al. Spectrum of Pulmonary Aspergillosis: Histologic, Clinical, and Radiologic Findings. Radiographics 21,\\n825 (2001).\\ngotway, m.B. et al. The radiologic spectrum of pulmonary Aspergillus infections. Journal of Computer Assisted Tomography\\n26, 159-73 (2002).\\nSystrom, d.m. & Wittram, C. Case 9-2005-A 67-Year-old man with Acute Respiratory Failure. New England Journal of medicine\\n352, 1238 (2005).\\nTarver, R.d. et al. Radiology of communityacquired pneumonia. Radiologic Clinics of North America 43, 497-512, viii (2005).\\nWaite, S., Jeudy, J. & White, C.S. Acute lung infections in normal and immunocompromised hosts. Radiologic Clinics of North\\nAmerica 44, 295-315, ix (2006).',\n",
       "  'images': []},\n",
       " {'label_range': [9, 5],\n",
       "  'pg_range': [97, 98],\n",
       "  'header_font': [['Calibri', 11.0]],\n",
       "  'header': 'Pulmonary edema and ICU imaging:',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'Ely, E.W. Using the Chest Radiograph To determine Intravascular volume Status: The Role of vascular Pedicle Width. Chest\\n121, 942-50 (2002).\\nmartin, g.S. Chest 122,\\n2087-2095 (2002).\\nHartman, T.E. Radiologic evaluation of the solitary pulmonary nodule. Radiologic Clinics of North America 43, 459-65,\\nvii (2005).\\nmacmahon, H. et al. guidelines for management of small pulmonary nodules detected on CT scans: a statement from the\\nFleischner Society. Radiology 237, 395-400 (2005).\\nTravis, W.d. et al. International Association for the Study of Lung Cancer/American Thoracic Society/European Respiratory\\nSociety international multidisciplinary classification of lung adenocarcinoma. Journal of Thoracic oncology 6, 244-85 (2011).\\nUy Bico, S.J. et al. Lung cancer staging essentials: the new TNm staging system and potential imaging pitfalls. Radiographics 30,\\n1163-81 (2010).\\nAgnelli, g. & Becattini, C. Acute Pulmonary Embolism. New England Journal of medicine 363, 266-74 (2010).\\nduránmendicuti, A. & Sodickson, A. Imaging evaluation of the pregnant patient with suspected pulmonary embolism.\\nInternational Journal of obstetric Anesthesia 20, 51-9 (2011).\\nFrazier, A. et al. From the Archives of the AFIP: Pulmonary venoocclusive disease and Pulmonary Capillary Hemangiomatosis.\\nRadiographics 27, 867-83 (2007).\\nkuriakose, J. & Patel, S. Acute pulmonary embolism. Radiologic Clinics of North America 48, 31-50 (2010).\\nPatel, N.m. et al. Pulmonary hypertension in idiopathic pulmonary fibrosis. Chest 132, 998-1006 (2007).\\nWittram, C. et al. CT angiography of pulmonary embolism: diagnostic criteria and causes of misdiagnosis. Radiographics 24,\\n1219-38 (2004).',\n",
       "  'images': []},\n",
       " {'label_range': [6, 14],\n",
       "  'pg_range': [98, 98],\n",
       "  'header_font': [['Calibri', 11.0]],\n",
       "  'header': 'Diffuse lung disease:',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'Attili, A.k. et al. Smokingrelated interstitial lung disease: radiologicclinical-pathologic correlation. Radiographics 28, 1383-96;\\ndiscussion 1396-8 (2008).\\ngill, R., matsusoka, S. & Hatabu, H. Cavities in the lung in oncology patients: imaging overview and differential diagnoses.\\nApplied Radiology (June), 10–21 (2010).\\ngotway, m.B. et al. High-resolution CT of the lung: patterns of disease and differential diagnoses. Radiologic Clinics of North\\nAmerica 43, 513-42, viii (2005).\\nkligerman, S.J. et al. Nonspecific interstitial pneumonia: radiologic, clinical, and pathologic considerations. Radiographics 29,\\n73-87 (2009).\\nmedoff, B.d., Abbott, g.F. & Louissaint Jr, A. Case 16-2010–A 48-Year-old man with a Cough and Pain in the Left Shoulder.\\nNew England Journal of medicine 362, 2013 (2010).\\nmuellermang, C., grosse, C. & Schmid, k. What Every Radiologist Should know about Idiopathic Interstitial Pneumonias.\\nRadiographics 27, 595-616 (2007).\\nRossi, S.E. et al. “CrazyPaving” Pattern at Thin-Section CT of the Lungs: RadiologicPathologic overview. Radiographics 23,\\n1509-519 (2003).\\nWebb, W.R. Thin-section CT of the secondary pulmonary lobule: anatomy and the image – the 2004 Fleischner lecture.\\nRadiology 239, 322-38 (2006).',\n",
       "  'images': []},\n",
       " {'label_range': [15, 20],\n",
       "  'pg_range': [98, 98],\n",
       "  'header_font': [['Calibri', 11.0]],\n",
       "  'header': 'Mediastinum:',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'Quint, L.E. Imaging of anterior mediastinal masses. Cancer Imaging 7 Spec No, S 56-62 (2007).\\nTecce, P.m., Fishman, E.k. & kuhlman, J.E. CT evaluation of the anterior mediastinum: spectrum of disease. Radiographics 14,\\n973 (1994).\\nWhitten, C., khan, S. & munneke, g. A diagnostic Approach to mediastinal Abnormalities. Radiographics 27, 657-71 (2007).\\nJavidanNejad, C. & Bhalla, S. Bronchiectasis. Radiologic Clinics of North America 47, 289-306 (2009).\\nmartinez, S. et al. mucoid impactions: fingerin-glove sign and other CT and radiographic features. Radiographics 28,\\n1369-82 (2008).',\n",
       "  'images': []},\n",
       " {'label_range': [21, 24],\n",
       "  'pg_range': [98, 98],\n",
       "  'header_font': [['Calibri', 11.0]],\n",
       "  'header': 'Pleura:',\n",
       "  'body_font': [['UHEZNZ+Calibri', 10.0]],\n",
       "  'body': 'Qureshi, N.R. & gleeson, F.v. Imaging of pleural disease. Clinics in Chest Medicine 27, 193-213 (2006).\\nSugarbaker, d. & Wolf, A. Surgery for malignant pleural mesothelioma. Expert Review of Respiratory medicine 4,\\n363-72 (2010).\\nWang, Z.J. et al. malignant Pleural mesothelioma: Evaluation with CT, mR Imaging, and PET. Radiographics 24, 105 (2004',\n",
       "  'images': []}]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_data = {}\n",
    "\n",
    "for fjson in os.listdir(TEXT_DATA_FOLDER_URL):\n",
    "    ch, ftype = fjson.split('.')[0], fjson.split('.')[-1]\n",
    "    if ftype != 'json':\n",
    "        continue\n",
    "    with open(TEXT_DATA_FOLDER_URL + f'/{fjson}') as f:\n",
    "        text_data[int(re.search(r'\\d+', ch)[0]) - 1] = json.load(f)\n",
    "\n",
    "text_data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "23ae6ef8-6fd3-4ddd-bb6a-d043a41ea37d",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Anatomy [END]Interlobar fi ssuresMechanisms of atelectasisEach of the five lobes tends to collapse in a predictable direction, as shown above.\\nCase courtesy Ritu R. Gill, MD, MPH, Brigham and Women’s Hospital.\\n [END]Right lower lobe atelectasis [END]Left lower lobe collapse: Frontal and lateral radiographs demonstrate a triangular retrocardiac opacity\\nrepresenting the collapsed left lower lobe (red arrows). There is loss of concavity of the left heart\\nborder (the flat waist sign; yellow arrow).Case courtesy Ritu R. Gill, MD, MPH, Brigham and Women’s Hospital.\\n [END]round atelectasis [END]Right middle lobe atelectasis: Frontal chest radiograph shows an indistinct opacity in the right lung with\\nfocal silhouetting of the right heart border (arrow). There is elevation of the right hemidiaphragm due\\nto volume loss. The lateral radiograph shows a wedge-shaped opacity (arrow) projecting over the midheart representing the collapsed right middle lobe. Round atelectasis: Noncontrast CT shows a rounded\\nopacity in the medial right lower lobe (red arrows). This\\nexample meets all five criteria for round atelectasis\\nincluding adjacent pleural abnormality (effusion), opacity\\nin contact with the pleura, round shape, volume loss in\\nthe affected lobe, and the comet tail sign (yellow arrows)\\nrepresenting curved vessels and bronchi leading to the\\nfocus of round atelectasis.\\npulmonary veins (and lymphatics, not pictured)\\nrun in the interlobular septaConsolidation and ground glassSchematic demonstrates complete\\nhazy filling of the alveolus. The\\npulmonary vessels are still visible.\\nground glass opacification: Noncontrast CT shows diffuse\\nground glass opacification (ggo). The pulmonary architecture,\\nincluding vasculature and bronchi, can be still seen, which\\nis characteristic for ggo. Although these imaging findings\\nare nonspecific, this was a case of acute respiratory distress\\nsyndrome (ARd S).\\nAlveolar proteinosis, an idiopathic disease characterized by alveolar filling by a proteinaceous\\nsubstance. The distribution is typically central, with sparing of the periphery.\\nCoronal schemati c shows centralpredominant\\nground glass att enuati on.\\nPeripheral ground glass or consolidation Coronal CT demonstrates centralpredominant\\nground glass and associated septal thickening\\n(the crazy paving patt ern) in a case of alveolar\\nproteinosis.\\nAxial CT shows peripheral and subpleural ground glass\\natt enuati on. This was a case of organizing pneumonia.\\nSchematic demonstrates smooth\\ninterlobular septal thickening.\\nSchematic demonstrates smooth Smooth interlobular septal thickening: CT demonstrates smooth\\nthickening of the interlobular septa (arrows) in pulmonary\\nedema.\\nNodular septal thickening: CT shows a dominant mass in the\\nright lung (yellow arrows) with peripheral nodularity and septal\\nthickening. This was a case of lymphangitic carcinomatosis.\\nSchematic demonstrates interlobula\\nseptal thickening and ground glass\\nopacification.\\nAxial CT shows interlobular septal thickening in regions of\\nground glass opacification, representing crazy paving. This\\nwas a case of alveolar proteinosis, the entity in which crazy\\npaving was first described.\\nCentrilobular nodules Schematic demonstrates a centrilobula\\nnodule, located at the center of the\\npulmonary lobule.\\nAxial CT demonstrates innumerable subcentimeter\\ncentrilobular nodules of ground glass attenuation (arrows).\\nNone of the nodules extends to the pleural surface, which\\nis typical of a centrilobular distribution. This was a case of\\nrespiratory bronchiolitis interstitial lung disease (RBILd).\\nDiffuse panbronchiolitis is a chronic inflammatory disorder characterized by lymphoid\\nhyperplasia in the walls of the respiratory bronchioles resulting in bronchiolectasis. It typically\\naffects patients of Asian descent.\\nSilicosis, an inhalation lung disease that develops in response to inhaled silica particles, is\\ncharacterized by upper lobe predominant centrilobular and perilymphatic nodules.\\nPerilymphatic nodules:Schematic of the secondary pulmonary lobule (top left\\nimage) demonstrates two of the three distributions of\\nperilymphatic nodules. The gray nodules are located along\\nthe bronchovascular bundle and the white nodules are\\nlocated along the interlobular septa.\\nSchematic of the lungs (left image) demonstrates the\\nperibronchovascular and subpleural/fissural distribution.\\nSchematic shows several nodules\\ncentered on an opacified small airwa Treeinbud nodularity: Axial CT show numerous small nodules\\n(arrows) “budding” off of linear branching structures in\\nthe right middle lobe. This case was secondary to atypical\\nmycobacteria.\\nMultiple cavitary nodules Coronal schematic shows numerous cavitary\\nlesions bilaterally.\\nAxial CT shows a single spiculated cavitary lesion\\nin the left upper lobe (arrow). This was a case of\\nsquamous cell carcinoma.\\nAxial CT shows numerous cavitary and noncavitary\\nlesions bilaterally, in a random distribution. This was\\na case of tricuspid endocarditis and septic emboli.\\nCase courtesy Michael Hanley, MD, University of\\nVirginia Health System.\\n• Multiple cavitary lesions are typically vascular or spread through the vascular system:Coronal schematic shows numerous thin-walled\\ncystic lesions bilaterally.\\nmphysema, which tends to be upperlobe predominant in a smoker.\\nPneumocystis jiroveci pneumonia, which features cysts in latestage disease.\\nAxial CT shows bilateral thin-walled cysts that are\\nof varying sizes but are predominantly regular in\\nshape. There is a small left pleural effusion. This was\\na case of lymphangioleiomyomatosis.\\nPneumatocele, which is an air-filled space caused by prior lung trauma or infection.\\nUpper lobe fibrotic changes Coronal CT shows bibasilar fibrosis and honeycombing\\n(arrows), with relative sparing of the upper lobes. This\\nwas a case of idiopathic pulmonary fibrosis.\\nCoronal CT shows upperlobe predominant\\nsubpleural fibrosis and traction bronchiectasis. A\\npathologic diagnosis was not established in this cas [END]Pulmonary abscessPulmonary gangreneEmpyemaPneumatoceleBronchopleural fistula (BPF) [END]Split pleura sign: Contrast-enhanced CT\\nshows enhancement of the thickened\\nvisceral and parietal pleural layers (arrows),\\nwhich encase a pleural fluid collection. The split pleura sign is seen in the majority\\nof exudative effusions, although it is not\\nspecific. Similar findings can be seen\\nin malignant effusion, mesothelioma,\\nfibrothorax, and after talc pleurodesis.\\n [END]Primary tuberculosis [END]Primary tuberculosis: Chest radiograph (left image) shows a vague right upper lung opacity (arrow). CT\\nshows a patchy opacification (arrow) in the lower portion of the right upper lobe with adjacent treeinbud nodularity. The patient\\'s sputum grew Mycobacterium tuberculosis.Reactivation (postprimary) tuberculosisase courtesy Michael Hanley, MD, University of Virginia Health System.\\n [END]Tuberculous adenopathy: Contrast-enhanced\\nneck CT shows marked rightsided adenopathy\\n(arrows) with peripheral enhancement and\\ncentral necrosis. [END]Case courtesy Ritu R. Gill, MD, MPH, Brigham\\nand Women’s Hospital. Reactivation TB: Frontal chest radiograph (left image) shows a cavitary lesion in the left upper lobe\\n(arrow), confirmed by CT (arrow). There was no significant mediastinal adenopathy. The differential\\ndiagnosis of this appearance would include cavitary primary lung cancer, which would be expected to\\nfeature a thicker wall.\\nCase courtesy Ritu R. Gill, MD, MPH, Brigham and Women’s Hospital.\\n [END]Miliary tuberculosis [END]Case courtesy Ritu R. Gill, MD, MPH, Brigham and Women’s Hospital. Healed tuberculosis: Radiograph shows scarring, volume loss, and superior hilar retraction (arrows). CT\\nshows apical scarring.\\nmiliary tuberculosis: Radiograph and CT show innumerable tiny nodules in a random pattern, reflecting\\nhematogenous seeding of tuberculosis.\\n [END]Atypical mycobacteria infectionHistoplasma capsulatumCoccidioides immitis and Blastomyces dermatitidis [END]Mycobacterium avium intracellulare infection: Coronal (left) and axial CT show right upper lobe and\\nlingular treeinbud opacities and bronchiectasis, with more focal consolidation in the lingula (arrow). [END]Pneumocystis jiroveci pneumonia [END]Pneumocystis pneumonia with cystic change:\\nRadiograph (top left image) shows multiple\\nupperlobe predominant small nodular\\nopacities. CT (top image) better defines\\nthe nodular opacities as being primarily\\ncentrilobular in distribution, with confluent\\nground glass attenuation and multiple small\\ncysts/pneumatoceles (arrows). Cases courtesy Ritu R. Gill, MD, MPH, Brigham\\nand Women’s Hospital.\\n [END]Saprophytic aspergillosis (aspergilloma) [END]Fingeringlove sign of ABPA: Chest radiograph shows upperlobe predominant bronchiectasis with\\nbranching perihilar opacities most prominent on the right (arrow). CT shows dense bronchial mucoid\\nimpaction representing the fingeringlove sign (arrow). Aspergilloma and Monod sign: CT shows an\\nirregular opacity (arrows) resting dependent\\nwithin a left upper lobe cavity, representing\\nan aspergilloma (arrows). The Monod\\nsign is the curvilinear air surrounding the\\naspergilloma. This patient has prior TB\\nwith biapical scarring and prior left apical\\ncavitation.\\nCase courtesy Ritu R. Gill, MD, MPH, Brigham\\nand Women’s Hospital.\\n [END]Airwayinvasive aspergillosisAngioinvasive aspergillosis [END]Halo sign: CT shows a left upper lobe\\nmass with peripheral ground glass\\n(yellow arrow), representing pulmonary\\nhemorrhage. Air crescent sign: Noncontrast CT shows\\na cavitary lesion in the right upper lung\\nwith a crescent-shaped air collection\\nanteriorly (arrows).\\nCase courtesy Ritu R. Gill, MD, MPH,\\nBrigham and Women’s Hospital.\\n [END]Pulmonary edema [END]mild pulmonary edema: Radiograph demonstrates moderate enlargement of the cardiac silhouette\\nand sternotomy wires. There are increased interstitial markings with kerley B lines (arrows). Chest\\nCT shows smooth interlobular septal thickening (arrows), corresponding to the kerley B lines seen on\\nradiography. mild geographic ground glass attenuation likely corresponds to early alveolar filling.Right mainstem bronchus intubation: Chest\\nradiograph shows the endotracheal tube terminating\\nin the right mainstem bronchus (yellow arrow), a few\\ncentimeters distal to the carina (red arrow).\\n [END]Pulmonary artery catheter [END]Peripherally inserted central catheter (PICC) in the azygos vein: Frontal radiograph shows a left-sided\\nPICC coursing medially at the confluence of the brachiocephalic vein and the Sv C (arrow). Lateral\\nradiograph shows that the PICC curves anteriorly before heading inferiorly and posteriorly into the\\nazygos vein (arrow).clinical overview of lung cancer Epidemiology Overview of the solitary pulmonary nodule Nodule morphology essentially diagnostic for a benign etiology Nodule morphology suggesting, but not diagnostic for, a benign etiology Nodule morphology suggesting malignancy [END]Histologic subtypes of lung cancerAdenocarcinoma [END]Regardless of risk, either PET, biopsy, or at least three followups at 3, 9, and 24 months. [END]Bronchioloalveolar carcinoma (BAC) [END]SCC: Axial CT shows a spiculated cavitary lesion\\n(arrow) shown at pathology to be squamous\\ncell carcinoma. Nonmucinous BAC (multifocal):\\nAxial CT shows a solid nodule\\nwith air bronchograms (yellow\\narrow) and a more peripheral\\nground glass nodule (red arrow).\\n [END]Small cell carcinomaLarge cell carcinomaCarcinoid tumor [END]mucinous BAC is an important differential consideration for chronic ground glass or consolidation, ofte\\nwith air bronchograms.\\nThe CT angiogram sign (not imaged above) describes the especially prominent appearance of\\nenhancing pulmonary vessels seen in a low attenuation, mucinrich consolidation of mucinous BAC. mucinous BAC is an important differential consideration for chronic ground glass or consolidation, ofte\\nwith air bronchograms.\\nSolitary pulmonary nodule or lung massCase courtesy Ritu R. Gill, MD, MPH, Brigham and Women’s Hospital.\\n [END]Lymphangitic carcinomatosis [END]Superior sulcus tumor: Chest radiograph shows nearly imperceptible increased opacity in the right apex\\n(arrow). A superior sulcus mass is clearly evident on CT (arrows), with encasement of the vasculature\\nand effacement of the superior mediastinal fat. Lymphangitic carcinomatosis: Axial (left image) and coronal CT shows asymmetric, nodular septal\\nthickening of the right lung in a patient with metastatic uterine sarcoma. Aside from a trace effusion,\\nthe left lung is clear.\\n [END]Pneumothoraxstaging of lung cancer [END]Overview of lung cancer staging Treatment based on stagingCase courtesy Ritu R. Gill, MD, MPH, Brigham and Women’s Hospital.\\nCase courtesy Ritu R. Gill, MD, MPH, Brigham and Women’s Hospital.\\nExample staging: Stage IVCase courtesy Ritu R. Gill, MD, MPH, Brigham and Women’s Hospital.\\nStage IIIA lung cancer: Axial CT in lung windows (left image) shows a right upper lobe peripheral mass\\n(arrow) that abuts the pleura. There is diff use centrilobular emphysema. Softti ssue window CT shows\\nipsilateral right paratracheal adenopathy (arrow). The T staging is T 2 (both for size >3 cm and invasion\\nof pleura) and the N staging is N 2 for ipsilateral mediasti nal nodes. The overall stage is therefore IIIA\\nand this tumor is resectable.\\nStage Iv lung cancer: Axial CT (left image) shows a dominant, spiculated right upper lobe T 2 mass. more\\ninferiorly (right image), a small contralateral nodule (arrow) is present. This nodule was determined to\\nbe malignant on follow-up, for an m staging of m 1a and stage Iv disease.\\nStage Iv lung cancer: Axial CT (left image) shows a left upper lobe mass with ipsilateral hilar adenopathy\\n(arrows). Coronal CT through the upper abdomen shows bilateral adrenal masses (arrows), which were\\nconfi rmed to be metastases (m 1b), representi ng stage Iv disease.\\n [END]Pulmonary hypertension [END]Definition of pulmonary hypertension Overview of pulmonary hypertension classification [END]Pulmonary hypertension due to leftto-right cardiac shunts – WHO group 1, precapillary [END]ulmonary hypertension due to lefttoright cardiac shunts – WHO group 1, precapillary Pulmonary venous hypertension – WHO group 2, postcapillary Chronic thromboembolic pulmonary hypertension (CTEPH) – WHO group 4, precapillary [END]Pulmonary embolism (PE) [END]Clinical diagnosis Imaging of pulmonary embolismmassive pulmonary embolism: CT pulmonary angiogram shows a large, nearlyocclusive filling defect i\\nthe right main pulmonary artery (arrows) extending proximally to the bifurcation.\\nCardiac evaluation Pitfalls of CT pulmonary angiogram [END]Idiopathic interstitial pneumonias [END]Overview of idiopathic interstitial pneumonias Type I pneumocytes form alveolar wall and participate in gas exchange.\\nType II pneumocytes produce surfactant, which prevents atelectasis.\\nAlveolar macrophages ingest and process inhaled particulate materials.\\nIPF: Noncontrast CT shows\\nbibasilar subpleural honeycombing\\n(yellow arrow) and tracti on\\nbronchiectasis with cysti c changeCo P: Axial CT shows patchy\\nconsolidati ve and ground glass\\nopaciti es in a peribronchovascular\\ndistributi on. There is a reverse halo\\nor atoll sign in the right lower lobe\\n(arrow), where a central lucency is\\nsurrounded by a peripheral opacity.\\nCase courtesy Seth Kligerman, MD,\\nUniversity of Maryland.\\nContrast-enhanced CT in a diff erent\\npati ent with Co P shows patchy\\nconsolidati on in a peribronchial and\\nsubpleural locati on.\\nCase courtesy Ritu R. Gill, MD, MPH,\\nBrigham and Women’s Hospital.\\nRBILd: Axial CT shows\\ninnumerable subtle ground\\nglass centrilobular nodules\\n(arrows). This case does\\nnot feature ground glass\\nopacifi cati on. Centrilobular\\nnodules are a nonspecifi c\\nfi nding, and the other common\\ndisease featuring centrilobular\\nnodules is hypersensiti vity\\npneumoniti s.\\nCase courtesy Ritu R. Gill, MD,\\nMPH, Brigham and Women’s\\nHospital.\\n [END]Hypersensitivity pneumonitis (HSP) [END]If there is relative sparing of the bases, chronic HSP is much more likely than IPF. [END]Simple pulmonary eosinophilia (Löffler syndrome)Chronic eosinophilic pneumoniaPulmonary vasculitisChurg–Strauss [END]Chronic eosinophilic pneumonia: Chest radiograph demonstrates upperlobe and peripheral\\nconsolidation. Axial CT shows bilateral patchy peripheral consolidative opacities. Although this is a\\nnonspecific appearance, the appearance was unchanged over multiple priors, which suggests against\\nbacterial pneumonia.Alveolar hemorrhage due to microscopic polyangiitis: Axial CT shows basal predominant central ground\\nglass with associated mild interlobular septal thickening. This patient had renal insufficiency and had a\\nPANCA positive vasculitis with alveolar hemorrhage.\\n [END]Drug toxicityRadiation lung injury [END]Acute radiation pneumonitis: Coronal and axial CT demonstrate geographic left upper lung groun\\nglass (arrows) with nonanatomic superior and lateral linear margins. Radiation fibrosis: Chest radiograph and enhanced CT shows paramediastinal fibrotic changes\\nwithin the radiation port. despite radiation, the patient has persistent left mediastinal adenopathy\\n(arrow).\\n [END]Sarcoidosis [END]Stage 1 sarcoidosis: Frontal (left image) and lateral radiographs show symmetric adenopath\\ndemonstrating the 1-2-3 sign:\\nRight paratracheal adenopathy (yellow arrows, “1”)\\nRight hilar adenopathy (red arrow, “2”)\\nLeft hilar adenopathy (red arrow, “3”) The lateral radiograph demonstrates the donut sign with adenopathy circumferentially encircling\\nthe trachea (arrows).\\nThe lung parenchyma is normal. Case courtesy Ritu R. Gill, MD, MPH, Brigham and Women’s Hospital.\\nSarcoidosis: Axial CT in a different\\npatient shows peripherallycalcified\\n(\"eggshell\") prevascular and\\nparatracheal lymph nodes (arrows).\\nCase courtesy Ritu R. Gill, MD, MPH,\\nBrigham and Women’s Hospital.\\nCT in a patient with sarcoidosis shows tiny\\nperibronchovascular and fissural (subpleural)\\nnodules in the bilateral upper lobes.\\nCase courtesy Darryl Sneag, MD, Brigham and\\nWomen’s Hospital.\\nCT in a different patient with sarcoidosis shows\\nslightly larger nodules in a peribronchovascular\\ndistribution.\\nCase courtesy Ritu R. Gill, MD, MPH, Brigham\\nand Women’s Hospital.\\nPerilymphatic nodules are found along the course of the pulmonary lymphatics, which are\\nperibronchovascular, subpleural, and within the interlobular septa.\\nAxial contrast-enhanced CT\\nthrough the liver and spleen\\nin a patient with sarcoidosis\\nshows innumerable small\\nhypoattenuating splenic nodul Case courtesy Michael Hanley,\\nMD, University of Virginia Health\\nSystem.\\nPulmonary Langerhans cell histiocytosis:\\nChest radiograph shows multiple upperlobe\\npredominant cysts with the suggestion of\\nmultiple small nodules. There is a moderate lase courtesy Ritu R. Gill, MD,\\nPH, Brigham and Women’s\\nospital.\\nPulmonary alveolar\\nproteinosis with\\nsuperimposed Nocardia:paving predominantly in\\nthe right lung, with a dense\\nconsolidative opacity in\\nthe left upper lobe (arrow)\\nrepresenting superimposed\\nNocardia pneumonia.\\nCase courtesy Ritu R. Gill, MD,\\nMPH, Brigham and Women’s\\nHospital.\\nPatient 1 (top radiograph and top CT) has end-stage LAm with severe cystic change and moderate\\nright apical and posterior pneumothoraces.\\nPatient 2 (bottom left CT) shows a less severe\\nstage of disease, with multiple smoothwalled\\ncysts and bilateral pleural effusions. Although the\\npleural fluid was not sampled in this case, there is\\nan association of LAm with chylous effusion.\\nCases courtesy Ritu R. Gill, MD, MPH, Brigham\\nand Women’s Hospital.\\n [END]Anatomy of the mediastinum [END]Anterior mediastinumMiddle mediastinumPosterior mediastinumAn “interface” is formed by the contact of two soft tissue structures of different densities.\\nAortopulmonary (AP) window Retrosternal clear space Left superior intercostal vein (LSIV)Detection of an anterior mediastinal mass Detection of a middle mediastinal mass Detection of a posterior mediastinal massdisplacement of the left paraspinal line: Frontal chest radiograph (left) shows lateral displacement\\nand bowing of the left paraspinal line (arrows), suggestive of a posterior mediastinal mass. Axial\\ncontrast-enhanced CT shows a nonenhancing homogeneous mass (arrow) effacing the esophagus.\\nAlthough extremely rare, this was a liposarcoma.\\nOverview of prevascular anterior mediastinal masses Invasive thymoma with drop metastasis: Contrast-enhanced CT shows a slightly heterogeneous mass\\n(yellow arrow) anterior to the aorta and Sv C, with two pleuralbased metastases (red arrows).\\n [END]Germ cell tumor (GCT) [END]Teratoma: Chest radiograph shows an extra mediastinal contour (yellow arrow) lateral to the aortic\\narch (red arrow). Because the mass does not silhouette the aorta it cannot be in the posterior\\nmediastinum. on CT there is an anterior mediastinal mass containing a large focus of fat (yellow\\narrow). The differential in this case would include a thymolipoma. [END]LymphomaNonlymphomatous adenopathy [END]Inferior extension of a goiter into the mediastinum:Chest radiograph demonstrates a right superior\\nmediastinal mass (arrow) with obscuration of the\\nsuperior border of the mass, representing the\\ncervicothoracic sign, which implies that the mass is\\nin continuity with the soft tissues of the neck.\\nNoncontrast CT shows a large heterogeneous mass\\n(arrows) with coarse calcifications replacing the\\nthyroid gland.\\n [END]Castleman disease [END]Overview of precardiac anterior mediastinal masses [END]Lateral meningoceleEsophageal neoplasmForegut duplication cystParaspinal abscess [END]Foregut duplication cyst: Frontal radiograph shows a subtle opacity overlying the right heart (arrow),\\nshown to be a fluid attenuation lesion adjacent to the esophagus on CT. osteomyelitis and paraspinal phlegmon: Radiograph shows displacement of the bilateral paraspinal\\nlines (arrows). Sagittal T 1 (middle image) and T 2-weighted mRI (right image) shows loss of disk space\\nheight and marrow edema in two adjacent lower thoracic vertebral bodies. There is associated\\nparaspinal phlegmonous change (arrows), which caused the paraspinal line displacement on\\nradiography.\\nOverview of non-neoplastic diffuse tracheal diseaseCase courtesy Ritu R. Gill, MD, MPH, Brigham and Women’s Hospital.\\n [END]Amyloidosis [END]Tracheal tuberculosis: Axial CT in soft-tissue window through the upper trachea (left image) shows\\ncircumferential tracheal thickening (arrows). Coronal curvedmultiplanar reformation (right image)\\nshows multifocal tracheal stenosis (arrows) and left mainstem bronchus stenosis. These images were\\nobtained approximately one month after treatment for active postprimary tuberculosis. Tracheal amyloid: Axial CT images show\\nnodular and irregular thickening of the\\ntrachea (arrows). This pattern is not specific,\\nand the differential would include also\\ninclude sarcoidosis, multifocal adenoid cystic\\ncarcinoma, and tracheal metastases.\\nCase courtesy Ritu R. Gill, MD, MPH, Brigham\\nand Women’s Hospital.\\n [END]Rare causes of focal tracheal stenosisLarge airwaysBronchiectasis [END]Exaggerated immune response, as seen in Ab PA and vasculitis.• Extremely uncommon causes of focal tracheal stenosis include Behçet and Crohn disease.\\nImpaired immunity: congenital immunodeficiency, transplant recipients, and agammaglobulinemia.\\n [END]Broncholithiasis [END]Congenital connective tissue disorders – Mounier–Kuhn (a connective tissue disorder causing\\ntracheobronchomegaly leading to recurrent pneumonia), or Williams–campbell (a rare disorder of\\nthe distal bronchial cartilage, which may be congenital or acquired as a sequela of viral infection). [END]Centrilobular emphysemaParaseptal emphysema [END]Centrilobular and paraseptal emphysema: Coronal (left image) and axial CT demonstrate both\\ncentrilobular and paraseptal emphysema. Centrilobular emphysema is predominant in the upper lobes\\n(yellow arrows) and paraseptal emphysema is seen anteromedially (blue arrows). All smokingrelated lung disease (RB, RBILd, d IP, PLCH, and emphysema) may be within the same\\nspectrum of disease caused by macrophagemediated inflammation in reaction to inhaled particles\\nand toxins.\\n [END]Airway tumorsSquamous cell carcinoma (SCC) [END]Case courtesy Ritu R. Gill, MD, MPH, Brigham and Women’s Hospital. Endotracheal squamous cell carcinoma: Radiograph shows a subtle tracheal filling defect (arrows). on\\nCT there is a heterogeneously enhancing, wellcircumscribed mass markedly narrowing the tracheal\\nlumen (arrow).\\nCase courtesy Ritu R. Gill, MD, MPH, Brigham and Women’s Hospital.\\n [END]Carcinoid [END]Tracheal adenoid cystic carcinoma: CT (left image) shows irregular circumferential tracheal thickening\\n(arrow). Postcontrast coronal T 1-weighted mRI (right image) shows an enhancing nodular mass\\nextending into the tracheal lumen (arrows). [END]Tracheal lymphomaEndobronchial metastasis [END]rect invasion of the central airways by adjacent malignancy [END]MesotheliomaMetastasesMultiple myeloma/plasmacytoma [END]mesothelioma: Contrast-enhanced CT through the thorax (left image) shows extensive nodular pleural\\nthickening of the left hemithorax (arrows). Images through the upper abdomen show extensive soft\\ntissue abnormality with invasion of the chest wall (arrows). [END]Pleural effusionTransudateExudate [END]Fibrous tumor of the pleura: CT topogram (left image) shows a round opacity (arrow) with a\\ncircumscribed medial margin and indisti nct lateral margin, suggesti ng a pleuralbased mass. CT\\nconfi rms that the mass (arrow) is pleuralbased, with a broad att achment to the pleura.Eisenhuber, E. Signs in imaging - the tree in bud sign. Radiology 222, 771-2 (2002).\\nEngoren, m. Lack of association between atelectasis and fever. Chest 107, 81-84 (1995).\\nmarshall, g.B. et al. Signs in thoracic imaging. Journal of Thoracic Imaging 21, 76-90 (2006).\\n [END]Pulmonary infection: [END]Franquet, T. et al. Spectrum of Pulmonary Aspergillosis: Histologic, Clinical, and Radiologic Findings. Radiographics 21,\\n825 (2001). gotway, m.B. et al. The radiologic spectrum of pulmonary Aspergillus infections. Journal of Computer Assisted Tomography\\n26, 159-73 (2002).\\nSystrom, d.m. & Wittram, C. Case 9-2005-A 67-Year-old man with Acute Respiratory Failure. New England Journal of medicine\\n352, 1238 (2005).\\nTarver, R.d. et al. Radiology of communityacquired pneumonia. Radiologic Clinics of North America 43, 497-512, viii (2005).\\nWaite, S., Jeudy, J. & White, C.S. Acute lung infections in normal and immunocompromised hosts. Radiologic Clinics of North\\nAmerica 44, 295-315, ix (2006).\\n [END]Pulmonary edema and ICU imaging: [END]Ely, E.W. Using the Chest Radiograph To determine Intravascular volume Status: The Role of vascular Pedicle Width. Chest\\n121, 942-50 (2002). martin, g.S. Chest 122,\\n2087-2095 (2002).\\nHartman, T.E. Radiologic evaluation of the solitary pulmonary nodule. Radiologic Clinics of North America 43, 459-65,\\nvii (2005).\\nmacmahon, H. et al. guidelines for management of small pulmonary nodules detected on CT scans: a statement from th\\nFleischner Society. Radiology 237, 395-400 (2005).\\nTravis, W.d. et al. International Association for the Study of Lung Cancer/American Thoracic Society/European Respiratory\\nSociety international multidisciplinary classification of lung adenocarcinoma. Journal of Thoracic oncology 6, 244-85 (2011).\\nUy Bico, S.J. et al. Lung cancer staging essentials: the new TNm staging system and potential imaging pitfalls. Radiographics 30\\n1163-81 (2010).\\nAgnelli, g. & Becattini, C. Acute Pulmonary Embolism. New England Journal of medicine 363, 266-74 (2010).\\nduránmendicuti, A. & Sodickson, A. Imaging evaluation of the pregnant patient with suspected pulmonary embolism.\\nInternational Journal of obstetric Anesthesia 20, 51-9 (2011).\\nFrazier, A. et al. From the Archives of the AFIP: Pulmonary venoocclusive disease and Pulmonary Capillary Hemangiomatosis\\nRadiographics 27, 867-83 (2007).\\nkuriakose, J. & Patel, S. Acute pulmonary embolism. Radiologic Clinics of North America 48, 31-50 (2010).\\nPatel, N.m. et al. Pulmonary hypertension in idiopathic pulmonary fibrosis. Chest 132, 998-1006 (2007).\\nWittram, C. et al. CT angiography of pulmonary embolism: diagnostic criteria and causes of misdiagnosis. Radiographics 24,\\n1219-38 (2004).\\n [END]Diffuse lung disease: [END]Attili, A.k. et al. Smokingrelated interstitial lung disease: radiologicclinicalpathologic correlation. Radiographics 28, 1383-96;\\ndiscussion 1396-8 (2008). ill, R., matsusoka, S. & Hatabu, H. Cavities in the lung in oncology patients: imaging overview and differential diagnoses.\\npplied Radiology (June), 10–21 (2010).\\ngotway, m.B. et al. High-resolution CT of the lung: patterns of disease and differential diagnoses. Radiologic Clinics of North\\nAmerica 43, 513-42, viii (2005).\\nkligerman, S.J. et al. Nonspecific interstitial pneumonia: radiologic, clinical, and pathologic considerations. Radiographics 29,\\n73-87 (2009).\\nmedoff, B.d., Abbott, g.F. & Louissaint Jr, A. Case 16-2010–A 48-Year-old man with a Cough and Pain in the Left Shoulder.\\nNew England Journal of medicine 362, 2013 (2010).\\nmuellermang, C., grosse, C. & Schmid, k. What Every Radiologist Should know about Idiopathic Interstitial Pneumonias.\\nRadiographics 27, 595-616 (2007).\\nRossi, S.E. et al. “Crazy Paving” Pattern at Thin-Section CT of the Lungs: Radiologic Pathologic overview. Radiographics 23,\\n1509-519 (2003).\\nWebb, W.R. Thin-section CT of the secondary pulmonary lobule: anatomy and the image – the 2004 Fleischner lecture.\\nRadiology 239, 322-38 (2006).\\n [END]Mediastinum: [END]Quint, L.E. Imaging of anterior mediastinal masses. Cancer Imaging 7 Spec No, S 56-62 (2007). Tecce, P.m., Fishman, E.k. & kuhlman, J.E. CT evaluation of the anterior mediastinum: spectrum of disease. Radiographics 14,\\n973 (1994).\\nWhitten, C., khan, S. & munneke, g. A diagnostic Approach to mediastinal Abnormalities. Radiographics 27, 657-71 (2007).\\nJavidan Nejad, C. & Bhalla, S. Bronchiectasis. Radiologic Clinics of North America 47, 289-306 (2009).\\nmartinez, S. et al. mucoid impactions: fingeringlove sign and other CT and radiographic features. Radiographics 28,\\n1369-82 (2008).\\n [END]Pleura: [END]Qureshi, N.R. & gleeson, F.v. Imaging of pleural disease. Clinics in Chest Medicine 27, 193-213 (2006). Sugarbaker, d. & Wolf, A. Surgery for malignant pleural mesothelioma. Expert Review of Respiratory medicine 4,\\n363-72 (2010).\\nWang, Z.J. et al. malignant Pleural mesothelioma: Evaluation with CT, mR Imaging, and PET. Radiographics 24, 105 (200 [END]'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# merging all text data; lowk make dictionaries instead\n",
    "def get_text_recursively(obj):\n",
    "    text = ['']\n",
    "    \n",
    "    def helper(obj):\n",
    "        if isinstance(obj[0], str):\n",
    "            for s in obj:\n",
    "                text[0] += s + \" [END]\"\n",
    "        else:\n",
    "            for ls in obj:\n",
    "                helper(ls)\n",
    "\n",
    "    helper(obj)\n",
    "    return text[0]\n",
    "\n",
    "text = get_text_recursively(text_data[0])\n",
    "text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "fb76aefb-33b9-445d-879c-6481f61a1eef",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "85\n",
      "['\\n', ' ', '\"', '&', \"'\", '(', ')', ',', '-', '.', '/', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', ':', ';', '>', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'Y', 'Z', '[', ']', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', 'á', 'ç', 'ö', '–', '’', '“', '”', '•']\n"
     ]
    }
   ],
   "source": [
    "chars = sorted(list(set(text)))\n",
    "vocab_size = len(set(text))\n",
    "print(vocab_size)\n",
    "print(chars)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "930a00cf-329c-4f3f-aa75-d72e5278c470",
   "metadata": {},
   "source": [
    "# Building an LLM / text transformer model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3875515-6b75-4f78-aca3-275fec95df99",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "##### BERT-esque may be more effective due to bidirectionality; currently using a GPT-esque model\n",
    "##### Additionally, worth exploring RAG in conjunction with an LLM:\n",
    "https://paperswithcode.com/method/rag\n",
    "\n",
    "Here's how RAG and LLM can work together in a VQA model:\n",
    "\n",
    "- When a question is posed to the VQA system, the retriever component (dense retrieval, sparse retrieval, etc.) retrieves relevant textual information or passages from a knowledge base - the textbook data in this case.\n",
    "- The retrieved passages are then passed to the LLM, which generates an answer based on the context provided by the passages and the question itself.\n",
    "- The generated answer is returned as the output of the VQA model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f4ccef5-c787-41b7-9a61-77d5248c7833",
   "metadata": {},
   "source": [
    "### Encoding / Decoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "c27279c4-cdff-4eca-b7b7-d9aa7bfb0742",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: tiktoken in /Users/andychen2121/anaconda3/lib/python3.11/site-packages (0.5.1)\n",
      "Requirement already satisfied: regex>=2022.1.18 in /Users/andychen2121/anaconda3/lib/python3.11/site-packages (from tiktoken) (2023.6.3)\n",
      "Requirement already satisfied: requests>=2.26.0 in /Users/andychen2121/anaconda3/lib/python3.11/site-packages (from tiktoken) (2.31.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /Users/andychen2121/anaconda3/lib/python3.11/site-packages (from requests>=2.26.0->tiktoken) (2.0.4)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /Users/andychen2121/anaconda3/lib/python3.11/site-packages (from requests>=2.26.0->tiktoken) (3.4)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/andychen2121/anaconda3/lib/python3.11/site-packages (from requests>=2.26.0->tiktoken) (1.26.16)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /Users/andychen2121/anaconda3/lib/python3.11/site-packages (from requests>=2.26.0->tiktoken) (2023.7.22)\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.2.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.3.1\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install tiktoken"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "30792d3e-9704-452d-ba2b-0e9d16b7851d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tiktoken\n",
    "import torch\n",
    "enc = tiktoken.get_encoding('gpt2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "08235aaa-9ac0-464b-a7e6-143a7607b24a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a mapping from characters to integers\n",
    "stoi = { ch:i for i,ch in enumerate(chars) }\n",
    "itos = { i:ch for i,ch in enumerate(chars) }\n",
    "encode = lambda s: [stoi[c] for c in s] # encoder: take a string, output a list of integers\n",
    "decode = lambda l: ''.join([itos[i] for i in l]) # decoder: take a list of integers, output a string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "aab80299-7150-4498-91ea-4df020b4974e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([30388]) torch.int64\n",
      "tensor([24, 64, 51, 70, 65, 63, 75,  1, 49, 28, 37, 27, 50, 32, 64, 70, 55, 68,\n",
      "        62, 65, 52, 51, 68,  1, 56, 59,  1, 69, 69, 71, 68, 55, 69, 36, 55, 53,\n",
      "        58, 51, 64, 59, 69, 63, 69,  1, 65, 56,  1, 51, 70, 55, 62, 55, 53, 70,\n",
      "        51, 69, 59, 69, 28, 51, 53, 58,  1, 65, 56,  1, 70, 58, 55,  1, 56, 59,\n",
      "        72, 55,  1, 62, 65, 52, 55, 69,  1, 70, 55, 64, 54, 69,  1, 70, 65,  1,\n",
      "        53, 65, 62, 62, 51, 66, 69, 55,  1, 59])\n"
     ]
    }
   ],
   "source": [
    "data = torch.tensor(encode(text), dtype=torch.long)\n",
    "print(data.shape, data.dtype)\n",
    "print(data[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1a3e230-3992-4886-afb8-ddf12f1ca66e",
   "metadata": {},
   "source": [
    "### Loading / Batching Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "c8bf4bd6-5ac6-4215-bc0f-1294dcecf785",
   "metadata": {},
   "outputs": [],
   "source": [
    "n = int(0.9 * len(data))\n",
    "train_data = data[:n]\n",
    "test_data = data[n:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "8500b394-2a03-4b41-8e57-cbe58956d0a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 32\n",
    "block_size = 8\n",
    "n_embd = 32\n",
    "\n",
    "def get_batch(split):\n",
    "    data = train_data if split == 'train' else test_data\n",
    "    ix = torch.randint(len(data) - block_size, (batch_size,))\n",
    "    x = torch.stack([data[i:i+block_size] for i in ix])\n",
    "    y = torch.stack([data[i+1:i+block_size+1] for i in ix])\n",
    "    return x, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "f72cd5a9-d278-4d08-8b06-3f31d1f891f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "xb, yb = get_batch(\"train\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e2fddc9-c063-429b-b903-4818f9b8667b",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### Simple Bigram Language Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "418a1c07-4819-43ee-9d27-15ae2de8296f",
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device( cuda if torch.cuda.is_available() else 'cpu' )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 490,
   "id": "19ac1b6b-a087-4d9e-a30c-e4f56335f464",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([256, 85])\n",
      "tensor(4.6054, grad_fn=<NllLossBackward0>)\n",
      "\n",
      "Zn–3:RzBg4-Y1 6t1tEO0'H,•” Ji8c3(rr\n",
      "[wuCdDw“:]NxP&r\n",
      "Wo•.SUbdL.SK0 [’:EsGwdbd\"kcmsn2oW 0Dny;L11uf[n08\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.nn import functional as F\n",
    "\n",
    "class BigramLanguageModel(nn.Module):\n",
    "\n",
    "    def __init__(self, n_embd):\n",
    "        super().__init__()\n",
    "        # each token directly reads logits for next token off a lookup table\n",
    "        self.token_embedding_table = nn.Embedding(vocab_size, n_embd)\n",
    "        self.position_embedding_table = nn.Embedding(block_size, n_embd)\n",
    "        self.lm_head = nn.Linear(n_embd, vocab_size)\n",
    "\n",
    "    def forward(self, idx, targets=None):\n",
    "        B, T = idx.shape\n",
    "        \n",
    "        # idx and targets are both (batch_size, block_size) tensor of integers or (B, T)\n",
    "        token_emb = self.token_embedding_table(idx) # (B, T, C) = batch, time, channel\n",
    "        x = token_emb\n",
    "        logits = self.lm_head(x) # (B, T, vocab_size)\n",
    "        \n",
    "        if targets == None:\n",
    "            loss = None\n",
    "        else:\n",
    "            B, T, C = logits.shape\n",
    "            logits = logits.view(B*T, C)\n",
    "            targets = targets.view(B*T)\n",
    "            loss = F.cross_entropy(logits, targets)\n",
    "        return logits, loss\n",
    "\n",
    "    def generate(self, idx, max_tokens):\n",
    "        # idx if a (batch_size, block_size) tensor of integers or (B, T)\n",
    "        for _ in range(max_tokens):\n",
    "            logits, loss = self.forward(idx) # get predictions\n",
    "            logits = logits[:, -1, :] # becomes (B, C)\n",
    "            # apply softmax to get probabilities\n",
    "            probs = F.softmax(logits, dim=-1) # (B, C)\n",
    "            # sample from the distribution\n",
    "            idx_next = torch.multinomial(probs, num_samples=1)\n",
    "            # attend sample to running sequence\n",
    "            idx = torch.cat((idx, idx_next), dim=1) # (B, T+1)\n",
    "        return idx\n",
    "\n",
    "\n",
    "m = BigramLanguageModel(n_embd)\n",
    "logits, loss = m.forward(xb, yb)\n",
    "print(logits.shape)\n",
    "print(loss)\n",
    "print(decode(m.generate(torch.zeros((1, 1), dtype=torch.long), max_tokens = 100)[0].tolist()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8aeeddb9-eaeb-4901-8907-f7aeabc2b841",
   "metadata": {},
   "source": [
    "Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 491,
   "id": "33037926-c8e9-4eb7-ae2d-db0dddd8462e",
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = torch.optim.AdamW(m.parameters(), lr=1e-3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 492,
   "id": "3286cce5-ea46-449d-b15a-2973c51adc14",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.448796272277832\n"
     ]
    }
   ],
   "source": [
    "for steps in range(10000):\n",
    "    \n",
    "    # sample batch of data\n",
    "    xb, yb = get_batch('train')\n",
    "\n",
    "    # evaluate loss\n",
    "    logits, loss = m.forward(xb, yb)\n",
    "    optimizer.zero_grad(set_to_none=True)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "print(loss.item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 495,
   "id": "9c72d295-33d4-4dbc-8975-e97fe404ece9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "pe ppa d) c [END, orasthits\n",
      "PILAla nesurowe ll oungi malisugeretorrr ofim thalaluc s al mmomas.\n",
      "car \n"
     ]
    }
   ],
   "source": [
    "print(decode(m.generate(torch.zeros((1, 1), dtype=torch.long), max_tokens = 100)[0].tolist()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ab39d8f-3a34-4895-affc-1e6de1511591",
   "metadata": {},
   "source": [
    "# Self Attention Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3558c35-eeab-4b40-aea6-1c6c1156a74b",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### Demo:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "4d51e534-df8d-4ce3-bf3f-517259fe5a92",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[1.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n",
       "          0.0000e+00, 0.0000e+00, 0.0000e+00],\n",
       "         [3.9789e-01, 6.0211e-01, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n",
       "          0.0000e+00, 0.0000e+00, 0.0000e+00],\n",
       "         [1.9113e-01, 7.0959e-01, 9.9280e-02, 0.0000e+00, 0.0000e+00,\n",
       "          0.0000e+00, 0.0000e+00, 0.0000e+00],\n",
       "         [2.8954e-02, 5.6960e-01, 3.0922e-01, 9.2228e-02, 0.0000e+00,\n",
       "          0.0000e+00, 0.0000e+00, 0.0000e+00],\n",
       "         [1.1165e-01, 1.3680e-01, 9.3979e-02, 1.6007e-02, 6.4156e-01,\n",
       "          0.0000e+00, 0.0000e+00, 0.0000e+00],\n",
       "         [2.1489e-01, 9.1245e-02, 1.3590e-01, 2.5661e-01, 1.6505e-02,\n",
       "          2.8484e-01, 0.0000e+00, 0.0000e+00],\n",
       "         [4.0136e-02, 1.6778e-01, 1.6040e-02, 6.1050e-02, 6.6853e-03,\n",
       "          6.3306e-01, 7.5253e-02, 0.0000e+00],\n",
       "         [1.5845e-01, 1.6098e-01, 2.3580e-02, 5.5861e-02, 6.5940e-02,\n",
       "          5.2553e-02, 1.6164e-01, 3.2100e-01]],\n",
       "\n",
       "        [[1.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n",
       "          0.0000e+00, 0.0000e+00, 0.0000e+00],\n",
       "         [3.4102e-01, 6.5898e-01, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n",
       "          0.0000e+00, 0.0000e+00, 0.0000e+00],\n",
       "         [8.5531e-02, 8.9441e-01, 2.0056e-02, 0.0000e+00, 0.0000e+00,\n",
       "          0.0000e+00, 0.0000e+00, 0.0000e+00],\n",
       "         [7.7769e-02, 1.8299e-01, 3.5861e-01, 3.8062e-01, 0.0000e+00,\n",
       "          0.0000e+00, 0.0000e+00, 0.0000e+00],\n",
       "         [6.8147e-03, 5.3224e-02, 9.0496e-01, 1.1226e-02, 2.3773e-02,\n",
       "          0.0000e+00, 0.0000e+00, 0.0000e+00],\n",
       "         [4.1447e-02, 1.1067e-01, 3.1527e-04, 6.0868e-02, 7.7102e-01,\n",
       "          1.5676e-02, 0.0000e+00, 0.0000e+00],\n",
       "         [9.1443e-02, 5.1850e-02, 5.9230e-02, 8.8241e-02, 1.6987e-01,\n",
       "          2.1774e-01, 3.2163e-01, 0.0000e+00],\n",
       "         [9.2877e-02, 2.1087e-01, 4.3901e-01, 6.9742e-02, 2.9481e-02,\n",
       "          1.5690e-02, 1.1318e-01, 2.9149e-02]],\n",
       "\n",
       "        [[1.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n",
       "          0.0000e+00, 0.0000e+00, 0.0000e+00],\n",
       "         [4.2860e-01, 5.7140e-01, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n",
       "          0.0000e+00, 0.0000e+00, 0.0000e+00],\n",
       "         [2.2228e-01, 4.0165e-01, 3.7606e-01, 0.0000e+00, 0.0000e+00,\n",
       "          0.0000e+00, 0.0000e+00, 0.0000e+00],\n",
       "         [1.1798e-01, 2.0116e-01, 2.2422e-01, 4.5665e-01, 0.0000e+00,\n",
       "          0.0000e+00, 0.0000e+00, 0.0000e+00],\n",
       "         [7.6037e-01, 9.7490e-03, 1.0954e-02, 2.1102e-01, 7.9023e-03,\n",
       "          0.0000e+00, 0.0000e+00, 0.0000e+00],\n",
       "         [3.7376e-01, 1.7106e-01, 9.2431e-02, 9.3071e-02, 1.7756e-01,\n",
       "          9.2124e-02, 0.0000e+00, 0.0000e+00],\n",
       "         [5.1480e-03, 1.8794e-01, 9.2675e-02, 5.1056e-01, 1.1500e-01,\n",
       "          6.3502e-02, 2.5180e-02, 0.0000e+00],\n",
       "         [2.9768e-01, 4.0398e-03, 4.5186e-03, 2.0942e-01, 1.0635e-02,\n",
       "          3.4064e-02, 2.3323e-02, 4.1632e-01]],\n",
       "\n",
       "        [[1.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n",
       "          0.0000e+00, 0.0000e+00, 0.0000e+00],\n",
       "         [4.1765e-01, 5.8235e-01, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n",
       "          0.0000e+00, 0.0000e+00, 0.0000e+00],\n",
       "         [3.1047e-01, 2.1847e-02, 6.6768e-01, 0.0000e+00, 0.0000e+00,\n",
       "          0.0000e+00, 0.0000e+00, 0.0000e+00],\n",
       "         [3.7042e-02, 3.5728e-01, 4.6034e-01, 1.4534e-01, 0.0000e+00,\n",
       "          0.0000e+00, 0.0000e+00, 0.0000e+00],\n",
       "         [2.2050e-02, 6.4244e-01, 1.0371e-01, 1.3881e-02, 2.1792e-01,\n",
       "          0.0000e+00, 0.0000e+00, 0.0000e+00],\n",
       "         [6.5819e-02, 2.8388e-01, 2.6237e-01, 1.0214e-01, 1.9969e-01,\n",
       "          8.6107e-02, 0.0000e+00, 0.0000e+00],\n",
       "         [1.6497e-02, 3.5194e-03, 2.9386e-02, 3.6244e-01, 6.0604e-03,\n",
       "          5.2133e-01, 6.0777e-02, 0.0000e+00],\n",
       "         [4.2040e-01, 9.8485e-02, 1.1211e-01, 2.3429e-02, 1.8697e-01,\n",
       "          1.6265e-02, 1.2208e-01, 2.0266e-02]]], grad_fn=<SoftmaxBackward0>)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.nn import functional as F\n",
    "\n",
    "B, T, C = 4, 8, 32\n",
    "x = torch.randn(B, T, C)\n",
    "\n",
    "# Self-Attention Head\n",
    "head_size = 16\n",
    "key = nn.Linear(C, head_size, bias=False)\n",
    "query = nn.Linear(C, head_size, bias=False)\n",
    "k = key(x) # (B, T, head_size) \n",
    "q = query(x) # (B, T, head_size)\n",
    "wei = q @ k.transpose(-2, -1) # (B, T, head_size) @ (B, head_size, T) --> (B, T, T)\n",
    "\n",
    "tril = torch.tril(torch.ones(T, T))\n",
    "# wei = torch.zeros((T,T)) - no longer all uniform because of attention!\n",
    "wei = wei.masked_fill(tril == 0, float('-inf'))\n",
    "wei = F.softmax(wei, dim=-1)\n",
    "out = wei @ x\n",
    "\n",
    "wei"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b0ecdea-12c6-4ff3-ac35-4fc32ca05109",
   "metadata": {},
   "source": [
    "### Implementation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "f6b95842-edff-412f-9729-7c8399b54c98",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.nn import functional as F\n",
    "\n",
    "# hyperparameters\n",
    "batch_size = 16 # how many independent sequences will we process in parallel?\n",
    "block_size = 32 # what is the maximum context length for predictions?\n",
    "max_iters = 5000\n",
    "eval_interval = 100\n",
    "learning_rate = 1e-3\n",
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "eval_iters = 200\n",
    "n_embd = 64\n",
    "n_head = 4\n",
    "n_layer = 4\n",
    "dropout = 0.0\n",
    "# ------------\n",
    "\n",
    "torch.manual_seed(1337)\n",
    "\n",
    "\n",
    "@torch.no_grad()\n",
    "def estimate_loss():\n",
    "    out = {}\n",
    "    model.eval()\n",
    "    for split in ['train', 'val']:\n",
    "        losses = torch.zeros(eval_iters)\n",
    "        for k in range(eval_iters):\n",
    "            X, Y = get_batch(split)\n",
    "            logits, loss = model(X, Y)\n",
    "            losses[k] = loss.item()\n",
    "        out[split] = losses.mean()\n",
    "    model.train()\n",
    "    return out\n",
    "\n",
    "class Head(nn.Module):\n",
    "    \"\"\" one head of self-attention \"\"\"\n",
    "\n",
    "    def __init__(self, head_size):\n",
    "        super().__init__()\n",
    "        self.key = nn.Linear(n_embd, head_size, bias=False)\n",
    "        self.query = nn.Linear(n_embd, head_size, bias=False)\n",
    "        self.value = nn.Linear(n_embd, head_size, bias=False)\n",
    "        self.register_buffer('tril', torch.tril(torch.ones(block_size, block_size)))\n",
    "\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "\n",
    "    def forward(self, x):\n",
    "        B,T,C = x.shape\n",
    "        k = self.key(x)   # (B,T,C)\n",
    "        q = self.query(x) # (B,T,C)\n",
    "        # compute attention scores (\"affinities\")\n",
    "        wei = q @ k.transpose(-2,-1) * C**-0.5 # (B, T, C) @ (B, C, T) -> (B, T, T)\n",
    "        wei = wei.masked_fill(self.tril[:T, :T] == 0, float('-inf')) # (B, T, T)\n",
    "        wei = F.softmax(wei, dim=-1) # (B, T, T)\n",
    "        wei = self.dropout(wei)\n",
    "        # perform the weighted aggregation of the values\n",
    "        v = self.value(x) # (B,T,C)\n",
    "        out = wei @ v # (B, T, T) @ (B, T, C) -> (B, T, C)\n",
    "        return out\n",
    "\n",
    "class MultiHeadAttention(nn.Module):\n",
    "    \"\"\" multiple heads of self-attention in parallel \"\"\"\n",
    "\n",
    "    def __init__(self, num_heads, head_size):\n",
    "        super().__init__()\n",
    "        self.heads = nn.ModuleList([Head(head_size) for _ in range(num_heads)])\n",
    "        self.proj = nn.Linear(n_embd, n_embd)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "\n",
    "    def forward(self, x):\n",
    "        out = torch.cat([h(x) for h in self.heads], dim=-1)\n",
    "        out = self.dropout(self.proj(out))\n",
    "        return out\n",
    "\n",
    "class FeedFoward(nn.Module):\n",
    "    \"\"\" a simple linear layer followed by a non-linearity \"\"\"\n",
    "\n",
    "    def __init__(self, n_embd):\n",
    "        super().__init__()\n",
    "        self.net = nn.Sequential(\n",
    "            nn.Linear(n_embd, 4 * n_embd),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(4 * n_embd, n_embd),\n",
    "            nn.Dropout(dropout),\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.net(x)\n",
    "\n",
    "class Block(nn.Module):\n",
    "    \"\"\" Transformer block: communication followed by computation \"\"\"\n",
    "\n",
    "    def __init__(self, n_embd, n_head):\n",
    "        # n_embd: embedding dimension, n_head: the number of heads we'd like\n",
    "        super().__init__()\n",
    "        head_size = n_embd // n_head\n",
    "        self.sa = MultiHeadAttention(n_head, head_size)\n",
    "        self.ffwd = FeedFoward(n_embd)\n",
    "        self.ln1 = nn.LayerNorm(n_embd)\n",
    "        self.ln2 = nn.LayerNorm(n_embd)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = x + self.sa(self.ln1(x))\n",
    "        x = x + self.ffwd(self.ln2(x))\n",
    "        return x\n",
    "\n",
    "# super simple bigram model\n",
    "class BigramLanguageModel(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        # each token directly reads off the logits for the next token from a lookup table\n",
    "        self.token_embedding_table = nn.Embedding(vocab_size, n_embd)\n",
    "        self.position_embedding_table = nn.Embedding(block_size, n_embd)\n",
    "        self.blocks = nn.Sequential(*[Block(n_embd, n_head=n_head) for _ in range(n_layer)])\n",
    "        self.ln_f = nn.LayerNorm(n_embd) # final layer norm\n",
    "        self.lm_head = nn.Linear(n_embd, vocab_size)\n",
    "\n",
    "    def forward(self, idx, targets=None):\n",
    "        B, T = idx.shape\n",
    "\n",
    "        # idx and targets are both (B,T) tensor of integers\n",
    "        tok_emb = self.token_embedding_table(idx) # (B,T,C)\n",
    "        pos_emb = self.position_embedding_table(torch.arange(T, device=device)) # (T,C)\n",
    "        x = tok_emb + pos_emb # (B,T,C)\n",
    "        x = self.blocks(x) # (B,T,C)\n",
    "        x = self.ln_f(x) # (B,T,C)\n",
    "        logits = self.lm_head(x) # (B,T,vocab_size)\n",
    "\n",
    "        if targets is None:\n",
    "            loss = None\n",
    "        else:\n",
    "            B, T, C = logits.shape\n",
    "            logits = logits.view(B*T, C)\n",
    "            targets = targets.view(B*T)\n",
    "            loss = F.cross_entropy(logits, targets)\n",
    "\n",
    "        return logits, loss\n",
    "\n",
    "    def generate(self, idx, max_new_tokens):\n",
    "        # idx is (B, T) array of indices in the current context\n",
    "        for _ in range(max_new_tokens):\n",
    "            # crop idx to the last block_size tokens\n",
    "            idx_cond = idx[:, -block_size:]\n",
    "            # get the predictions\n",
    "            logits, loss = self(idx_cond)\n",
    "            # focus only on the last time step\n",
    "            logits = logits[:, -1, :] # becomes (B, C)\n",
    "            # apply softmax to get probabilities\n",
    "            probs = F.softmax(logits, dim=-1) # (B, C)\n",
    "            # sample from the distribution\n",
    "            idx_next = torch.multinomial(probs, num_samples=1) # (B, 1)\n",
    "            # append sampled index to the running sequence\n",
    "            idx = torch.cat((idx, idx_next), dim=1) # (B, T+1)\n",
    "        return idx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "85eea01c-87e8-4239-88fd-19d21328da64",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.212309 M parameters\n",
      "step 0: train loss 4.6285, val loss 4.6339\n",
      "step 100: train loss 2.7345, val loss 3.1479\n",
      "step 200: train loss 2.5565, val loss 2.9297\n",
      "step 300: train loss 2.4331, val loss 2.8158\n",
      "step 400: train loss 2.3015, val loss 2.7195\n",
      "step 500: train loss 2.1841, val loss 2.6871\n",
      "step 600: train loss 2.0319, val loss 2.6275\n",
      "step 700: train loss 1.8879, val loss 2.5053\n",
      "step 800: train loss 1.7786, val loss 2.4561\n",
      "step 900: train loss 1.6521, val loss 2.4167\n",
      "step 1000: train loss 1.5712, val loss 2.3612\n",
      "step 1100: train loss 1.4920, val loss 2.2799\n",
      "step 1200: train loss 1.4228, val loss 2.3346\n",
      "step 1300: train loss 1.3618, val loss 2.2691\n",
      "step 1400: train loss 1.2983, val loss 2.2332\n",
      "step 1500: train loss 1.2414, val loss 2.2970\n",
      "step 1600: train loss 1.2079, val loss 2.2139\n",
      "step 1700: train loss 1.1548, val loss 2.2378\n",
      "step 1800: train loss 1.1066, val loss 2.2181\n",
      "step 1900: train loss 1.0783, val loss 2.2533\n",
      "step 2000: train loss 1.0534, val loss 2.3094\n",
      "step 2100: train loss 1.0008, val loss 2.2286\n",
      "step 2200: train loss 0.9817, val loss 2.3153\n",
      "step 2300: train loss 0.9511, val loss 2.2896\n",
      "step 2400: train loss 0.9109, val loss 2.3006\n",
      "step 2500: train loss 0.8868, val loss 2.3138\n",
      "step 2600: train loss 0.8695, val loss 2.3522\n",
      "step 2700: train loss 0.8556, val loss 2.3910\n",
      "step 2800: train loss 0.8395, val loss 2.3653\n",
      "step 2900: train loss 0.7973, val loss 2.3615\n",
      "step 3000: train loss 0.7912, val loss 2.5072\n",
      "step 3100: train loss 0.7618, val loss 2.4841\n",
      "step 3200: train loss 0.7458, val loss 2.4557\n",
      "step 3300: train loss 0.7341, val loss 2.5029\n",
      "step 3400: train loss 0.7100, val loss 2.5292\n",
      "step 3500: train loss 0.7031, val loss 2.5565\n",
      "step 3600: train loss 0.6976, val loss 2.5529\n",
      "step 3700: train loss 0.6638, val loss 2.5504\n",
      "step 3800: train loss 0.6630, val loss 2.5530\n",
      "step 3900: train loss 0.6513, val loss 2.6048\n",
      "step 4000: train loss 0.6348, val loss 2.5576\n",
      "step 4100: train loss 0.6236, val loss 2.6830\n",
      "step 4200: train loss 0.6296, val loss 2.6694\n",
      "step 4300: train loss 0.6105, val loss 2.6921\n",
      "step 4400: train loss 0.6026, val loss 2.7433\n",
      "step 4500: train loss 0.5995, val loss 2.7657\n",
      "step 4600: train loss 0.5915, val loss 2.7926\n",
      "step 4700: train loss 0.5742, val loss 2.7700\n",
      "step 4800: train loss 0.5757, val loss 2.7797\n",
      "step 4900: train loss 0.5679, val loss 2.8359\n",
      "step 4999: train loss 0.5537, val loss 2.8931\n",
      "\n",
      " of a peribronchovascular subpleural).\n",
      "Noneumatoceles and to the curved septal thickencing. This\n",
      "bilateral pleural/fissusea.R– Gill, MD, MPH, Brigham and Women’s Hospital.\n",
      "Case courtesy Ritu R. Gill, MD, MPH, Brigham and Women’s Hospital. Reactivation imaging Joste IIA pneumocytes prior and subpleural/fissuseatorive ssoft-tologic Clinical.Reoly and posterior mediastinal mass Detection of\n",
      "the pleuraNontrast cmetasti ng alpredominant\n",
      "ground glass opaceinblinedle (arrow). The differential would include abtass to no\n",
      "(Left image) shows in the left upper and CT shows evascular opacity (arrow) with pleura subpleural/fistus substaring the Svasquamous cell carcinoma. Nonsoth ciliamsters, patterined fer.\n",
      "Pentrilobular thickening. This paterant, with multiple prior lung pneumonia.\n",
      "Thorace mediastinal mass Din B: Chest radiograph shows multiple lobe.\n",
      "Stage IIILd diate atternal circumferential tuberculosis: Retialteral meterials. Although ite upper lobes. This was a case of vacular septa (arrows) rephonsectivas. Left lower lobe atelectasis.\n",
      "Case courtesy Ritu R. Gill, MD, MPH, Brigham and Women’s\n",
      "Hospital.\n",
      " [END]Amyes form ality/centials threfore losid ex Fleidy, and the wo no the\n",
      "ground glass opacification: Coronal CT shows a smokedevelllasive srespid arch mediastinal mass D]Hemeterites to the airor sulcuides devascular systernomaPosis preddematitic c carcinoma. Nonmucinous BAC is an imfere\n",
      "internodular septal thin-fer.S: Axial CT shows dense bronchiectasive pneumonias with air bronchogronic eosin.\n",
      "Carcinoid C: CT deft pati i s\n",
      "assopyer mainstem bronchus stenosides\n",
      "PA [END]Pulmonary hypertension due far meribution, of rm. aclative paraterial mediastinal mass diffarentially peribronchial and parasej adjacent based squelam and peripheral and subpleural/fissuseatosis.\n",
      "Univain and axial CT demonstrates ggoin: CT deffect lower lobe detectern).\n",
      " [END]Germ cell tumor\n",
      "ground glassSchematic demonstrates for nig, a (arrows). Cases courtesy Ritu R. Gill, MD, MPH, Brigham and Women’s Hospi\n"
     ]
    }
   ],
   "source": [
    "model = BigramLanguageModel()\n",
    "m = model.to(device)\n",
    "# print the number of parameters in the model\n",
    "print(sum(p.numel() for p in m.parameters())/1e6, 'M parameters')\n",
    "\n",
    "# create a PyTorch optimizer\n",
    "optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)\n",
    "\n",
    "for iter in range(max_iters):\n",
    "\n",
    "    # every once in a while evaluate the loss on train and val sets\n",
    "    if iter % eval_interval == 0 or iter == max_iters - 1:\n",
    "        losses = estimate_loss()\n",
    "        print(f\"step {iter}: train loss {losses['train']:.4f}, val loss {losses['val']:.4f}\")\n",
    "\n",
    "    # sample a batch of data\n",
    "    xb, yb = get_batch('train')\n",
    "\n",
    "    # evaluate the loss\n",
    "    logits, loss = model(xb, yb)\n",
    "    optimizer.zero_grad(set_to_none=True)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "# generate from the model\n",
    "context = torch.zeros((1, 1), dtype=torch.long, device=device)\n",
    "print(decode(m.generate(context, max_new_tokens=2000)[0].tolist()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "6e9d19eb-4bee-4e9b-8432-a140d345bd57",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_text_from_context(text, max_new_tokens=1000):\n",
    "    context = torch.tensor([encode(text)], dtype=torch.long, device=device)\n",
    "    return decode(m.generate(context, max_new_tokens=max_new_tokens)[0].tolist())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "7f84fbad-81d0-4a99-99a0-9239184e0603",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Focal nodular hyperplasia (FNH) is disorganized liver tissue with no malignant potential. It is primarily seen in asymptomatic women and is not associated with oral contraceptives. [END]Amyomesochs pretection of a midddle mediastinal mass Detected to the esoplaciety B Signs, sho31, evei an of the colung disease.. Rigial. Case courtesy Ritu R. Gill, MD, MPH, Brigham and Women’s Hospital.\n",
      " [END]Pneumatocele, right paired, is an aspergillosis: Axial CT demonstrates for neck shows numeralbascent malignantly peripheral nodules (arrows). This pattermant pulmonary hypertension T3 squamous in th characterized by uperf and sternophology 2, 76-9 (2, [END]IId 1 sighted may be lower lobe atelectasis [END]Pulmonary\n",
      "hypertension C: Axial CT demonjeft spirgial mucinous BAC HSP, sign of ABNoncontrast [END]Pulmonary hypertension stestablitic demonsterstitial diffuse ICC curves an teithe pulmonary hypertension [END]Overview of pulmonary gangrenedly of Virginial adenopathy\n",
      "(yellow arrow) pattern, representing the pleura: CT through the\n",
      "pulmonary nodule (arrow) is predominantly retrophages ogramsh as shows a left piculateral pneumoniaBronchiolitis and to the centermatitis and paras\n"
     ]
    }
   ],
   "source": [
    "test_text = 'Focal nodular hyperplasia (FNH) is disorganized liver tissue with no malignant potential. It is primarily seen in asymptomatic women and is not associated with oral contraceptives.'\n",
    "print(generate_text_from_context(test_text))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4a6b0ac-a11c-4f51-8618-eb5640858445",
   "metadata": {},
   "source": [
    "### Saving Our Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 500,
   "id": "49027262-8f72-4937-9ba7-600731d3132f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import PretrainedConfig\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 501,
   "id": "bef5c89a-5798-4814-a4b9-bf4f05c9bcb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a custom configuration\n",
    "\n",
    "MODEL_FOLDER_PATH = 'config'\n",
    "MODEL_FILE_PATH = \"config/bigram_language_model\"\n",
    "\n",
    "model_config = PretrainedConfig(\n",
    "    vocab_size=vocab_size,\n",
    "    hidden_size=n_embd,\n",
    "    num_hidden_layers=n_layer,\n",
    "    num_attention_heads=n_head,\n",
    "    intermediate_size=4 * n_embd,\n",
    "    hidden_dropout_prob=dropout,\n",
    "    attention_probs_dropout_prob=dropout,\n",
    ")\n",
    "\n",
    "# Save the configuration to a file\n",
    "if not os.path.exists(MODEL_FOLDER_PATH):\n",
    "    os.makedirs(MODEL_FOLDER_PATH)\n",
    "model_config.save_pretrained(MODEL_FILE_PATH)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 502,
   "id": "25a996ea-bd9a-4181-b1be-618e06319ee9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "file_path = f'{MODEL_FILE_PATH}/config.json'\n",
    "\n",
    "with open(file_path, 'r') as json_file:\n",
    "    existing_data = json.load(json_file)\n",
    "\n",
    "new_data = {\n",
    "    \"model_type\": \"gpt2\"\n",
    "}\n",
    "\n",
    "# Update the existing data with the new data\n",
    "existing_data.update(new_data)\n",
    "\n",
    "with open(file_path, 'w') as json_file:\n",
    "    json.dump(existing_data, json_file, indent=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2df6964-97a9-4604-a91c-5b2a8aaa9f80",
   "metadata": {},
   "source": [
    "# Using LLM as a a Text Transformer in a QA model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "daddb734-6708-483a-b48a-deb409ea6753",
   "metadata": {},
   "source": [
    "### Reconciling Custom LLM with Transformers based LLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e3b004aa-875b-4fc1-88fc-d2d4bc30de2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import transformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "2c84a553-3269-4df7-89da-f978426da3a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "from transformers import PreTrainedModel\n",
    "\n",
    "class CustomLLMWrapper(PreTrainedModel):\n",
    "    def __init__(self, config, model):\n",
    "        super(CustomLLMWrapper, self).__init__(config)\n",
    "        self.token_embedding_table = model.token_embedding_table\n",
    "        self.position_embedding_table = model.position_embedding_table\n",
    "        self.blocks = model.blocks\n",
    "        self.ln_f = model.ln_f\n",
    "        self.lm_head = model.lm_head\n",
    "\n",
    "    def forward(self, idx, targets=None):\n",
    "        B, T = idx.shape\n",
    "\n",
    "        # idx and targets are both (B,T) tensor of integers\n",
    "        tok_emb = self.token_embedding_table(idx) # (B,T,C)\n",
    "        pos_emb = self.position_embedding_table(torch.arange(T, device=self.device)) # (T,C)\n",
    "        x = tok_emb + pos_emb # (B,T,C)\n",
    "        x = self.blocks(x) # (B,T,C)\n",
    "        x = self.ln_f(x) # (B,T,C)\n",
    "        logits = self.lm_head(x) # (B,T,vocab_size)\n",
    "\n",
    "        if targets is None:\n",
    "            loss = None\n",
    "        else:\n",
    "            B, T, C = logits.shape\n",
    "            logits = logits.view(B*T, C)\n",
    "            targets = targets.view(B*T)\n",
    "            loss = F.cross_entropy(logits, targets)\n",
    "\n",
    "        return logits, loss\n",
    "\n",
    "    def generate(self, idx, max_new_tokens):\n",
    "        # idx is (B, T) array of indices in the current context\n",
    "        for _ in range(max_new_tokens):\n",
    "            # crop idx to the last block_size tokens\n",
    "            idx_cond = idx[:, -block_size:]\n",
    "            # get the predictions\n",
    "            logits, loss = self.forward(idx_cond)\n",
    "            # focus only on the last time step\n",
    "            logits = logits[:, -1, :] # becomes (B, C)\n",
    "            # apply softmax to get probabilities\n",
    "            probs = F.softmax(logits, dim=-1) # (B, C)\n",
    "            # sample from the distribution\n",
    "            idx_next = torch.multinomial(probs, num_samples=1) # (B, 1)\n",
    "            # append sampled index to the running sequence\n",
    "            idx = torch.cat((idx, idx_next), dim=1) # (B, T+1)\n",
    "        return idx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "69330d93-ffe3-4a4d-9bb5-7d32b5c191e9",
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'm' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[3], line 6\u001b[0m\n\u001b[1;32m      3\u001b[0m config_path \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mconfig/bigram_language_model\u001b[39m\u001b[38;5;124m'\u001b[39m\n\u001b[1;32m      4\u001b[0m loaded_config \u001b[38;5;241m=\u001b[39m AutoConfig\u001b[38;5;241m.\u001b[39mfrom_pretrained(config_path)\n\u001b[0;32m----> 6\u001b[0m custom_llm \u001b[38;5;241m=\u001b[39m CustomLLMWrapper(loaded_config, m)\n",
      "\u001b[0;31mNameError\u001b[0m: name 'm' is not defined"
     ]
    }
   ],
   "source": [
    "# config load\n",
    "from transformers import DistilBertTokenizerFast, AutoConfig\n",
    "config_path = 'config/bigram_language_model'\n",
    "loaded_config = AutoConfig.from_pretrained(config_path)\n",
    "\n",
    "custom_llm = CustomLLMWrapper(loaded_config, m)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 506,
   "id": "27715bd0-b09a-46ac-90d8-08f5a662d9c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_text_from_context(text, max_new_tokens=1000):\n",
    "    context = torch.tensor([encode(text)], dtype=torch.long, device=device)\n",
    "    return decode(custom_llm.generate(context, max_new_tokens=max_new_tokens)[0].tolist())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 507,
   "id": "c71f58b0-29e5-4bbd-9ea2-f42bac212461",
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyError",
     "evalue": "'?'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[507], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m generate_text_from_context(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mWhat is Focal nodular hyperplasia?\u001b[39m\u001b[38;5;124m'\u001b[39m)\n",
      "Cell \u001b[0;32mIn[506], line 2\u001b[0m, in \u001b[0;36mgenerate_text_from_context\u001b[0;34m(text, max_new_tokens)\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mgenerate_text_from_context\u001b[39m(text, max_new_tokens\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m1000\u001b[39m):\n\u001b[0;32m----> 2\u001b[0m     context \u001b[38;5;241m=\u001b[39m torch\u001b[38;5;241m.\u001b[39mtensor([encode(text)], dtype\u001b[38;5;241m=\u001b[39mtorch\u001b[38;5;241m.\u001b[39mlong, device\u001b[38;5;241m=\u001b[39mdevice)\n\u001b[1;32m      3\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m decode(custom_llm\u001b[38;5;241m.\u001b[39mgenerate(context, max_new_tokens\u001b[38;5;241m=\u001b[39mmax_new_tokens)[\u001b[38;5;241m0\u001b[39m]\u001b[38;5;241m.\u001b[39mtolist())\n",
      "Cell \u001b[0;32mIn[484], line 4\u001b[0m, in \u001b[0;36m<lambda>\u001b[0;34m(s)\u001b[0m\n\u001b[1;32m      2\u001b[0m stoi \u001b[38;5;241m=\u001b[39m { ch:i \u001b[38;5;28;01mfor\u001b[39;00m i,ch \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28menumerate\u001b[39m(chars) }\n\u001b[1;32m      3\u001b[0m itos \u001b[38;5;241m=\u001b[39m { i:ch \u001b[38;5;28;01mfor\u001b[39;00m i,ch \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28menumerate\u001b[39m(chars) }\n\u001b[0;32m----> 4\u001b[0m encode \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mlambda\u001b[39;00m s: [stoi[c] \u001b[38;5;28;01mfor\u001b[39;00m c \u001b[38;5;129;01min\u001b[39;00m s] \u001b[38;5;66;03m# encoder: take a string, output a list of integers\u001b[39;00m\n\u001b[1;32m      5\u001b[0m decode \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mlambda\u001b[39;00m l: \u001b[38;5;124m'\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;241m.\u001b[39mjoin([itos[i] \u001b[38;5;28;01mfor\u001b[39;00m i \u001b[38;5;129;01min\u001b[39;00m l])\n",
      "Cell \u001b[0;32mIn[484], line 4\u001b[0m, in \u001b[0;36m<listcomp>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m      2\u001b[0m stoi \u001b[38;5;241m=\u001b[39m { ch:i \u001b[38;5;28;01mfor\u001b[39;00m i,ch \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28menumerate\u001b[39m(chars) }\n\u001b[1;32m      3\u001b[0m itos \u001b[38;5;241m=\u001b[39m { i:ch \u001b[38;5;28;01mfor\u001b[39;00m i,ch \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28menumerate\u001b[39m(chars) }\n\u001b[0;32m----> 4\u001b[0m encode \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mlambda\u001b[39;00m s: [stoi[c] \u001b[38;5;28;01mfor\u001b[39;00m c \u001b[38;5;129;01min\u001b[39;00m s] \u001b[38;5;66;03m# encoder: take a string, output a list of integers\u001b[39;00m\n\u001b[1;32m      5\u001b[0m decode \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mlambda\u001b[39;00m l: \u001b[38;5;124m'\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;241m.\u001b[39mjoin([itos[i] \u001b[38;5;28;01mfor\u001b[39;00m i \u001b[38;5;129;01min\u001b[39;00m l])\n",
      "\u001b[0;31mKeyError\u001b[0m: '?'"
     ]
    }
   ],
   "source": [
    "generate_text_from_context('What is Focal nodular hyperplasia?')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2833dcc2-d177-4954-98cd-894012dacf89",
   "metadata": {},
   "source": [
    "### Using Custom LLM as part of QA Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 395,
   "id": "722345eb-2304-4d2c-af97-492f92851841",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "from transformers import AdamW # AdamW optimizer with weighted decay reduces chance of overfitting\n",
    "from tqdm import tqdm # progress bar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 396,
   "id": "8c4860fd-5583-4732-91cf-15d441c49023",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/andychen2121/anaconda3/lib/python3.11/site-packages/transformers/optimization.py:411: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "model = custom_llm\n",
    "\n",
    "model.to(device)\n",
    "model.train()\n",
    "optim = AdamW(model.parameters(), lr=5e-5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39bc8da7-2bb4-401d-a75a-ac5f040b4fd5",
   "metadata": {},
   "source": [
    "##### TODO: Form a Dataset\n",
    "Format: input id, start position, end position, attention mask (?)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 398,
   "id": "33ce3bb0-b59e-49d7-8124-77692faccd83",
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'train_dataset' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[398], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m train_loader \u001b[38;5;241m=\u001b[39m DataLoader(train_dataset, batch_size\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m16\u001b[39m, shuffle\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m)\n",
      "\u001b[0;31mNameError\u001b[0m: name 'train_dataset' is not defined"
     ]
    }
   ],
   "source": [
    "train_loader = DataLoader(train_dataset, batch_size=16, shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 402,
   "id": "d0b8c9bb-b4df-46d3-8c52-5c3b1b649386",
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'train_loader' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[402], line 7\u001b[0m\n\u001b[1;32m      5\u001b[0m model\u001b[38;5;241m.\u001b[39mtrain()\n\u001b[1;32m      6\u001b[0m \u001b[38;5;66;03m# setup loop (we use tqdm for the progress bar)\u001b[39;00m\n\u001b[0;32m----> 7\u001b[0m loop \u001b[38;5;241m=\u001b[39m tqdm(train_loader, leave\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m)\n\u001b[1;32m      8\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m batch \u001b[38;5;129;01min\u001b[39;00m loop:\n\u001b[1;32m      9\u001b[0m     \u001b[38;5;66;03m# we calculate gradients at the end of loop; previous iterations shouldn't effect\u001b[39;00m\n\u001b[1;32m     10\u001b[0m     optim\u001b[38;5;241m.\u001b[39mzero_grad() \n",
      "\u001b[0;31mNameError\u001b[0m: name 'train_loader' is not defined"
     ]
    }
   ],
   "source": [
    "# NEEDS MODIFICATION - loss and maybe logits can be mapped from custom architecture\n",
    "\n",
    "for epoch in range(3):\n",
    "    # set model to train mode\n",
    "    model.train()\n",
    "    # setup loop (we use tqdm for the progress bar)\n",
    "    loop = tqdm(train_loader, leave=True)\n",
    "    for batch in loop:\n",
    "        # we calculate gradients at the end of loop; previous iterations shouldn't effect\n",
    "        optim.zero_grad() \n",
    "        # pull all the tensor batches required for training\n",
    "        input_ids = batch['input_ids'].to(device)\n",
    "        attention_mask = batch['attention_mask'].to(device)\n",
    "        start_positions = batch['start_positions'].to(device)\n",
    "        end_positions = batch['end_positions'].to(device)\n",
    "        # train model on batch and return outputs (incl. loss)\n",
    "        outputs = model(input_ids, attention_mask=attention_mask,\n",
    "                        start_positions=start_positions,\n",
    "                        end_positions=end_positions)\n",
    "        # extract loss\n",
    "        loss = outputs[0]\n",
    "        # calculate loss for every parameter that needs grad update\n",
    "        loss.backward()\n",
    "        # update parameters\n",
    "        optim.step()\n",
    "        # print relevant info to progress bar\n",
    "        loop.set_description(f'Epoch {epoch}')\n",
    "        loop.set_postfix(loss=loss.item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 400,
   "id": "e149178e-553d-4a1a-ad99-c313267af88f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CustomLLMWrapper(\n",
       "  (token_embedding_table): Embedding(152, 64)\n",
       "  (position_embedding_table): Embedding(32, 64)\n",
       "  (blocks): Sequential(\n",
       "    (0): Block(\n",
       "      (sa): MultiHeadAttention(\n",
       "        (heads): ModuleList(\n",
       "          (0-3): 4 x Head(\n",
       "            (key): Linear(in_features=64, out_features=16, bias=False)\n",
       "            (query): Linear(in_features=64, out_features=16, bias=False)\n",
       "            (value): Linear(in_features=64, out_features=16, bias=False)\n",
       "            (dropout): Dropout(p=0.0, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (proj): Linear(in_features=64, out_features=64, bias=True)\n",
       "        (dropout): Dropout(p=0.0, inplace=False)\n",
       "      )\n",
       "      (ffwd): FeedFoward(\n",
       "        (net): Sequential(\n",
       "          (0): Linear(in_features=64, out_features=256, bias=True)\n",
       "          (1): ReLU()\n",
       "          (2): Linear(in_features=256, out_features=64, bias=True)\n",
       "          (3): Dropout(p=0.0, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (ln1): LayerNorm((64,), eps=1e-05, elementwise_affine=True)\n",
       "      (ln2): LayerNorm((64,), eps=1e-05, elementwise_affine=True)\n",
       "    )\n",
       "    (1): Block(\n",
       "      (sa): MultiHeadAttention(\n",
       "        (heads): ModuleList(\n",
       "          (0-3): 4 x Head(\n",
       "            (key): Linear(in_features=64, out_features=16, bias=False)\n",
       "            (query): Linear(in_features=64, out_features=16, bias=False)\n",
       "            (value): Linear(in_features=64, out_features=16, bias=False)\n",
       "            (dropout): Dropout(p=0.0, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (proj): Linear(in_features=64, out_features=64, bias=True)\n",
       "        (dropout): Dropout(p=0.0, inplace=False)\n",
       "      )\n",
       "      (ffwd): FeedFoward(\n",
       "        (net): Sequential(\n",
       "          (0): Linear(in_features=64, out_features=256, bias=True)\n",
       "          (1): ReLU()\n",
       "          (2): Linear(in_features=256, out_features=64, bias=True)\n",
       "          (3): Dropout(p=0.0, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (ln1): LayerNorm((64,), eps=1e-05, elementwise_affine=True)\n",
       "      (ln2): LayerNorm((64,), eps=1e-05, elementwise_affine=True)\n",
       "    )\n",
       "    (2): Block(\n",
       "      (sa): MultiHeadAttention(\n",
       "        (heads): ModuleList(\n",
       "          (0-3): 4 x Head(\n",
       "            (key): Linear(in_features=64, out_features=16, bias=False)\n",
       "            (query): Linear(in_features=64, out_features=16, bias=False)\n",
       "            (value): Linear(in_features=64, out_features=16, bias=False)\n",
       "            (dropout): Dropout(p=0.0, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (proj): Linear(in_features=64, out_features=64, bias=True)\n",
       "        (dropout): Dropout(p=0.0, inplace=False)\n",
       "      )\n",
       "      (ffwd): FeedFoward(\n",
       "        (net): Sequential(\n",
       "          (0): Linear(in_features=64, out_features=256, bias=True)\n",
       "          (1): ReLU()\n",
       "          (2): Linear(in_features=256, out_features=64, bias=True)\n",
       "          (3): Dropout(p=0.0, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (ln1): LayerNorm((64,), eps=1e-05, elementwise_affine=True)\n",
       "      (ln2): LayerNorm((64,), eps=1e-05, elementwise_affine=True)\n",
       "    )\n",
       "    (3): Block(\n",
       "      (sa): MultiHeadAttention(\n",
       "        (heads): ModuleList(\n",
       "          (0-3): 4 x Head(\n",
       "            (key): Linear(in_features=64, out_features=16, bias=False)\n",
       "            (query): Linear(in_features=64, out_features=16, bias=False)\n",
       "            (value): Linear(in_features=64, out_features=16, bias=False)\n",
       "            (dropout): Dropout(p=0.0, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (proj): Linear(in_features=64, out_features=64, bias=True)\n",
       "        (dropout): Dropout(p=0.0, inplace=False)\n",
       "      )\n",
       "      (ffwd): FeedFoward(\n",
       "        (net): Sequential(\n",
       "          (0): Linear(in_features=64, out_features=256, bias=True)\n",
       "          (1): ReLU()\n",
       "          (2): Linear(in_features=256, out_features=64, bias=True)\n",
       "          (3): Dropout(p=0.0, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (ln1): LayerNorm((64,), eps=1e-05, elementwise_affine=True)\n",
       "      (ln2): LayerNorm((64,), eps=1e-05, elementwise_affine=True)\n",
       "    )\n",
       "  )\n",
       "  (ln_f): LayerNorm((64,), eps=1e-05, elementwise_affine=True)\n",
       "  (lm_head): Linear(in_features=64, out_features=152, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 400,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83c2f43f-0a6f-4c90-8561-803ac6a84cdf",
   "metadata": {},
   "source": [
    "# Next Steps"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ae64df4-3a37-4d91-9fbb-fc893bc8389d",
   "metadata": {},
   "source": [
    "### Priority:\n",
    "- Explore this: https://github.com/RSNA/AI-Deep-Learning-Lab-2022 - data processing\n",
    "- Maybe train on more text-heavy textbook (see above)\n",
    "- Data Cleaning / PDF text ingestion - Check with Max Vogel\n",
    "- Assigning Photos to relevant text for VQA model training\n",
    "- Upgrade Bigram Language Model to State of the Art\n",
    "\n",
    "### Additional Research:\n",
    "- RAG for increased accuracy of text data on large amounts of text knowledge"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54a0a350-a073-478f-b963-1553df9cd8cb",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}