day1/4 Parsing Metadata from Filenames.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %pip install pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extracting Metadata from strings\n",
    "\n",
    "\n",
    "\n",
    "In neuroscience, we often work with large datasets where file naming conventions encode crucial metadata, helping to find the relavant files for a given analysis. String manipulation--the extraction of structured data from text written in a machine-readable pattern-- makes it possible to extract this information efficiently, streamlining data processing workflows.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Extracting Metadata from Fixed-Length Strings using String Slicing\n",
    "\n",
    "| Code | Description |\n",
    "| :--- | :--- |\n",
    "| **Indexing by Position (i.e. \"Slicing\" a String)** |   |\n",
    "| **`\"BonnKölnAachen\"[:4]`** | Extracts the first four characters 'Bonn' |\n",
    "| **`\"BonnKölnAachen\"[4:8]`** | Extracts the characters from position 4 to 7, resulting in 'Köln' |\n",
    "| **`\"BonnKölnAachen\"[8:]`** | Extracts all characters from position 8 onwards, resulting in 'Aachen' |\n",
    "| **`\"BonnKölnAachen\"[4:6]`** | Extracts the characters from position 4 to 5, resulting in 'Kö' |\n",
    "| **`\"BonnKölnAachen\"[6:8]`** | Extracts the characters from position 6 to 7, resulting in 'ln' |\n",
    "| **`\"BonnKölnAachen\"[:4]`** | Extracts the first four characters, resulting in 'Bonn' |\n",
    "| **`\"BonnKölnAachen\"[-6:]`** | Extracts the last six characters, resulting in 'Aachen' |\n",
    "\n",
    "These examples provide a clear understanding of how to use slicing to extract specific substrings from a larger string based on their positions. This is a powerful tool in string manipulation, often used in data processing and analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercises**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This researcher had a rule for her filenames: she would store session metadata in **fixed-length** strings, with information always in the same order:\n",
    "  - **Subject Name**: 6 Characters\n",
    "  - **Date**: 8 Characters\n",
    "  - **Treatmet Group**: 7 Characters:\n",
    "  - **Session Number**: 5 Characters (\"sess\" and then the number)\n",
    "\n",
    "That way, when she later needed the information, she could extract it from the filename just by slicing it!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Example**: What subject name's data is in this file?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Arthur'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fname = \"Arthur20241008controlsess1.txt\"   # Filename convention: Subject, Date, Group, Session\n",
    "fname[:6]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What group is this subject in?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What Session number was this?  (Note: after extracting the number, turn it from a string into an int with the `int()` function.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extract all four metadata variables from the following file and put them into their own variables (note that the subject has fewer than 6 characters in their name.  After slicing the data, you can replace the underscore characters with \"empty strings\" by using the `replace()` method on strings (e.g. `\"name__\".replace('_', '')`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fname = \"Joe___20241009experimsess1.txt\"  # Filename convention: Subject, Date, Group, Session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make a dictionary with the keys \"Subject\", \"Date\", \"Group\", and \"SessionNum\" with the data from this filename:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fname = \"Arthur20241008controlsess1.txt\"   # Filename convention: Subject, Date, Group, Session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Building a table of metadata usually has the following steps, which can be done in a loop:\n",
    "\n",
    "1. Extract data into a dictionary\n",
    "2. Append the dictionary into a list of dictionaries\n",
    "3. Change the list of dictionaries into a data frame (the table)\n",
    "\n",
    "**Example**: Fill in the missing data extraction code for the filenames below to make a session table.  Include the original filename in its own column, to make finding the file later simpler:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "fnames = [\"a2.txt\", \"b3.txt\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'Letter': 'a', 'Number': 2, 'Filename': 'a2.txt'},\n",
       " {'Letter': 'b', 'Number': 3, 'Filename': 'b3.txt'}]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_sessions = []\n",
    "for fname in fnames:\n",
    "    session = {\n",
    "        \"Letter\": fname[0],\n",
    "        \"Number\": int(fname[1]),\n",
    "        \"Filename\": fname,\n",
    "    }\n",
    "    all_sessions.append(session)\n",
    "\n",
    "all_sessions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Example**: Use the Pandas library to turn this list of dictionaries into a table:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Letter</th>\n",
       "      <th>Number</th>\n",
       "      <th>Filename</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>a</td>\n",
       "      <td>2</td>\n",
       "      <td>a2.txt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>b</td>\n",
       "      <td>3</td>\n",
       "      <td>b3.txt</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Letter  Number Filename\n",
       "0      a       2   a2.txt\n",
       "1      b       3   b3.txt"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "df = pd.DataFrame(all_sessions)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "**Exercise**: Fill in the missing data extraction code for the filenames below to make a session table. Include the original filename in its own column, to make finding the file later simpler:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Arthur20241008controlsess1.txt',\n",
       " 'Joseph20241009controlsess1.txt',\n",
       " 'Arthur20241010treatmesess2.txt',\n",
       " 'Joseph20241011controlsess2.txt']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fnames = [\"Arthur20241008controlsess1.txt\", \"Joseph20241009controlsess1.txt\", \"Arthur20241010treatmesess2.txt\", \"Joseph20241011controlsess2.txt\"]\n",
    "fnames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Subject</th>\n",
       "      <th>Date</th>\n",
       "      <th>Group</th>\n",
       "      <th>SessionNum</th>\n",
       "      <th>Filename</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Arthur</td>\n",
       "      <td>20241008</td>\n",
       "      <td>control</td>\n",
       "      <td>1</td>\n",
       "      <td>Arthur20241008controlsess1.txt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Joseph</td>\n",
       "      <td>20241009</td>\n",
       "      <td>control</td>\n",
       "      <td>1</td>\n",
       "      <td>Joseph20241009controlsess1.txt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Arthur</td>\n",
       "      <td>20241010</td>\n",
       "      <td>treatme</td>\n",
       "      <td>2</td>\n",
       "      <td>Arthur20241010treatmesess2.txt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Joseph</td>\n",
       "      <td>20241011</td>\n",
       "      <td>control</td>\n",
       "      <td>2</td>\n",
       "      <td>Joseph20241011controlsess2.txt</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Subject      Date    Group  SessionNum                        Filename\n",
       "0  Arthur  20241008  control           1  Arthur20241008controlsess1.txt\n",
       "1  Joseph  20241009  control           1  Joseph20241009controlsess1.txt\n",
       "2  Arthur  20241010  treatme           2  Arthur20241010treatmesess2.txt\n",
       "3  Joseph  20241011  control           2  Joseph20241011controlsess2.txt"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Technique: Variable-Length, Character-Seperated Strings (string splitting)\n",
    "\n",
    "In this section, we explore a flexible and practical approach to handling filenames in data management: variable-length, character-separated strings. This method is particularly useful in scenarios where the length of data attributes varies significantly, such as with names of different lengths. By adopting a convention where each piece of metadata in the filename is separated by a specific character (like an underscore \"_\"), researchers can accommodate varying data lengths effortlessly. This technique is common in many fields, including neuroscience, where data files often need to contain detailed, yet neatly organized, metadata.  For example:\n",
    "\n",
    "`<Subject>_<Date>_<SessionCondition>_<SessionNum>.<FileExtension>`\n",
    "\n",
    "The filename convention here uses underscores to separate different data elements and a dot to denote the file extension. For example, a filename like \"Joe_20230101_Control_01.txt\" is easily parsed into its constituent parts: subject name, date, session condition, and session number. You'll learn to use the `split` method in Python, which is a straightforward way to divide a string into a list of substrates based on a specified separator.\n",
    "\n",
    "\n",
    "| Code | Description |\n",
    "| :--- | :--- |\n",
    "| values = \"hello_world\".split('_') | Splits the string \"hello_world\" at underscores, resulting in a list: ['hello', 'world'] |\n",
    "| hello = \"hello_world\".split('_')[0] | Splits \"hello_world\" at underscores and takes the first element, resulting in 'hello' |\n",
    "| world = \"hello world\".split(' ')[1] | Splits \"hello world\" at spaces and takes the second element, resulting in 'world' |\n",
    "| hello, world = \"hello world\".split(' ') | Splits \"hello world\" at spaces and assigns the elements to variables 'hello' and 'world' |\n",
    "| basename, extension = \"filename.txt\".split('.') | Splits \"filename.txt\" at the dot and assigns the elements to 'basename' and 'extension', resulting in 'filename' and 'txt' |\n",
    "| hello, *rest = \"hello dog cat bunny cow\".split(' ') | Splits \"hello dog cat bunny cow\" at spaces, assigns 'hello' to the first variable and the rest of the elements to 'rest' as a list |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercises**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Example**: The filename convention here is `<Subject>_<Date>_<Group>_<SessionNum>.<FileExtension>`.  Extract the date this filename into its own variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'20241008'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fname = \"Arthur_20241008_control_1.txt\"\n",
    "base, ext = fname.split('.')\n",
    "data = base.split('_')\n",
    "date = data[1]\n",
    "date"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extract the Group from this filename into its own variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fname = \"Arthur_20241008_control_1.txt\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extract all the data from this filename into a dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fname = \"Arthur_20241008_control_1.txt\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the filenames below to extract data into a session metadata table in a for-loop (feel free to copy-paste and adjust the solution from the previous section!) Include the original filename in its own column, to make finding the file later simpler:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fnames = [\"Arthur_20241008_control_1.txt\", \"Josephine_20241009_control_1.txt\", \"Arthur_20241010_treatment_2.txt\", \"Joseph_20241011_control_2.txt\"]\n",
    "fnames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Self-Describing Metadata: Getting Key-Values Directly from a String\n",
    "\n",
    "### Searching the String for Patterns using index()\n",
    "\n",
    "In this section, we focus on extracting self-describing metadata from strings using pattern searching, a technique especially useful in scenarios where data is embedded within a string in a predictable manner. This method is crucial when dealing with filenames or text data where specific metadata follows a known pattern or a set keyword. Neuroscience researchers often encounter such situations, for instance, when filenames or data entries include coded information like session numbers or participant IDs embedded within them.\n",
    "\n",
    "Certainly! Here's the completed table with additional examples demonstrating how to use the `index()` method for finding specific patterns in strings and extracting relevant information:\n",
    "\n",
    "| Code | Description |\n",
    "| :--- | :--- |\n",
    "| idx = \"JoeSess1\".index(\"Sess\") | Finds the index of the substring \"Sess\" in the string \"JoeSess1\", storing the position in `idx` |\n",
    "| sessNum = \"JoeSess1\"[idx+4 : idx+5] | Extracts the session number following \"Sess\" by slicing from `idx+4` to `idx+5`, resulting in '1' |\n",
    "| idx = \"Data202302_experiment\".index(\"2023\") | Finds the index of the year \"2023\" in the string, useful for extracting the year data |\n",
    "| year = \"Data202302_experiment\"[idx : idx+4] | Extracts the year \"2023\" from the string by slicing from the found index |\n",
    "| idx = \"experiment_control_groupB\".index(\"group\") | Finds the index of \"group\" in the string, indicating the start of group information |\n",
    "| group = \"experiment_control_groupB\"[idx+5:] | Extracts the group identifier 'B' from the string after \"group\" |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercises**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following Filenames have a different file naming convention:\n",
    "\n",
    "`<SessionID>_<BrainRegion>-d1=<ImageHeightInPixels>,d2=<ImageWidthInPixels>.<FileExtension>`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the index to find the `d1=` section from this filename, extract the image height:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "720"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fname = \"242_CA1-d1=720,d2=1080.tif\"\n",
    "start_idx = fname.index(\"d1=\") + len(\"d1=\")\n",
    "end_idx = fname.index(\",\")\n",
    "height = int(fname[start_idx:end_idx])\n",
    "height"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the index to find the `d2=` section from this filename, extract the image width:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fname = \"2045_CA3-d1=1080,d2=720.tif\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the index to find the `_` section from this filename, extract the brain region:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fname = \"24_DG-d1=720,d2=720.tif\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extract all the data from the following filenames in a loop to build a session table. Include the original filename in its own column, to make finding the file later simpler:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fnames = [\"242_CA1-d1=720,d2=1080.tif\", \"2045_CA3-d1=1080,d2=720.tif\", \"24_DG-d1=720,d2=720.tif\", \"52313_CA1-d1=720,d2=720.tif\", \"4_DG-d1=1080,d2=1080.tif\"]\n",
    "fnames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variable-Length Data on Variable Keys: Using a Double-Seperator to Store Keys Directly in the Filename\n",
    "\n",
    "**Introduction:**\n",
    "Extracting the key-value pairs in a filename can be fully automated when they use a double-separator method. This technique is particularly useful when dealing with variable-length data and keys, a common scenario in scientific data management, including neuroscience research. By embedding key-value pairs in the filename itself, researchers can create self-descriptive files that contain crucial metadata in an organized and accessible format.\n",
    "\n",
    "**`\"sess=232_subj=Bill_grp=Control.txt\"`**\n",
    "\n",
    "In this method, filenames are constructed using two separators: one to separate different metadata elements (e.g., '_') and another to distinguish between keys and their corresponding values (e.g., '='). For example, in the filename above, each underscore separates different metadata items, and the equals sign distinguishes the key (e.g., 'sess', 'subj', 'grp') from its value. \n",
    "\n",
    "Here, we'll practice splitting these filenames to extract each key-value pair and store them in a Python dictionary. This practice is invaluable for organizing data in a way that is both human-readable and easily parsed programmatically, streamlining data analysis and retrieval.\n",
    "\n",
    "**Reference Table:**\n",
    "\n",
    "| Code | Description |\n",
    "| :--- | :--- |\n",
    "| **`base, ext = fname.split('.')`** | Splits the filename at the dot to separate the base name from the file extension |\n",
    "| **`for item in items:`** | start a for-loop, iterating over each item in a sequence. |\n",
    "| **`data = {}`** | Initializes an empty dictionary to store the extracted metadata |\n",
    "| **`data[key] = value`** | Assigns the value to its respective key in the dictionary |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercises**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Example**: Extract all the data from the filename:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "fname = \"sess=232_subj=Bill_grp=Control.txt\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'sess': '232', 'subj': 'Bill', 'grp': 'Control'}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "base, ext = fname.split('.')\n",
    "data = {}\n",
    "for item in base.split('_'):\n",
    "    key, value = item.split('=')\n",
    "    data[key] = value\n",
    "\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extract all the data from the filename"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "fname = \"day-22 clinic-Tuebingen room-3.dat\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extract all the data from the following filenames in a loop to build a session table. Include the original filename in its own column, to make finding the file later simpler:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fnames = [\"sessId-11_height-720_width-1028_region-DG.tif\", \"sessId-13_height-720_width-720.tif\", \"height-720_width-1028_region-DG_sessId-110.tif\", \"height-720_width-1028_region-DG_sessId-110_quality-bad.tif\"]\n",
    "fnames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Extra Demo) Making Data Model Contracts Explicit With Schemas\n",
    "\n",
    "In scientific research, particularly in fields like neuroscience, it's crucial to have a clear understanding of the data structure you're working with. A schema, or a data model contract, serves as a blueprint for the data, outlining its format and the relationships between different data elements. By defining these contracts explicitly, you ensure that your data adheres to a specific structure, which facilitates more efficient and error-free data processing.\n",
    "\n",
    "\n",
    "In this demonstration, we explore the use of Python's built-in `namedtuple` feature from the `collections` module to create explicit schemas. A `namedtuple` allows you to create tuple-like objects that are accessible via named fields, making your code more self-documenting and easy to understand.  \n",
    "\n",
    "The example below shows an example of how this would work:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MetadataModel(subject='Arthur', date='20241008', group='control', sess_num='1')"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import namedtuple\n",
    "\n",
    "# The Schema\n",
    "MetadataModel = namedtuple(\"MetadataModel\", \"subject date group sess_num\")\n",
    "\n",
    "# Extracting the data\n",
    "fname = \"Arthur_20241008_control_1.txt\"\n",
    "base, ext = fname.split('.')\n",
    "\n",
    "# Putting the data into the schema\n",
    "data_tuple = MetadataModel(*base.split('_'))\n",
    "data_tuple\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Named tuples can be converted to dictionaries using the `_asdict()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'subject': 'Arthur', 'date': '20241008', 'group': 'control', 'sess_num': '1'}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_dict = data_tuple._asdict()\n",
    "data_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python comes with several built-in utilities for making th these schemas: below is a reference comparing three of them.  Very handy for writing well-documented code! Each of these is a way to create structured data types, but they have different features and use cases:\n",
    "\n",
    "| Feature/Tool | `collections.namedtuple` | `typing.NamedTuple` | `dataclasses.dataclass` |\n",
    "| :--- | :--- | :--- | :--- |\n",
    "| **Module** | `collections` | `typing` | `dataclasses` |\n",
    "| **Basic Use** | Creates tuple-like objects with named fields | Extends `namedtuple` with type hints | Creates classes with built-in methods for handling data |\n",
    "| **Syntax** | `Point = namedtuple('Point', ['x', 'y'])` | `class Point(NamedTuple): x: int; y: int` | `@dataclass class Point: x: int; y: int` |\n",
    "| **Mutability** | Immutable | Immutable | Mutable by default, can be made immutable |\n",
    "| **Type Annotations** | Not supported natively | Supports type annotations | Supports type annotations |\n",
    "| **Default Values** | Not supported natively | Supports default values | Supports default values |\n",
    "| **Inheritance** | Can't inherit from other classes | Can inherit from other classes | Can inherit from other classes |\n",
    "| **Field Ordering** | Maintains order of fields | Maintains order of fields | Maintains order of fields |\n",
    "| **Methods** | Limited to tuple methods | Can define additional methods | Can define methods, and comes with built-in methods like `__init__`, `__repr__`, etc. |\n",
    "| **Use Case** | Simple use cases where a lightweight, immutable container is needed | When you need immutable containers with type hinting | Ideal for more complex data structures requiring mutability and additional functionality |\n",
    "\n",
    "Each of these tools serves a different purpose:\n",
    "\n",
    "- `collections.namedtuple` is great for when you need a simple, lightweight container with named fields.\n",
    "- `typing.NamedTuple` is useful for a similar purpose but with the added benefit of type hints.\n",
    "- `dataclasses.dataclass` is more suited for complex data structures where you might need mutability, default values, and built-in methods for common tasks.\n",
    "\n",
    "Choosing the right tool depends on your specific needs, especially in terms of complexity, mutability, and the requirement for type hinting."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "duckdb",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}