-
Notifications
You must be signed in to change notification settings - Fork 0
Home
Welcome to the Two-photon-data-analysis wiki!
This documents describes the analysis pipeline of the data coming from the two photon microscopes in the Murthy lab Some steps are specific to the VR2P (virtual reality two-photon) setup, but most of the pipeline is unified, simply skipping irrelevant steps when used on the 2P upstairs. There are three main steps in the analysis: (1) Organizing folders: creating one sub-folder and one structure per session (2) Manually drawing ROIs (in the case of the VR2P - also ROI in fly head to detect laser-on) (3) Finding which folders for a single experiment still don't have DF/F calculating - creating a list. (4) Using the list of folders missing DF/F - sent to Spock: on worker per session - so number of sessions from the list can run in paeallel. (5) 1-4 is the same for all experiment. From this point (e.g., create tuning curves and other analysis) - write Im2P_Analysis_XXX for each experiment separately - not a part of the general pipeline.
There are two machines in the experimental setup: Imaging computer – stores the Tiff files, one per session. Control software: scanImage (different versions in different machines) Stimulus computer – controls the inputs and outputs via NI-DAQ. Control software prvGreyPlayback (written by Jan Clemens) The raw data at the end of the experiment includes the manually inserted data in an excel file (there is one such file per experiment, for example Z:\Dudi\Imaging\2Photon\Diego_setup\DSX_2P_Playback\DSX_2P_Playback.xlsx), and the following files: tif files created by the imaging computer and XXXbin.mat/XXXvDat.mat/XXXlog.txt files created by the stimulus computer - one set of 3 files per session. More files (.avi and MASK.png) in the VR2P setup;
Organize and prepare the data (some parts are for both setups, others are VR2P only): Move all the files from both the imaging and stimulus computers to a single folder in Bucket at the end of each day. For VR2P [assuming open-loop at this point] – On the stimulus computer, make a MASK for each session, and then track all the relevant videos using FicTrac to get .dat files. Then rename the .dat files and move all to bucket (this part is described in more details later) Run locally the following three steps (same for both setups): Create one folder per relevant session, and move non relevant files to a separate folder called “probablyJunk” Im2P_BatchCreateFolder.m is running over the master folder (one full experiment), and for each folder (single session), runs the function Im2P_CreateFolders_OnePerSession.m Im2P_Redo_CreateFolders_OnePerSession.m - reverse the function Im2P_CreateFolders_OnePerSession.m – removes the one subfolder per session and have all the files from one date in a single folder.
Create sample Tiff image (for both setups) and sample AVI image (only for VR2P) - Im2P_SampleTIFFandAVI_forROI.m is adding two fields to sOneSession: TiffFrame (single imaging frame for the whole session, used later by the user to define ROIs in the imaging data) and AVIFrame (single avi frame from the video - one for the whole session, used later by the user to define ROI between the fly eyes to laser ON/OFF detection, used for synchronization). Define ROI in the imaging data and (in the VR2P only) also in the tracking-video - Im2P_BatchfindROIs.m - is running over the master folder (one full experiment), and for each folder (single session), runs the function Im2P_findROIs.m who prompts the user to define ROIs (a polygon, name and side of ROI).
Up to this point: sOneSession was created for each session in the experiment, ROIs (Tiff and AVI if exist) are defined manually and the ball is tracked (in the VR2P). Each session was moved to a single separate folder, so now all the info for one session including the structure sOneSession are in one folder.
Comments: When there are new sessions, can run the whole pipeline again without overriding sessions that are already pre-analyzed by setting IsOverride=0. Currently missing: get the relevant tracking info from the FicTrac output file .dat and add to sOneSession. Also – run the pre-analysis on Spock. The function Im2P_Batch_OneSession_PreAnalysis.m is running over each folder (session) under the master folder (experiment, e.g., playback) and for each folder calling the function Im2P_OneSession_PreAnalysis.m who is running the pre-analysis pipeline for a single session. Im2P_OneSession_PreAnalysis.m can be called directly if one wants to run only one folder (session) through the pipeline. The function Im2P_OneSession_PreAnalysis.m is running in a pipeline the following functions, adding relevant fields to the structure sOneSession : Im2P_find_LaserONOFF_inAVI.m; %Find laser on/off frame in the movie of fly on ball [relevant only for VR2P, otherwise will skip]
Im2P_sync_samplesTifs.m; %Sample number for each TiffFrame
Im2P_Check_timestamps.m; %Check timestamps and look for jumps in the .avi files [relevant only for VR2P, otherwise will skip]
Im2P_sync_samplesAvi.m; %sample number for each AviFrame [relevant only for VR2P, otherwise will skip]
Im2P_AddStimData.m; %Adds the info regarding stimuli
Im2P_DFF.m %Finds DF/F in each ROI
- Still missing: move the .dat tracking data from FicTrac to sOneSession
At this point the structure sOneSession has all the information needed for analyzing the data. Next steps (‘post-analysis’) are experiment specific. Im2P_ROIs_OneDate.m - ROIs can be added to the structure sOneSession later for one date using, and than need to run In2P_DFF.m again.Im2P_VisualizeSession.m - visualize a single session (all responses color coded, and then can see trial by trial)
In a given day, a given setup there can be multiple sessions. Typically, but not always, there is one (long..) session per fly. At the end of the pre-analysis step, there is one structure that includes all the relevant info about this session. In the NB2P it currently includes:
fieldnames(sOneSession)
'Folder‘ – Full path to the folder with all the session data (one folder per session, with no subfolders in it)
'Info_from_Excell‘ – data that was inserted by the user in the excel table during the session (e.g., fly are, genotype)
'ImMeta‘ – meta data from the Multi-page Tiff file
'TiffFrame‘ – a single Tif frame, used for manual ROI/s segmentation by the user (that will be used for analyzing calcium imaging response)
'ROIname‘ – ROI name chosen by the user when creating the ROI (one name per ROI)
'ROISide‘ - ROI hemisphere as entered by the user when creating the ROI (one value per ROI)
'TiffROI‘ – zeros and ones – defining the ROI (one matrix per ROI; size of matrix X_Pixels * Y_Pixels, e.g., 256 * 256)
'AVIFrame‘ - a single avi frame, used for manual ROI segmentation by the user (used to detect laser ON/OFF in the movie). VR2P only, otherwise empty.
'AVIROI‘ - zeros and ones – defining the AVIROI
'NumberOfTifs‘ – No. of individual tiff files in the multi-page tiff (if two channels, number of tiffs = pages/2). Ignores corrupted (not finished) acquisitions
'LaserOnSample‘ – The sample of two photon laser on
'LaserOffSample‘ – The sample of two
'SamplesStart_ImFrames‘ – 3 columns: sample number at the beginning of each tiff image, multi-page tiff number, number of image file in the multi-tiff file
'SamplesTiffs_fields‘ – Titles of columns in ‘SamplesStart_ImFrames’
'StartStim‘ – The frame of StimOn in for each stimuli (note: one Tiff multi-page file per stimulus!)
'stimAll‘ – The list of stimuli used in the session (each entry is the actual stimulus at 10,000Hz)
'stimOrder‘ – The order of the stimuli as presented in the experiment (just numbers between 1: size(sOneSession.stimAll,2))
'LOGfile‘ – Table class where row j is stim number j presented in the session (e.g., 'pulseTrain_36IPI_100Hz‘ as parsed by the prvPlayback function)
'StimParams‘ – list of parameters for each stim. Parsed from the LOGfile using Im2P_parseParametersFromFileName.m
'GreenSignal‘ – DF/F in the green channel (GCaMP) for each one of the TiffROIs
'RedSignal‘ - DF/F in the red channel (currently TdTom, but could also be red Ca indicator) for each one of the TiffROIs if a red channel exist
First run locally in Matlab the function Im2P_FindFoldersToTrack that will create a list of folders to run the pipe line through on Spock. Please make sure that ROIs are marked before creating the list.
login to spock
if not in home directory:
cd ~
(will change directory to: /usr/people/ddeutsch)
./submitImageFolderJobs.sh /jukebox/murthy/Dudi/Imaging/2Photon/Diego_setup/DSX_2P_Playback 0 0
Explaination:
./submitImageFolderJobs.sh - the function that goes over lines in the list, and makes an array of jobs to run in Matlab. The function that is running in Matlab is sOneSession = Im2P_OneSession_PreAnalysis(Folder, IsOverride, IsOverrideDFF)
Argument 1 - /jukebox/murthy/Dudi/Imaging/2Photon/Diego_setup/DSX_2P_Playback
In this folder there are: (1) subfolders with imaging data (2) A file named FoldersToProcess.txt that has a list of folders to send to Im2P_OneSession_PreAnalysis(Folder, IsOverride, IsOverrideDFF) ** Each worker in the cluster gets one folder from the list, therefore running Im2P_OneSession_PreAnalysis on a single folder
Arguments 2,3 -
IsOverride = 0 IsOverrideDFF = 0
The output flies will be saved in this example in /jukebox/murthy/Dudi/Imaging/2Photon/Diego_setup/DSX_2P_Playback
[ to run MATLAB interactively for debugging proposes: module load matlab/R2016b matlab ]
Add a field StimParams to sOneSession with the parameters for each stimulus (for example inter-pulse-interval of frequency) - Im2P_parseParametersFromFileName.m IntensityIm2P_Analysis_Playback_StimGroups.m - using the parameters defined in StimParams, to divide the trials into grops based on the stimuli for further analysis. For example: Pulse trains with different pulse parameters or Sine with different freq. Im2P_AnalyzePlayback.m - creates the folder res (results) under the master folder. Than creates one structure per group of stimuli with the relevant stim parameter and the calcium response. Im2P_FiguresPlayback.m - create summary analysis figures for the playback experiment
This part is not full yes, but that’s the idea: On the stimulus computer, build a mask per session. Run the relevant files (as defined in the excel table) though FicTrac for analysis in a batch (so will run all the movies from this date). Visualize the tracking and check for skips. Once OK, move all the files from both computers to Bucket, and the rest is the same as for the NB2P.