Differences

This shows you the differences between two versions of the page.

--- ephys_pipeline:1_pipeline_overview [2021/12/16 13:01] – [Associated code] lukas.schneider
+++ ephys_pipeline:1_pipeline_overview [2022/12/29 07:15] (current) – external edit 127.0.0.1
@@ Line 1: / Line 1: @@
 ====== Ephys pipeline overview ======
+monkeypsych <-> TDT => https://github.com/dagdpz/Phys_preprocessing
 ===== Folder structure =====
-  * Y:\Data\TDTtanks\Monkey_phys contains the Raw TDT data
+^Inputs                         ^Info^
-  * Y:\Data\Sortcodes\Monkey_phys contains sorting related files
+|Y:\Data\TDTtanks\Monkey_phys| Raw TDT data|
-  * Y:\Data\Monkey contains the behavioral data
+|Y:\Data\Sortcodes\Monkey_phys| sorting related files|
-  * Y:\Data\Monkey_phys_mat_from_TDT contains recorded data, arranged in the same trial structure as the behavioral data
+|Y:\Data\Monkey| behavioral data|
-  * Y:\Data\Monkey_phys_combined_monkeypsych_TDT contains trial structures with combined behavioral and ephys data
+|\Dropbox\DAG\phys\Monkey_phys_dpz\Electrode_depths.mat| electrode depths documentation|
-  * Y:\Data\All_phys_preprocessing_log\Monkey_phys contains automatically created log files of all preprocessing performed
+|\Dropbox\DAG\phys\Monkey_phys_dpz\Same_cells.mat| same cells across blocks assignment|
-  * \Dropbox\DAG\phys\Monkey_phys_dpz contains
+^Outputs                         ^Info^
-     -Electrode_depths matfile
+|Y:\Data\Monkey_phys_mat_from_TDT| recorded data, arranged in the same trial structure as the behavioral data|
-     -Same_cells matfile
+|Y:\Data\Monkey_phys_combined_monkeypsych_TDT| trial structures with combined behavioral and ephys data|
-     -plx files excel table
+|Y:\Data\All_phys_preprocessing_log\Monkey_phys| automatically created log files of all preprocessing performed|
-     -Sorted_neurons excel table
+|\Dropbox\DAG\phys\Monkey_phys_dpz\plx_files.xls| (for multiple sortings:) used plx file documentation|
+|\Dropbox\DAG\phys\Monkey_phys_dpz\Mon_sorted_neurons.xls|block/run/unit electrode locations, IDs, ratings, and procedures information|
 ===== General Workflow =====
-  * Update Electrode depths matfile. Only block/channel combinations defined here will be processed (?)
-  * (Optional - Synapse only) Reformat Synapse tank folder format to match previous conventions
-  * (Optional - WC only) Run waveclus automatic sorting (Channels with electrodes in the same depth across blocks will be combined across blocks)
-  * (Optional - WC only) manually refine WC sorting
-  * (Optional - Plexon only) Create Plexon sortcodes
-  * (Optional - Plexon only) Manual Plexon sorting
-  * Update plx files excel table
-  * Combine TDT and behavioral data
-    - Create TDT trial structure
-    - Combine
-  * Update Same_cells matfile
-  * Update sorted_neurons excel table
-Most of the steps above are automatized, check out phys_gui page for details.
+  - Fill in [[ephys_pipeline:5_Meta-info#electrode_depths file|Electrode depths file]] (required as first step for WC pipeline)
+  - WC: Create waveclus pre-clustering files (implemented in [[ephys_pipeline:3_phys_gui|phys_gui]])
+  - WC: Run waveclus and sort for all channels, don’t forget to save (wave_clus3new3, MATLAB 2014+)
+    * Preferentially separate spike shapes in difficult cases
+  - Create plxfiles (implemented in [[ephys_pipeline:3_phys_gui|phys_gui]])
+  - Check and clean PLX file and **TAKE NOTES** (single/SNR/stability rating)
+  - Now we need to assess which units are the same within and across blocks (based on channel, electrode depths, spike shapes and potentially tuning, carefully check notes for this). For that purpose you might want to do the following steps in any order (potentially even going back and forth):
+     * Create combined (phys+behavior) files (implemented in [[ephys_pipeline:3_phys_gui|phys_gui]])
+     * Assign same cells across blocks in [[ephys_pipeline:5_Meta-info#same_cells file|Same cells file]]
+     * Plot single unit tuning assuming cells are unique in every block (implemented in [[ephys_pipeline:3_phys_gui|phys_gui]]) - This is meant to help assessing if two "units" in the same block might actually be the same (if they have very similar tuning properties)
+     * Re-sort PLX file (combining same cells sort codes of same block same channel)
+  - If you have finalized the sorting, create combined (phys+behavior) files (implemented in [[ephys_pipeline:3_phys_gui|phys_gui]]) (yes, once again, now that you have a different amount of cells)
+  - Complete [[ephys_pipeline:5_Meta-info#same_cells file|Same cells file]]
+  - Automatically update the automatic_sorting sheet from the [[ephys_pipeline:5_Meta-info#sorted_neurons table|Excel sorting table]] using [[ephys_pipeline:3_phys_gui|phys_gui]]
+  - Copy the automatic_soting entries to the final_sorting sheet of the [[ephys_pipeline:5_Meta-info#sorted_neurons table|Excel sorting table]] and complete with manual entries.
-===== Synchronization =====
-  * TDT data is stored as a continuous data (starting from start of recording)
-  * behavioral data is stored per trial (starting from first trial)
-For combining both, we first convert TDT data to trial format (function TDT_trial_struct.m) preferentially using epocs store (Tnum and SVal) information.
-\\  Note: There is a flag for using the continous state information stream (stream_state_info), but it is permanently turned off, as the epoc information is already extracted from the continuous state information within the TDT circuit.
-\\ Importantly, trial start reference is STATE 2 (fixation acquisition). Trial end is STATE 2 of the next trial.
-\\ Therefore, ITI between trial 1 and 2 as well as STATE 1 (initiation) of trial 2 are found at the end of trial 1.
-\\ This also means that data recorded before 1st trial's state 2 can not be stored in the trial structure and is saved in a separate variable (First_trial_INI) instead.
-===== Associated code =====
-Data conversion functions are located in Github External_modified/PLXTDT repository
-  * PLX2SPK
-  * PLX2TDT
-  * SEV2mat_working
-  * SPK2PLX
-  * TDT2PLX
-  * TDTbin2mat_working
-  * WC32SPK
-  * WC32SPK_concatenated
-  * WC32SPK_directly
+===== Associated code =====
+For loading and writing plx files we use an external toolbox located in Github External_unmodified/Matlab Offline Files SDK repository
 The remaining code related to phys preprocessing (not including Waveclus and Plexon) can be found on Github Phys_preprocessing repository
+  * Data conversion functions
+    - TDTbin2mat_working.m             Debugged and modified version of TDTbin2mat
+    - SEV2mat_working.m                Debugged and modified version of SEV2mat
+    - PLX2SPK.m                        For reading in PLX files (Stefan Schaffelhofer)
+    - SPK2PLX.m                        For writing PLX files (Stefan Schaffelhofer)
+    - WC32SPK.m
+    - WC32SPK_concatenated.m
+    - WC32SPK_directly.m
   * Core functions
     - phys_gui_working.m
@@ Line 73: / Line 67: @@
     - ph_debugging_GUI.m
     - DAG_take_over_sortcode_PLX2PLX.m
-  * Only of historical relevance
+  * Only of historical relevance (legacy subfolder)
     - DAG_move_sorting_files.m      Was used to transfer sortcodes to their current destination
     - ph_derive_electrode_depth.m   Was used to create the first electrode_depths file extracting from sorted_neurons table
     - ph_derive_same_cells.m        Was used to create the first same_cells file extracting from sorted_neurons table
     - ph_simulate_history.m         Was used to create the first log files
+    - PLX2TDT.m                     Was used to create TDT sortcodes from PLX files
+    - TDT2PLX.m                     Was used to create PLX files from TDT sortcodes