====== Ephys pipeline overview ======
monkeypsych <-> TDT => https://github.com/dagdpz/Phys_preprocessing
===== Folder structure =====
^Inputs                         ^Info^
|Y:\Data\TDTtanks\Monkey_phys| Raw TDT data|
|Y:\Data\Sortcodes\Monkey_phys| sorting related files|
|Y:\Data\Monkey| behavioral data|
|\Dropbox\DAG\phys\Monkey_phys_dpz\Electrode_depths.mat| electrode depths documentation|
|\Dropbox\DAG\phys\Monkey_phys_dpz\Same_cells.mat| same cells across blocks assignment|
^Outputs                         ^Info^
|Y:\Data\Monkey_phys_mat_from_TDT| recorded data, arranged in the same trial structure as the behavioral data|
|Y:\Data\Monkey_phys_combined_monkeypsych_TDT| trial structures with combined behavioral and ephys data|
|Y:\Data\All_phys_preprocessing_log\Monkey_phys| automatically created log files of all preprocessing performed|
|\Dropbox\DAG\phys\Monkey_phys_dpz\plx_files.xls| (for multiple sortings:) used plx file documentation|
|\Dropbox\DAG\phys\Monkey_phys_dpz\Mon_sorted_neurons.xls|block/run/unit electrode locations, IDs, ratings, and procedures information|
===== General Workflow =====

  - Fill in [[ephys_pipeline:5_Meta-info#electrode_depths file|Electrode depths file]] (required as first step for WC pipeline)
  - WC: Create waveclus pre-clustering files (implemented in [[ephys_pipeline:3_phys_gui|phys_gui]])
  - WC: Run waveclus and sort for all channels, don’t forget to save (wave_clus3new3, MATLAB 2014+)
    * Preferentially separate spike shapes in difficult cases
  - Create plxfiles (implemented in [[ephys_pipeline:3_phys_gui|phys_gui]])
  - Check and clean PLX file and **TAKE NOTES** (single/SNR/stability rating)
  - Now we need to assess which units are the same within and across blocks (based on channel, electrode depths, spike shapes and potentially tuning, carefully check notes for this). For that purpose you might want to do the following steps in any order (potentially even going back and forth):
     * Create combined (phys+behavior) files (implemented in [[ephys_pipeline:3_phys_gui|phys_gui]])
     * Assign same cells across blocks in [[ephys_pipeline:5_Meta-info#same_cells file|Same cells file]]
     * Plot single unit tuning assuming cells are unique in every block (implemented in [[ephys_pipeline:3_phys_gui|phys_gui]]) - This is meant to help assessing if two "units" in the same block might actually be the same (if they have very similar tuning properties)
     * Re-sort PLX file (combining same cells sort codes of same block same channel)
  - If you have finalized the sorting, create combined (phys+behavior) files (implemented in [[ephys_pipeline:3_phys_gui|phys_gui]]) (yes, once again, now that you have a different amount of cells)
  - Complete [[ephys_pipeline:5_Meta-info#same_cells file|Same cells file]]
  - Automatically update the automatic_sorting sheet from the [[ephys_pipeline:5_Meta-info#sorted_neurons table|Excel sorting table]] using [[ephys_pipeline:3_phys_gui|phys_gui]]
  - Copy the automatic_soting entries to the final_sorting sheet of the [[ephys_pipeline:5_Meta-info#sorted_neurons table|Excel sorting table]] and complete with manual entries.




===== Associated code =====
For loading and writing plx files we use an external toolbox located in Github External_unmodified/Matlab Offline Files SDK repository 

The remaining code related to phys preprocessing (not including Waveclus and Plexon) can be found on Github Phys_preprocessing repository
  * Data conversion functions 
    - TDTbin2mat_working.m             Debugged and modified version of TDTbin2mat
    - SEV2mat_working.m                Debugged and modified version of SEV2mat
    - PLX2SPK.m                        For reading in PLX files (Stefan Schaffelhofer)
    - SPK2PLX.m                        For writing PLX files (Stefan Schaffelhofer)
    - WC32SPK.m
    - WC32SPK_concatenated.m
    - WC32SPK_directly.m
  * Core functions
    - phys_gui_working.m
    - phys_gui_execute.m
    - TDT_trial_struct.m
    - ph_combine_MP_and_TDT_data
    - DAG_update_sorting_table.m
  * Waveclus pipeline specific
    - DAG_WC3_preprocessing.m
    - DAG_parse_data_tdt.m
    - DAG_SpikefilterChan.m
    - WC32SPK_directly.m
    - ph_readout_broadband_lag.m
  * Plexon pipeline specific
    - DAG_create_PLX.m
    - DAG_update_plx_file_table.m
    - ph_get_new_plx_extension.m
  * Extra standalone functions
    - DAG_derive_TDT_streamer_broadband_lag.m
    - ph_debugging_GUI.m
    - DAG_take_over_sortcode_PLX2PLX.m
  * Only of historical relevance (legacy subfolder)
    - DAG_move_sorting_files.m      Was used to transfer sortcodes to their current destination
    - ph_derive_electrode_depth.m   Was used to create the first electrode_depths file extracting from sorted_neurons table
    - ph_derive_same_cells.m        Was used to create the first same_cells file extracting from sorted_neurons table
    - ph_simulate_history.m         Was used to create the first log files
    - PLX2TDT.m                     Was used to create TDT sortcodes from PLX files
    - TDT2PLX.m                     Was used to create PLX files from TDT sortcodes