notebooks/ephys_element/00-Set-up-configuration.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Configure the `custom` field in `dj.config` for the element-array-ephys\n",
    "\n",
    "The major component of the current workflow is the [DataJoint Array Ephys Element](https://github.com/datajoint/element-array-ephys). Array Ephys Element requires configurations in the field `custom` in `dj.config`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Local configuration file found !!, no need to run the configuration (unless configuration has changed)\n"
     ]
    }
   ],
   "source": [
    "from scripts.conf_file_finding import try_find_conf_file\n",
    "try_find_conf_file()\n",
    "import os\n",
    "import getpass\n",
    "import datajoint as dj\n",
    "import u19_pipeline.utility as utility"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should be able to connect to the database at this stage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Connecting alvaros@datajoint00.pni.princeton.edu:3306\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "DataJoint connection (connected) alvaros@datajoint00.pni.princeton.edu:3306"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dj.conn()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Root directories for raw ephys data and kilosort 2 processed results\n",
    "\n",
    "+ `ephys_root_data_dir` field indicates the root directory for the **ephys raw data** from SpikeGLX or OpenEphys (e.g. `*imec0.ap.bin`, `*imec0.ap.meta`, `*imec0.lf.bin`, `imec0.lf.meta`) or the **clustering results** from kilosort2 (e.g. `spike_times.npy`, `spike_clusters.npy`). The root path typically **do not** contain information of subjects or sessions, all data from subjects/sessions should be subdirectories in the root path."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ In the database, every path for the ephys raw data is **relative to this root path**. The benefit is that the absolute path could be configured for each machine, and when data transfer happens, we just need to change the root directory in the config file.\n",
    "+ The workflow supports **multiple root directories**. If there are multiple possible root directories, specify the `ephys_root_data_dir` as a list.\n",
    "+ The root path(s) are **specific to each machine**, as the name of drive mount could be different for different operating systems or machines.\n",
    "+ In the context of the workflow, all the paths saved into the database or saved in the config file need to be in the **POSIX standards** (Unix/Linux), with `/`. The path conversion for machines of any operating system is taken care of inside the elements.\n",
    "\n",
    "### Examples\n",
    "+ U19 mesoscope directory for Linux systems: `/mnt/Bezos-center/Manuel/neuropixel/manuel`\n",
    "+ U19 mesoscope directory for Mac systems:   `/Volumes/Bezos-center/Manuel/neuropixel/manuel`\n",
    "+ U19 mesoscope directory for Windows systems: `//bucket.pni.princeton.edu/Bezos-center/Manuel/neuropixel/manuel`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If there is only one root path.\n",
    "dj.config['custom']['ephys_root_data_dir'] = utility.get_network_path('Bezos') + '/Manuel/neuropixel/manuel'\n",
    "# If there are multiple possible root paths:\n",
    "#dj.config['custom']['ephys_root_data_dir'] = ['/mnt/bucket/labs/tank']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'database.prefix': 'u19_',\n",
       " 'imaging_root_data_dir': '/Volumes/braininit/RigData/mesoscope/imaging/',\n",
       " 'ephys_root_data_dir': '/Volumes/Bezos-center/Manuel/neuropixel/manuel'}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dj.config['custom']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Save the configuration as a json file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the proper configurations, we could save this as a file, either as a local json file, or a global file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "dj.config.save_local()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Local configuration file is saved as `dj_local_conf.json` in the root directory of this package `U19-pipeline_python`. Next time if you change your directory to `U19-pipeline_python` before importing datajoint and the pipeline packages, the configurations will get properly loaded.\n",
    "\n",
    "If saved globally, there will be a hidden configuration file saved in your root directory. The configuration will be loaded no matter where the directory is."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Next Step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the configuration, we will be able to review the ephys related schemas and tables structure with [01-ephys-workflow-structure](01-ephys-workflow-structure.ipynb)."
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "ad2050938946b9745fdc6507c7561603613194cae6ce583d2036ae212150e70f"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}