Integrating Spike Sorted data (kilosort) into NWB format?

[DailyDreaming]

Hi,

Disclaimer: I am new to working with NWB datasets and to DANDI.

I've been tasked with uploading a dandiset for our lab. I've successfully converted our Maxwell dataset files (".raw.h5") to NWB format using NeuroConv (awesome tool!). I was wondering how to include associated spike sorting data (which seem to mostly be ".npy" and ".tsv" files) outputted from kilosort.

Can these derived files be included separately? Or must they be integrated into the NWB file?

I found this example of a data integration: https://github.com/NeurodataWithoutBorders/nwb_hackathons/blob/main/HCK04_2018_Seattle/Projects/Neuropixels/notebooks/Packaging%20Kilosort%20%2B%20phy%20outputs%20using%20pynwb.ipynb

However, it does not use all of the data I've been supplied with:

derived/kilosort2/
├── amplitudes.npy
├── chanMap.mat
├── channel_map.npy
├── channel_positions.npy
├── cluster_Amplitude.tsv
├── cluster_ContamPct.tsv
├── cluster_KSLabel.tsv
├── cluster_group.tsv
├── cluster_info.tsv
├── ops.mat
├── pc_feature_ind.npy
├── pc_features.npy
├── similar_templates.npy
├── spike_clusters.npy
├── spike_templates.npy
├── spike_times.npy
├── template_feature_ind.npy
├── template_features.npy
├── templates.npy
├── templates_ind.npy
├── whitening_mat.npy
└── whitening_mat_inv.npy

Before I go through the labor of going into each file, finding its associated meaning, and then finding and packaging it into an associated Class of Organization within the NWB file, I'd thought I'd ask here if there is an easier way.

Ideally, is there either:

1. A method to package derived files (i.e. kilosort files) as-is with submitted NWB files?
2. OR a conversion tool for kilosort file integration into the NWB format?

Thank you in advance!

[CodyCBakerPhD]

I've successfully converted our Maxwell dataset files (".raw.h5") to NWB format using NeuroConv (awesome tool!)

Glad to hear the MaxOneRecordinglInterface worked for you; out of curiosity, did you have any trouble with the compression algorithm for reading that data or did our auto-installer do the trick?

a conversion tool for kilosort file integration into the NWB format?

NeuroConv also supports Kilsort, which is really just a subset of the Phy format

Check out the gallery for all our spike sorting interfaces: https://neuroconv.readthedocs.io/en/main/conversion_examples_gallery/conversion_example_gallery.html#sorting

As well as tutorials for combining data streams into a single file: https://neuroconv.readthedocs.io/en/main/conversion_examples_gallery/combinations/spikeglx_and_phy.html

Can these derived files be included separately? Or must they be integrated into the NWB file?

As far as whether or not to combine both data streams into the same file, or keep 'raw' and 'processed' separate is an ongoing debate within the DANDI archive community. If your raw files are especially large (~tens to hundreds of GB), it can be more efficient to store them separately; but if it's not too inconvenient to have a single large file, it can be nice to keep all data streams bundled nicely in a single container. Your decision at the end of the day, whichever works best for you

[DailyDreaming]

Glad to hear the MaxOneRecordinglInterface worked for you; out of curiosity, did you have any trouble with the compression algorithm for reading that data or did our auto-installer do the trick?

The auto-installer did the trick and worked as advertised, automagically.

Thank you so much for the links! That's incredibly convenient. Really appreciate it!