SpikeInterface fails with spykingcircus2 spike sorting tetrode data

[dervinism]

This is the full error output:

Exception has occurred: SpikeSortingError
Spike sorting error trace:
Traceback (most recent call last):
  File "/home/m329786/anaconda3/envs/si_env/lib/python3.11/site-packages/spikeinterface/sorters/basesorter.py", line 270, in run_from_folder
    SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)
  File "/home/m329786/anaconda3/envs/si_env/lib/python3.11/site-packages/spikeinterface/sorters/internal/spyking_circus2.py", line 199, in _run_from_folder
    prototype, waveforms, _ = get_prototype_and_waveforms_from_recording(
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/m329786/anaconda3/envs/si_env/lib/python3.11/site-packages/spikeinterface/sortingcomponents/tools.py", line 191, in get_prototype_and_waveforms_from_recording
    prototype = np.nanmedian(waveforms[:, :, 0] / (np.abs(waveforms[:, nbefore, 0][:, np.newaxis])), axis=0)
                                                          ~~~~~~~~~^^^^^^^^^^^^^^^
IndexError: index 0 is out of bounds for axis 1 with size 0

Spike sorting failed. You can inspect the runtime trace in /mnt/c/Users/m329786/Data/spikeinterface_output/spykingcircus2/sorter_output/sub-MSEL02698_ses-stim01_task-none_ieeg/spikeinterface_log.json.
  File "/mnt/c/Users/m329786/code_repositories/nwb2spikesort/master_file.py", line 93, in <module>
    folder=si_output_folder_specific, \

                        docker_image=docker_image, verbose=True, \

                        remove_existing_folder=True, **params)

print(sorting)
spikeinterface.sorters.utils.misc.SpikeSortingError: Spike sorting error trace:
Traceback (most recent call last):
  File "/home/m329786/anaconda3/envs/si_env/lib/python3.11/site-packages/spikeinterface/sorters/basesorter.py", line 270, in run_from_folder
    SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)
  File "/home/m329786/anaconda3/envs/si_env/lib/python3.11/site-packages/spikeinterface/sorters/internal/spyking_circus2.py", line 199, in _run_from_folder
    prototype, waveforms, _ = get_prototype_and_waveforms_from_recording(
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/m329786/anaconda3/envs/si_env/lib/python3.11/site-packages/spikeinterface/sortingcomponents/tools.py", line 191, in get_prototype_and_waveforms_from_recording
    prototype = np.nanmedian(waveforms[:, :, 0] / (np.abs(waveforms[:, nbefore, 0][:, np.newaxis])), axis=0)
                                                          ~~~~~~~~~^^^^^^^^^^^^^^^
IndexError: index 0 is out of bounds for axis 1 with size 0

Spike sorting failed. You can inspect the runtime trace in /mnt/c/Users/m329786/Data/spikeinterface_output/spykingcircus2/sorter_output/sub-MSEL02698_ses-stim01_task-none_ieeg/spikeinterface_log.json.

The parameters I am using are given below:

{'apply_motion_correction': (False,),
 'apply_preprocessing': True,
 'cache_preprocessing': {'delete_cache': True,
                         'memory_limit': 0.5,
                         'mode': 'memory'},
 'clustering': {'legacy': True},
 'debug': False,
 'detection': {'detect_threshold': 5, 'peak_sign': 'neg'},
 'filtering': {'filter_order': 2,
               'freq_max': 7000,
               'freq_min': 150,
               'ftype': 'bessel',
               'margin_ms': 10},
 'general': {'ms_after': 0.001, 'ms_before': 0.001, 'radius_um': 70.0},
 'job_kwargs': {'n_jobs': 0.5},
 'matched_filtering': True,
 'matching': {'method': 'circus-omp-svd'},
 'merging': {'max_distance_um': 50},
 'motion_correction': {'preset': 'dredge_fast'},
 'multi_units_only': False,
 'seed': 42,
 'selection': {'method': 'uniform',
               'min_n_peaks': 100000,
               'n_peaks_per_channel': 5000,
               'seed': 42,
               'select_per_channel': False},
 'sparsity': {'amplitude_mode': 'peak_to_peak',
              'method': 'snr',
              'threshold': 1},
 'whitening': {'mode': 'local', 'regularize': False}}

I attach a probegroup to the recording based on the attached kilosort4 probe file.

probe_ks4.json

Do you have any suggestions on how to address this issue? I can provide the binary or NWB file that I use to produce the error.

[zm711]

Just so we have all the info. What version of spikeinterface were you using? @yger works on SC2 so he can comment on that.

[dervinism]

spikeinterface            0.102.2
spikeinterface-gui        0.10.0

[yger]

If I can have the data (rec+probes+anything related that might be necessary) to reproduce the error, I'll fix it. This is strange because this implies that no waveforms are found..

[dervinism]

This is the ks4_probe file
probe_ks4.json

And this is the link to download the recording
https://drive.google.com/file/d/1KQV-RKrONYZJ0WHUaSGJCotSRG7ZhAi4/view?usp=sharing

Let me know if you need anything else.
Thanks

[dervinism]

The data is in int32 format and contains stimulation artifacts.

[yger]

Can I get a probe in the probeinterface file format ? Or alternatively give the script you are using to process the raw recording with spikeinterface

[dervinism]

Here it is

si_probegroup.zip

[dervinism]

I have also tried using a different file and loading from a binary file instead of NWB and I get a very similar output:

Error running spykingcircus2
Traceback (most recent call last):
  File "/mnt/c/Users/m329786/code_repositories/nwb2spikesort/run_spykingcircus2.py", line 115, in <module>
    sorting = ss.run_sorter(sorter_name=sorter, recording=recording, # type: ignore \
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/m329786/anaconda3/envs/si_env/lib/python3.11/site-packages/spikeinterface/sorters/runsorter.py", line 198, in run_sorter
    return run_sorter_local(**common_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/m329786/anaconda3/envs/si_env/lib/python3.11/site-packages/spikeinterface/sorters/runsorter.py", line 260, in run_sorter_local
    SorterClass.run_from_folder(folder, raise_error, verbose)
  File "/home/m329786/anaconda3/envs/si_env/lib/python3.11/site-packages/spikeinterface/sorters/basesorter.py", line 310, in run_from_folder
    raise SpikeSortingError(
spikeinterface.sorters.utils.misc.SpikeSortingError: Spike sorting error trace:
Traceback (most recent call last):
  File "/home/m329786/anaconda3/envs/si_env/lib/python3.11/site-packages/spikeinterface/sorters/basesorter.py", line 270, in run_from_folder
    SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)
  File "/home/m329786/anaconda3/envs/si_env/lib/python3.11/site-packages/spikeinterface/sorters/internal/spyking_circus2.py", line 193, in _run_from_folder
    prototype, waveforms, _ = get_prototype_and_waveforms_from_recording(
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/m329786/anaconda3/envs/si_env/lib/python3.11/site-packages/spikeinterface/sortingcomponents/tools.py", line 191, in get_prototype_and_waveforms_from_recording
    prototype = np.nanmedian(waveforms[:, :, 0] / (np.abs(waveforms[:, nbefore, 0][:, np.newaxis])), axis=0)
                                                          ~~~~~~~~~^^^^^^^^^^^^^^^
IndexError: index 0 is out of bounds for axis 1 with size 0

Spike sorting failed. You can inspect the runtime trace in /mnt/c/Users/m329786/Data/spikeinterface_output/spykingcircus2/sorter_output/msel_02750_ICU02a_micro_data/dump/spikeinterface_log.json.

The link to the binary file is below:
https://drive.google.com/file/d/17tyA0VQNA1gdLC88uNia0PC5wvVNx-9k/view?usp=sharing

[alejoe91]

Let us check the raw data and get back to you @dervinism !

[alejoe91]

Also, can you try to view the data with sw.plot_traces? You can also use the backend="ipywidgets" for an interactive plot in Jupyter. Since many sorters fail, I suspect something is wrong in the data loading/parsing

[yger]

SOrry, I was away last week for conferences. I have your data and the probe, but you just sent me a raw binary. So what are the metadata info (dtype, nchannels I can guess, but also sampling rate) ? Or at least a script to show me how do you load the data to reproduce the bug

[alejoe91]

@yger you can find the probe here: #3909 (comment)

It's KS4 json, but easy to convert. I was able to run SC2 and it found some units, but mainly noisy. @dervinism here's my script:

import spikeinterface.full as si
import json
import numpy as np

rec = si.read_nwb_recording("/home/alessio/Downloads/msel_02698_ICU03.nwb",
                            electrical_series_path="acquisition/TimeSeries_32000_Hz")
rec.reset_times()

# load probe and groups
with open("/home/alessio/Downloads/probe_ks4.json") as f:
    probe_info = json.load(f)

groups = probe_info["kcoords"]

locations = np.zeros((rec.get_num_channels(), 2))
locations[:, 0] = probe_info["xc"]
locations[:, 1] = probe_info["yc"]

rec.set_channel_locations(locations)
rec.set_channel_groups(groups)

# preprocess
rec_pre = si.bandpass_filter(rec)
rec_cmr = si.common_reference(rec_pre)

# run MS5
sort_ms5 = si.run_sorter_by_property("mountainsort5", rec_cmr, grouping_property="group", folder="by_prop_ms5")
print(sort_ms5)

sort_sc2 = si.run_sorter_by_property("spykingcircus2", rec_cmr, grouping_property="group", folder="by_prop_sc2")
print(sort_sc2)

So both sorters run, but they find bad units, probably due to the high contamination of the recording. Did KS4 find any good unit?

[dervinism]

You are running it differently though. What is the difference between run_sorter and run_sorter_by_property?