todo

---

• remote-h5-file: arrange for benchmarks

• dandi-vis: timing-benchmarks

• Dendro: approve selected jobs

• dendro: list of apps to conveniently add to a compute resource

• dandi-nwb-meta: create index file - do not gzip - load from neurosift

• Neurosift: PSTH button even when units is not in standard place. Pop up with message that they should select a TimeIntervals. Same with TimeAlignedSeries

• Neurosift: Dendro tab

• Dendro: figure out how to support projects with very large number of files/jobs

• Neurosift: Trial Aligned ROI Series: flatironinstitute/neurosift#115

• Neurosift: dandiset selector: flatironinstitute/neurosift#104

• Dendro: don't refresh the whole file view when a job status changes

• Dendro: Private projects - do not show on neurosift

• Dendro compute resource: trigger reload of apps from gui (in case spec.json has changed) - do not crash on fail

• Dendro - quotas for api requests

• Dendro: maintain list of system apps

• Dendro: projects list - make sortable

• Generate composite views for dandi-vis

• raster plot nh5 - include binned data array

• Dendro: users bring their own s3 bucket

• Dendro: pruning intermediate files

• Dendro frontent: plugins - create a plugin interface

• Dendro: compute resource - provide configuration option to determine which jobs require approval

---

• remfile: improve benchmarks

---

• Dendro: in an app, somehow specify what requirements there are for nwb input file

---

• Kachery: revisit docs
• Kachery: remove fallback bucket stuff
  • Need to think about how to do this. There is ~500GB on wasabi. A lot of this is spikeforest stuff. Maybe should be moved to a spikeforest zone? Then how to redirect existing links?

---

• Figure out how to dynamically compose figurl views, sharing a context

---

• Dendro: isomorphic git - handle case where history has changed

• Dendro: isomorphic git - create own cors server

• Improve spike sorting analysis figurl (show snippets)

• Clean up github home page

• Dendro: Compute receptive fields figurl

• Dendro: Set up free default resource with limited cpu/ram/disk for basic tasks

• Audio/video view

• Neurosift zoom improvement suggestion from Cody: flatironinstitute/neurosift#120

---

2024-04-17

• neurosift: change log in app
• neurosift: specifications view
• neurosift: view inheritance from specifications
• neurosift: test page

2024-04-12

• dendro: use fsbucket

2024-04-10

• Resurrect spyndle UI pointing to franklab database
• Dendro: Fix parameter bug
• Dendro: Install predefined apps from dropdown

2024-04-09

• Neurosift: advanced query

• MP4AVCCodec

LINDI...

LINDI...

2024-03-13

• LINDI
• SortingView curation kbd shortcuts

2024-03-13

• LINDI

2024-03-12

• neurosift-kerchunker

2024-03-11

• neurosift-kerchunker

2024-03-08

• try_spikeforest_new (with ab)
• hdmf-zarr, zarr, nwb, kerchunk...

2024-03-07

• fix neurosift psth groupby categories for NaN case
• Create neurosift-dendro-apps
• neurosift: instruct to set dandi api key if unable to load file and suspect embargoed

2024-03-06

• Move neurosift app to neurosift.app
• neurosift: improve annotations login
• Expand neurosift-annotations to have an aggregated database
• neurosift-annotations: rework naming within repos
• neurosift psth: select groupby categories: flatironinstitute/neurosift#133 (comment)

2024-03-04

• measure entropy of ephys recording to get an idea of compression potential

2024-03-01

• neurosift-annotations
• polishing qfc

2024-02-27 - 2024-02-29

• pre-COSYNE Brainhack
• rastervis
• h5_to_zarr_for_nwb
• ophys visualization
• qfc codec

2024-02-26

• dandi-vis: attach processing scripts to dendro project
• dandi-vis: workflow to generate dandiset md files

2024-02-23

• QFC - Quantized Fourier Compression

2024-02-22

• Brainhack: RasterVis slack channel

2024-02-21

• Dendro SI MC pipeline
• figurl/RasterVis: https://figurl.org/f?v=https://figurl-raster-vis.surge.sh/index.html&d=sha1://6960bfacd9d975c8aab448b80713e2e5c440f69e&label=figurl_data.json
• dendro: rename project script

2024-02-20

• dendro processor: Extract frame slice -> NWB
• dendro: skipCloudUpload option

2024-02-19

• dendroextractors
• troubleshooting error on writing binary recording

2024-02-14

• Dendro: projects scripts

2024-02-13

• Dendro: file caching on compute resource
• Dendro: option to store outputs local to compute resource
• Dendro: implementing project scripts

2024-02-12

• si-dendro-apps: working on si_preprocessing_dev
• dendro: get project file uri
• Figure out cloudflare tunnel

2024-02-10

• Neurosift: TimeAlignedSeries

2024-02-09

• Neurosift: dandiset page
• Dendro: changes to api supporting SI json files for processed recordings
• Neurosift: show CHANGES file for dandiset

2024-04-08

• dandi-vis: ecephys_summary processor

2024-02-07

• dendro: spike_sorting_summary (SSS)
• move neurosift source code to fi-sci (this was actually done sometime last week)

2024-02-06

• prog rep
• dandi-vis: spike_sorting_summary
• Dendro: file download button
• dendro si-pipelines: stub_test_duration_sec parameter
• dandi-vis links to neurosift
• neurosift - load stuff from dendro project
• figurl-dandi-vis: raster plot: enforce max vert spacing between rows

2024-02-01

• nh5 js: implement canceler
• dandi-vis
• dendro app: dandi-vis-1
• fi-sci app: figurl-dandi-vis

2024-01-31

• dandiset-vis-generator: 000582
• dandiset-vis-generator: 000784
• NwbSortingExtractor: allow units_path
• NwbSortingExtractor: allow no elec series
• dendro: approve all jobs button
• Create dandiset-vis-generator
• Prepend Project ID on Job ID

2024-01-30

• MountainSort5: created Quip
• Collecting blurbs for p. eval

2024-01-29

• Create FSBucket
• FSBucket: Deployed on popeye

Misc

• Dendro: add project without starting with dandiset
• Dendro: create example analyses
• Dendro: Fix CDK stack for attaching large disk to instance
• Archive old repos

2024-01-26

• spyndle - initial version of spyglass interface
• spyndle - kubernetes cluster
• dandi-nwb-meta - use h5tojson from rly
• h5tojson - use pydantic models

2024-01-22

• dandi-nwb-meta: include namespace (e.g., core)
• dandi-nwb-meta
• Dendro: support folders in gui

2024-01-19

• Dendro: change landing page
• Neurosift - fixed display problem for time intervals
• Dendro - folders as inputs and outputs of jobs

2024-01-18

• Dendro - pipelining of jobs

2024-01-17

• letter for simons neuro collab
• Create cebra dendro app
• Create safepyeval: https://github.com/magland/safepyeval

2024-01-16

• simplypyeval: in js
• Denro: Dropdown for options parameters
• Dendro: import/export job parameters from json file

2024-01-15

• Backup all github repos, create dev repo, transfer kachery-gateway repo to magland
• Dendro: save parameters -- flatironinstitute/dendro-old#115
• Dendro: render figurl in notebook in linked analysis (this was actually completed last week)

2024-01-13

• remfile/pynwb: Respond to Ryan
• remfile/pynwb: created a PR to update remfile verbiage in the docs.
• remfile/pynwb: Created pynwb_streaming_benchmark repo
• Dendro: jobs now require approval

On slack to Luiz: I have implemented and deployed a new 'feature' where you need to manually approve jobs before they will be picked up by the compute resource. Jobs can only be approved by the github owner of the compute resource. It should be evident in the gui how you do the approval (it's just clicking on an "approve" link). We can modify this mechanism in the future, but for now this will allow us to safely open up our aws compute resources to other groups. And it also avoids issues caused by a very large number of jobs accidentally queued by a runaway python script.

2024-01-12

• Create nh5 python package, and release to pypi
• Kachery: update log processing job to include all zones
• Dendro: analysis view: render markdown
• Dendro analysis: create app for dendro job
• Description of Dendro vision (posted on CN slack)
• Dendro: create script to submit dendro jobs to project
• Dendro/figurl: created tuning_curves_2d figurl view that reads nh5

2024-01-11

• Dendro: console output PR - merge and publish to pypi
• Kachery: allow users to create zones (subdirectories of default)
• Dendro analyses: use poetry to manage dependencies for project analyses
• Dendro: Exploring AWS ec2 options

2024-01-10

• Dendro: Demo 2D tuning curve notebooks for dandisets 000128 and 000582
• fi-sci: figurl-tuning-curves -- deployed to surge.sh

2024-01-09

• Kachery: multiple zones in a bucket
• Dendro: on import, add dandiset tag to project ***
• Dendro: recent projects list - update project names
• Dendro: scroll bar for projects list
• Dendro/neurosift: Associate dendro project with dandi asset - reverse lookup
• Dendro: Show current project name in title bar

2024-01-08

• Dendro: Fix CI tests

2024-01-05

• Dendro: fix gpu docker issue flatironinstitute/dendro-old#116
• Dendro: fix console output upload issue (needs to be merged) flatironinstitute/dendro-old#117
• Dendro frontend: support null parameter values for Optional[int] and Optional[float]

2024-01-04

• Dendro: nwb -> spike sorting analysis figurl
• Develop dendro_apps - autodeploy, etc
• Linked analysis: render jupyter notebooks!
• Add allow_float64 to figurl and sortingview to be able to handle very long recordings
• Dendro: linked analysis view

2024-01-03

• Figurl: Support json in d query parameter. Here's an example.
• Export from Neurosift to Dendro

It is now possible to export nwb files from neurosift into Dendro. See the "Export to Dendro" link in the screenshot. The user is redirected to the dendro app and can select whether to import into an existing project or create a new project. This provides a route from dandi to dendro: browse dandi -> export to neurosift -> export to dendro. It opens possibilities for running non-spike-sorting tasks from neurosift, such as computing autocorrelograms or performing downstream analyses.