plans to split out the dvid module as a package?

[xiuliren]

thanks for making this great package!

I am very interested in trying the DVID module.
you mentioned splitting the DVID module as a standalone package, do you have any plan to do it?
I can also help if you feel appropriate.

currently, the conda install is pretty heavy and a lot of packages have fixed version and it is hard to integrate with other packages. I would prefer a requirements.txt file and make it a little bit light weight.

[stuarteberg]

Sorry for the delayed response.

you mentioned splitting the DVID module as a standalone package, do you have any plan to do it?

I do want to extract the neuclease.dvid subpackage into its own top-level package at some point. Ideally, I want to make the following changes in the process:

1. Rename neuclease.dvid to pydvid, create a new repo, copy test suite, etc.
2. Move most of neuclease.util into a new repo. (Most of those functions don't belong in pydvid per se, but pydvid needs a number of them.)
3. Switch from using uint64 types for labels to int64. (See issue Stop using uint64 types #3 in this repo.)
4. Upgrade to the pandas 1.x API. (Currently, we're stuck on 0.2x -- I don't remember exactly why.)
5. Drop the dependency on libdvid-cpp. Instead implement labelmap segmentation compression functions in one of the following places:
   • in dvidutils
   • in a new C++ library
   • in pydvid using numba
6. Consider making some dependencies optional, by implementing two conda packages : pydvid and pydvid-base. The pydvid-base package will not install the following dependencies. (s a result some functionality will not be supported, but most users won't notice. (Many of the other dependencies in neuclease will be dropped entirely, since they aren't necessary for any functions in neuclease.dvid.
   • protobuf
   • librdkafka
   • ujson
   • vigra
   • pillow
   • dvidutils (maybe -- depends on how I address the previous point)
7. Make an API enhancement. Right now, all dvid API functions accept a Session object. When dvid eventually supports authentication, I may need to replace that with a custom Client object (which manages the Session internally, along with some other state). I will also need to give some thought to how the new mechanism behaves correctly in a multi-threaded or multi-process context. The details are not yet settled.

currently, the conda install is pretty heavy and a lot of packages have fixed version and it is hard to integrate with other packages.

As mentioned above, I the pydvid package will have fewer dependencies, many of which could be made optional (for users who don't need only basic functions). I suspect the biggest issue you're running into is the out-dated pandas requirement.

I would prefer a requirements.txt file

Currently, a few of the dependencies are conda-only. They are not available on PyPI, so requirements.txt is not an option. If step 5 above is implemented, then perhaps a pip-based installation will be possible. Personally, I don't see an advantage to using pip instead of conda, so it's not something I usually put much effort into.

---

Most of those issues won't take too long to address, except for items 5 and 7. Also, I'm unsure if the test suite is really comprehensive enough to catch mistakes introduced by item 3.

For now, I think the place to start is with items 1, 2, 4, and 6. We'll be stuck with the libdvid-cpp dependency a bit longer (along with all of its upstream dependencies). We can implement items 3, 5, and 7 later. When we do, we'll likely break the API. Oh well, it's probably better to make incremental progress than no progress at all.

Unfortunately, I can't commit to a particular timeline for any of this. But I think by reducing the scope of what needs to be done initially, there's hope that it won't take too long.

I can also help if you feel appropriate.

Thanks. Can you think of any ways to do that? I'm not sure it makes sense at this stage, but I'm open to suggestions.

[xiuliren]

@stuarteberg I just split the code to a new package. It is still not working since it requires dvidutils

The requirement of dvidutils is a bit outdated and is not compatible with my environment.
I would prefer to make it installable using pip.

I can move this repo to your github organization if you prefer.