Use entry points for services and service proxies

[ehpor]

This PR converts service type discovery from explicit paths to an entry-point-based solution.

This has the advantage of not having to import proxies defined outside of catkit2, since they are discovered and installed as plugins at installation time rather than runtime.

Furthermore, this allows us to use separate packages for installation of services and service proxies. That remove dependency of catkit2 on hardware libraries which are unnecessary to run on specific testbeds. Actually splitting out services into separate packages is left for a separate PR.

Finally, it allows us to put services and service proxies at any arbitrary location in the library. So we are free to bunch up cameras into separate folders, and other similar groupings.

This PR supersedes #166.

[ivalaginja]

Hey @ehpor, so what is this supposed to look like once the splitting is done? Will all the entry points be removed from catkit2's setup.py and instead defined in the downstream package's one, but selectively? How is it possible to control the selection of package installations with that?

Other than that this PR looks good, except for the failing CI.

[ehpor]

@ivalaginja Not all entry points will be removed. I think just the ones related to hardware. They will be separate packages in the same repo, with their own requirements in a pyproject.toml file. Since they will be separate packages, you can pick and choose which hardware you want to control on your conda environment.

[ehpor]

Failing CI is due to mismatched API from importlib.metadata vs importlib_metadata packages?! I thought they were strict about their API. I'll have to investigate more.

[ivalaginja]

@ivalaginja Not all entry points will be removed. I think just the ones related to hardware. They will be separate packages in the same repo, with their own requirements in a pyproject.toml file. Since they will be separate packages, you can pick and choose which hardware you want to control on your conda environment.

Ok, I'll wait for an example by you for that 😬 .

[ehpor]

@ivalaginja Example:

File structure:

root
|- catkit2-bmc
|  |- catkit2-bmc
|  |  |- __init__.py
|  |  |- bmc_deformable_mirror_hardware.py
|  |  |- bmc_deformable_mirror_sim.py
|  |  |- bmc_deformable_mirror.py
|  |  |- proxy.py
|  |- pyproject.toml
|- catkit2-aimtti
|  etc...  

pyproject.toml:

[build-system]
requires = ['setuptools']
build-backend = 'setuptools.build_meta'

[project]
name = 'catkit2_bmc'
version = '0.1.0'
description = 'Catkit2 package for controlling Boston Micromachines Corporation devices'
authors = [
    { name="Emiel Por", email="[email protected]" },
    { name="Iva Laginja", email="[email protected]" }
]

[project.entry-points."catkit2.services"]
bmc_deformable_mirror_hardware = "catkit2_bmc.bmc_deformable_mirror_hardware"
bmc_deformable_mirror_sim = "catkit2_bmc.bmc_deformable_mirror_sim"

[project.entry-points."catkit2.proxies"]
some_proxy = "catkit2_bmc.proxy:SomeProxyClass

Installation would be something like:

cd catkit2
conda env create --file environment.yml --name hicat
cd ../hicat-package2
conda activate hicat
conda env update --file environment.yml
cd ../catkit2
python setup.py develop
cd catkit2-bmc
pip install -e .
cd ../catkit2-thorlabs
pip install -e .
etc for all other hardware devices that you have...

[ehpor]

@a-sevin This PR is a first with the ultimate goal of separating dependencies of catkit into separately installable packages. This will allow different testbeds to use different versions of Python and be able to install mutually incompatible SDKs for hardware devices. I'd like your opinion on this.

[ehpor]

@RemiSoummer I'd like your input on this as well, to make sure that this is indeed the direction you want to take.

[ehpor]

One small thing that probably wasn't make clear enough, for the reviewer: this PR requires the use of a Python file to start the service. A C++ service would need to be started by the Python script, if we want one, rather than started directly by the testbed server. I'm guessing that the process priorities and CPU pinning (set by the testbed server after launching) are inherited by the subprocess, but this needs to be checked.

[ehpor]

@ivalaginja See https://github.com/spacetelescope/hicat-package2/pull/673 as an example for what this looks like on a downstream package.

[ehpor]

@a-sevin @RemiSoummer Any comment on this? I was hoping to get this, and two other PRs that depend on this, merged before a talk on catkit2 at the CfAO Fall Retreat in about a week from now.

[ivalaginja]

Installation would be something like:

cd catkit2
conda env create --file environment.yml --name hicat
cd ../hicat-package2
conda activate hicat
conda env update --file environment.yml
cd ../catkit2
python setup.py develop
cd catkit2-bmc
pip install -e .
cd ../catkit2-thorlabs
pip install -e .
etc for all other hardware devices that you have...

Is this something that can still be automated on the downstream repo with conda or would we have to do it differently?

[ivalaginja]

@ehpor this looks good to me, thank you for all the examples! So do I understand correctly that this PR here enables the flexibility on service and proxy discovery, while a future PR will change the catkit2 structure for the possibility to install the different packages selectively?

[RemiSoummer]

This is fine with me. I understand this is a first step, and then we'll be able to filter the packages and install them automatically later. Go ahead with the PR.

[RemiSoummer]

LGTM

[ehpor]

Okay, I'll merge after fixing the conflict and receiving another approval.

Note that this PR breaks all downstream packages (ie. hicat-package2, thd-controls, etc...) until they update their pyproject.toml or setup.py file.

[a-sevin]

It seems great! I will take some time to take a deeper look at the code ASAP

[ehpor]

@ehpor this looks good to me, thank you for all the examples! So do I understand correctly that this PR here enables the flexibility on service and proxy discovery, while a future PR will change the catkit2 structure for the possibility to install the different packages selectively?

Yes. Except for "possibility". After this PR it is already possible to create separate packages that contain services that will be used by catkit2. This PR however does not split up the current catkit2 package into multiple packages, each of which targeted to a particular manufacturer. That is a separate PR with less high priority.

Okay, merging. Note that this breaks compatibility and requires downstream changes to fix. See e.g. https://github.com/spacetelescope/hicat-package2/pull/673.