Proof-of-concept: Add network support (IPv4Transport)

[hperrey]

This PR adds basic support for a network transport by adding the necessary classes IPv4Transport and IPv4TransportHandle that communicate over socket to the pyseabreeze backend.

It supports multicast to discover devices on the local network. Tests were done using an HDX spectrometer connected via ethernet (direct link).

Please note that this is a best-effort proof-of-concept at this point:

• no few exceptions from the socket library are handled yet
• there is no discovery of devices, only registration of known devices via api.add_ipv4_device_location; this could be replaced by multicast discovery
• the HDX device is hard-coded to IPv4Transport; having both transports there caused issues
• some methods are basically placeholders and still require implementation
• no extensive tests beyond reading intensities has been done yet nor have any unit tests been implemented

Feedback on proper integration into the framework and best-practices welcome!

Sample run:

In [1]: import seabreeze.pyseabreeze as psb
   ...: # specifying a network adapter will enable multicast:
   ...: api = psb.SeaBreezeAPI(network_adapter="192.168.254.200")
   ...: # alternatively: add device and address manually: 
   ...: # api.add_ipv4_device_location("HDX", "192.168.254.254", 57357)
   ...: devices = api.list_devices()
Waiting to receive multicast response(s)

In [2]: devices
Out[2]: [<SeaBreezeDevice QE65000:QEPB01234>, <SeaBreezeDevice HDX:HDX01234>]

In [3]: import seabreeze
   ...: seabreeze.use('pyseabreeze')
   ...: from seabreeze.spectrometers import Spectrometer

In [4]: spec = Spectrometer(device=devices[1])

In [5]: spec.intensities(0,1)
Out[5]: 
array([1978.05266071, 1332.67455752, 1433.36236484, ..., 1415.41637616,
       1417.41034014, 1416.41335706])

In [6]: spec.model
Out[6]: 'HDX'

(note that I replaced the serial in the output above)

Left to be done:

• fix remaining typing errors
• test that both USB and ethernet work with the HDX
• consider to refactorize the packet generation for the multicast request and the model name matching
• implement shutdown and initialize of IPv4Transport (or remove TODO comment)
• multicast feature: investigate why requests for multicast group and port do not work (or remove implementation)

Addresses #240.

[hperrey]

I have been playing around with multicast, both the feature accessible via OBP and as a means to discover connected spectrometers.

With the feature implementation, I ran into unexpected issues: the message types in my version of the "Firmware and Advanced Communications: OCEAN HDX Firmware" for retrieving multicast settings from the device are only triggering a "unknown message type" error. The only command of those working is get_multicast_enable_state.

The discovery via multicast looks promising and I hope to add a commit implementing a first version of that soon.

Then I will be looking into the failing tests and pre-commit CI checks...

[hperrey]

Ok, I believe that I have multicast discovery working now :)

[ap--]

Thank you so much for contributing @hperrey ❤️

This looks great already. I don't have access to an HDX spectrometer (or any ocean optics/insight spectrometer with ethernet support), so I won't be able to test this, but it seems you have the important parts already running 👏 🎉

I'm currently traveling, but will allocate some time tomorrow afternoon to review your PR!

Cheers,
Andreas

[ap--]

This looks really great already. I made several comments.

We'd still need to expose this in seabreeze.spectrometers.list_devices or as a new. I wonder what the nicest interface would be.

[hperrey]

All type errors are now fixed :)

[hperrey]

I have changed how the socket is managed and now the interface works almost as expected:

In [1]: import seabreeze
   ...: seabreeze.use('pyseabreeze')
   ...: import seabreeze.pyseabreeze as psb
   ...: api = psb.SeaBreezeAPI(network_adapter="192.168.254.200")
   ...: from seabreeze.spectrometers import Spectrometer

In [2]: api.list_devices()
Out[2]: [<SeaBreezeDevice QE65000:QEPB0123>, <SeaBreezeDevice HDX:HDX01234>]

In [3]: spec = Spectrometer.from_serial_number("HDX01234")

In [4]: spec.intensities(0,1)
Out[4]: 
array([1970.06733923, 1338.65538922, 1418.40732539, ..., 1409.43453638,
       1409.43453638, 1409.43453638])

The call to api.list_devices() is needed, or from_serial_number() will not see the HDX (see comment above). However, after that, things seem to be working :)

[ap--]

backend config

The way to do the configuration of a backend in the current design would be the following:

python-seabreeze/src/seabreeze/backends.py

Lines 60 to 67 in a8fef98

	if backend == "pyseabreeze":
	if "pyusb_backend" in kwargs:
	pyusb_backend = kwargs.pop("pyusb_backend")
	BackendConfig.api_kwargs["pyusb_backend"] = pyusb_backend
	if kwargs:
	raise TypeError(
	f"unknown keyword arguments {set(kwargs)!r} for backend {backend!r}"
	)

here we would need to add code like:

    if backend == "pyseabreeze":
        if "pyusb_backend" in kwargs:
            pyusb_backend = kwargs.pop("pyusb_backend")
            BackendConfig.api_kwargs["pyusb_backend"] = pyusb_backend
        if "network_adapter" in kwargs:
            network_adapter = kwargs.pop("network_adapter")
            BackendConfig.api_kwargs["network_adapter"] = network_adapter
    ...

Then using the ethernet discovery would look like this in code:

import seabreeze
seabreeze.use("pyseabreeze", network_adapter="192.168.1.200")

from seabreeze.spectrometers import Spectrometer

spec = Spectrometer.from_serial_number("...")

adding lazy backend retrieval

With a small change to seabreeze.spectrometers we should also be able to not have to split the imports:

python-seabreeze/src/seabreeze/spectrometers.py

Lines 22 to 23 in a8fef98

	# get the backend and add some functions/classes to this module
	_lib: SeaBreezeBackend = seabreeze.backends.get_backend()

should be lazy:

# get the backend and add some functions/classes to this module
_lib: SeaBreezeBackend

def __getattr__(name):
    global _lib
    if name == "_lib":
        _lib = seabreeze.backends.get_backend()
        return _lib
    elif name == "SeabreezeDevice":
        ...  # same for this one (right now they are eagerly created)
    elif name == "SeabreezeError":
        ...  # same for this one
    else:
        raise AttributeError(name)

This should allow us to write code like this:

import seabreeze
from seabreeze.spectrometer import Spectrometer

seabreeze.use("pyseabreeze", network_adapter="192.168.1.200")

spec = Spectrometer.from_serial_number("...")

[hperrey]

backend config

The way to do the configuration of a backend in the current design would be the following:

python-seabreeze/src/seabreeze/backends.py

Lines 60 to 67 in a8fef98

	if backend == "pyseabreeze":
	if "pyusb_backend" in kwargs:
	pyusb_backend = kwargs.pop("pyusb_backend")
	BackendConfig.api_kwargs["pyusb_backend"] = pyusb_backend
	if kwargs:
	raise TypeError(
	f"unknown keyword arguments {set(kwargs)!r} for backend {backend!r}"
	)

here we would need to add code like:

    if backend == "pyseabreeze":
        if "pyusb_backend" in kwargs:
            pyusb_backend = kwargs.pop("pyusb_backend")
            BackendConfig.api_kwargs["pyusb_backend"] = pyusb_backend
        if "network_adapter" in kwargs:
            network_adapter = kwargs.pop("network_adapter")
            BackendConfig.api_kwargs["network_adapter"] = network_adapter
    ...

Then using the ethernet discovery would look like this in code:

import seabreeze
seabreeze.use("pyseabreeze", network_adapter="192.168.1.200")

from seabreeze.spectrometers import Spectrometer

spec = Spectrometer.from_serial_number("...")

Yes, this approach works!

In [1]: import seabreeze
   ...: seabreeze.use("pyseabreeze", network_adapter="192.168.254.200")
   ...: 
   ...: from seabreeze.spectrometers import Spectrometer

In [2]: spec = Spectrometer.from_serial_number("HDX01234")

In [3]: spec.intensities(0,1)
Out[3]: 
array([1967.07287765, 1325.69702018, 1418.40732539, ..., 1398.4680335 ,
       1406.44364585, 1397.47109175])

The approach with lazy loading would require to create get_device and get_error similarly to get_backend? The interface does look nicer. However, I don't quite understand that part of the code well enough.

For now, I think I am satisfied with what 0e5a59b brings to the table.

The tests still fail though, mainly due to the multicast feature that unfortunately doesn't even work -- I am not sure whether it is a mistake on my side, or that the specs are not correct for that FW version or something like that. Anyway, that feature is not really necessary, so I could remove everything but the get_multicast_enabled (iirc) command which seem to work.

[hperrey]

I have tried connecting to the HDX using my Ubuntu laptop but ran into #133 .. ethernet, on the other hand, works fine :)

[ap--]

This is great! I am happy to merge this. For now I would remove the extra functionality you added to the Multicast Feature (the 3 methods the tests complain about), to keep feature parity with cseabreeze. (Although it might not really be enabled at all there...) We can add it back later down the line. I will add some text to the readme with a code snippet example.

fyi, the multicast get state / set state feature in cseabreeze uses these two command codes:

python-seabreeze/src/libseabreeze/include/vendors/OceanOptics/protocols/obp/constants/OBPMessageTypes.h

Lines 86 to 87 in a8fef98

	static const unsigned int OBP_GET_GBE_ENABLE_STATE = 0x00000920;
	static const unsigned int OBP_SET_GBE_ENABLE_STATE = 0x00000930;

python-seabreeze/src/libseabreeze/src/vendors/OceanOptics/protocols/obp/exchanges/OBPGetMulticastEnableExchange.cpp

Lines 38 to 44 in a8fef98

	OBPGetMulticastEnableExchange::OBPGetMulticastEnableExchange() {
	this->messageType = OBPMessageTypes::OBP_GET_GBE_ENABLE_STATE;
	this->hints->push_back(new OBPControlHint());
	this->payload.resize(sizeof(unsigned char));
	this->payload[0] = 0; /* default state of device on startup */
	}

python-seabreeze/src/libseabreeze/src/vendors/OceanOptics/protocols/obp/exchanges/OBPSetMulticastEnableExchange.cpp

Lines 39 to 47 in a8fef98

	OBPSetMulticastEnableExchange::OBPSetMulticastEnableExchange()
	{
	this->hints->push_back(new OBPControlHint());
	this->messageType = OBPMessageTypes::OBP_SET_GBE_ENABLE_STATE;
	this->payload.resize(sizeof(unsigned char)+ sizeof(unsigned char)); // two bytes in immediate data
	}

Let me know if you still want to make changes, or if I should take over and merge and prepare a new release!

Cheers,
Andreas 😃

[hperrey]

Great! I'd be more than fine with you going forward with the merge and release -- thank you!

I might revisit the multicast feature, but more importantly I would like to work with the buffered readout that the HDX allows. But that is for another day and PR :)