Support parallel connect

[coretl]

DiamondLightSource/dodal#415 (comment) summarizes a discussion on Device connection.
bluesky/ophyd-async#265 will implement the ophyd-async supporting features of idempotent connect.
DiamondLightSource/dodal#483 will implement introspectable device factories.

The strategy we would like for blueapi is:

• Import the dodal.beamlines.<bl> module
• Call dodal.beamlines.beamline_utils.get_device_factories() to get all the device factories and whether they are lazy or not
• Create devices using all the factories
• For the non-lazy factories kick off connection in a background task (parallel to start off, revisit this if there are performance problems)
• Allow plans to be run
• When a plan is run, call ophyd_async.plan_stubs.ensure_connected(*devices) on all plan args and default args that are devices

Time sensitivity

Only needed when a beamline starts having so many devices that the startup connection time is delayed enough to be problematic
#440 (comment)

[stan-dot]

    def with_dodal_module(self, module: ModuleType, **kwargs) -> None:
        devices, exceptions = make_all_devices(module, **kwargs)

        for device in devices.values():
            self.device(device)

        # If exceptions have occurred, we log them but we do not make blueapi
        # fall over
        if len(exceptions) > 0:
            LOGGER.warning(
                f"{len(exceptions)} exceptions occurred while instantiating devices"
            )
            LOGGER.exception(NotConnected(exceptions))

change the call to make_all_devices inside context.py

[stan-dot]

not sure if we should support the old and new instantiation behaviour or make it behind a flag and go beamline-by-beamline

[callumforrester]

@stan-dot I would favour supporting both, deprecating one and making issues to migrate as-and-when. Migrating beamline-by-beamline is still ideal but not mandated.

[stan-dot]

same

[stan-dot]

@callumforrester now that isolated_device_factory got merged we can presumably resume work on this. At the moment we could only lazy connect for i22, do you have any amendments to the strategy since the issue definition in April?

[callumforrester]

In general I am against lazy connection unless it is needed for performance reasons. If a device isn't available, the earlier you know the better!

[stan-dot]

ok but we still want the parallel connection, right?

[callumforrester]

We do, also isolated_device_factory has now been reverted

[stan-dot]

tracking the location of the desired change

# context.py:115
 def with_dodal_module(self, module: ModuleType, **kwargs) -> None:
        devices, exceptions = make_all_devices(module, **kwargs)

        for device in devices.values():
            self.device(device)

        # If exceptions have occurred, we log them but we do not make blueapi
        # fall over
        if len(exceptions) > 0:
            LOGGER.warning(
                f"{len(exceptions)} exceptions occurred while instantiating devices"
            )
            LOGGER.exception(NotConnected(exceptions))

and after device registration step presumably

[stan-dot]

@callumforrester is the location of change comment correct?

still I assume that to support parallel connect we'll need to wait for the bluesky / dodal / device factory shakeup to be resolved, so some weeks, right?

[callumforrester]

@stan-dot I'm unsure without further investigation. Yes it depends on the factory shakeup. However even then it is not a high priority. The work so far has to make sure we don't design parallel connect out when we need it, which will happen when beamlines start having many devices and slowing down blueapi's startup time. At that point we can prioritise this issue.