Create directory provider that can use the RunEngine's scan number hook

[callumforrester]

This is a proposal/RFC

Replace attach_metadata wrapper with a DirectoryProvider that can be hooked into the RunEngine's scan_number_source. Meaning:

• We control the scan_number in each start document, and can potentially get it from GDA
• The directory provider also shares the scan_number, so each run should match the detector file names
• The logic for getting the scan number into the metadata already exists inside bluesky so we don't need to duplicate it

Still need to figure out if this can be made compatible with the UDC/panda directory providers or if RunEngineCompatibleDirectoryProvider and UpdatingDirectoryProvider would need to coexist.

Corresponding changes to blueapi: DiamondLightSource/blueapi#476

Instructions to reviewer on how to test:

1. Confirm unit tests pass

Checks for reviewer

• Would the PR title make sense to a scientist on a set of release notes
• If a new device has been added does it follow the standards
• If changing the API for a pre-existing device, ensure that any beamlines using this device have updated their Bluesky plans accordingly

[DominicOram]

Not done a very in-depth review but I think the idea is generally good and I think we can probably make use of it for UDC, with some tweaking. Some initial comments:

• Can we have some documentation on how you would expect this to be used? e.g. attaching it to a RE, giving it an initial value, using it as a directory provider for a detector
• The RE doesn't actually have a scan_number_source it has a scan_id_source though. I assume this is what you mean, in which case I would suggest we stick to id throughout? Even though I think I prefer number I value the consistency more

[DominicOram]

@olliesilvester, you've looked at run numbers etc. a bit, any thoughts?

[olliesilvester]

This looks neat, I agree that documentation with examples (maybe in the dodal wiki?) would be useful. Also, I assume run_number and scan_number are interchangeable?

Just out of curiosity, why is it useful for the RunEngine to keep track of this specific parameter? Is there a general guideline of what type of things the RunEngine should know and what can be an outside-parameter?

[callumforrester]

Thanks! I'll neaten this up and add docs etc.

I agree we need to be consistent, will stick to scan_id

@olliesilvester Not sure, here's the relevant docs page: https://blueskyproject.io/bluesky/main/metadata.html
It says "not necessarily unique", but that's with the default function. When making our own, uniqueness is our responsibility.

[callumforrester]

If I'm turning this into a proper PR, I've made an issue to track it: #576

[dperl-dls]

Closing as stale, please feel free to reopen if this is still needed