adds support for configuring commands and channels using YAML #1034

elmjag · 2024-10-02T14:11:09Z

Adds support for configuring commands and channels using YAML. Currently it's possible to configure Tango, exporter and EPICS commands and channels.

This implements roughly the format discussed in: #1023
The actual format implement is documented here: https://mxcubecore--1034.org.readthedocs.build/en/1034/dev/commands_channels.html

This is still work in progress. Still on the todo list:

write some tests
do a bit more "manual" testing

I would like to get some feedback, before spending more time on this.

fabcor-maxiv

Good stuff.

A first review pass, a first batch of comments and improvement suggestions.

fabcor-maxiv · 2024-10-03T07:32:05Z

docs/source/dev/commands_channels.md

+| property  | purpose                           | default             |
+|-----------|-----------------------------------|---------------------|
+| attribute | tango attribute name              | MXCuBE channel name |
+| poll      | polling periodicity, milliseconds | polling is disabled |


changed to polling_period

fabcor-maxiv · 2024-10-03T07:42:52Z

docs/source/dev/commands_channels.md

+
+The mxcubecore provides a hardware object-level abstraction
+for communicating using various flavors of control system protocols.
+A hardware object can utilize instances of `Command` and `Channel` objects.


Link to the documentation for the actual classes, please.
(I always forget how to do it, look up the Myst doc: https://myst-parser.readthedocs.io/en/latest/syntax/cross-referencing.html).

Also isn't it mxcubecore.CommandContainer.ChannelObject and mxcubecore.CommandContainer.CommandObject?

Yes, it's mxcubecore.CommandContainer.ChannelObject and mxcubecore.CommandContainer.ChannelObject. Added links to API docs for these objects.

fabcor-maxiv · 2024-10-03T07:43:50Z

docs/source/dev/commands_channels.md

+for communicating using various flavors of control system protocols.
+A hardware object can utilize instances of `Command` and `Channel` objects.
+These objects provide a uniform API for accessing a specific control system.
+Mxcubecore provides support for using protocols such as _tango_, _EPICS_, _exporter_ and more.


Links, links, links! Documentation should have lots of links. Typically the first occurrence of every term should have a link to its definition.

Suggested change

Mxcubecore provides support for using protocols such as _tango_, _EPICS_, _exporter_ and more.

Mxcubecore provides support for using protocols such as [_tango_](https://pytango.readthedocs.io/), _EPICS_, _exporter_ and more.

https://docs.epics-controls.org/en/latest/ for the official EPICS docs (that is being reworked now as far as I know, but at least it's not the APS website from the 80's anymore 😄)

Added links to Tango and EPICS web sites.

fabcor-maxiv · 2024-10-03T07:45:12Z

docs/source/dev/commands_channels.md

+The general format for specifying `Command` and `Channel` objects is as follows:
+
+```yaml
+class: MegaCommunicator


To keep consistent with the rest of the example:

Suggested change

class: MegaCommunicator

class: <hardware-object-class>

fabcor-maxiv · 2024-10-03T07:46:06Z

docs/source/dev/commands_channels.md

+The format for specifying _tango_ `Command` and `Channel` objects is as follows:
+
+```yaml
+class: TangoCommunicator


Suggested change

class: TangoCommunicator

class: <hardware-object-class>

fabcor-maxiv · 2024-10-03T08:09:27Z

mxcubecore/model/protocols/epics.py

+    """
+    an EPICS channel configuration
+    """


Suggested change

"""

an EPICS channel configuration

"""

"""EPICS channel configuration."""

reluctantly changed!

fabcor-maxiv · 2024-10-03T08:10:13Z

mxcubecore/model/protocols/epics.py

+"""
+Models the `epics` section of YAML hardware configuration file.
+
+Provides an API to read configured EPICS channels.
+"""


Suggested change

"""

Models the `epics` section of YAML hardware configuration file.

Provides an API to read configured EPICS channels.

"""

"""Models for the `epics` section of YAML hardware configuration file.

Provides an API to read configured EPICS channels.

"""

stupid google!

fabcor-maxiv · 2024-10-03T08:10:59Z

mxcubecore/model/protocols/tango.py

+        """
+        get all tango devices specified in this section
+        """


Suggested change

"""

get all tango devices specified in this section

"""

"""Get all Tango devices specified in this section."""

mxcubecore/protocols_config.py

henrique-silva · 2024-10-04T12:55:46Z

docs/source/dev/commands_channels.md

+```yaml
+class: EpicsCommunicator
+epics:
+  "FOO:B:x1.":
+    channels:
+      State:
+        suffix: stat
+      Vol:
+```
+
+We have an `FOO:B:x1.` prefix specified, with two `Channel` objects `State` and `Vol`.
+`State` will use `FOO:B:x1.stat` PV name, specified by section's prefix and the `suffix` configuration property.
+`Vol` will use `FOO:B:x1.Vol` PV name, specified by section's prefix and channels MXCuBE name.


So, I think this might be a bit problematic, because the modeling of PV is usually $(PREFIX):$(DEVICE).$(PROPERTY), so one record - or PV - ( $(PREFIX):$(DEVICE)) will have multiple properties, for example an analog out has, .DESC (description), .VAL (value), .ASLO (slope).
But in this case that you're modelling it is way more likely that this will be separated into multiple PVs, so you'd likely have:

class: EpicsCommunicator epics: "FOO:B:": channels: State: suffix: pv_1.STAT Vol: suffix: volume.VAL

In the end I think the code will handle just fine if you're just appending, but the prefix is only the part before teh last colon (in most naming conventions of EPICS that I've seen)

I am not sure I am understanding you correctly, but would #1023 (comment) answer your question?

Yes, that's pretty much it.

It's just the non-obvious separation between PV/record and properties in EPICS that is a bit weird for modelling, but the simple concatenation should handle all cases

@elmjag I think it is worth modifying the epics example to show a more complex and complete example, that shows how to combine all those things.

Will this example make things more clear?

class: EpicsCommunicator epics: "FOO:B:": channels: State: suffix: pv_1.STAT Vol: suffix: volume.VAL Freq:

Regarding Freq channel, I want to include an example channel, without the optional config property suffix.

Yeah, I think it's clearer, but take my opinion with a grain of salt since I don't really know much about MxCube 😅

But yes, it does show that you can map PVs and fields to the mxcube channels

updated example

fabcor-maxiv · 2024-10-15T11:08:27Z

mxcubecore/model/protocols/epics.py

+
+    channels: Optional[Dict[str, Optional[Channel]]]
+
+    def get_channels(self) -> Iterable[Tuple[str, Channel]]:


I think it should be Iterator and not Iterable.

Suggested change

def get_channels(self) -> Iterable[Tuple[str, Channel]]:

def get_channels(self) -> Iterator[Tuple[str, Channel]]:

It is valid, generators are Iterables:

>>> from typing import Iterable >>> def foo(): ... yield 1 ... yield 3 >>> isinstance(foo(), Iterable) True

Here I want to specify a more generic return type. It is some kind of iterable. Today it's generator, but in the future perhaps it is a list.

I see, I guess it is fine... I do not know for sure.

The rules I had read some time ago were like this:

When you produce you should use as precise a type you can.

When you consume you should use as generic a type as you can that still fulfils your usage.

so that is what I always follow and I never had any issue.

So for example, I would do something like this:

def producer() -> list[int]: return [1, 2, 3] def consumer(things: typing.Iterable[int]) -> None for thing in things: print(thing) some_things = producer() consumer(some_things)

If later the producer function is refactored into a generator (or iterator or any other iterable), I still do not need to change anything in consumer.

A question purely for my education (you guys have studied more than I have). Reading you example one might be tempted to think that producer promises to return a list. Would it be OK to write code like

list2 = list1 + producer()

Or should one instead write

list2 = list1 + list(producer())

in case producer was later refactored to return something else? Or is this principle only valid for certain contexts?

(I know you could write something like

list2 = []; list2.extend(list1); list2.extend(producer())

but the question is whether you need to assume that the return type of producer() might be(come) any iterable at some later date.

If you have

def producer() -> list[int]: return [1, 2, 3]

Then, it explicitly promises to return a list. Thus you can use return value as a list.

However, if you have:

def producer() -> Iterable[int]: return [1, 2, 3]

Then it does not promise a list. This can any type of iterable value. Then you should explicitly convert this to a list, if you a doing list operations on it.

Here is documentation on Iterable type annotation: https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable

However, bear in mind that none of this is enforced during runtime. Nothing in python stops you from doing this:

def producer() -> Iterable[int]: return False

Python will happily execute this and return a boolean to you.

The only way to find you that your code is not following the type annotations is to run a static checker tool, such as mypy.

@elmjag Thanks, that makes sense. I know that Python does not respect type annotations, but I want to be a good boy and follow our rules, so I was trying to make sure I understood what our rules were.

I suggest that we try out pyright: https://github.com/microsoft/pyright

I suggest that we try out pyright: https://github.com/microsoft/pyright

Yes, I agree. I wrote a new issue: #1053

This way, it's possible to link to generated API documentation for CommandContainer and ChannelObject classes. Without this change, sphix refuses to create a link for CommandContainer and ChannelObject classes, from other section of the documentation.

Adds a section the describes how to configure hardware object's commands and channels using YAML configuration files. Documents the format for Tango, exporter and EPICS protocols.

Adds support for 'tango', 'exporter' and 'epics' sections to YAML configuration files. These section are used to configure Command and Channel objects for hardware objects.

elmjag added the YAML Work on supporting YAML configuration files. label Oct 2, 2024

elmjag mentioned this pull request Oct 2, 2024

channel and commands in YAML files #1023

Open

fabcor-maxiv reviewed Oct 3, 2024

View reviewed changes

henrique-silva reviewed Oct 4, 2024

View reviewed changes

elmjag force-pushed the yaml-commands-channels branch from a7bf566 to 617b67a Compare October 14, 2024 08:48

elmjag requested a review from marcus-oscarsson October 14, 2024 08:48

elmjag force-pushed the yaml-commands-channels branch 2 times, most recently from 138f62e to 0becc48 Compare October 15, 2024 11:00

fabcor-maxiv reviewed Oct 15, 2024

View reviewed changes

elmjag added 3 commits October 21, 2024 17:01

add docs on commands and channels YAML configuration format

82006ef

Adds a section the describes how to configure hardware object's commands and channels using YAML configuration files. Documents the format for Tango, exporter and EPICS protocols.

adds support for configuring commands and channels using YAML

e7ba036

Adds support for 'tango', 'exporter' and 'epics' sections to YAML configuration files. These section are used to configure Command and Channel objects for hardware objects.

elmjag force-pushed the yaml-commands-channels branch from 0becc48 to e7ba036 Compare October 21, 2024 15:01

elmjag merged commit e7ba036 into mxcube:xml_yaml_conversion Oct 21, 2024
9 checks passed

elmjag deleted the yaml-commands-channels branch October 21, 2024 15:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

adds support for configuring commands and channels using YAML #1034

adds support for configuring commands and channels using YAML #1034

elmjag commented Oct 2, 2024 •

edited

Loading

fabcor-maxiv left a comment

fabcor-maxiv Oct 3, 2024

elmjag Oct 14, 2024

fabcor-maxiv Oct 3, 2024

elmjag Oct 14, 2024

fabcor-maxiv Oct 3, 2024

henrique-silva Oct 4, 2024

elmjag Oct 14, 2024

fabcor-maxiv Oct 3, 2024

elmjag Oct 14, 2024

fabcor-maxiv Oct 3, 2024

elmjag Oct 14, 2024

fabcor-maxiv Oct 3, 2024

elmjag Oct 14, 2024

fabcor-maxiv Oct 3, 2024

elmjag Oct 14, 2024

fabcor-maxiv Oct 3, 2024

elmjag Oct 14, 2024

henrique-silva Oct 4, 2024 •

edited

Loading

fabcor-maxiv Oct 4, 2024

henrique-silva Oct 4, 2024

fabcor-maxiv Oct 4, 2024

elmjag Oct 14, 2024

henrique-silva Oct 15, 2024

elmjag Oct 15, 2024

fabcor-maxiv Oct 15, 2024

elmjag Oct 15, 2024

fabcor-maxiv Oct 15, 2024

rhfogh Oct 15, 2024

elmjag Oct 16, 2024

rhfogh Oct 16, 2024

marcus-oscarsson Oct 22, 2024

elmjag Oct 22, 2024

	\| poll \| polling periodicity, milliseconds \| polling is disabled \|
	\| polling_period \| polling periodicity, milliseconds \| polling is disabled \|

	Mxcubecore provides support for using protocols such as _tango_, _EPICS_, _exporter_ and more.
	Mxcubecore provides support for using protocols such as [_tango_](https://pytango.readthedocs.io/), _EPICS_, _exporter_ and more.


		channels: Optional[Dict[str, Optional[Channel]]]

		def get_channels(self) -> Iterable[Tuple[str, Channel]]:

	def get_channels(self) -> Iterable[Tuple[str, Channel]]:
	def get_channels(self) -> Iterator[Tuple[str, Channel]]:

adds support for configuring commands and channels using YAML #1034

adds support for configuring commands and channels using YAML #1034

Conversation

elmjag commented Oct 2, 2024 • edited Loading

fabcor-maxiv left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

henrique-silva Oct 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elmjag commented Oct 2, 2024 •

edited

Loading

henrique-silva Oct 4, 2024 •

edited

Loading