libyang wildcard support

[breifman]

I am using libyang for validating gNMI xpaths for OpenConfig and proprietary YANG models. We have requirements to support wildcard patterns as described in
https://github.com/openconfig/reference/blob/master/rpc/gnmi/gnmi-path-conventions.md#wildcards-in-paths

To summarize

1. Wildcards are allowed to indicate all elements at a given subtree in the schema.
2. A single-level wildcard is indicated by specifying the name of a PathElem to be an asterisk.
3. A multi-level wildcard is indicated by specifying the name of a PathElem to be the an ellipsis "....".
4. Wildcards may be used in multiple levels of the path.
5. Wildcards may also appear as list key values.
6. A single path element, including keyed fields, maybe be replaced by "..." to select all matching descendents.

Examples:

/components/component[name=*]
/components/component[name=*]/optical-channel/state
/components/component[name=*]/.../state
/terminal-device/logical-channels/channel[index=*]/ethernet/state
/terminal-device/logical-channels/channel[index=*]/otn/state
/components/component[name=*]/transceiver/physical-channels[index=*]/state/input-power
/system/logging/remote-servers/remote-server[host=*]/selectors/selector[facility=*][severity=*]/facility
/interfaces///*/state
/interfaces/interface[name=if0-2-1]/.../state

The validation function uses lys_find_path().
struct lysc_node* lys_find_path(const struct ly_ctx *ctx, const struct lysc_node *ctx_node, const char *path, ly_bool output)

I am able to validate paths with keys having wildcard values, i.e. [key=*] by replacing an asterisk with a quoted string '@'. For example, if an input xpath is /terminal-device/logical-channels/channel[index=*]/ethernet/state what is presented to lys_find_path() is /terminal-device/logical-channels/channel[index='@']/ethernet/state.
This works great...I can validate a proper xpath and also detect and reject an ill-formed xpath like /terminal-device/logical-channels/channel[index='@']/state (ethernet container is missing).

What I am unable to do is to validate xpaths in which path elements are asterisks, ellipses or simply omitted as in ///

/interfaces/interface[name=if0-2-1]/*/state
/interfaces/interface[name=if0-2-1]/.../state
/interfaces///*/state

I tried different replacement patterns by following examples on the web in https://gerrit.cesnet.cz/plugins/gitiles/github/CESNET/libyang/+/46180b50f25a408db86aba33701493994c932dc7/tests/api/test_xpath.c but nothing seems to work for me :(

Any advice would be greatly appreciated!
Thanks.

[michalvasko]

What you want to do, in short, is transform those paths into XPath. That should work in the following way:

Predicates such as [name=*] can simply be [name] I think but using an arbitrary string works fine, too.

Skipping path elements /.../ should have its XPath equivalent as //.

Not sure what exactly /*/ is supposed to mean but it is valid in XPath and means a single arbitrary element.

However, lys_find_path() is probably not the function to use because it accepts only path (simplified XPath) and the last 2 cases are not supported for that. You should probably switch to lys_find_xpath() that accepts full XPath (to the extent supported by libyang).

[breifman]

Thank you, Michal. For someone relatively new to libyang, what is the difference between lys_find_xpath() and lyd_find_xpath() or in general lys_ vs. lyd_ functions? The former's description says "Evaluate an xpath expression on schema nodes.", the latter "Search in the given data for instances of nodes matching the provided XPath."

[michalvasko]

In short, lys_find_xpath() evaluates on struct lysc_node structures (schema nodes) and lyd_find_xpath() on struct lyd_node structures. Of course, XPath is defined only for data and so schema evaluation has some restrictions but in practice works quite well.

[breifman]

Hi Michal,

I have wildcards working well, but have problems validating XPaths which have multiple keys. I am using the code that someone else wrote, so I am on the learning curve. Here is in a nutshell what the code does:

Assume input to a validation function is XPath
/components/component[name=XCVR-3-1]/transceiver/physical-channels/channel[index=1]/state

bool validateXPath(std::string& xpath)
{
   struct ly_set                     *lySet = NULL;
   struct lys_node                *lysNode = NULL;
   struct lys_module const  *lysModule = NULL;
   std::string                         qualXpath, firstElem;

    // The following 'for' loop iterates through loaded modules to find the module of the top-level container
   for (uint32_t i = 0; (lys_module = ly_ctx_get_module_iter(ly_ctx, &i)); )
   {
       // Code not shown: firstElem will contain parsed out name of top-level container.

       qualXpath = "/" + std::string(lys_module->name) + ":" + firstElem;
       if ((lySet = ly_ctx_find_path(ly_ctx, qualXpath.c_str())) == NULL)
       {
           continue;
       }
       // Found the module, so from it we capture its node.
       lysNode = lySet->set.s[0];
       ly_set_clean(lySet);
       ly_set_free(lySet);

       // Code not shown: the module name has been added to xpath, so at this point xpath is
       //     /openconfig-platform:components/component[name=XCVR-3-1]/transceiver/physical-channels/channel[index=1]/state.

       // Here we validate the entire qualified xpath using the node of the module and null lysModule. I think documentation
       // says use either module or node.
       if (lySet = lys_find_path(NULL, lysNode, xpath.c_str()))
       {
            ly_set_free(ly_set);
            return true;
       }
   }
   return false;
}

These XPaths work (single key):
/openconfig-platform:components/component[name=XCVR-3-1]/transceiver/physical-channels/channel
/openconfig-platform:components/component[name=*]/transceiver/physical-channels/channel

These do not (multiple keys):
/openconfig-platform:components/component[name=XCVR-3-1]/transceiver/physical-channels/channel[index=1]/state
/openconfig-platform:components/component[name=*]/transceiver/physical-channels/channel[index=*]/state

Realizing that the 2nd list, the channel list, is not the part of the context tree node used by the example code above, my questions are:

1. Is there another way to handle XPath validation in a single function call?
2. If not, how do I traverse nodes of the context tree to match elements of multiple lists? Do I need to divide xpath into two parts, let's say
   xpath1 = /openconfig-platform:components/component[name=XCVR-3-1]/transceiver/physical-channels
xpath2 = channel[index=*]/state
   where xpath2 is a child of xpath1 and then use lys_find_path() with the corresponding node?

Thanks in advance for your help.
Regards,
Boris

[michalvasko]

I do not know why it would not work for multiple keys. But seeing that lys_find_path() returns a set means that you are using libyang v1. And it may not work there, I really do not know anymore. XPath handling was changed to some extent in libyang v2 and I am pretty sure multiple keys should not cause any problems.

[breifman]

Thanks, Michal, for the reply.

I found a workaround. Adding the namespace to a 2nd key name, makes it work...
/openconfig-platform:components/component[name=XCVR-3-1]/transceiver/physical-channels/channel[openconfig-platform-transceiver:index=1]/state

Yes, we're still using libyang v1.x; however, plans are being drawn to move to v2 in foreseeable future.

[michalvasko]

Well, you have not mentioned that some nodes are from a different module. Generally, you can either specify the module explicitly for a node or it is inherited from its parent, which was the wrong module in this case.

[breifman]

Hello Michal,

How do I transform a container-level XPath plus a JSON value into a root-level JSON, so it can be then parsed into LY data schema tree?
E.g.
A SET request is /a/b/c/d/e/f="XYZ"
The client sends the request as:

path : /a/b/c
json_val : {"d": {"e": {"f" : "XYZ" }}}

I want to transform it into

path : /
json_val : {"a": {"b": {"c": {"d": {"e": {"f" : "XYZ" }}}

What would be the most efficient way to do this either in v1 or v2 libyang?

Thanks

[michalvasko]

Use lyd_new_path().