Implement new JSON Jinja filters

[nmaludy]

This PR implements several new Jinja filters:

• from_json_str Parses a JSON string into a dict/list/string/int/bool, just does json.loads under the hood. This is useful, and really necessary, when storing complex data structures in the datastore. Since everything in the datastore is a string, in order to use the data it needs to be parsed out. Now we can have an expression like: {{ st2kv.system.my_complex_value | from_json_str }} and be able to use the result like a regular object.
• from_yaml_str Parses a YAML string into a dict/list/string/int/bool, just does yaml.safe_load under the hood.
• jsonpath_query Similar to Ansible's json_query filter, except this version uses JSONPath where ansible uses JMESPath. This allows complex data queries in a straight forward syntax that would otherwise require complex and hard to read chains of jinja filters.

TODO

• Port modifications to st2mistral - DONE Implement new JSON Jinja filters st2mistral#33
• Add documentation to st2docs - DONE Implement new JSON Jinja filters st2docs#617
• Migrate from JMESPath to JSONPath (thanks @Kami !)

[Kami]

We already use jsonpath-rw dependency and JSONPath in other places.

I think it would be a good idea to also use that for your new filter, if possible, to avoid additional dependencies in the standard distribution.

I would be fine with this dependency if it was part of a custom filter which is not shipped with the default distribution.

[Kami]

Thanks.

I have one concern about additional dependency, but besides that I don't see any other blockers.

One thing I still don't like is just copy and pasting code over to Mistral. I know we did this in the past, but I already wasn't a big fan of it back then.

Imo, we should split those filters in a common package (e.g. st2_jinja_filters or similar) and then re-use that in both places.

[Kami]

We actually already have pack actions for that, but I guess I still see some value in having those operations available as filters as well.

Having said that, we need to be careful down the road and come up with some rules / conventions on what goes in a filter and what in an action.

[nmaludy]

Makes sense.

My use case for from_json_string is reading values out of the datastore. Commonly in our action metadata files we have default values set to Jinja expressions that read from datastore and contain JSON strings. In order for us to utilize this data effectively we need to parse the JSON out into objects.

Part of this problem stems from the fact that the datastore only stores information in string format, so you're forced to serialize on either end. If typed loading/unloading was supported, then the need for this filter (in our use case) would go away.

[Kami]

On a related note - we actually have action for that which handles serialization / de-serialization (st2kv.get_object st2kv.set_object or something along those lines) and could maybe also work for some of your use cases.

[nmaludy]

@Kami I'm looking into JSONPath, honestly didn't see it when i was coding this. It might be a good alternative since it's already in the distribution. Keeping the dependencies low is always a good thing!

Also, i 100% agree that the right thing to do is have an external st2jinjafilters package (or similar). The only problem is that, currently, the filters in st2mistral have a different signature than the ones in st2. This is due to a bug/implementation-detail in Mistral upstream and closely relates to this bug StackStorm/st2mistral#31 reported by @Mierdin .

[nmaludy]

@Kami i took your advice and ported the code to use JSONPath. Any other comments are appreciated!

One thing to note that's lacking in the implementation of JSONPath we currently have included jsonpath-rw is that it can't do complex 'where' matching. For example, checkout the JSONPath documentation here http://goessner.net/articles/JsonPath/ there is an expression $..book[?(@.price<10)]. Our JSONPath implementation fails on this expression.

The same expression in JMESPath would be store.book[?price < `10`] and actually works according to my tests.

[cognifloyd]

Cool. At a glance, I wonder if this would make it so I don't need StackStorm-Exchange/stackstorm-ansible/pull/14 to get complex data passing working.

[arm4b]

Awesome stuff here! 🥇

StackStorm is definitely missing more useful Jinja filters.

[Kami]

Thanks - looks good to me.

I'm still not fan of thise copy and paste between st2 and mistral repo so let's try to come up with a solution for that soon (would appreciate if you can open issue for that so we can track it).