Idea: Lint / Validate JEXL expressions

[czosel]

Currently it's quite easy to confuse == and in when asserting values of single / multiple choice questions. This is particularily annoying because some errors only surface in PyJEXL and not in the frontend (see #1263).

I'd love to have something like this:

This could be done with a function

validate(expression, all_questions) -> "ok" | "potentially wrong because (explanation)"

that would traverse the JEXL AST and cross-check it with the existing questions according to rules like:

• easy: are all subjects of |answer transforms existing questions
• advanced: find expressions where a literal (plain strings "foo" or lists of strings ["foo", "bar"]) is compared to an answer value ("foo"|answer):
  • "foo" == "bar"|answer -> complain if "bar" is multiple choice
  • "foo" in "bar"|answer -> complain if "bar" is single choice
  • ["foo", "bar"] intersects "bar"|answer -> complain if "bar" is single choice
  • "bar"|answer in ["foo", "bar"] -> complain if "bar" is multiple choice
• there are several more cases one could catch, e.g. check types based on other clues
  • "foo"| answer > 5 -> "foo" should be numeric question
  • "foo"| answer|mapby("bar") -> "foo" should be table question
  • ...

The function would probably be most suitable for implementation in the backend (such that no list of all questions needs to be fetched), exposed via an API endpoint, and potential complaints could be displayed in the form builder (e.g. on blur of the JEXL field). Until we'd be 1000% sure that our validations are always correct, I'd treat them as warnings, i.e. the user could ignore them and save the JEXL anyway.

Curious to hear your feedback @winged @open-dynaMIX @kaldras @anehx @luytena

[winged]

I like the whole idea a lot!

I already started once with some low-hanging fruits in this context - See #1254 from a while ago :)

The type checking is a good idea, but may be difficult pulling off. I feel like having different return data types in the |answer filter is already kind of a code smell in itself. It could be a table that needs to be transformed by mapby into a column/list, or could be a string (where Python would allow the in operator but JS wouldn't) or even a number usable for computation.

The validation will fall short if the LHS of the in operator is not a literal (ohai halting problem). What if someone goes for "foo"|answer == "bar"|answer or something the like?

Then, the |answer could refer to a question in another form, so checking for pure existence is not entirely enough, as you still wouldn't know (without checking all form structures) that the referenced question will exist in context.

HOWEVER, all those doubts should not keep us from pushing this (Perfect is the enemy of good, after all). Your suggestion of treating things as warnings until proven "bomb proof" sounds great!