Support Vizzu data filter

[simzer]

Data filter can be specified for Vizzu on the JS API as a JS function:

chart.animate({ data: { filter: record => expression ) }});

where data series can be referenced through the record object, e.g.:

record => { return record["foo"] == "blabla" || record["baz"] > 5; } 

It would be good to have two possibilities to define this filter on the ipyvizzu API

• Simple solution, supplying the JS expression in a string:

  Python:

  filter = JSExpression("string-of-JS-code")

  where ipyvizzu should generate the following JS code:
  JS:

  filter: record => { return (string-of-JS-code); }

  e.g.:

  Python:

  filter = JSExpression("record['foo'] == 'blabla' || record['baz'] > 5")

  =>
  JS:

  filter: record => { return record['foo'] == 'blabla' || record['baz'] > 5; }

• Through Python wrapper object with overloaded operators building the js code under the hood.
  Python:

  filter = Series("foo") == "blabla" || Series("baz" > 5)

  where ipyvizzu should generate the following:
  JS:

  filter: record => { return record["foo"] == "blabla" || record["baz"] > 5; }

  Operators, which should be supported:
  ==, !=, <, <=, >=, >, ||, &&

[simzer]

@nyirog Could you please check this proposal? Thanks!

[nyirog]

The JS expression as string would be easy to implement. But it would be good to detect syntax error. If the vizzu chart would have an on_failure callback hook then we could report the syntax error back to the user. Or we could evaluate the filter function on python side for syntax check (e.g.: js2py).

Filter("record['foo'] == 'blabla' || record['baz'] > 5")

The operator overload looks nice, but how do you describe parentheses?

Filter(Record("foo") == "blabla" | Record("baz") > 5)

How about some dot snake:

Filter(Record("foo").eq("blabla").or(Record("baz").gt(5)))

[simzer]

The JS interpreter would fail on syntax error before vizzu can even receive the faulty filter function. The error will be printed on the js console log. If it is not enough, we could check the syntax on the Python side.

About the operator overload solution:
I don't think we should describe parentheses if each operator generates the string this way:
A && B => "(A && B)"
This way the parenthesis nesting in the generated string will follow the execution order of the pyton code, so it will be the same as in the python code:

(A && B) && C => "((A && B) && C)"
A && (B && C) => "(A && (B && C))"

[nyirog]

Of course the user defined parentheses would more practical if we would support negation. But we might be good with python side parentheses if the overloaded operators would accept scalar values (int, str, ...) and Record instances.

For example in Record.__eq_(self, other) other could be scalar or Record and the Record.__eq__ would return with a Record instance so the expression could be chained.

Should we support comparition between record fields?

Record("foo") < `Record("bar")

Should we support basic arithmetics?

Record("foo") < Record("bar") + 5

[simzer]

The far most frequent use case is comparing dimension values to strings, so the most important operators to support would be ==, !=, &&, ||.
Comparison of record fields with each other is nice to have, and if it's not too complicated, it would be good to support basic arithmetic as well.

[simzer]

@veghdev did you want to close this issue? Should we create a new issue for the filter using the operator overload method?

[veghdev]

Raw Js string filter implemented and will be released in 0.6.0.

@simzer I reopened this (operator overload method implementation), but removed 0.6.0 milestone.

[simzer]

An alternative method for supporting python filter functions would be to use pyodide. The basic idea would be something like this:

Python code:

filter = "record["foo"] == "blabla" | record["baz"] > 5"

Generated JS code:

      <script src="https://cdn.jsdelivr.net/pyodide/v0.22.1/full/pyodide.js"></script>

let pyodide = await loadPyodide();
...
let filter = (record) => { return pyodide.runPython(`
  f = lambda record: record["foo"] == "blabla" || record["baz"] > 5
  f(record)
`)); };

[simzer]

or micropython
https://micropython.org/

[simzer]

or transcript, rapidscript, pyjs

[simzer]

There is a solution in sqlalchemy similar to the operator overloading proposal above:
https://docs.sqlalchemy.org/en/20/core/operators.html#comparison-operators

Drawbacks: logical and/or/not cannot be overloaded in Python, bitwise operators can be used instead, but they have a higher precedence then comparison operators, which is confusing. e.g. parenthesis needed in this expr: (Year==2023)&(Month=='Jun')

[simzer]

Regardless, I have made a draft implementation for the operator overloading solution:
https://90b6028f-b426-4390-ab84-febd297bdf96.pyscriptapps.com/8df0cb69-f132-4e34-aa71-245c9c7f34ba/latest/

(see link to the source in the bottom right corner)