Limited number of arguments for queries

[153957]

Due to some (arbitrary?) limit in NumPy (numpy/numpy#4398) there is a limit to the number of inputs in a query in PyTables (PyTables/PyTables#286).

This mostly affects the coincidence_queries module where queries may include many station columns.
For instance, if looking for coincidences that include all stations in the Amsterdam cluster, more than 32 columns will be queried. This will fail.

Example using all station columns:

>>> coln = data.root.coincidences.coincidences.colnames
>>> c = data.root.coincidences.coincidences.read_where(' & '.join(coln[11:]))
........
ValueError: too many inputs

We could work around this by evaluating the query in python instead of in-kernel.
However that would slow execution..
Perhaps we can work around this limit or see if we can also vote for an increase in the limit.

[153957]

Some more discussion at numpy/numpy#4840

[153957]

See this as well numpy/numpy#4398

[tomkooij]

Won't fix.

[153957]

Why not?

[tomkooij]

@153957: We cannot change the numpy query limit and a work-around (in kernel query for large querys) is too much hassle.

I'm happy with leaving this open, but I was trying to close issues that seem too much hassle to fix. (Too low return on investment)

[153957]

It would actually be nice if this is fixed, currently simply selecting 'Amsterdam' as cluster in the Coincidence download form returns the 'to many stations' error:
http://data.hisparc.nl/data/network/coincidences/?end=2016-09-11+00%3A01&start=2016-09-11&stations=None&n=2&cluster=Amsterdam&download=False

Is it to much trouble to convince NumPy to change it on their side? I know various attempts have been made, with little progress..

[tomkooij]

Ok, that make sense. Let's at least keep this open and hope numpy fixes this. (Or do the workaround thing...).

[153957]

I think this should at least be kept open as long as the numpy issue above is still open.

[153957]

The value was increased in NumPy 2, you can even manually set it higher it seems.
https://numpy.org/devdocs/release/2.0.0-notes.html#larger-npy-maxdims-and-npy-maxargs-npy-ravel-axis-introduced

However, PyTables does not fully support NumPy 2 yet PyTables/PyTables#1172

[153957]

Awaiting the next release of PyTables which includes this fix; PyTables/PyTables#1208