fix #6070: hierarchical sorting #7163

rorour · 2024-07-08T21:43:04Z

No description provided.

rorour · 2024-07-08T21:45:47Z

@moellep This is a bit hacky. Interested to see your feedback. Map seemed like the best structure in JavaScript to contain the sortColumns, but required encoding/decoding for requests

e-carlin · 2024-07-09T15:46:55Z

I'm getting errors when I try to perform certain sorts. For example, select csx, add fccd_intersection, nanop_setup, detectors. The clear sort, sort fccd_intersection, sort nano_setup, sort detectors, see error.

Traceback (most recent call last):
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/tornado/web.py", line 1790, in _execute
    result = await result
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/raydata/scan_monitor.py", line 257, in post
    self.write(await self._incoming(PKDict(pkjson.load_any(self.request.body))))
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/raydata/scan_monitor.py", line 254, in _incoming
    return getattr(self, "_request_" + body.method)(body.data.get("args"))
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/raydata/scan_monitor.py", line 421, in _request_get_scans
    l, s = self._databroker_search(req_data)
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/raydata/scan_monitor.py", line 355, in _databroker_search
    l = [
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/raydata/scan_monitor.py", line 355, in <listcomp>
    l = [
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/databroker/_drivers/mongo_normalized.py", line 70, in __iter__
    for run_start_doc in cursor:
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/pymongo/cursor.py", line 1264, in next
    if len(self.__data) or self._refresh():
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/pymongo/cursor.py", line 1181, in _refresh
    self.__send_message(q)
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/pymongo/cursor.py", line 1060, in __send_message
    response = client._run_operation(
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/pymongo/_csot.py", line 107, in csot_wrapper
    return func(self, *args, **kwargs)
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1394, in _run_operation
    return self._retryable_read(
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1492, in _retryable_read
    return self._retry_internal(
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/pymongo/_csot.py", line 107, in csot_wrapper
    return func(self, *args, **kwargs)
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1453, in _retry_internal
    return _ClientConnectionRetryable(
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/pymongo/mongo_client.py", line 2315, in run
    return self._read() if self._is_read else self._write()
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/pymongo/mongo_client.py", line 2445, in _read
    return self._func(self._session, self._server, conn, read_pref)  # type: ignore
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1390, in _cmd
    return server.run_operation(
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/pymongo/helpers.py", line 322, in inner
    return func(*args, **kwargs)
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/pymongo/server.py", line 167, in run_operation
    _check_command_response(first, conn.max_wire_version)
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/pymongo/helpers.py", line 230, in _check_command_response
    raise OperationFailure(errmsg, code, response, max_wire_version)
pymongo.errors.OperationFailure: multiplanner encountered a failure while selecting best plan :: caused by :: cannot sort with keys that are parallel arrays, full error: {'ok': 0.0, 'errmsg': 'multiplanner encountered a failure while selecting best plan :: caused by :: cannot sort with keys\
 that are parallel arrays', 'code': 2, 'codeName': 'BadValue'}

I also got this error but I couldn't figure out how to reproduce. Looks like it is a known error so maybe you could backtrack from the stack trace with guesses to figure out how it happened.

HTTPServerRequest(protocol='http', host='127.0.0.1:9001', method='POST', uri='/scan-monitor', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/tornado/web.py", line 1790, in _execute
    result = await result
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/raydata/scan_monitor.py", line 257, in post
    self.write(await self._incoming(PKDict(pkjson.load_any(self.request.body))))
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/raydata/scan_monitor.py", line 254, in _incoming
    return getattr(self, "_request_" + body.method)(body.data.get("args"))
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/raydata/scan_monitor.py", line 421, in _request_get_scans
    l, s = self._databroker_search(req_data)
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/raydata/scan_monitor.py", line 355, in _databroker_search
    l = [
  File "/home/vagrant/src/radiasoft/sirepo/sirepo/raydata/scan_monitor.py", line 355, in <listcomp>
    l = [
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/databroker/_drivers/mongo_normalized.py", line 70, in __iter__
    for run_start_doc in cursor:
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/pymongo/cursor.py", line 1264, in next
    if len(self.__data) or self._refresh():
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/pymongo/cursor.py", line 1181, in _refresh
    self.__send_message(q)
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/pymongo/cursor.py", line 1060, in __send_message
    response = client._run_operation(
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/pymongo/_csot.py", line 107, in csot_wrapper
    return func(self, *args, **kwargs)
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1394, in _run_operation
    return self._retryable_read(
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1492, in _retryable_read
    return self._retry_internal(
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/pymongo/_csot.py", line 107, in csot_wrapper
    return func(self, *args, **kwargs)
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1453, in _retry_internal
    return _ClientConnectionRetryable(
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/pymongo/mongo_client.py", line 2315, in run
    return self._read() if self._is_read else self._write()
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/pymongo/mongo_client.py", line 2445, in _read
    return self._func(self._session, self._server, conn, read_pref)  # type: ignore
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1390, in _cmd
    return server.run_operation(
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/pymongo/helpers.py", line 322, in inner
    return func(*args, **kwargs)
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/pymongo/server.py", line 167, in run_operation
    _check_command_response(first, conn.max_wire_version)
  File "/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/pymongo/helpers.py", line 230, in _check_command_response
    raise OperationFailure(errmsg, code, response, max_wire_version)
pymongo.errors.OperationFailure: FieldPath cannot be constructed with empty string, full error: {'ok': 0.0, 'errmsg': 'FieldPath cannot be constructed with empty string', 'code': 40352, 'codeName': 'Location40352'}

e-carlin

Haven't reviewed fully but one comment on using a 2d array instead of a string for the encoded values.

sirepo/package_data/static/js/raydata.js

moellep

I can get the "FieldPath cannot be constructed with empty string" error by selecting "Clear Sort" and letting the UI refresh without selecting a new sort field.

For the sorting behavior, I think it is confusing to select a column and not have it become the primary sort. For example, if I'm viewing the list and it is already sorted by "start", and I select "acquire period", it doesn't change the sorting. It does do that if I select "Clear Sort" first and then select "acquire period" but I don't think that is very intuitive.

Instead, you could remove the "Clear Sort" button and keep track of the last 3 sort columns, applied in order. For example, I view the page initially, and it is sorted by "start" by default. Then I select "acquire period" and it sorts by "acquire period", "start". Then I select "sequence id" and it sorts by "sequence id", "acquire period", "start". If you select a forth column, then only the last three selected columns would be used for sorting.

rorour · 2024-07-10T22:24:37Z

I'm getting errors when I try to perform certain sorts. For example, select csx, add fccd_intersection, nanop_setup, detectors. The clear sort, sort fccd_intersection, sort nano_setup, sort detectors, see error.

This seems to be some kind of restriction with Mongo sort.

https://www.mongodb.com/community/forums/t/sorting-on-multiple-fields-that-may-or-may-not-be-arrays-cannot-sort-with-keys-that-are-parallel-arrays/251689

https://jira.mongodb.org/browse/SERVER-826

Any suggestions on how to handle this? I guess we could check the type of data in each column and only allow one array type column to be added to sort.

e-carlin · 2024-07-11T15:28:38Z

Interesting. For now let's just not allow sorting of any array field and see if anyone asks for it.

rorour · 2024-07-25T19:15:14Z

Ready for re-review. I went back and forth on whether to remove the "Clear Sort" button but I did remove it.

sirepo/package_data/static/js/raydata.js

rorour · 2024-07-31T23:33:15Z

@moellep @e-carlin ping

moellep · 2024-08-02T21:26:02Z

I find I can get the sorting in an odd state, where selecting "acquire period" doesn't always sort the same. If I open the CHX page, and sort a few times on "acquire period", it behaves as expected, where empty fields either appear at the top or the bottom of the sort.

But if I open a fresh CHX page and then sort on "T_sample_", "acquire period", "T_yoke", and then back to "acquire period", it always keeps the empty rows at the top of the sorted list even if I keep selecting "acquire period".

rorour · 2024-08-02T23:17:48Z

Thanks for catching that. When reversing the order of a column that was already being sorted on, it was remaining at the same sort priority. It makes more sense to make that column become priority again. Should be fixed.

rorour · 2024-08-02T23:46:59Z

Taking screenshots for the report got me thinking about whether it would be nice to have some visual indicator of sort order. Currently a user has to remember which column they clicked most recently, or study the column values.

I think color gradient of the arrows would work, with the most saturated color being the top priority. Or just an indicator on the top priority column. On the other hand, sort order isn't preserved in appState so maybe it's unlikely that a user would forget which column they clicked last.
Any thoughts?

moellep · 2024-08-02T23:55:33Z

Yes, I like the color saturation idea - show the top priority darker than the sub priorities.

e-carlin

Worked well. Some final style comments.

sirepo/package_data/static/js/raydata.js

rorour added 7 commits July 8, 2024 15:19

ckp

1ed9778

front end

f8da118

decode columns

5e01100

working on Run Analysis tab

aa87925

works

8ed787c

fix compare function

db8240e

cleanup

5a6a8da

rorour requested a review from moellep July 8, 2024 21:43

semicolon

b95963c

e-carlin reviewed Jul 9, 2024

View reviewed changes

sirepo/package_data/static/js/raydata.js Outdated Show resolved Hide resolved

moellep requested changes Jul 9, 2024

View reviewed changes

fix FieldPath error

5ebd69f

rorour added 2 commits July 24, 2024 17:18

columns with array values not sortable

70a6527

comments

67876a7

rorour requested review from moellep and e-carlin July 25, 2024 19:11

remove clear sort button

2fa0212

moellep reviewed Jul 25, 2024

View reviewed changes

sirepo/package_data/static/js/raydata.js Outdated Show resolved Hide resolved

handle null values

b98f2e9

rorour requested a review from moellep July 25, 2024 21:44

move column to top of list when reversing sort

6ba3a96

moellep approved these changes Aug 5, 2024

View reviewed changes

rorour added 3 commits August 5, 2024 15:58

color gradient arrows

ed7c3a4

fix indent

2c03022

comments

7322d19

e-carlin requested changes Aug 6, 2024

View reviewed changes

rorour requested a review from e-carlin August 6, 2024 23:06

Merge branch 'master' into 6070-hierarchical-sorting

d2e6c4f

e-carlin approved these changes Aug 8, 2024

View reviewed changes

e-carlin enabled auto-merge (squash) August 8, 2024 16:33

Merge branch 'master' into 6070-hierarchical-sorting

382f118

e-carlin merged commit 9e60ffd into master Aug 8, 2024
3 checks passed

e-carlin deleted the 6070-hierarchical-sorting branch August 8, 2024 16:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix #6070: hierarchical sorting #7163

fix #6070: hierarchical sorting #7163

rorour commented Jul 8, 2024

rorour commented Jul 8, 2024

e-carlin commented Jul 9, 2024

e-carlin left a comment

moellep left a comment •

edited

Loading

rorour commented Jul 10, 2024

e-carlin commented Jul 11, 2024

rorour commented Jul 25, 2024

rorour commented Jul 31, 2024

moellep commented Aug 2, 2024

rorour commented Aug 2, 2024

rorour commented Aug 2, 2024

moellep commented Aug 2, 2024

e-carlin left a comment

fix #6070: hierarchical sorting #7163

fix #6070: hierarchical sorting #7163

Conversation

rorour commented Jul 8, 2024

rorour commented Jul 8, 2024

e-carlin commented Jul 9, 2024

e-carlin left a comment

Choose a reason for hiding this comment

moellep left a comment • edited Loading

Choose a reason for hiding this comment

rorour commented Jul 10, 2024

e-carlin commented Jul 11, 2024

rorour commented Jul 25, 2024

rorour commented Jul 31, 2024

moellep commented Aug 2, 2024

rorour commented Aug 2, 2024

rorour commented Aug 2, 2024

moellep commented Aug 2, 2024

e-carlin left a comment

Choose a reason for hiding this comment

moellep left a comment •

edited

Loading