Never choose a "group" value in dropdowns

[CasperWA]

See issues: aiidalab/aiidalab-qe#135 and aiidalab/aiidalab-qe#134.

Don't allow the application to auto-choose an index of a dropdown that corresponds to a grouping header.

[csadorf]

Maybe as an additional comment, the group headers are currently not visually distinguished in my environment. I'm not sure what the reason for that could be.

[CasperWA]

Maybe as an additional comment, the group headers are currently not visually distinguished in my environment. I'm not sure what the reason for that could be.

Could you show an example?

I'm not very diverse in my browser choices for testing, using only Chrome, so this might be the issue? Although I am using nothing else but the standard HTML tag options, I believe.

[CasperWA]

Can you check whether the latest version of develop works? Or should I release a new version for you to more easily test? @csadorf

[csadorf]

Can you check whether the latest version of develop works? Or should I release a new version for you to more easily test? @csadorf

@CasperWA Thanks for proposing a fix. I'm gonna test this, no release or prerelease needed.

[csadorf]

@CasperWA I've tested it and this is what it looks like on my developer machine:

$ pip freeze | grep ipywidgets
ipywidgets==7.6.5
ipywidgets-extended==1.1.1

Selecting "General" does not crash the app anymore, but still kind of unexpected that it is even possible to select the headers. I tested it on Firefox and Chrome, same issue in both browsers.

[csadorf]

@CasperWA You should be able to reproduce the issue with the following steps:

1. In a terminal, run:

git clone https://github.com/aiidalab/aiidalab-docker-stack
cd ./aiidalab-docker-stack/
pip install -r requirements-manage.txt  # for docker-compose and click
./manage.py configure --app quantum-espresso==21.10.0b1
./manage.py --develop -v up

2. Open the QE app.
3. Select the "OPTIMADE" tab for the structure selection.
4. Select "Materials Cloud" as the provider.
5. Open the list of databases.

qe-app-issue-134_how2reproduce.mp4

[unkcpz]

I can reproduce this on my computer
OS: Ubuntu 20.04.3 LTS
Browser: Chrome and Firefox

[unkcpz]

Downgrade the ipywidgets-extended version to ipywidgets-extended==1.1.0 and selecting "General" DOES crash the app again.

[csadorf]

@unkcpz noticed that on a different deployment, the widget works as expected, however we are not able to figure out what is causing that.

Diff of pip list --format=freeze between the non-working (left) and working (right) environment:

							      >	aiida-bands-inspect==0.4.0
							      >	aiida-bigdft==0.2.6
							      >	aiida-codtools==2.1.0
aiida-core==1.6.4						aiida-core==1.6.4
							      >	aiida-cp2k==1.3.0
							      >	aiida-crystal17==0.11.0
							      >	aiida-ddec==1.1.0
							      >	aiida-diff==1.2.0
							      >	aiida-fleur==1.2.1
							      >	aiida-gaussian==1.4.1
							      >	aiida-lammps==0.8.0
							      >	aiida-lsmo==1.0.0
							      >	aiida-nanotech-empa==0.4.1
							      >	aiida-optimize==0.4.0
aiida-pseudo==0.6.3						aiida-pseudo==0.6.3
							      >	aiida-qeq==1.0.0a1
aiida-quantumespresso==3.4.2					aiida-quantumespresso==3.4.2
							      >	aiida-raspa==1.2.0
							      >	aiida-sssp-workflow==0.1.0
							      >	aiida-tbmodels==0.3.0
							      >	aiida-tools==0.3.3
							      >	aiida-vasp==2.1.1
							      >	aiida-wannier90==2.0.1
							      >	aiida-yambo==1.3.0
							      >	aiida-zeopp==1.1.2
aiidalab==21.10.1						aiidalab==21.10.1
aiidalab-eln==0.1.1						aiidalab-eln==0.1.1
aiidalab-qe==21.10.0b1						aiidalab-qe==21.10.0b1
aiidalab-qe-workchain==1.0					aiidalab-qe-workchain==1.0
aiidalab-widgets-base==1.0.0					aiidalab-widgets-base==1.0.0
aio-pika==6.8.0							aio-pika==6.8.0
aiormq==3.3.1							aiormq==3.3.1
							      >	alabaster==0.7.12
aldjemy==0.9.1							aldjemy==0.9.1
alembic==1.6.5							alembic==1.6.5
							      >	aniso8601==9.0.1
ansi2html==1.6.0						ansi2html==1.6.0
appdirs==1.4.4							appdirs==1.4.4
archive-path==0.2.1						archive-path==0.2.1
argcomplete==1.12.3						argcomplete==1.12.3
argon2-cffi==21.1.0						argon2-cffi==21.1.0
arrow==1.2.0						      |	arrow==1.1.1
ase==3.19.3							ase==3.19.3
async-generator==1.10						async-generator==1.10
attrs==21.2.0							attrs==21.2.0
							      >	Babel==2.9.1
backcall==0.2.0							backcall==0.2.0
backports.entry-points-selectable==1.1.0		      |	bands-inspect==0.3.1
bcrypt==3.2.0							bcrypt==3.2.0
binaryornot==0.4.4						binaryornot==0.4.4
black==21.9b0						      <
bleach==4.1.0							bleach==4.1.0
bokeh==2.4.1						      |	bokeh==2.4.0
bqplot==0.12.31						      |	bqplot==0.12.30
brotlipy==0.7.0							brotlipy==0.7.0
							      >	bump2version==1.0.1
CacheControl==0.12.6						CacheControl==0.12.6
cachetools==4.2.4					      |	cached-property==1.5.2
certifi==2021.5.30					      |	cachetools==4.2.2
							      >	calc-pe==1.0.1
							      >	cclib==1.7.1
							      >	certifi==2021.10.8
cffi==1.14.4							cffi==1.14.4
cfgv==3.3.1						      |	cftime==1.5.1
chardet==4.0.0							chardet==4.0.0
charset-normalizer==2.0.7				      |	charset-normalizer==2.0.6
cheminfopy==0.7.0						cheminfopy==0.7.0
circus==0.17.1							circus==0.17.1
click==7.1.2							click==7.1.2
click-completion==0.5.2						click-completion==0.5.2
click-config-file==0.6.0					click-config-file==0.6.0
click-spinner==0.1.10						click-spinner==0.1.10
							      >	colorama==0.4.4
conda==4.9.2							conda==4.9.2
conda-package-handling==1.7.2					conda-package-handling==1.7.2
configobj==5.0.6						configobj==5.0.6
cookiecutter==1.7.3						cookiecutter==1.7.3
cryptography==3.3.1						cryptography==3.3.1
cycler==0.10.0							cycler==0.10.0
dacite==1.6.0							dacite==1.6.0
debugpy==1.5.0						      |	debugpy==1.4.3
decorator==4.4.2						decorator==4.4.2
							      >	deepdiff==5.6.0
defusedxml==0.7.1						defusedxml==0.7.1
deprecation==2.1.0						deprecation==2.1.0
distlib==0.3.3						      <
Django==2.2.24							Django==2.2.24
dnspython==2.1.0						dnspython==2.1.0
							      >	docutils==0.15.2
dulwich==0.20.25						dulwich==0.20.25
elementpath==2.3.2						elementpath==2.3.2
email-validator==1.1.3						email-validator==1.1.3
entrypoints==0.3						entrypoints==0.3
ete3==3.1.2							ete3==3.1.2
fastapi==0.65.3							fastapi==0.65.3
filelock==3.3.1						      |	fileupload==0.1.5
							      >	Flask==1.1.4
							      >	Flask-Cors==3.0.10
							      >	Flask-RESTful==0.3.9
							      >	fsc.export==1.2.0
							      >	fsc.hdf5-io==1.0.3
future==0.18.2							future==0.18.2
graphviz==0.16							graphviz==0.16
identify==2.3.0						      |	h5py==3.5.0
							      >	humanfriendly==10.0
idna==2.10							idna==2.10
							      >	imagesize==1.2.0
importlib-metadata==4.6.0					importlib-metadata==4.6.0
importlib-resources==5.3.0				      |	importlib-resources==3.3.1
							      >	iniconfig==1.1.1
ipydatawidgets==4.2.0						ipydatawidgets==4.2.0
ipykernel==6.4.1						ipykernel==6.4.1
ipython==7.25.0							ipython==7.25.0
ipython-genutils==0.2.0						ipython-genutils==0.2.0
ipytree==0.2.1							ipytree==0.2.1
ipywidgets==7.6.5						ipywidgets==7.6.5
ipywidgets-extended==1.1.1				      |	ipywidgets-extended==1.1.0
							      >	itsdangerous==1.1.0
jedi==0.18.0							jedi==0.18.0
Jinja2==2.11.3							Jinja2==2.11.3
jinja2-time==0.2.0						jinja2-time==0.2.0
joblib==1.1.0						      |	joblib==1.0.1
							      >	jsonextended==0.7.11
jsonpointer==2.1						jsonpointer==2.1
jsonref==0.2							jsonref==0.2
jsonschema==3.2.0						jsonschema==3.2.0
jupyter-client==6.1.12						jupyter-client==6.1.12
jupyter-core==4.8.1						jupyter-core==4.8.1
jupyterlab-widgets==1.0.2					jupyterlab-widgets==1.0.2
kiwipy==0.7.4							kiwipy==0.7.4
kiwisolver==1.3.1						kiwisolver==1.3.1
lark-parser==0.12.0						lark-parser==0.12.0
lockfile==0.12.2						lockfile==0.12.2
							      >	lxml==4.6.3
Mako==1.1.4							Mako==1.1.4
Markdown==3.3.4							Markdown==3.3.4
MarkupSafe==2.0.1						MarkupSafe==2.0.1
							      >	masci-tools==0.6.2
matplotlib==3.4.2						matplotlib==3.4.2
matplotlib-inline==0.1.2					matplotlib-inline==0.1.2
							      >	mendeleev==0.7.0
mistune==0.8.4							mistune==0.8.4
monty==2021.6.10						monty==2021.6.10
more-itertools==8.10.0						more-itertools==8.10.0
mpmath==1.2.1							mpmath==1.2.1
msgpack==1.0.2							msgpack==1.0.2
multidict==5.1.0						multidict==5.1.0
mypy-extensions==0.4.3					      <
nbconvert==5.6.1						nbconvert==5.6.1
nbformat==5.1.3							nbformat==5.1.3
nest-asyncio==1.4.3						nest-asyncio==1.4.3
							      >	netCDF4==1.5.7
networkx==2.5.1							networkx==2.5.1
nglview==3.0.3							nglview==3.0.3
nodeenv==1.6.0						      <
notebook==6.4.4							notebook==6.4.4
numpy==1.21.1							numpy==1.21.1
olefile==0.46							olefile==0.46
optimade==0.16.4					      |	optimade==0.16.6
optimade-client==2021.8.24					optimade-client==2021.8.24
							      >	ordered-set==4.0.2
packaging==20.9							packaging==20.9
palettable==3.3.0						palettable==3.3.0
pamqp==2.3.0							pamqp==2.3.0
pandas==1.3.1							pandas==1.3.1
pandocfilters==1.5.0						pandocfilters==1.5.0
paramiko==2.7.2							paramiko==2.7.2
							      >	parsevasp==2.0.1
parso==0.8.2							parso==0.8.2
pathspec==0.9.0						      |	pathlib2==2.3.6
pathtools==0.1.2						pathtools==0.1.2
pbr==5.6.0							pbr==5.6.0
							      >	periodictable==1.6.0
pexpect==4.8.0							pexpect==4.8.0
pgsu==0.2.0							pgsu==0.2.0
pickleshare==0.7.5						pickleshare==0.7.5
Pillow==8.3.1							Pillow==8.3.1
Pint==0.16.1							Pint==0.16.1
pip==21.3							pip==21.3
platformdirs==2.4.0					      <
plotly==5.0.0							plotly==5.0.0
							      >	pluggy==1.0.0
plumpy==0.19.0							plumpy==0.19.0
poyo==0.5.0							poyo==0.5.0
pre-commit==2.15.0					      <
prometheus-client==0.11.0					prometheus-client==0.11.0
prompt-toolkit==3.0.19						prompt-toolkit==3.0.19
psutil==5.8.0							psutil==5.8.0
psycopg2-binary==2.8.6						psycopg2-binary==2.8.6
ptyprocess==0.7.0						ptyprocess==0.7.0
							      >	py==1.10.0
pycairo==1.20.1							pycairo==1.20.1
PyCifRW==4.4.3							PyCifRW==4.4.3
pycosat==0.6.3							pycosat==0.6.3
pycparser==2.20							pycparser==2.20
pydantic==1.8.2							pydantic==1.8.2
							      >	pydata-sphinx-theme==0.4.3
							      >	pyfiglet==0.8.post1
Pygments==2.9.0							Pygments==2.9.0
pymatgen==2022.0.9					      |	pyhull==2015.2.1
							      >	pyiast==1.4.3
							      >	pymatgen==2020.12.3
PyMySQL==0.9.3							PyMySQL==0.9.3
PyNaCl==1.4.0							PyNaCl==1.4.0
pyOpenSSL==20.0.1						pyOpenSSL==20.0.1
pyparsing==2.4.7						pyparsing==2.4.7
pyrsistent==0.17.3						pyrsistent==0.17.3
pysmiles==1.0.1							pysmiles==1.0.1
PySocks==1.7.1							PySocks==1.7.1
							      >	pytest==6.2.5
python-dateutil==2.8.2						python-dateutil==2.8.2
python-editor==1.0.4						python-editor==1.0.4
							      >	python-memcached==1.59
python-slugify==5.0.2						python-slugify==5.0.2
pythreejs==2.3.0						pythreejs==2.3.0
pytojcamp==0.1.0						pytojcamp==0.1.0
pytray==0.3.2							pytray==0.3.2
pytz==2019.3							pytz==2019.3
PyYAML==5.1.2							PyYAML==5.1.2
pyzmq==22.1.0							pyzmq==22.1.0
qe-tools==2.0.0rc2						qe-tools==2.0.0rc2
reentry==1.3.2							reentry==1.3.2
regex==2021.10.8					      <
requests==2.26.0						requests==2.26.0
requests-cache==0.5.2						requests-cache==0.5.2
requests-mock==1.8.0						requests-mock==1.8.0
rfc3987==1.3.8							rfc3987==1.3.8
ruamel.yaml==0.16.13						ruamel.yaml==0.16.13
ruamel.yaml.clib==0.2.2						ruamel.yaml.clib==0.2.2
ruamel-yaml-conda==0.15.80					ruamel-yaml-conda==0.15.80
scikit-learn==0.24.2						scikit-learn==0.24.2
scipy==1.7.0							scipy==1.7.0
							      >	seaborn==0.11.2
seekpath==1.9.7							seekpath==1.9.7
Send2Trash==1.8.0						Send2Trash==1.8.0
setuptools==49.6.0.post20210108					setuptools==49.6.0.post20210108
shellingham==1.4.0						shellingham==1.4.0
shortuuid==1.0.1						shortuuid==1.0.1
simplejson==3.17.2						simplejson==3.17.2
six==1.15.0							six==1.15.0
spglib==1.16.1						      |	snowballstemmer==2.1.0
							      >	spglib==1.16.2
							      >	Sphinx==3.2.1
							      >	sphinx-copybutton==0.3.3
							      >	sphinx-notfound-page==0.7.1
							      >	sphinx-panels==0.5.2
							      >	sphinxcontrib-applehelp==1.0.2
							      >	sphinxcontrib-details-directive==0.1.0
							      >	sphinxcontrib-devhelp==1.0.2
							      >	sphinxcontrib-htmlhelp==2.0.0
							      >	sphinxcontrib-jsmath==1.0.1
							      >	sphinxcontrib-qthelp==1.0.3
							      >	sphinxcontrib-serializinghtml==1.1.5
							      >	sphinxext-rediraffe==0.2.7
SQLAlchemy==1.3.24						SQLAlchemy==1.3.24
SQLAlchemy-Utils==0.36.8					SQLAlchemy-Utils==0.36.8
sqlparse==0.4.1							sqlparse==0.4.1
starlette==0.14.2						starlette==0.14.2
strict-rfc3339==0.7						strict-rfc3339==0.7
							      >	subprocess32==3.5.4
sympy==1.8							sympy==1.8
tabulate==0.8.9							tabulate==0.8.9
tenacity==7.0.0							tenacity==7.0.0
terminado==0.12.1						terminado==0.12.1
testpath==0.5.0							testpath==0.5.0
text-unidecode==1.3						text-unidecode==1.3
threadpoolctl==3.0.0					      |	threadpoolctl==2.2.0
toml==0.10.2							toml==0.10.2
tomli==1.2.1						      <
tornado==6.1							tornado==6.1
tqdm==4.56.1							tqdm==4.56.1
traitlets==5.0.5						traitlets==5.0.5
traittypes==0.2.1						traittypes==0.2.1
typed-ast==1.4.3					      |	typing-extensions==3.10.0.2
typing-extensions==3.10.0.0				      <
tzlocal==2.1							tzlocal==2.1
uncertainties==3.1.5						uncertainties==3.1.5
upf-to-json==0.9.3						upf-to-json==0.9.3
urllib3==1.26.3							urllib3==1.26.3
Vapory==0.1.2							Vapory==0.1.2
virtualenv==20.8.1					      |	voluptuous==0.12.2
watchdog==0.10.4						watchdog==0.10.4
wcwidth==0.2.5							wcwidth==0.2.5
webcolors==1.11.1						webcolors==1.11.1
webencodings==0.5.1						webencodings==0.5.1
							      >	Werkzeug==1.0.1
wheel==0.36.2							wheel==0.36.2
widget-bandsplot==0.2.5					      |	widget-bandsplot==0.2.4
widget-periodictable==2.1.5					widget-periodictable==2.1.5
widgetsnbextension==3.5.1					widgetsnbextension==3.5.1
wrapt==1.11.2							wrapt==1.11.2
xmlschema==1.8.1						xmlschema==1.8.1
yarl==1.6.3							yarl==1.6.3
zipp==3.4.1							zipp==3.4.1

[CasperWA]

Hmm... this is very peculiar.
So to me this might come down to two things, and both things might be wrong for where to go to try and find the issue:

• It's an OS/browser support related.
  The idea here is that it might be caused by some incompatibilities or short-comings (uninstalled fluff plugins) for a bare-bones linux OS for any given browser.
• It's an installation issue with the ipywidgets-extended package into the Jupyter environment.
  It's not uncommon that a widget is not properly installed as a notebook extension (nbextension). It could be that the installation protocol for the package isn't rigorous enough and doesn't find the proper place to install itself to enable it as a proper extension or that it doesn't have the rights upon installation to place itself in some critical Jupyter locations?
  Using virtuelenv this seems to work fine, it uses the nbconfig file under the respective virtual environment (e.g., ~/.venvs/ipywidgets/etc/jupyter/nbconfig, where ipywidgets is the name of the venv). However, it may be that this fails under other conditions?

The latter is what I'd check first, however, usually if they're not loaded properly it just doesn't show the widget at all. It simply doesn't show up. But it may still be that this is the issue and we're just seeing a different error than normal... It's all quite confusing to be honest 😅 😟

[csadorf]

* It's an OS/browser support related.
  The idea here is that it might be caused by some incompatibilities or short-comings (uninstalled fluff plugins) for a bare-bones linux OS for any given browser.

I highly doubt that, since we were able to reproduce the issues on Mac OS-X/Linux with both Firefox and Chrome and on top of that, it did work fine in one of the deployments using the same system. So I would say it's almost impossible to be OS or browser-related.

[csadorf]

@CasperWA Would you mind trying to reproduce the issue with the steps shown above (it should only take 5-10mins) and at least confirm that you observe it in your system? Maybe you can check the browser console or so and see if you recognize any of the errors or warnings?

[CasperWA]

I get the same thing happening when I use the approach you have mentioned above.

I tried installing the OPTIMADE Client application as well, and have been looking into the Chrome DevTools to try and see what's going on.
There are several things I could point out, but I am completely unaware whether they are the culprits or not.

One is this "find-and-replace" that's missing. I don't know what it is. Find and replace widgets/code or just something else completely?

Then there's this entry, which I don't see if I simply open the "clean" Voilà edition of the client. I cannot see whether it thinks this is an error or it's just printing stuff out to the console. It might simply be the latter.

It should be noted it loads the ipywidgets_extended/extension fine. And this also makes sense, because the list of providers is the same type of widget, but it's only using the "disable options" feature.

Comparing the HTML code for the widgets in my two client editions (the one from your Docker stack and my local version) it's clear the dropdown with the databases doesn't receive any information about what to disable, the style, anything... it just seems to instantiate the widget and then... die.

I don't see other significant things in the DevTools.

One other thing I just noticed (again, might not be in any way significant) is the presence of lm-Widget in the classes of all my widgets in the local instance, while the lm-* class names are not present whatsoever in the widgets for the Docker stack edition. Now, this might simply be a Voilà thing that it adds to the classes. But it might also be a Jupyter versioning / ipywidgets versioning thing. But as you listed the pip diff above.. well... it's just weird, and maybe just a Voilà-specific thing and doesn't actually have anything to do with this issue.

My reasoning here is that I'm trying to find clues to why the information about disabling and grouping is not getting through to the dropdown widget.

[csadorf]

My reasoning here is that I'm trying to find clues to why the information about disabling and grouping is not getting through to the dropdown widget.

How is the information provided in the first place?

[CasperWA]

In any case. I unfortunately do not have more time to spend on this at this moment.
If you want a (possibly) working setup, you can try to do the following:

1. Setup a new virtual env. using minimum Python 3.7.
2. Run:

   $ pip install -U pip setuptools wheel
   $ pip install optimade-client[dev]

   Note, it might be that one needs to first run pip install jupyter. I'm unsure.
3. Run the server via $ optimade-client and then go to the shown URL to test it out.

[CasperWA]

My reasoning here is that I'm trying to find clues to why the information about disabling and grouping is not getting through to the dropdown widget.

How is the information provided in the first place?

You mean to the widget? Well it's this JS/Python-hybrid schema.
I don't remember the exact timing. Whether it's on initialization of the Python object that represents the widget that the grouping is done. I don't think so though. Because the list of options needs to be updated, so this is most likely triggered when the database is chosen and can be found in the Python code.
Then when/how the actual HTML is updated should be from the JS. It should be fed a variable from Python that contains a dictionary/JS object, which it then runs through to populate the dropdown options and stylize them.

[csadorf]

I was able to follow the code up until roughly here:

voila-optimade-client/optimade_client/subwidgets/provider_database.py

Lines 336 to 340 in a013cce

	child_dbs = (
	{"": []}
	if self.providers.label not in self.child_db_groupings
	else deepcopy(self.child_db_groupings[self.providers.label])
	)

and then I get lost. Maybe you can help me understand where exactly the hand-over happens to the JS object?

[CasperWA]

Sure, we can have a look at some point - possibly next week.

In the mean time, it might be interesting to see whether the issue is with the extension itself or with the implementation in the client.
I.e., see if you can reproduce it by just installing the ipywidgets-extended package from GitHub. Then you can test it by running the examples/examples.ipynb notebook :)

[csadorf]

In the mean time, it might be interesting to see whether the issue is with the extension itself or with the implementation in the client.
I.e., see if you can reproduce it by just installing the ipywidgets-extended package from GitHub. Then you can test it by running the examples/examples.ipynb notebook :)

I just ran the example and it seems to fail:

[csadorf]

@CasperWA Based on my last experiment it seems that the issue is with the extension, would you agree?

[CasperWA]

@CasperWA Based on my last experiment it seems that the issue is with the extension, would you agree?

Yes indeed. I thought I commented on this, but I must have missed pushing the button.
So we should attack this issue on the side of the extension instead: CasperWA/ipywidgets-extended.

[csadorf]

@CasperWA Based on my last experiment it seems that the issue is with the extension, would you agree?

Yes indeed. I thought I commented on this, but I must have missed pushing the button. So we should attack this issue on the side of the extension instead: CasperWA/ipywidgets-extended.

@CasperWA Would you mind transferring #369 to https://github.com/CasperWA/ipywidgets-extended? I lack permissions to do so.

[csadorf]

Auxilliary issue now tracked in #369 .