ldap in qt configuration

[rhfogh]

How are you supposed to install the qt branch using the configuration files?

The following fails:

conda env create -f ~/pycharm/mxcubeqt/conda-environment.yml --name mxcubeqt
conda activate mxcubeqt
cd ~/pycharm/mxcubecore/mxcubecore
poetry install

with

  ImportError

  cannot import name 'Python2Supports' from 'virtualenv.create.describe' (/home/rhfogh/Software/miniconda3/envs/mxcubeqt/lib/python3.8/site-packages/virtualenv/create/describe.py)

  at ~/Software/miniconda3/envs/mxcubeqt/lib/python3.8/site-packages/virtualenv/create/via_global_ref/builtin/python2/python2.py:6 in <module>
        2│ import json
        3│ import os
        4│ from pathlib import Path
        5│ 
    →   6│ from virtualenv.create.describe import Python2Supports
        7│ from virtualenv.create.via_global_ref.builtin.ref import PathRefToDest
        8│ from virtualenv.info import IS_ZIPAPP
        9│ from virtualenv.util.zipapp import read as read_from_zipapp
       10│ 

Cannot install python-ldap.

NB, for mxcubeweb, the following (which should be equivalent) succeeds:

conda env remove -n mxcubeweb
# COMMENT OUT mxcubecore in pyproject.toml
conda env create -f ~/pycharm/mxcubeweb/mxcubeweb/conda-environment.yml --name mxcubeweb
conda activate mxcubeweb

cd ~/pycharm/mxcubeweb/mxcubeweb/
poetry install
pnpm --prefix ui install
pnpm --prefix ui build
# in mxcubeweb:
pip install -e .
# in mxcubecore:
cd ../mxcubecore/
pip install -e .

[fabcor-maxiv]

I have seen this error before, but I do not recall how I got past it. I think that eventually after re-creating the environment a bunch of times from scratch it worked.

---

As I must have mentioned a few times already, there are some relatively serious problems with the content of our conda environment YAML files. The biggest one being that we end up having the project itself (mxcubecore, mxcubeqt, or mxcubeweb) and its runtime dependencies installed in the same environment as things like poetry and virtualenv. And that is not how things should be.

See for example this big red warning:

Poetry should always be installed in a dedicated virtual environment to isolate it from the rest of your system. It should in no case be installed in the environment of the project that is to be managed by Poetry.

I would be in favour of getting rid of those conda environment YAML files, as I already discussed in mxcube/mxcubeweb#1383 (comment).

Or maybe we find a different solution to prevent the situation mentioned above and situations like #849 (comment). I attempted a few times but never managed to finish my investigation, I might give it another shot soon-ish...

[fabcor-maxiv]

Ah, I just tried to re-create my conda environment for mxcubecore and got a similar error when running poetry install:

$ micromamba create -f conda-environment-dev.yml                                                      
Found conda-prefix at '/home/fabcor/.local/share/mamba/envs/mxcubecore'. Overwrite?: [y/N] y

[...]

$ micromamba activate mxcubecore

$ poetry install

[...]

  - Updating python-ldap (3.4.3 /home/conda/feedstock_root/build_artifacts/python-ldap_1698435803608/work -> 3.4.3): Failed

  ImportError

  cannot import name 'Python2Supports' from 'virtualenv.create.describe' (/home/fabcor/.local/share/mamba/envs/mxcubecore/lib/python3.11/site-packages/virtualenv/create/describe.py)

  at ~/.local/share/mamba/envs/mxcubecore/lib/python3.11/site-packages/virtualenv/create/via_global_ref/builtin/python2/python2.py:6 in <module>
        2│ import json
        3│ import os
        4│ from pathlib import Path
        5│ 
    →   6│ from virtualenv.create.describe import Python2Supports
        7│ from virtualenv.create.via_global_ref.builtin.ref import PathRefToDest
        8│ from virtualenv.info import IS_ZIPAPP
        9│ from virtualenv.util.zipapp import read as read_from_zipapp
       10│ 

Cannot install python-ldap.

[...]

[rhfogh]

@marcus-oscarsson, @fabcor-maxiv I am not sure that getting rid of the conda yaml files would solve the problem, at least for me. As I understand you would still need some kind of virtual environment to install in, and you would still want to know what was in it and be able to control that. As I think you said yourself, one would want a documented way of installing that is known to work. Surely that is the point of these systems - that not every developer needs to know about the peculiarities of poetry, venv, micromamba, conda, ..., or about the technical issues of installing ldap, but can leave these issues to the people who are changing the dependencies.

On the original point: It may well be that the mxcubeqt conda-environment.yml file is wrong and needs changing. But does anyone know how it should be changed?

[marcus-oscarsson]

I think its a virtualenv related issue,

cannot import name 'Python2Supports' from 'virtualenv.create.describe' 

Could it be the version of virutalenv that causes this (or some other dependency ?)

[fabcor-maxiv]

you would still need some kind of virtual environment to install in, and you would still want to know what was in it and be able to control that

That is exactly what Poetry (and its lockfile) does. In my opinion having both conda and Poetry is the problem and more work needs to be done to make them cooperate harmoniously without all these paper cuts.

[fabcor-maxiv]

I think its a virtualenv related issue,

cannot import name 'Python2Supports' from 'virtualenv.create.describe' 

Could it be the version of virutalenv that causes this (or some other dependency ?)

One could try to put a version constraint on virtualenv (or poetry) in the YAML file.

[marcus-oscarsson]

I agree that for production we should ideally build a package with poetry that is installed in an environment/image/container, if that is what you meant.

I don't know if its practical to separate the two environments for development purpose. I think we are happy to try it out if there is some good way to do it. The conda environment make sure that the right system libraries are in place and that is not managed by Poetry.

[fabcor-maxiv]

Things like Poetry and pre-commit should preferably be kept out of the environment where mxcube is meant to be installed. Ideally there should be only runtime dependencies in mxcube's environment and nothing else.

Typically, you would:

1. install conda/mamba/micromamba and Poetry and pre-commit and tools like this directly on your system or whatever you prefer
2. create an empty environment (with conda, or virtualenv or docker or whatever you prefer)
3. install only the runtime dependencies in that environment (no nodejs, no redis-server, definitely no pre-commit, poetry or virtualenv)
   • note that some things like pytest are tricky, because they need to be in the runtime environment to be able to do their job, but luckily they are marked as "dev" dependencies in pyproject.toml and can easily be skipped

But for convenience (fair enough, I am not comfortable with that but I get it) we put all of this in one environment, and then sometimes things break.

Now so far it has apparently not broken too much or too often, so we can keep living with that. I am investigating better solutions as a low priority task.

[rhfogh]

Would it be possible to generate requirements.txt files in parallel with / from the pyproject.toml and poetry.lock files? This would allow people to work with non-poetry installations and make it easier (for some, at least) to deal with problems like the one I came across here.

[rhfogh]

Issue closed. The problem has been solved by @fabcor-maxiv (see issue #1063 and mxcubeqt PR #462), and nothing further is needed.