desiInstall creates incorrect module file for QuasarNP

[sbailey]

QuasarNP is a semi-external package hosted in desihub. It doesn't have a boilerplate etc/quasarnp.module file, but it also doesn't require anything custom so I was expecting the desiInstall default module file to work. desiInstall does correctly identify the "py" install and pip installs QuasarNP, but the resulting module file doesn't add $PRODUCT_DIR/lib/python3.8/site-packages to $PYTHONPATH.

[cori06 ~] desiInstall -v -r $SCRATCH/desi/test QuasarNP 0.1.0
WARNING:install.py:183:get_options:2021-06-03T14:21:29: The environment variable LANG is not set!
DEBUG:install.py:251:get_options:2021-06-03T14:21:29: Set log level to DEBUG.
DEBUG:install.py:320:get_product_version:2021-06-03T14:21:29: Detected GitHub install.
DEBUG:install.py:351:identify_branch:2021-06-03T14:21:29: Using https://github.com/desihub/QuasarNP/archive/0.1.0.tar.gz as the URL of this product.
INFO:install.py:412:get_code:2021-06-03T14:21:29: Detected old working directory, /global/u2/s/sjbailey/QuasarNP-0.1.0. Deleting...
DEBUG:install.py:414:get_code:2021-06-03T14:21:29: shutil.rmtree('/global/u2/s/sjbailey/QuasarNP-0.1.0')
DEBUG:install.py:638:start_modules:2021-06-03T14:21:30: Initializing Modules with MODULESHOME=/opt/cray/pe/modules/3.2.11.4.
DEBUG:install.py:538:build_type:2021-06-03T14:21:30: Detected build type: py
DEBUG:install.py:700:install_module:2021-06-03T14:21:30: configure_module(QuasarNP, 0.1.0, working_dir=/global/u2/s/sjbailey/QuasarNP-0.1.0, dev=False)
DEBUG:install.py:720:install_module:2021-06-03T14:21:30: process_module('/global/common/software/desi/cori/desiconda/20200801-1.4.0-spec/code/desiutil/3.2.1/lib/python3.8/site-packages/desiutil/data/desiutil.module', self.module_keywords, '/global/cscratch1/sd/sjbailey/desi/test/modulefiles')
DEBUG:install.py:757:prepare_environment:2021-06-03T14:21:31: module('switch', 'QuasarNP/0.1.0')
DEBUG:install.py:538:build_type:2021-06-03T14:21:31: Detected build type: py
DEBUG:install.py:538:build_type:2021-06-03T14:21:31: Detected build type: py
DEBUG:install.py:802:install:2021-06-03T14:21:31: os.makedirs('/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib/python3.8/site-packages')
DEBUG:install.py:824:install:2021-06-03T14:21:31: /global/common/software/desi/cori/desiconda/20200801-1.4.0-spec/conda/bin/python -m pip install --no-deps --disable-pip-version-check --ignore-installed --prefix=/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0 .
DEBUG:install.py:833:install:2021-06-03T14:21:33: Processing /global/u2/s/sjbailey/QuasarNP-0.1.0
Building wheels for collected packages: quasarnp
  Building wheel for quasarnp (setup.py): started
  Building wheel for quasarnp (setup.py): finished with status 'done'
  Created wheel for quasarnp: filename=quasarnp-0.1.0-py3-none-any.whl size=13407 sha256=17d7dbdf89520f3a0e5d751edd4b7591b45563425a190550ff31c71e48b1b855
  Stored in directory: /global/u2/s/sjbailey/.cache/pip/wheels/b3/5d/19/08d052aecb141666e9fca7cef889a0c9393b18766c47f69300
Successfully built quasarnp
Installing collected packages: quasarnp
Successfully installed quasarnp-0.1.0

DEBUG:install.py:538:build_type:2021-06-03T14:21:33: Detected build type: py
DEBUG:install.py:538:build_type:2021-06-03T14:21:33: Detected build type: py
DEBUG:install.py:957:permissions:2021-06-03T14:21:33: fix_permissions.sh -v /global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0
DEBUG:install.py:962:permissions:2021-06-03T14:21:35: Fixing permissions on /global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0 ...
/usr/bin/find /global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0 -user sjbailey -not -group desi -exec chgrp -c -h desi {} ;
changed group of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0' from sjbailey to desi
changed group of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib' from sjbailey to desi
changed group of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib/python3.8' from sjbailey to desi
changed group of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib/python3.8/site-packages' from sjbailey to desi
changed group of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib/python3.8/site-packages/quasarnp' from sjbailey to desi
changed group of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib/python3.8/site-packages/quasarnp/model.py' from sjbailey to desi
changed group of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib/python3.8/site-packages/quasarnp/io.py' from sjbailey to desi
changed group of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib/python3.8/site-packages/quasarnp/__init__.py' from sjbailey to desi
changed group of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib/python3.8/site-packages/quasarnp/__pycache__' from sjbailey to desi
changed group of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib/python3.8/site-packages/quasarnp/__pycache__/layers.cpython-38.pyc' from sjbailey to desi
changed group of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib/python3.8/site-packages/quasarnp/__pycache__/model.cpython-38.pyc' from sjbailey to desi
changed group of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib/python3.8/site-packages/quasarnp/__pycache__/__init__.cpython-38.pyc' from sjbailey to desi
changed group of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib/python3.8/site-packages/quasarnp/__pycache__/utils.cpython-38.pyc' from sjbailey to desi
changed group of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib/python3.8/site-packages/quasarnp/__pycache__/io.cpython-38.pyc' from sjbailey to desi
changed group of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib/python3.8/site-packages/quasarnp/layers.py' from sjbailey to desi
changed group of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib/python3.8/site-packages/quasarnp/utils.py' from sjbailey to desi
changed group of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib/python3.8/site-packages/quasarnp-0.1.0.dist-info' from sjbailey to desi
changed group of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib/python3.8/site-packages/quasarnp-0.1.0.dist-info/INSTALLER' from sjbailey to desi
changed group of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib/python3.8/site-packages/quasarnp-0.1.0.dist-info/WHEEL' from sjbailey to desi
changed group of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib/python3.8/site-packages/quasarnp-0.1.0.dist-info/LICENSE' from sjbailey to desi
changed group of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib/python3.8/site-packages/quasarnp-0.1.0.dist-info/top_level.txt' from sjbailey to desi
changed group of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib/python3.8/site-packages/quasarnp-0.1.0.dist-info/METADATA' from sjbailey to desi
changed group of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib/python3.8/site-packages/quasarnp-0.1.0.dist-info/RECORD' from sjbailey to desi
/usr/bin/find /global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0 -user sjbailey -type f -not -perm /g+r -exec chmod -c g+r {} ;
/usr/bin/find /global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0 -user sjbailey -type d -not -perm -g+rxs -exec chmod -c g+rxs {} ;
mode of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0' changed from 0755 (rwxr-xr-x) to 2755 (rwxr-sr-x)
mode of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib' changed from 0755 (rwxr-xr-x) to 2755 (rwxr-sr-x)
mode of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib/python3.8' changed from 0755 (rwxr-xr-x) to 2755 (rwxr-sr-x)
mode of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib/python3.8/site-packages' changed from 0755 (rwxr-xr-x) to 2755 (rwxr-sr-x)
mode of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib/python3.8/site-packages/quasarnp' changed from 0755 (rwxr-xr-x) to 2755 (rwxr-sr-x)
mode of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib/python3.8/site-packages/quasarnp/__pycache__' changed from 0755 (rwxr-xr-x) to 2755 (rwxr-sr-x)
mode of '/global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0/lib/python3.8/site-packages/quasarnp-0.1.0.dist-info' changed from 0755 (rwxr-xr-x) to 2755 (rwxr-sr-x)

DEBUG:install.py:970:permissions:2021-06-03T14:21:35: chmod -R a-w /global/cscratch1/sd/sjbailey/desi/test/code/QuasarNP/0.1.0
DEBUG:install.py:975:permissions:2021-06-03T14:21:35: 
DEBUG:install.py:987:cleanup:2021-06-03T14:21:35: os.chdir('/global/u2/s/sjbailey')
DEBUG:install.py:991:cleanup:2021-06-03T14:21:35: shutil.rmtree('/global/u2/s/sjbailey/QuasarNP-0.1.0')
DEBUG:install.py:1024:run:2021-06-03T14:21:35: run() complete.
[cori06 ~] tail -15 $SCRATCH/desi/test/modulefiles/QuasarNP/0.1.0 
#
setenv [string toupper $product] $PRODUCT_DIR
#
# The lines below set various other environment variables.  They assume the
# template product layout.  These will be set or commented as needed by the
# desiInstall script.
#
# prepend-path PATH $PRODUCT_DIR/bin
# prepend-path PYTHONPATH $PRODUCT_DIR/lib/python3.8/site-packages
# prepend-path PYTHONPATH $PRODUCT_DIR/py
# prepend-path LD_LIBRARY_PATH $PRODUCT_DIR/lib
# prepend-path IDL_PATH +$PRODUCT_DIR/pro
#
# Add any non-standard Module code below this point.
#

[weaverba137]

@sbailey, is $SCRATCH/desi/test/modulefiles/... readable by me? Is $SCRATCH your home directory? If it is not possible for me to inspect this test install area, can you post the entire, generated module file, not just the last few lines?

[weaverba137]

@sbailey, never mind, I know exactly what is happening. This is a corner case we've never encountered before.

We have successfully installed software products that do not have a py/ directory before because the case of the name of the product matches the name of the Python package directory inside the product. For example, specsim and speclite follow this pattern. However, QuasarNP breaks this pattern.

The fastest possible way to fix this would be to rename the GitHub repo QuasarNP -> quasarnp.

[sbailey]

I dislike capitals in github product names, especially when that causes the python package to have a different case-sensitive name from its github product name, but I will defer to @dylanagreen and @dkirkby as the maintainers of that package for how strongly they feel about the capitalization, which I suspect they chose to mimic QuasarNET from which QuasarNP is adapted.

Either way, @weaverba137 I'll submit a PR with an additional check for this case.

[weaverba137]

OK, but keep in mind, that mixed-case to lower-case is not the only possible combination. So you have to check for all possible combinations of upper and lower case, and yet still retain the correct information about the name of the package and the name of the directory.

And, in principle, the alternate-case package name could be inside a py/ directory. So this is not as straightforward as it might seem.

Still, if you're able to add unit tests for all these possible cases (in both senses of the word), then we can be reasonably confident going forward.

Finally, the check for these directories happens in desiutil.modules.configure_module(), not in desiutil.install.