Inconsistent behavior when using tox

[b1quint]

The tests in GitHub Actions are failing while the same tests pass when using Jenkin. You can see the traceback from the latest errors in GitHub Actions here. The traceback itself is a bit obscure.

When calling tox locally, the tests also fail. After trying several options, I realized that they fail when tox is called without any arguments. For example:

$ tox -e py37-unit                                                                                                                                                                                                           

Sometimes the error is the same reported on GitHub Actions, but most of the time the test failures have a different traceback. I am not completely sure that both issues are the same, if they are correlated somehow, or if they are simply two different issues.

The major requirement for our Tox setup is that we can run the tests using the command line above locally. So I will focus on this particular issue. The main error shows as:

_ ERROR collecting .tox/py37-unit/lib/python3.7/site-packages/recipe_system/cal_service/tests/test_caldb.py _
../../.tox/py37-unit/lib/python3.7/site-packages/recipe_system/cal_service/tests/test_caldb.py:12: in <module>
    from geminidr.gmos.primitives_gmos_longslit import GMOSLongslit
<frozen importlib._bootstrap>:983: in _find_and_load
    ???
<frozen importlib._bootstrap>:967: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:677: in _load_unlocked
    ???
../../.tox/py37-unit/lib/python3.7/site-packages/_pytest/assertion/rewrite.py:170: in exec_module
    exec(co, module.__dict__)
../../.tox/py37-unit/lib/python3.7/site-packages/geminidr/gmos/primitives_gmos_longslit.py:21: in <module>
    from geminidr.core.primitives_spect import _transpose_if_needed
<frozen importlib._bootstrap>:983: in _find_and_load
    ???
<frozen importlib._bootstrap>:967: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:677: in _load_unlocked
    ???
../../.tox/py37-unit/lib/python3.7/site-packages/_pytest/assertion/rewrite.py:170: in exec_module
    exec(co, module.__dict__)
../../.tox/py37-unit/lib/python3.7/site-packages/geminidr/core/__init__.py:8: in <module>
    from .primitives_spect import Spect
<frozen importlib._bootstrap>:983: in _find_and_load
    ???
<frozen importlib._bootstrap>:967: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:677: in _load_unlocked
    ???
../../.tox/py37-unit/lib/python3.7/site-packages/_pytest/assertion/rewrite.py:170: in exec_module
    exec(co, module.__dict__)
../../.tox/py37-unit/lib/python3.7/site-packages/geminidr/core/primitives_spect.py:42: in <module>
    from geminidr.interactive.fit.aperture import interactive_find_source_apertures
<frozen importlib._bootstrap>:983: in _find_and_load
    ???
<frozen importlib._bootstrap>:967: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:677: in _load_unlocked
    ???
../../.tox/py37-unit/lib/python3.7/site-packages/_pytest/assertion/rewrite.py:170: in exec_module
    exec(co, module.__dict__)
../../.tox/py37-unit/lib/python3.7/site-packages/geminidr/interactive/fit/aperture.py:4: in <module>
    import holoviews as hv
../../.tox/py37-unit/lib/python3.7/site-packages/holoviews/__init__.py:9: in <module>
    reponame="holoviews"))
../../.tox/py37-unit/lib/python3.7/site-packages/param/version.py:278: in __str__
    known_stale = self._known_stale()
../../.tox/py37-unit/lib/python3.7/site-packages/param/version.py:223: in _known_stale
    commit = self.commit
../../.tox/py37-unit/lib/python3.7/site-packages/param/version.py:133: in commit
    return self.fetch()._commit
../../.tox/py37-unit/lib/python3.7/site-packages/param/version.py:162: in fetch
    self.git_fetch(cmd)
../../.tox/py37-unit/lib/python3.7/site-packages/param/version.py:212: in git_fetch
    self._update_from_vcs(output)
../../.tox/py37-unit/lib/python3.7/site-packages/param/version.py:259: in _update_from_vcs
    self._commit_count = int(split[1])
E   ValueError: invalid literal for int() with base 10: 'dev'

The full traceback is here:
py37-unit.log

[b1quint]

This error is being caused by holoviews param. To be more specific, the param.version._update_from_vcs tries to update holoviews (not sure here) from GitHub using the latest Tag and commit hash. The expected tag-hash format within this function is not compatible with our current 3.0.0-dev+####### tag (###### is the commit hash).

It turns out that, although our tag does follow the standard semantic versioning, the holoviews.param package does not allow pre-release versions different from alpha, beta, rc.

We can either adapt to their versioning scheme or I can try to submit a PR fixing this.

[b1quint]

It turns out that their version template follows PEP440. @KathleenLabrie Could we follow PEP440 as well? If not, I could submit a Pull Request, but it would slow us down quite a bit since it would require some discussion on something that seems pretty well established for them.

For example, our git describe is currently returning something like this: v3.0.0-dev-1232-gd60b9ff98. Could we try to change v3.0.0-dev to v3.0.0.dev and see if this solves this issue?

[b1quint]

I tried to add a new v3.0.0.dev tag to my local branch, and I got a similar issue. Thinking a bit better, it does not make sense to me how an imported package extracts its version from our git repository. This seems wrong. Now, how should I fix this?

I can import holoviews using conda normally. This error only happens from within tox. So what are we doing wrong?

[saimn]

I don't think this has anything to do with tox. holoviews is trying to get its own version number from git if you use a git checkout (many packages do the same, e.g. astropy and others use setuptools_scm for this), but holoviews/param is doing something very wrong if it happens to read DRAGONS' version instead (there is no reason for holoviews/param to do that).
For why it happens only in some environments, it may have something to do with the way Github Actions / Jenkins checkout the code from git. Remember the difficulties we had to find the current branch.

[b1quint]

It is true that holoviews should not try to get its own version number from git, but:

1. it only works in some circumstances, and
2. I doubt that they will do something because of these particular cases.

Yesterday, I stepped back and started to isolate the problem. The following command runs the test locally successfully:

$ python -m coverage run -m pytest -v --dragons-remote-data --durations=20 -m "not integration_test and not gmosls and not regression and not slow" astrodata geminidr gemini_instruments gempy recipe_system

This is the command called within Tox to run the tests. When calling from Tox, they fail with the traceback above. When called from the command line, they pass (actually they fail due to the lack of input files, but this is expected). So there is something within Tox that is causing holoviews to check its version via git describe. The question is what? And how can we modify our tox file to prevent this behavior?

[b1quint]

Now GitHub Actions seems happy with it.

I did not do anything. In any case, this behavior still shows up when I run Tox on my machine but not when running with the command above.

[saimn]

param's version detection is just broken : holoviz/param#468

[b1quint]

I will leave this issue open until the release holoviews v1.10.2 since it affects our documentation. At least, it is not affecting Jenkins/Github Actions.