Install failure on CI

[quinnhornblow]

• whitebox version:
• Python version: 3.11.6
• Operating System: Linux

Description

I am trying to test a function that uses whitebox with pytest. Tests pass locally but when I run the tests on a CI pipeline (bitbucket) they fail. It appears to be an error with the download and setup. I have tried with different combinations of musl and reset but all fail. CI pipeline is run on linux.

whitebox.download_wbt(
    linux_musl=False,
    reset=False,
    verbose=True,
)

What I Did

See below for snip from CI output.

Downloading WhiteboxTools pre-compiled binary for first time use ...
Downloading WhiteboxTools binary from https://www.whiteboxgeo.com/WBT_Linux/WhiteboxTools_linux_amd64.zip
Decompressing WhiteboxTools_linux_amd64.zip ...
Unexpected error: <class 'EOFError'>

.venv/lib/python3.11/site-packages/_pytest/runner.py:341: in from_call
    result: Optional[TResult] = func()
.venv/lib/python3.11/site-packages/_pytest/runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
.venv/lib/python3.11/site-packages/_pytest/python.py:531: in collect
    self._inject_setup_module_fixture()
.venv/lib/python3.11/site-packages/_pytest/python.py:545: in _inject_setup_module_fixture
    self.obj, ("setUpModule", "setup_module")
.venv/lib/python3.11/site-packages/_pytest/python.py:310: in obj
    self._obj = obj = self._getobj()
.venv/lib/python3.11/site-packages/_pytest/python.py:528: in _getobj
    return self._importtestmodule()
.venv/lib/python3.11/site-packages/_pytest/python.py:617: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
.venv/lib/python3.11/site-packages/_pytest/pathlib.py:567: in import_path
    importlib.import_module(module_name)
/usr/local/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1204: in _gcd_import
    ???
<frozen importlib._bootstrap>:1176: in _find_and_load
    ???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:690: in _load_unlocked
    ???

.venv/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:186: in exec_module
    exec(co, module.__dict__)

    whitebox.download_wbt(
.venv/lib/python3.11/site-packages/whitebox/whitebox_tools.py:138: in download_wbt
    with zipfile.ZipFile(zip_name, "r") as zip_ref:
/usr/local/lib/python3.11/zipfile.py:1302: in __init__
    self._RealGetContents()
/usr/local/lib/python3.11/zipfile.py:1369: in _RealGetContents
    raise BadZipFile("File is not a zip file")
E   zipfile.BadZipFile: File is not a zip file

[giswqs]

Yes, this is a known issue. Downloaing WBT binaries automatically from whiteboxgeo.com sometimes gets a timeout error. That's whey I placed some backup links in download_wbt just to pass the CI tests.

whitebox-python/whitebox/download_wbt.py

Line 58 in 117f08e

backup_links = {

[quinnhornblow]

It's not a timeout error though... it happens here when unzipping the download.

I see that line 108 checks if zip_ext == ".zip": # Decompress Windows/Mac OS zip - but all of the links and backup links are .zip?

Also line 87 sets the zip_name from the url, then the url is potentially updated in line 99. I'm not sure what impact this has (if any) but why not set the zip_name after line 101?

[giswqs]

It works fine on Colab. The issue is probably caused by the corrupted file downloaded from whiteboxgeo.com when the download is not complete. It is not a valid zip file, so it throws errors when trying to unzip it.

[quinnhornblow]

Ok, so it sounds like you are aware of the issue and know what's causing it. It's not clear what your proposed solution is though. Since it's a timeout issue, does the timeout parameter in the urllib request need to be increased? or do I need to change something with my CI pipeline?

[giswqs]

The timeout vlaue is already set to 500 seconds, which should be long enough. The issue is probably because whiteboxgeo.com rejects automated download request from time to time. I don't know why.

Here is a previous discussion on this issue: giswqs/whitebox-bin#1

I don't have a solution to this. Suggestions are welcome.

[quinnhornblow]

Finally got to the bottom of this, and error is not what I originally thought.

I am running pytest on the CI pipeline, and it was configured with '-n logical' to run tests in parallel. This appears to cause failure because multiple tests are trying to download and setup the whitebox binary at once. Changing to '-n 1' so the tests run in sequence has fixed this issue for me.

[giswqs]

Glad to hear that you sorted things out.