Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pathology/nuclick/nuclick_infer.ipynb #1542

Open
wyli opened this issue Oct 5, 2023 · 8 comments · Fixed by #1544
Open

pathology/nuclick/nuclick_infer.ipynb #1542

wyli opened this issue Oct 5, 2023 · 8 comments · Fixed by #1544
Assignees

Comments

@wyli
Copy link
Contributor

wyli commented Oct 5, 2023

https://warwick.ac.uk/fac/cross_fac/tia/data/hovernet/consep_dataset.zip not available

[2023-10-05T01:48:32.818Z] Running ./pathology/nuclick/nuclick_infer.ipynb
[2023-10-05T01:48:32.818Z] Checking PEP8 compliance...
[2023-10-05T01:48:33.381Z] Running notebook...
[2023-10-05T01:48:33.942Z] Before:
[2023-10-05T01:48:33.942Z]     "max_epochs = 1"
[2023-10-05T01:48:34.197Z] After:
[2023-10-05T01:48:34.197Z]     "max_epochs = 1"
[2023-10-05T01:48:44.136Z] MONAI version: 1.3.0rc2+14.g141bcf0a
[2023-10-05T01:48:44.136Z] Numpy version: 1.22.2
[2023-10-05T01:48:44.136Z] Pytorch version: 2.1.0a0+29c30b1
[2023-10-05T01:48:44.136Z] MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
[2023-10-05T01:48:44.136Z] MONAI rev id: 141bcf0a20c46c505cdd1f722bdd5cb5c856033d
[2023-10-05T01:48:44.136Z] MONAI __file__: /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/__init__.py
[2023-10-05T01:48:44.136Z] 
[2023-10-05T01:48:44.136Z] Optional dependencies:
[2023-10-05T01:48:44.136Z] Pytorch Ignite version: 0.4.11
[2023-10-05T01:48:44.136Z] ITK version: 5.3.0
[2023-10-05T01:48:44.136Z] Nibabel version: 5.1.0
[2023-10-05T01:48:44.136Z] scikit-image version: 0.22.0
[2023-10-05T01:48:44.136Z] scipy version: 1.11.1
[2023-10-05T01:48:44.136Z] Pillow version: 9.2.0
[2023-10-05T01:48:44.136Z] Tensorboard version: 2.9.0
[2023-10-05T01:48:44.136Z] gdown version: 4.7.1
[2023-10-05T01:48:44.136Z] TorchVision version: 0.16.0a0
[2023-10-05T01:48:44.136Z] tqdm version: 4.65.0
[2023-10-05T01:48:44.136Z] lmdb version: 1.4.1
[2023-10-05T01:48:44.136Z] psutil version: 5.9.4
[2023-10-05T01:48:44.136Z] pandas version: 1.5.2
[2023-10-05T01:48:44.136Z] einops version: 0.6.1
[2023-10-05T01:48:44.136Z] transformers version: 4.21.3
[2023-10-05T01:48:44.136Z] mlflow version: 2.7.1
[2023-10-05T01:48:44.136Z] pynrrd version: 1.0.0
[2023-10-05T01:48:44.136Z] clearml version: 1.13.1
[2023-10-05T01:48:44.136Z] 
[2023-10-05T01:48:44.136Z] For details about installing the optional dependencies, please visit:
[2023-10-05T01:48:44.136Z]     https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
[2023-10-05T01:48:44.136Z] 
[2023-10-05T01:48:45.500Z] papermill  --progress-bar -k python3
[2023-10-05T01:48:46.060Z] /usr/local/lib/python3.10/dist-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2023-10-05T01:48:46.060Z]   warnings.warn(
[2023-10-05T01:49:00.131Z] 
Executing:   0%|          | 0/18 [00:00<?, ?cell/s]
Executing:   6%|▌         | 1/18 [00:01<00:21,  1.25s/cell]
Executing:  17%|█▋        | 3/18 [00:02<00:09,  1.57cell/s]
Executing:  28%|██▊       | 5/18 [00:07<00:22,  1.77s/cell]
Executing:  44%|████▍     | 8/18 [00:10<00:12,  1.25s/cell]
Executing:  50%|█████     | 9/18 [00:10<00:09,  1.03s/cell]
Executing:  67%|██████▋   | 12/18 [00:12<00:05,  1.09cell/s]
Executing:  67%|██████▋   | 12/18 [00:14<00:07,  1.18s/cell]
[2023-10-05T01:49:00.387Z] /usr/local/lib/python3.10/dist-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2023-10-05T01:49:00.387Z]   warnings.warn(
[2023-10-05T01:49:00.387Z] Traceback (most recent call last):
[2023-10-05T01:49:00.387Z]   File "/usr/local/bin/papermill", line 8, in <module>
[2023-10-05T01:49:00.387Z]     sys.exit(papermill())
[2023-10-05T01:49:00.387Z]   File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1157, in __call__
[2023-10-05T01:49:00.387Z]     return self.main(*args, **kwargs)
[2023-10-05T01:49:00.387Z]   File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1078, in main
[2023-10-05T01:49:00.387Z]     rv = self.invoke(ctx)
[2023-10-05T01:49:00.387Z]   File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1434, in invoke
[2023-10-05T01:49:00.387Z]     return ctx.invoke(self.callback, **ctx.params)
[2023-10-05T01:49:00.387Z]   File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 783, in invoke
[2023-10-05T01:49:00.387Z]     return __callback(*args, **kwargs)
[2023-10-05T01:49:00.387Z]   File "/usr/local/lib/python3.10/dist-packages/click/decorators.py", line 33, in new_func
[2023-10-05T01:49:00.387Z]     return f(get_current_context(), *args, **kwargs)
[2023-10-05T01:49:00.387Z]   File "/usr/local/lib/python3.10/dist-packages/papermill/cli.py", line 250, in papermill
[2023-10-05T01:49:00.387Z]     execute_notebook(
[2023-10-05T01:49:00.387Z]   File "/usr/local/lib/python3.10/dist-packages/papermill/execute.py", line 128, in execute_notebook
[2023-10-05T01:49:00.387Z]     raise_for_execution_errors(nb, output_path)
[2023-10-05T01:49:00.387Z]   File "/usr/local/lib/python3.10/dist-packages/papermill/execute.py", line 232, in raise_for_execution_errors
[2023-10-05T01:49:00.387Z]     raise error
[2023-10-05T01:49:00.387Z] papermill.exceptions.PapermillExecutionError: 
[2023-10-05T01:49:00.387Z] ---------------------------------------------------------------------------
[2023-10-05T01:49:00.387Z] Exception encountered at "In [7]":
[2023-10-05T01:49:00.387Z] ---------------------------------------------------------------------------
[2023-10-05T01:49:00.387Z] BadZipFile                                Traceback (most recent call last)
[2023-10-05T01:49:00.387Z] Cell In[7], line 5
[2023-10-05T01:49:00.387Z]       2 consep_dir = os.path.join(workspace_path, "CoNSeP")
[2023-10-05T01:49:00.387Z]       4 if not os.path.exists(consep_dir):
[2023-10-05T01:49:00.387Z] ----> 5     download_and_extract(consep_zip, output_dir=workspace_path)
[2023-10-05T01:49:00.387Z] 
[2023-10-05T01:49:00.387Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/apps/utils.py:329, in download_and_extract(url, filepath, output_dir, hash_val, hash_type, file_type, has_base, progress)
[2023-10-05T01:49:00.387Z]     327 filename = filepath or Path(tmp_dir, _basename(url)).resolve()
[2023-10-05T01:49:00.387Z]     328 download_url(url=url, filepath=filename, hash_val=hash_val, hash_type=hash_type, progress=progress)
[2023-10-05T01:49:00.387Z] --> 329 extractall(filepath=filename, output_dir=output_dir, file_type=file_type, has_base=has_base)
[2023-10-05T01:49:00.387Z] 
[2023-10-05T01:49:00.387Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/apps/utils.py:283, in extractall(filepath, output_dir, hash_val, hash_type, file_type, has_base)
[2023-10-05T01:49:00.387Z]     281 _file_type = file_type.lower().strip()
[2023-10-05T01:49:00.387Z]     282 if filepath.name.endswith("zip") or _file_type == "zip":
[2023-10-05T01:49:00.387Z] --> 283     zip_file = zipfile.ZipFile(filepath)
[2023-10-05T01:49:00.387Z]     284     zip_file.extractall(output_dir)
[2023-10-05T01:49:00.387Z]     285     zip_file.close()
[2023-10-05T01:49:00.387Z] 
[2023-10-05T01:49:00.387Z] File /usr/lib/python3.10/zipfile.py:1269, in ZipFile.__init__(self, file, mode, compression, allowZip64, compresslevel, strict_timestamps)
[2023-10-05T01:49:00.387Z]    1267 try:
[2023-10-05T01:49:00.387Z]    1268     if mode == 'r':
[2023-10-05T01:49:00.387Z] -> 1269         self._RealGetContents()
[2023-10-05T01:49:00.387Z]    1270     elif mode in ('w', 'x'):
[2023-10-05T01:49:00.387Z]    1271         # set the modified flag so central directory gets written
[2023-10-05T01:49:00.387Z]    1272         # even if no files are added to the archive
[2023-10-05T01:49:00.387Z]    1273         self._didModify = True
[2023-10-05T01:49:00.387Z] 
[2023-10-05T01:49:00.387Z] File /usr/lib/python3.10/zipfile.py:1336, in ZipFile._RealGetContents(self)
[2023-10-05T01:49:00.387Z]    1334     raise BadZipFile("File is not a zip file")
[2023-10-05T01:49:00.387Z]    1335 if not endrec:
[2023-10-05T01:49:00.387Z] -> 1336     raise BadZipFile("File is not a zip file")
[2023-10-05T01:49:00.387Z]    1337 if self.debug > 1:
[2023-10-05T01:49:00.387Z]    1338     print(endrec)
[2023-10-05T01:49:00.387Z] 
[2023-10-05T01:49:00.387Z] BadZipFile: File is not a zip file
@KumoLiu KumoLiu self-assigned this Oct 7, 2023
@KumoLiu
Copy link
Contributor

KumoLiu commented Oct 7, 2023

The data doesn't appear to be accessible to the public. Can we skip the tutorials on this data and just add some comments in the tutorial?

@wyli
Copy link
Contributor Author

wyli commented Oct 7, 2023

sure, please help skip the tests for now, and if the datasets are permanently removed, we'll need to redesign the tutorials.

@KumoLiu
Copy link
Contributor

KumoLiu commented Oct 8, 2023

I just noticed this issue Project-MONAI/MONAI#5412.
Do we need to confirm with Warwick University, Tissue Image Analytics (TIA) Centre again?

cc @drbeh

@drbeh
Copy link
Member

drbeh commented Oct 8, 2023

I checked their website and it seems that this dataset is temporarily not available!
image

I will check with the Warwick university to see what's going on.
Meanwhile, we should disable the check. Anyways, we should think of creating a more reliable mirror for this dataset.

@KumoLiu
Copy link
Contributor

KumoLiu commented Oct 8, 2023

Thanks for the confirmation. Sure, I will skip the test for now. If there is no problem with the license, we may be able to put it in extra data. I remember that the zip size is only 154 MB.

KumoLiu added a commit to KumoLiu/tutorials that referenced this issue Oct 8, 2023
Signed-off-by: KumoLiu <[email protected]>
@KumoLiu KumoLiu mentioned this issue Oct 8, 2023
5 tasks
@wyli wyli closed this as completed in #1544 Oct 8, 2023
wyli pushed a commit that referenced this issue Oct 8, 2023
Fixes #1542.

### Description
Temporarily skip the tutorials related to the CoNSep dataset.
After confirming with the Warwick University, [Tissue Image Analytics
(TIA) Centre](https://warwick.ac.uk/fac/cross_fac/tia/), may remove the
dataset to extra data repo or google drive.

### Checks
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [ ] Avoid including large-size files in the PR.
- [ ] Clean up long text outputs from code cells in the notebook.
- [ ] For security purposes, please check the contents and remove any
sensitive info such as user names and private key.
- [ ] Ensure (1) hyperlinks and markdown anchors are working (2) use
relative paths for tutorial repo files (3) put figure and graphs in the
`./figure` folder
- [ ] Notebook runs automatically `./runner.sh -t <path to .ipynb file>`

Signed-off-by: KumoLiu <[email protected]>
@KumoLiu
Copy link
Contributor

KumoLiu commented Oct 9, 2023

Just skip the tests for now at #1544.
May need to have another update after checking with Warwick university.

@KumoLiu KumoLiu reopened this Oct 9, 2023
@KumoLiu
Copy link
Contributor

KumoLiu commented Nov 9, 2023

Hi @drbeh, have you had a chance to get in touch with Warwick University and can we move the data?
Thanks!

@drbeh
Copy link
Member

drbeh commented Nov 9, 2023

Hi @KumoLiu, I have talked to them a while ago about the CoNSeP dataset but it was not clear what was the plan for this dataset. @shaneahmed do you have any update here? Is this dataset going to be available on TIA homepage, or should we make a mirror for this?

KumoLiu added a commit to KumoLiu/tutorials that referenced this issue Apr 16, 2024
boneseva pushed a commit to boneseva/MONAI-tutorials that referenced this issue Apr 21, 2024
Fixes Project-MONAI#1542.

### Description
Temporarily skip the tutorials related to the CoNSep dataset.
After confirming with the Warwick University, [Tissue Image Analytics
(TIA) Centre](https://warwick.ac.uk/fac/cross_fac/tia/), may remove the
dataset to extra data repo or google drive.

### Checks
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [ ] Avoid including large-size files in the PR.
- [ ] Clean up long text outputs from code cells in the notebook.
- [ ] For security purposes, please check the contents and remove any
sensitive info such as user names and private key.
- [ ] Ensure (1) hyperlinks and markdown anchors are working (2) use
relative paths for tutorial repo files (3) put figure and graphs in the
`./figure` folder
- [ ] Notebook runs automatically `./runner.sh -t <path to .ipynb file>`

Signed-off-by: KumoLiu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants