Skip to content

Commit

Permalink
Merge branch 'representation' into evaluate_computed_features
Browse files Browse the repository at this point in the history
  • Loading branch information
ziw-liu committed Sep 27, 2024
2 parents 3e215b5 + 17a2e48 commit f7160c9
Show file tree
Hide file tree
Showing 47 changed files with 13,543 additions and 1,531 deletions.
5 changes: 4 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -42,5 +42,8 @@ coverage.xml
.hypothesis/
.pytest_cache/

# SLURM
slurm*.out

#lightning_logs directory
lightning_logs/
lightning_logs/
126 changes: 66 additions & 60 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,87 +1,94 @@
# VisCy

VisCy is a deep learning pipeline for training and deploying computer vision models for image-based phenotyping at single-cell resolution.

The following methods are being developed:
VisCy (abbreviation of `vision` and `cyto`) is a deep learning pipeline for training and deploying computer vision models for image-based phenotyping at single-cell resolution.

This repository provides a pipeline for the following.
- Image translation
- Robust virtual staining of landmark organelles
- Image classification
- Supervised learning of of cell state (e.g. state of infection)
- Image representation learning
- Self-supervised learning of the cell state and organelle phenotypes

<div style="border: 2px solid orange; padding: 10px; border-radius: 5px; background-color: #fff8e1;">
<strong>Note:</strong><br>
VisCy is currently considered alpha software and is under active development. Frequent breaking changes are expected.
</div>
> **Note:**
> VisCy has been extensively tested for the image translation task. The code for other tasks is under active development. Frequent breaking changes are expected in the main branch as we unify the codebase for above tasks. If you are looking for a well-tested version for virtual staining, please use release `0.2.1` from PyPI.
## Virtual staining
### Pipeline
A full illustration of the virtual staining pipeline can be found [here](docs/virtual_staining.md).

### Library of virtual staining (VS) models
The robust virtual staining models (i.e *VSCyto2D*, *VSCyto3D*, *VSNeuromast*), and fine-tuned models can be found [here](https://github.com/mehta-lab/VisCy/wiki/Library-of-virtual-staining-(VS)-Models)
## Virtual staining

### Demos
#### Image-to-Image translation using VisCy
- [Guide for Virtual Staining Models](https://github.com/mehta-lab/VisCy/wiki/virtual-staining-instructions):
Instructions for how to train and run inference on ViSCy's virtual staining models (*VSCyto3D*, *VSCyto2D* and *VSNeuromast*)
- [Virtual staining exercise](https://github.com/mehta-lab/VisCy/blob/46beba4ecc8c4f312fda0b04d5229631a41b6cb5/examples/virtual_staining/dlmbl_exercise/solution.ipynb):
Notebook illustrating how to use VisCy to train, predict and evaluate the VSCyto2D model. This notebook was developed for the [DL@MBL2024](https://github.com/dlmbl/DL-MBL-2024) course and uses UNeXt2 architecture.

- [Image translation Exercise](./dlmbl_exercise/solution.py):
Example showing how to use VisCy to train, predict and evaluate the VSCyto2D model. This notebook was developed for the [DL@MBL2024](https://github.com/dlmbl/DL-MBL-2024) course.
- [Image translation demo](https://github.com/mehta-lab/VisCy/blob/92215bc1387316f3af49c83c321b9d134d871116/examples/virtual_staining/img2img_translation/solution.ipynb): Fluorescence images can be predicted from label-free images. Can we predict label-free image from fluorescence? Find out using this notebook.

- [Virtual staining exercise](./img2img_translation/solution.py): exploring the label-free to fluorescence virtual staining and florescence to label-free image translation task using VisCy UneXt2.
More usage examples and demos can be found [here](https://github.com/mehta-lab/VisCy/blob/b7af9687c6409c738731ea47f66b74db2434443c/examples/virtual_staining/README.md)
- [Training Virtual Staining Models via CLI](https://github.com/mehta-lab/VisCy/wiki/virtual-staining-instructions):
Instructions for how to train and run inference on ViSCy's virtual staining models (*VSCyto3D*, *VSCyto2D* and *VSNeuromast*).

### Gallery
Below are some examples of virtually stained images (click to play videos).
See the full gallery [here](https://github.com/mehta-lab/VisCy/wiki/Gallery).

| VSCyto3D | VSNeuromast | VSCyto2D |
|:---:|:---:|:---:|
| [![HEK293T](docs/figures/svideo_1.png)](https://github.com/mehta-lab/VisCy/assets/67518483/d53a81eb-eb37-44f3-b522-8bd7bddc7755) | [![Neuromast](docs/figures/svideo_3.png)](https://github.com/mehta-lab/VisCy/assets/67518483/4cef8333-895c-486c-b260-167debb7fd64) | [![A549](docs/figures/svideo_5.png)](https://github.com/mehta-lab/VisCy/assets/67518483/287737dd-6b74-4ce3-8ee5-25fbf8be0018) |
| [![HEK293T](https://github.com/mehta-lab/VisCy/blob/dde3e27482e58a30f7c202e56d89378031180c75/docs/figures/svideo_1.png?raw=true)](https://github.com/mehta-lab/VisCy/assets/67518483/d53a81eb-eb37-44f3-b522-8bd7bddc7755) | [![Neuromast](https://github.com/mehta-lab/VisCy/blob/dde3e27482e58a30f7c202e56d89378031180c75/docs/figures/svideo_3.png?raw=true)](https://github.com/mehta-lab/VisCy/assets/67518483/4cef8333-895c-486c-b260-167debb7fd64) | [![A549](https://github.com/mehta-lab/VisCy/blob/dde3e27482e58a30f7c202e56d89378031180c75/docs/figures/svideo_5.png?raw=true)](https://github.com/mehta-lab/VisCy/assets/67518483/287737dd-6b74-4ce3-8ee5-25fbf8be0018) |

### Reference

The virtual staining models and training protocols are reported in our recent [preprint on robust virtual staining](https://www.biorxiv.org/content/10.1101/2024.05.31.596901):

```bibtex
@article {Liu2024.05.31.596901,
author = {Liu, Ziwen and Hirata-Miyasaki, Eduardo and Pradeep, Soorya and Rahm, Johanna and Foley, Christian and Chandler, Talon and Ivanov, Ivan and Woosley, Hunter and Lao, Tiger and Balasubramanian, Akilandeswari and Liu, Chad and Leonetti, Manu and Arias, Carolina and Jacobo, Adrian and Mehta, Shalin B.},
title = {Robust virtual staining of landmark organelles},
elocation-id = {2024.05.31.596901},
year = {2024},
doi = {10.1101/2024.05.31.596901},
publisher = {Cold Spring Harbor Laboratory},
URL = {https://www.biorxiv.org/content/early/2024/06/03/2024.05.31.596901},
eprint = {https://www.biorxiv.org/content/early/2024/06/03/2024.05.31.596901.full.pdf},
journal = {bioRxiv}
}
```

This package evolved from the [TensorFlow version of virtual staining pipeline](https://github.com/mehta-lab/microDL), which we reported in [this paper in 2020](https://elifesciences.org/articles/55502):

```bibtex
@article {10.7554/eLife.55502,
article_type = {journal},
title = {Revealing architectural order with quantitative label-free imaging and deep learning},
author = {Guo, Syuan-Ming and Yeh, Li-Hao and Folkesson, Jenny and Ivanov, Ivan E and Krishnan, Anitha P and Keefe, Matthew G and Hashemi, Ezzat and Shin, David and Chhun, Bryant B and Cho, Nathan H and Leonetti, Manuel D and Han, May H and Nowakowski, Tomasz J and Mehta, Shalin B},
editor = {Forstmann, Birte and Malhotra, Vivek and Van Valen, David},
volume = 9,
year = 2020,
month = {jul},
pub_date = {2020-07-27},
pages = {e55502},
citation = {eLife 2020;9:e55502},
doi = {10.7554/eLife.55502},
url = {https://doi.org/10.7554/eLife.55502},
keywords = {label-free imaging, inverse algorithms, deep learning, human tissue, polarization, phase},
journal = {eLife},
issn = {2050-084X},
publisher = {eLife Sciences Publications, Ltd},
}
```
The virtual staining models and training protocols are reported in our recent [preprint on robust virtual staining](https://www.biorxiv.org/content/10.1101/2024.05.31.596901).


This package evolved from the [TensorFlow version of virtual staining pipeline](https://github.com/mehta-lab/microDL), which we reported in [this paper in 2020](https://elifesciences.org/articles/55502).

<details>
<summary>Liu, Hirata-Miyasaki et al., 2024</summary>

<pre><code>
@article {Liu2024.05.31.596901,
author = {Liu, Ziwen and Hirata-Miyasaki, Eduardo and Pradeep, Soorya and Rahm, Johanna and Foley, Christian and Chandler, Talon and Ivanov, Ivan and Woosley, Hunter and Lao, Tiger and Balasubramanian, Akilandeswari and Liu, Chad and Leonetti, Manu and Arias, Carolina and Jacobo, Adrian and Mehta, Shalin B.},
title = {Robust virtual staining of landmark organelles},
elocation-id = {2024.05.31.596901},
year = {2024},
doi = {10.1101/2024.05.31.596901},
publisher = {Cold Spring Harbor Laboratory},
URL = {https://www.biorxiv.org/content/early/2024/06/03/2024.05.31.596901},
eprint = {https://www.biorxiv.org/content/early/2024/06/03/2024.05.31.596901.full.pdf},
journal = {bioRxiv}
}
</code></pre>
</details>

<details>
<summary>Guo, Yeh, Folkesson et al., 2020</summary>

<pre><code>
@article {10.7554/eLife.55502,
article_type = {journal},
title = {Revealing architectural order with quantitative label-free imaging and deep learning},
author = {Guo, Syuan-Ming and Yeh, Li-Hao and Folkesson, Jenny and Ivanov, Ivan E and Krishnan, Anitha P and Keefe, Matthew G and Hashemi, Ezzat and Shin, David and Chhun, Bryant B and Cho, Nathan H and Leonetti, Manuel D and Han, May H and Nowakowski, Tomasz J and Mehta, Shalin B},
editor = {Forstmann, Birte and Malhotra, Vivek and Van Valen, David},
volume = 9,
year = 2020,
month = {jul},
pub_date = {2020-07-27},
pages = {e55502},
citation = {eLife 2020;9:e55502},
doi = {10.7554/eLife.55502},
url = {https://doi.org/10.7554/eLife.55502},
keywords = {label-free imaging, inverse algorithms, deep learning, human tissue, polarization, phase},
journal = {eLife},
issn = {2050-084X},
publisher = {eLife Sciences Publications, Ltd},
}
</code></pre>
</details>

### Library of virtual staining (VS) models
The robust virtual staining models (i.e *VSCyto2D*, *VSCyto3D*, *VSNeuromast*), and fine-tuned models can be found [here](https://github.com/mehta-lab/VisCy/wiki/Library-of-virtual-staining-(VS)-Models)

### Pipeline
A full illustration of the virtual staining pipeline can be found [here](https://github.com/mehta-lab/VisCy/blob/dde3e27482e58a30f7c202e56d89378031180c75/docs/virtual_staining.md).


## Installation

Expand Down Expand Up @@ -118,8 +125,7 @@ publisher = {eLife Sciences Publications, Ltd},
viscy --help
```

## Contributing
For development installation, see [the contributing guide](CONTRIBUTING.md).
For development installation, see [the contributing guide](https://github.com/mehta-lab/VisCy/blob/main/CONTRIBUTING.md).

## Additional Notes
The pipeline is built using the [PyTorch Lightning](https://www.pytorchlightning.ai/index.html) framework.
Expand Down
113 changes: 113 additions & 0 deletions applications/contrastive_phenotyping/contrastive_cli/fit_ctc_mps.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
# See help here on how to configure hyper-parameters with config files:
# https://lightning.ai/docs/pytorch/stable/cli/lightning_cli_advanced.html
seed_everything: 42
trainer:
accelerator: gpu
strategy: auto
devices: 1
num_nodes: 1
precision: 32-true
logger:
class_path: lightning.pytorch.loggers.TensorBoardLogger
# Nesting the logger config like this is equivalent to
# supplying the following argument to `lightning.pytorch.Trainer`:
# logger=TensorBoardLogger(
# "/hpc/projects/intracellular_dashboard/viral-sensor/infection_classification/models/contrastive_tune_augmentations",
# log_graph=True,
# version="vanilla",
# )
init_args:
save_dir: /Users/ziwen.liu/Projects/test-time
# this is the name of the experiment.
# The logs will be saved in `save_dir/lightning_logs/version`
version: time_interval_1
log_graph: True
callbacks:
- class_path: lightning.pytorch.callbacks.LearningRateMonitor
init_args:
logging_interval: step
- class_path: lightning.pytorch.callbacks.ModelCheckpoint
init_args:
monitor: loss/val
every_n_epochs: 1
save_top_k: 4
save_last: true
fast_dev_run: false
max_epochs: 100
log_every_n_steps: 10
enable_checkpointing: true
inference_mode: true
use_distributed_sampler: true
# synchronize batchnorm parameters across multiple GPUs.
# important for contrastive learning to normalize the tensors across the whole batch.
sync_batchnorm: true
model:
encoder:
class_path: viscy.representation.contrastive.ContrastiveEncoder
init_args:
backbone: convnext_tiny
in_channels: 1
in_stack_depth: 1
stem_kernel_size: [1, 4, 4]
stem_stride: [1, 4, 4]
embedding_dim: 768
projection_dim: 32
drop_path_rate: 0.0
loss_function:
class_path: torch.nn.TripletMarginLoss
init_args:
margin: 0.5
lr: 0.0002
log_batches_per_epoch: 3
log_samples_per_batch: 2
example_input_array_shape: [1, 1, 1, 128, 128]
data:
data_path: /Users/ziwen.liu/Downloads/Hela_CTC.zarr
tracks_path: /Users/ziwen.liu/Downloads/Hela_CTC.zarr
source_channel:
- DIC
z_range: [0, 1]
batch_size: 16
num_workers: 4
initial_yx_patch_size: [256, 256]
final_yx_patch_size: [128, 128]
time_interval: 1
normalizations:
- class_path: viscy.transforms.NormalizeSampled
init_args:
keys: [DIC]
level: fov_statistics
subtrahend: mean
divisor: std
augmentations:
- class_path: viscy.transforms.RandAffined
init_args:
keys: [DIC]
prob: 0.8
scale_range: [0, 0.2, 0.2]
rotate_range: [3.14, 0.0, 0.0]
shear_range: [0.0, 0.01, 0.01]
padding_mode: zeros
- class_path: viscy.transforms.RandAdjustContrastd
init_args:
keys: [DIC]
prob: 0.5
gamma: [0.8, 1.2]
- class_path: viscy.transforms.RandScaleIntensityd
init_args:
keys: [DIC]
prob: 0.5
factors: 0.5
- class_path: viscy.transforms.RandGaussianSmoothd
init_args:
keys: [DIC]
prob: 0.5
sigma_x: [0.25, 0.75]
sigma_y: [0.25, 0.75]
sigma_z: [0.0, 0.0]
- class_path: viscy.transforms.RandGaussianNoised
init_args:
keys: [DIC]
prob: 0.5
mean: 0.0
std: 0.2
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
seed_everything: 42
trainer:
accelerator: gpu
strategy: auto
devices: auto
num_nodes: 1
precision: 32-true
callbacks:
- class_path: viscy.representation.embedding_writer.EmbeddingWriter
init_args:
output_path: /Users/ziwen.liu/Projects/test-time/predict/time_interval_1.zarr
inference_mode: true
model:
encoder:
class_path: viscy.representation.contrastive.ContrastiveEncoder
init_args:
backbone: convnext_tiny
in_channels: 1
in_stack_depth: 1
stem_kernel_size: [1, 4, 4]
stem_stride: [1, 4, 4]
embedding_dim: 768
projection_dim: 32
drop_path_rate: 0.0
example_input_array_shape: [1, 1, 1, 128, 128]
data:
data_path: /Users/ziwen.liu/Downloads/Hela_CTC.zarr
tracks_path: /Users/ziwen.liu/Downloads/Hela_CTC.zarr
source_channel: DIC
z_range: [0, 1]
batch_size: 16
num_workers: 4
initial_yx_patch_size: [128, 128]
final_yx_patch_size: [128, 128]
time_interval: 1
normalizations:
- class_path: viscy.transforms.NormalizeSampled
init_args:
keys: [DIC]
level: fov_statistics
subtrahend: mean
divisor: std
return_predictions: false
ckpt_path: /Users/ziwen.liu/Projects/test-time/lightning_logs/time_interval_1/checkpoints/last.ckpt
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
font.family: sans-serif
font.sans-serif: Arial
font.size: 10
figure.titlesize: 12
axes.titlesize: 10
xtick.labelsize: 8
ytick.labelsize: 8
text.usetex: True
Loading

0 comments on commit f7160c9

Please sign in to comment.