From a557201e470557bad0c5f6c688aa7967a900a963 Mon Sep 17 00:00:00 2001 From: Niklas Schmacke Date: Fri, 29 Nov 2024 15:39:57 +0100 Subject: [PATCH 01/10] Updated walkthrough notebook headers --- .../notebooks/example_scPortrait_project.ipynb | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/pages/notebooks/example_scPortrait_project.ipynb b/docs/pages/notebooks/example_scPortrait_project.ipynb index 54a7ee48..2ddcc731 100644 --- a/docs/pages/notebooks/example_scPortrait_project.ipynb +++ b/docs/pages/notebooks/example_scPortrait_project.ipynb @@ -5,7 +5,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# A walk through the scPortrait Ecosystem\n", + "# A Walk Through The scPortrait Ecosystem\n", "\n", "This notebook will introduce you to the scPortrait ecosystem and give you a complete working example of how to use scPortrait. It will walk you through the following steps of the scPortrait workflow:\n", "\n", @@ -15,7 +15,7 @@ "\n", "3. **classification**: The image-based phenotype of each individual cell in the extracted single-cell dataset is classified using the specified classification method. Multiple classification runs can be performed on the same dataset using different classification methods. Here we utilize the pretrained binary classifier from the original [SPARCS manuscript](https://doi.org/10.1101/2023.06.01.542416) that identifies individual cells defective in a biological process called \"autophagy\". \n", "\n", - "4. **selection**: Cutting instructions for the isolation of selected individual cells by laser microdissection are generated. The cutting shapes are written to an ``.xml`` file which can be loaded on a leica LMD microscope for automated cell excision.\n", + "4. **selection**: Cutting instructions for the isolation of selected individual cells by laser microdissection are generated. The cutting shapes are written to an ``.xml`` file which can be loaded on a Leica LMD7 microscope for automated cell excision.\n", "\n", "The data used in this notebook was previously stitched using the stitching workflow in [SPARCStools](https://github.com/MannLabs/SPARCStools). Please see the notebook [here](https://mannlabs.github.io/SPARCStools/html/pages/notebooks/example_stitching_notebook.html)." ] @@ -1059,7 +1059,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### Looking at Segmentation Results\n", + "### Inspecting Segmentation Results\n", "\n", "The Segmentation Results are written to a hdf5 file called `segmentation.h5` located in the segmentation directory of our scPortrait project.\n", "\n", @@ -1180,7 +1180,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Extracting single-cell images\n", + "## Extracting single cell images\n", "\n", "Once we have generated a segmentation mask, the next step is to extract single-cell images of segmented cells in the dataset.\n", "\n", @@ -1295,7 +1295,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### Look at extracted single-cell images\n", + "### Look at extracted single cell images\n", "\n", "The extracted single-cell images are written to a h5py file `single_cells.h5` located under `extraction\\data` within the project folder.\n", "\n", @@ -1383,7 +1383,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Classification of extracted single-cells\n", + "## Classification of extracted single cells\n", "\n", "Next we can apply a pretained model to classify our cells within the scPortrait project. \n", "\n", @@ -1470,7 +1470,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### looking at the generated results\n", + "### Looking at the generated results\n", "\n", "The results are written to a csv file which we can load with pandas.\n", "\n", @@ -1663,7 +1663,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Exporting Cutting contours for excision on the LMD7\n", + "## Exporting Cutting contours for excision on a Leice LMD7\n", "\n", "scPortrait directly interfaces with our other open-source python library [py-lmd](https://github.com/MannLabs/py-lmd) to easily select and export cells for excision on a Leica LMD microscope. \n", "\n", From 6a54fd3882aebff8eeb3211441f58275913005a3 Mon Sep 17 00:00:00 2001 From: Niklas Schmacke Date: Sun, 1 Dec 2024 10:48:03 +0100 Subject: [PATCH 02/10] Capitalized workflow step names in walkthrough notebook --- docs/pages/notebooks/example_scPortrait_project.ipynb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/pages/notebooks/example_scPortrait_project.ipynb b/docs/pages/notebooks/example_scPortrait_project.ipynb index 2ddcc731..f92bb783 100644 --- a/docs/pages/notebooks/example_scPortrait_project.ipynb +++ b/docs/pages/notebooks/example_scPortrait_project.ipynb @@ -9,13 +9,13 @@ "\n", "This notebook will introduce you to the scPortrait ecosystem and give you a complete working example of how to use scPortrait. It will walk you through the following steps of the scPortrait workflow:\n", "\n", - "1. **segmentation**: Generates masks for the segmentation of input images into individual cells\n", + "1. **Segmentation**: Generates masks for the segmentation of input images into individual cells\n", "\n", - "2. **extraction**: The segmentation masks are applied to extract single-cell images for all cells in the input images. Images of individual cells are rescaled to [0, 1] per channel.\n", + "2. **Extraction**: The segmentation masks are applied to extract single-cell images for all cells in the input images. Images of individual cells are rescaled to [0, 1] per channel.\n", "\n", - "3. **classification**: The image-based phenotype of each individual cell in the extracted single-cell dataset is classified using the specified classification method. Multiple classification runs can be performed on the same dataset using different classification methods. Here we utilize the pretrained binary classifier from the original [SPARCS manuscript](https://doi.org/10.1101/2023.06.01.542416) that identifies individual cells defective in a biological process called \"autophagy\". \n", + "3. **Classification**: The image-based phenotype of each individual cell in the extracted single-cell dataset is classified using the specified classification method. Multiple classification runs can be performed on the same dataset using different classification methods. Here we utilize the pretrained binary classifier from the original [SPARCS manuscript](https://doi.org/10.1101/2023.06.01.542416) that identifies individual cells defective in a biological process called \"autophagy\". \n", "\n", - "4. **selection**: Cutting instructions for the isolation of selected individual cells by laser microdissection are generated. The cutting shapes are written to an ``.xml`` file which can be loaded on a Leica LMD7 microscope for automated cell excision.\n", + "4. **Selection**: Cutting instructions for the isolation of selected individual cells by laser microdissection are generated. The cutting shapes are written to an ``.xml`` file which can be loaded on a Leica LMD7 microscope for automated cell excision.\n", "\n", "The data used in this notebook was previously stitched using the stitching workflow in [SPARCStools](https://github.com/MannLabs/SPARCStools). Please see the notebook [here](https://mannlabs.github.io/SPARCStools/html/pages/notebooks/example_stitching_notebook.html)." ] From cd1957f101ef8b46c148e73172dadaa02e1f858a Mon Sep 17 00:00:00 2001 From: Niklas Schmacke Date: Sun, 1 Dec 2024 14:02:51 +0100 Subject: [PATCH 03/10] Markdown instead of comment in walkthrough notebook --- docs/pages/notebooks/example_scPortrait_project.ipynb | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/docs/pages/notebooks/example_scPortrait_project.ipynb b/docs/pages/notebooks/example_scPortrait_project.ipynb index f92bb783..e2f5c47d 100644 --- a/docs/pages/notebooks/example_scPortrait_project.ipynb +++ b/docs/pages/notebooks/example_scPortrait_project.ipynb @@ -775,14 +775,19 @@ "fig.tight_layout()" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Alternatively, you can also visualize the input images as well as all other objects saved in a spatialdata object" + ] + }, { "cell_type": "code", - "execution_count": 7, + "execution_count": null, "metadata": {}, "outputs": [], "source": [ - "# alternatively you can also visualize the input images as well as all other objects saved in spatialdata object\n", - "\n", "project.view_sdata()" ] }, From 275d35e6b0a6b9e40f8e681515aa7cbd199e918a Mon Sep 17 00:00:00 2001 From: Niklas Schmacke Date: Thu, 5 Dec 2024 11:36:28 +0100 Subject: [PATCH 04/10] Clarifications in stitching notebook --- docs/index.rst | 2 +- .../parsing/example_parsing_notebook.ipynb | 2 +- .../example_stitching_notebook.ipynb | 29 ++++++++++++------- 3 files changed, 20 insertions(+), 13 deletions(-) diff --git a/docs/index.rst b/docs/index.rst index f8064fe8..c7e73835 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -1,4 +1,4 @@ -scPortrait - image-based single cell analysis at scale in Python +scPortrait – image-based single cell analysis at scale in Python ================================================================ scPortrait is a scalable toolkit to analyse single-cell image datasets. This Python implementation efficiently segments individual cells, generates single-cell datasets and provides tools for the efficient deep learning classification of their phenotypes for downstream applications. diff --git a/docs/pages/tools/parsing/example_parsing_notebook.ipynb b/docs/pages/tools/parsing/example_parsing_notebook.ipynb index 7314f53f..d7b8165b 100644 --- a/docs/pages/tools/parsing/example_parsing_notebook.ipynb +++ b/docs/pages/tools/parsing/example_parsing_notebook.ipynb @@ -5,7 +5,7 @@ "id": "6fc618c5", "metadata": {}, "source": [ - "# Example Parsing Notebook to rename phenix experiments" + "# Example Notebook to parse and rename files from experiments imaged on an Opera Phenix microscope" ] }, { diff --git a/docs/pages/tools/stitching/example_stitching_notebook.ipynb b/docs/pages/tools/stitching/example_stitching_notebook.ipynb index 0b0098f8..beb95c24 100644 --- a/docs/pages/tools/stitching/example_stitching_notebook.ipynb +++ b/docs/pages/tools/stitching/example_stitching_notebook.ipynb @@ -8,6 +8,13 @@ "# Example Stitching Notebook" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "scPortrait uses [Ashlar](https://labsyspharm.github.io/ashlar/) for stitching images. When stitching from `.tif` files, Ashlar reads channel and tile position information from filenames according to a predefined `pattern`. Hence, filenames matter when stitching from `.tif` files." + ] + }, { "cell_type": "code", "execution_count": 1, @@ -43,12 +50,12 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### initializing the stitcher object" + "### Initializing the `Stitcher` object" ] }, { "cell_type": "code", - "execution_count": 3, + "execution_count": null, "metadata": {}, "outputs": [ { @@ -64,12 +71,12 @@ "slidename = \"stitching_test\"\n", "outdir = os.path.join(input_dir.replace(\"stitching_example\", \"example_projects/stitching\"), slidename)\n", "\n", - "row = str(2).zfill(2) # specify the row of the well you want to stitch\n", - "well = str(4).zfill(2) # specifc the well number you wish to stitch\n", + "row = str(2).zfill(2) # specify the row of the well you want to stitch, here = 2\n", + "well = str(4).zfill(2) # specifc the well number you wish to stitch, here = 4\n", "zstack_value = str(1).zfill(\n", " 3\n", ") # specify the zstack you want to stitch. for multiple zstacks please make a loop and iterate through each of them.\n", - "timepoint = str(1).zfill(3) # specifz the timepoint you wish to stitch\n", + "timepoint = str(1).zfill(3) # specify the timepoint you wish to stitch\n", "\n", "pattern = f\"Timepoint{timepoint}_Row{row}_Well{well}_{{channel}}_zstack{zstack_value}_r{{row:03}}_c{{col:03}}.tif\"\n", "\n", @@ -460,19 +467,19 @@ "source": [ "## Multi-threaded Stitching\n", "\n", - "Using the ParallelStitcher class stitching can be speed up by using multiple threads. The code to perform stitching remains more or less the same." + "Using the `ParallelStitcher` class stitching can be speed up by using multiple threads. The code to perform stitching remains the same, but `ParallelStitcher` takes an additional argument `threads`, specifying the number of parallel threads to use." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "### initializing the stitcher object" + "### Initializing the `ParallelStitcher` object" ] }, { "cell_type": "code", - "execution_count": 14, + "execution_count": null, "metadata": {}, "outputs": [ { @@ -489,12 +496,12 @@ "outdir_parallel = os.path.join(input_dir.replace(\"stitching_example\", \"example_projects/stitching\"), slidename)\n", "\n", "\n", - "row = str(2).zfill(2) # specify the row of the well you want to stitch\n", - "well = str(4).zfill(2) # specifc the well number you wish to stitch\n", + "row = str(2).zfill(2) # specify the row of the well you want to stitch, here = 2\n", + "well = str(4).zfill(2) # specifc the well number you wish to stitch, here = 4\n", "zstack_value = str(1).zfill(\n", " 3\n", ") # specify the zstack you want to stitch. for multiple zstacks please make a loop and iterate through each of them.\n", - "timepoint = str(1).zfill(3) # specifz the timepoint you wish to stitch\n", + "timepoint = str(1).zfill(3) # specify the timepoint you wish to stitch\n", "\n", "pattern = f\"Timepoint{timepoint}_Row{row}_Well{well}_{{channel}}_zstack{zstack_value}_r{{row:03}}_c{{col:03}}.tif\"\n", "\n", From 6c6fd015e8f475f856093dd5eb7bc2914649d961 Mon Sep 17 00:00:00 2001 From: Niklas Schmacke Date: Thu, 5 Dec 2024 12:15:33 +0100 Subject: [PATCH 05/10] Fixed stitching notebook parallel speed up description --- docs/pages/tools/stitching/example_stitching_notebook.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/pages/tools/stitching/example_stitching_notebook.ipynb b/docs/pages/tools/stitching/example_stitching_notebook.ipynb index beb95c24..cc2acd39 100644 --- a/docs/pages/tools/stitching/example_stitching_notebook.ipynb +++ b/docs/pages/tools/stitching/example_stitching_notebook.ipynb @@ -467,7 +467,7 @@ "source": [ "## Multi-threaded Stitching\n", "\n", - "Using the `ParallelStitcher` class stitching can be speed up by using multiple threads. The code to perform stitching remains the same, but `ParallelStitcher` takes an additional argument `threads`, specifying the number of parallel threads to use." + "The `ParallelStitcher` class can speed up stitching by using multiple threads. The code to perform stitching remains the same, but `ParallelStitcher` takes an additional argument `threads`, specifying the number of parallel threads to use." ] }, { From 5ebff706832095a83516d29b5e5261978ba74b26 Mon Sep 17 00:00:00 2001 From: Niklas Schmacke Date: Thu, 5 Dec 2024 12:20:01 +0100 Subject: [PATCH 06/10] Fixed stitching notebook parallel speed up description --- docs/pages/tools/stitching/example_stitching_notebook.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/pages/tools/stitching/example_stitching_notebook.ipynb b/docs/pages/tools/stitching/example_stitching_notebook.ipynb index cc2acd39..d9b3eb05 100644 --- a/docs/pages/tools/stitching/example_stitching_notebook.ipynb +++ b/docs/pages/tools/stitching/example_stitching_notebook.ipynb @@ -467,7 +467,7 @@ "source": [ "## Multi-threaded Stitching\n", "\n", - "The `ParallelStitcher` class can speed up stitching by using multiple threads. The code to perform stitching remains the same, but `ParallelStitcher` takes an additional argument `threads`, specifying the number of parallel threads to use." + "The `ParallelStitcher` class can speed up stitching by using multiple threads. The code to start stitching remains the same, but `ParallelStitcher` takes an additional argument `threads`, specifying the number of parallel threads to use." ] }, { From 8d526fa953c9d073e85581f4c051eefb1846242d Mon Sep 17 00:00:00 2001 From: namsaraeva Date: Tue, 10 Dec 2024 13:12:12 +0100 Subject: [PATCH 07/10] change marker file --- src/scportrait/tools/ml/pretrained_models.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/scportrait/tools/ml/pretrained_models.py b/src/scportrait/tools/ml/pretrained_models.py index 8b26bf3b..92dd3d7d 100644 --- a/src/scportrait/tools/ml/pretrained_models.py +++ b/src/scportrait/tools/ml/pretrained_models.py @@ -54,7 +54,7 @@ def find_root_by_file(marker_file: str, current_path: Path) -> Path | None: return parent return None - src_code_dir = find_root_by_file("README.md", Path(__file__)) + src_code_dir = find_root_by_file("pyproject.toml", Path(__file__)) if src_code_dir is None: raise FileNotFoundError("Could not find scPortrait root directory") From 50a26276dfb23e4934254786948a64af79a90a2f Mon Sep 17 00:00:00 2001 From: namsaraeva Date: Tue, 10 Dec 2024 13:18:15 +0100 Subject: [PATCH 08/10] add debugging msg --- src/scportrait/tools/ml/pretrained_models.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/scportrait/tools/ml/pretrained_models.py b/src/scportrait/tools/ml/pretrained_models.py index 92dd3d7d..8ad27e06 100644 --- a/src/scportrait/tools/ml/pretrained_models.py +++ b/src/scportrait/tools/ml/pretrained_models.py @@ -50,11 +50,12 @@ def get_data_dir() -> Path: def find_root_by_file(marker_file: str, current_path: Path) -> Path | None: for parent in current_path.parents: + print(parent) if (parent / marker_file).exists(): return parent return None - src_code_dir = find_root_by_file("pyproject.toml", Path(__file__)) + src_code_dir = find_root_by_file("README.md", Path(__file__)) if src_code_dir is None: raise FileNotFoundError("Could not find scPortrait root directory") From b04f1d71671ce31292a366b3b9bc9419acdb1821 Mon Sep 17 00:00:00 2001 From: namsaraeva Date: Tue, 10 Dec 2024 15:15:26 +0100 Subject: [PATCH 09/10] modify code to look for src dir --- src/scportrait/tools/ml/pretrained_models.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/src/scportrait/tools/ml/pretrained_models.py b/src/scportrait/tools/ml/pretrained_models.py index 8ad27e06..6d2a8f88 100644 --- a/src/scportrait/tools/ml/pretrained_models.py +++ b/src/scportrait/tools/ml/pretrained_models.py @@ -48,18 +48,18 @@ def get_data_dir() -> Path: Path to data directory """ - def find_root_by_file(marker_file: str, current_path: Path) -> Path | None: + def find_root_by_folder(marker_folder: str, current_path: Path) -> Path | None: for parent in current_path.parents: - print(parent) - if (parent / marker_file).exists(): + if (parent / marker_folder).is_dir(): return parent return None - src_code_dir = find_root_by_file("README.md", Path(__file__)) - if src_code_dir is None: - raise FileNotFoundError("Could not find scPortrait root directory") + src_code_dir = find_root_by_folder("io", Path(__file__)) + if src_code_dir is None: + raise FileNotFoundError("Could not find scPortrait source directory") data_dir = src_code_dir / "scportrait_data" + return data_dir.absolute() From 1177aa6fde3e954cdaf455265a9413ceca55e6b7 Mon Sep 17 00:00:00 2001 From: namsaraeva Date: Tue, 10 Dec 2024 20:17:32 +0100 Subject: [PATCH 10/10] silence dask warning --- src/scportrait/io/daskmmap.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/scportrait/io/daskmmap.py b/src/scportrait/io/daskmmap.py index e6d2a77d..0e0a27a6 100644 --- a/src/scportrait/io/daskmmap.py +++ b/src/scportrait/io/daskmmap.py @@ -1,3 +1,6 @@ +import warnings + +warnings.filterwarnings("ignore", message=".*`dataframe.query-planning`.*") import dask import dask.array as da import h5py