Skip to content

Commit

Permalink
update
Browse files Browse the repository at this point in the history 
Signed-off-by: YunLiu <[email protected]>
  • Loading branch information
@KumoLiu
KumoLiu committed Dec 5, 2024
1 parent 25f7a81 commit 1812df3
Show file tree
Hide file tree
Showing 2 changed files with 77 additions and 111 deletions.
Binary file added modules/omniverse/omniverse.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
188 changes: 77 additions & 111 deletions modules/omniverse/omniverse_integration.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,13 +20,21 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# From 3D Segmentation to Omniverse: A Complete Workflow for Mesh Conversion, USD Export, and NVIDIA Omniverse Integration\n",
"# From 3D Segmentation to USD: A Complete Workflow for Hierarchical Mesh Conversion, USD Export, and NVIDIA Omniverse Integration\n",
"In this tutorial, we’ll cover:\n",
"\n",
"- Utilizing 3D Segmentation Results: How to extract and prepare segmentation data from VISTA-3D or MAISI for mesh conversion.\n",
"- Converting to Mesh Format: Step-by-step instructions on transforming segmentation results into mesh models.\n",
"- Exporting to USD: A guide to exporting meshes as Universal Scene Description (USD) files, optimized for Omniverse workflows.\n",
"- Visualizing in NVIDIA Omniverse: Instructions on importing USD files into Omniverse for high-quality 3D visualization and manipulation.\n",
"- **Download the Model Bundle**: First, we download a pre-trained model bundle from the MONAI Model Zoo. This model bundle contains the necessary models and configurations for medical image analysis, which can accelerate our development process.\n",
"\n",
"- **Run the Inference Workflow of the Bundle**: Using the downloaded model bundle, we run its built-in inference workflow to automatically segment or analyze the input medical imaging data and obtain the desired results.\n",
"\n",
"- **Convert NIfTI/DICOM to Mesh**: The inference results are usually in NIfTI or DICOM format. We need to convert this volumetric data into a 3D mesh model for visualization and further processing.\n",
"\n",
"- **Save the Mesh as OBJ and GLTF Formats**:\n",
" - Save Single Mesh as OBJ Format: For a single organ or structure mesh, we save it in OBJ format, which is convenient to open and view in various 3D software.\n",
" - Save Combined Mesh as GLTF Format: When we have meshes of multiple organs, we save them in GLTF format. This format preserves the hierarchical structure of the organs, making it easier to reflect the relationships between different organs during visualization.\n",
"\n",
"- **Visualization in the Omniverse**: Finally, we import the GLTF format mesh into NVIDIA Omniverse. In Omniverse, we can utilize its powerful rendering and interactive capabilities to perform high-quality 3D visualization of medical imaging data, exploring the structures and spatial relationships of organs.\n",
"\n",
"This end-to-end process enables efficient, high-quality visualization in NVIDIA Omniverse from raw segmentation data."
]
},
Expand Down Expand Up @@ -107,7 +115,6 @@
"\n",
"import vtk\n",
"import vtkmodules\n",
"from vtkmodules.vtkRenderingCore import vtkRenderWindow, vtkRenderer\n",
"from ipyvtklink.viewer import ViewInteractiveWidget\n",
"\n",
"from utility import convert_to_mesh, convert_mesh_to_usd\n",
Expand Down Expand Up @@ -157,7 +164,8 @@
"source": [
"## Generate segmentation from MAISI\n",
"\n",
"In this section, we download the MAISI bundle and run the inference workflow to generate large CT images with paired segmentation masks. This involves downloading the necessary model and running inference."
"### Download the Model Bundle\n",
"In this section, we download the MAISI bundle from monai model-zoo."
]
},
{
Expand Down Expand Up @@ -205,6 +213,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### Run the Inference Workflow of the Bundle\n",
"We use the `create_workflow` API from MONAI to streamline the inference process directly from the bundle.\n",
"\n",
"Key input details for inference, such as the body region and target anatomy, are specified in [./configs/inference.json]. For a comprehensive explanation of the parameters, refer to [./docs/README.md] in the bundle directory. Additionally, we adjust the `output_size`, `spacing` and `num_splits` parameters to prevent out-of-memory issues during inference."
Expand Down Expand Up @@ -255,6 +264,8 @@
")\n",
"\n",
"# uncomment this line to run the inference workflow\n",
"# then you will get the generated CT images and paired masks which can be used for the following steps.\n",
"# In this tutorial, we just use the tested data (IntegrationTest-AbdomenCT.nii.gz) from bundle for demonstration.\n",
"# workflow.run()"
]
},
Expand All @@ -268,86 +279,26 @@
"\n",
"We define a function `nii_to_mesh` to handle the conversion of NIfTI files to OBJ files. The workflow is as follows:\n",
"\n",
"- Preprocessing:\n",
"The function uses a series of transformations (Compose, LoadImaged, BorderPadd, and SqueezeDimd) to load and preprocess the input NIfTI file, ensuring it is ready for segmentation.\n",
"- Organ Label Mapping:\n",
"- **Preprocessing**:\n",
"The function uses a series of transformations (`Compose`, `LoadImaged`, `BorderPadd`, and `SqueezeDimd`) to load and preprocess the input NIfTI file, ensuring it is ready for segmentation.\n",
"\n",
"- **Organ Label Mapping**:\n",
"It iterates over a dictionary mapping organ names to their respective label values. For each organ:\n",
"A binary mask (single_organ) is created to isolate the organ by assigning its corresponding label value.\n",
"The segmented organ is saved as a NIfTI file.\n",
"- Mesh Conversion:\n",
"\n",
"- **Mesh Conversion**:\n",
"Each segmented NIfTI file is converted into an OBJ file using the `convert_to_mesh` function.\n",
"- Combined Mesh:\n",
"\n",
"- **Combined Mesh**:\n",
"A combined segmentation file is created by merging all organ segmentations into a single NIfTI file. This file is then converted into a GLTF file, preserving the hierarchical structure of the organs.\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Assigning index 1 to label Liver\n",
"2024-12-03 09:13:00,689 INFO image_writer.py:197 - writing: /workspace/Data/maisi_ct_generative/datasets/monai/nii/Liver.nii.gz\n",
"Mesh successfully exported to /workspace/Data/maisi_ct_generative/datasets/monai/obj/Liver.obj\n",
"Assigning index 2 to label Spleen\n",
"2024-12-03 09:13:07,086 INFO image_writer.py:197 - writing: /workspace/Data/maisi_ct_generative/datasets/monai/nii/Spleen.nii.gz\n",
"Mesh successfully exported to /workspace/Data/maisi_ct_generative/datasets/monai/obj/Spleen.obj\n",
"Assigning index 3 to label Pancreas\n",
"2024-12-03 09:13:10,982 INFO image_writer.py:197 - writing: /workspace/Data/maisi_ct_generative/datasets/monai/nii/Pancreas.nii.gz\n",
"Mesh successfully exported to /workspace/Data/maisi_ct_generative/datasets/monai/obj/Pancreas.obj\n",
"Assigning index 4 to label Heart\n",
"2024-12-03 09:13:14,755 INFO image_writer.py:197 - writing: /workspace/Data/maisi_ct_generative/datasets/monai/nii/Heart.nii.gz\n",
"Mesh successfully exported to /workspace/Data/maisi_ct_generative/datasets/monai/obj/Heart.obj\n",
"Assigning index 5 to label Body\n",
"2024-12-03 09:13:18,516 INFO image_writer.py:197 - writing: /workspace/Data/maisi_ct_generative/datasets/monai/nii/Body.nii.gz\n",
"Mesh successfully exported to /workspace/Data/maisi_ct_generative/datasets/monai/obj/Body.obj\n",
"Assigning index 6 to label Gallbladder\n",
"2024-12-03 09:14:50,151 INFO image_writer.py:197 - writing: /workspace/Data/maisi_ct_generative/datasets/monai/nii/Gallbladder.nii.gz\n",
"Mesh successfully exported to /workspace/Data/maisi_ct_generative/datasets/monai/obj/Gallbladder.obj\n",
"Assigning index 7 to label Stomach\n",
"2024-12-03 09:14:53,691 INFO image_writer.py:197 - writing: /workspace/Data/maisi_ct_generative/datasets/monai/nii/Stomach.nii.gz\n",
"Mesh successfully exported to /workspace/Data/maisi_ct_generative/datasets/monai/obj/Stomach.obj\n",
"Assigning index 8 to label Small_bowel\n",
"2024-12-03 09:14:57,531 INFO image_writer.py:197 - writing: /workspace/Data/maisi_ct_generative/datasets/monai/nii/Small_bowel.nii.gz\n",
"Mesh successfully exported to /workspace/Data/maisi_ct_generative/datasets/monai/obj/Small_bowel.obj\n",
"Assigning index 9 to label Colon\n",
"2024-12-03 09:15:05,387 INFO image_writer.py:197 - writing: /workspace/Data/maisi_ct_generative/datasets/monai/nii/Colon.nii.gz\n",
"Mesh successfully exported to /workspace/Data/maisi_ct_generative/datasets/monai/obj/Colon.obj\n",
"Assigning index 10 to label Kidney\n",
"2024-12-03 09:15:11,283 INFO image_writer.py:197 - writing: /workspace/Data/maisi_ct_generative/datasets/monai/nii/Kidney.nii.gz\n",
"Mesh successfully exported to /workspace/Data/maisi_ct_generative/datasets/monai/obj/Kidney.obj\n",
"Assigning index 11 to label Veins\n",
"2024-12-03 09:15:16,689 INFO image_writer.py:197 - writing: /workspace/Data/maisi_ct_generative/datasets/monai/nii/Veins.nii.gz\n",
"Mesh successfully exported to /workspace/Data/maisi_ct_generative/datasets/monai/obj/Veins.obj\n",
"Assigning index 12 to label Lungs\n",
"2024-12-03 09:15:22,560 INFO image_writer.py:197 - writing: /workspace/Data/maisi_ct_generative/datasets/monai/nii/Lungs.nii.gz\n",
"Mesh successfully exported to /workspace/Data/maisi_ct_generative/datasets/monai/obj/Lungs.obj\n",
"Assigning index 13 to label Spine\n",
"2024-12-03 09:15:30,198 INFO image_writer.py:197 - writing: /workspace/Data/maisi_ct_generative/datasets/monai/nii/Spine.nii.gz\n",
"Mesh successfully exported to /workspace/Data/maisi_ct_generative/datasets/monai/obj/Spine.obj\n",
"Assigning index 14 to label Ribs\n",
"2024-12-03 09:15:40,221 INFO image_writer.py:197 - writing: /workspace/Data/maisi_ct_generative/datasets/monai/nii/Ribs.nii.gz\n",
"Mesh successfully exported to /workspace/Data/maisi_ct_generative/datasets/monai/obj/Ribs.obj\n",
"Assigning index 15 to label Shoulders\n",
"2024-12-03 09:15:46,055 INFO image_writer.py:197 - writing: /workspace/Data/maisi_ct_generative/datasets/monai/nii/Shoulders.nii.gz\n",
"No points found for label 15. Skipping...\n",
"Mesh successfully exported to /workspace/Data/maisi_ct_generative/datasets/monai/obj/Shoulders.obj\n",
"Assigning index 16 to label Hips\n",
"2024-12-03 09:15:49,647 INFO image_writer.py:197 - writing: /workspace/Data/maisi_ct_generative/datasets/monai/nii/Hips.nii.gz\n",
"Mesh successfully exported to /workspace/Data/maisi_ct_generative/datasets/monai/obj/Hips.obj\n",
"Assigning index 17 to label Back_muscles\n",
"2024-12-03 09:15:57,623 INFO image_writer.py:197 - writing: /workspace/Data/maisi_ct_generative/datasets/monai/nii/Back_muscles.nii.gz\n",
"Mesh successfully exported to /workspace/Data/maisi_ct_generative/datasets/monai/obj/Back_muscles.obj\n",
"2024-12-03 09:16:11,489 INFO image_writer.py:197 - writing: /workspace/Data/maisi_ct_generative/datasets/monai/nii/all_organs.nii.gz\n",
"No points found for label 15. Skipping...\n",
"Mesh successfully exported to /workspace/Data/maisi_ct_generative/datasets/monai/obj/all_organs.gltf\n",
"Saved whole segmentation /workspace/Data/maisi_ct_generative/datasets/monai/nii/all_organs\n"
]
}
],
"outputs": [],
"source": [
"# 17 groupings that cover 101 segments/regions out of 140\n",
"labels = {\n",
Expand Down Expand Up @@ -387,7 +338,7 @@
" \"right_lung_lower_lobe\": 32,\n",
" },\n",
" \"Spine\": {\n",
" # \"vertebrae_L6\": 131,\n",
" \"vertebrae_L6\": 131,\n",
" \"vertebrae_L5\": 33,\n",
" \"vertebrae_L4\": 34,\n",
" \"vertebrae_L3\": 35,\n",
Expand Down Expand Up @@ -524,13 +475,42 @@
"out = nii_to_mesh(input_nii_path, output_nii_path, output_obj_path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Convert 3D model contain all organs to OpenUSD format\n",
"\n",
"[Universal Scene Description (OpenUSD)](https://openusd.org/release/index.html) is an extensible ecosystem of file formats, compositors, renderers, and other plugins for comprehensive 3D scene description."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"USD file successfully exported to /workspace/Data/maisi_ct_generative/datasets/monai/obj/all_organs.usd\n"
]
}
],
"source": [
"obj_filename = \"/workspace/Data/maisi_ct_generative/datasets/monai/obj/all_organs.gltf\"\n",
"usd_filename = \"/workspace/Data/maisi_ct_generative/datasets/monai/obj/all_organs.usd\"\n",
"\n",
"convert_mesh_to_usd(obj_filename, usd_filename)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Visualize one single organ mesh\n",
"\n",
"Here we randomly select one organ to visualize the mesh"
"Here we randomly select one organ to visualize the mesh using `ViewInteractiveWidget`"
]
},
{
Expand All @@ -557,19 +537,19 @@
"# Step 4: Create a renderer\n",
"renderer = vtk.vtkRenderer()\n",
"renderer.AddActor(actor)\n",
"renderWindow = vtk.vtkRenderWindow()\n",
"renderWindow.AddRenderer(renderer)\n",
"renderWindow.SetSize(800, 600)\n",
"renderWindow.SetOffScreenRendering(1)\n",
"render_window = vtk.vtkRenderWindow()\n",
"render_window.AddRenderer(renderer)\n",
"render_window.SetSize(800, 600)\n",
"render_window.SetOffScreenRendering(1)\n",
"\n",
"# Step 5: Create a render window interactor\n",
"renderWindowInteractor = vtk.vtkRenderWindowInteractor()\n",
"renderWindowInteractor.SetRenderWindow(renderWindow)\n",
"renderWindowInteractor.SetRenderWindow(render_window)\n",
"interactorStyle = vtk.vtkInteractorStyleTrackballCamera()\n",
"renderWindowInteractor.SetInteractorStyle(interactorStyle)\n",
"renderWindow.Render()\n",
"render_window.Render()\n",
"\n",
"interactive_widget = ViewInteractiveWidget(renderWindow)\n",
"interactive_widget = ViewInteractiveWidget(render_window)\n",
"\n",
"# Uncomment the following line to display the interactive widget\n",
"# interactive_widget"
Expand All @@ -586,44 +566,30 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Visualization of All Organs\n",
"## Visualization in the Omniverse\n",
"\n",
"Download the [NVIDIA Omniverse](https://www.nvidia.com/en-us/omniverse/) launcher to explore applications such as USD Composer for viewing and manipulating the OpenUSD output file.\n",
"\n",
"You can view the 3D models using online viewer such as https://3dviewer.net/#"
"![omniverse](./omniverse.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Visualization of All Organs\n",
"\n",
"You can view the 3D models using online viewer such as https://3dviewer.net/#\n",
"\n",
"![all organs](result.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Convert OBJ to USD"
]
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"USD file successfully exported to /workspace/Data/maisi_ct_generative/datasets/monai/obj/all_organs.usd\n"
]
}
],
"source": [
"obj_filename = \"/workspace/Data/maisi_ct_generative/datasets/monai/obj/all_organs.gltf\"\n",
"usd_filename = \"/workspace/Data/maisi_ct_generative/datasets/monai/obj/all_organs.usd\"\n",
"\n",
"convert_mesh_to_usd(obj_filename, usd_filename)"
]
"outputs": [],
"source": []
}
],
"metadata": {
Expand Down

0 comments on commit 1812df3

Please sign in to comment.