Skip to content

Latest commit

 

History

History
94 lines (62 loc) · 4.92 KB

README.md

File metadata and controls

94 lines (62 loc) · 4.92 KB
Raw

DDPS: Denoising Diffusion Prior for Segmentation

Zeqiang Lai*, Yuchen duan*, Jifeng Dai, Ziheng Li, Ying Fu, Hongsheng Li, Yu Qiao, and Wenhai Wang

Denoising Diffusion Semantic Segmentation with Mask Prior Modeling.

DDPS explore the mask prior modeled by a recently developed denoising diffusion generative model for ameliorating the semantic segmentation quality of existing discriminative approaches. DDPS focuses on a discrete instantiation of a unified architecture that adapts diffusion models for mask prior modeling, with two novel training and inference strategies, i.e., Diffusion-on-First-Prediction diffusion strategy and Free-Re-noising denoising strategy.

Hugging Face Models

arch

Working in progress.

News

  • [TBD] Integrate an online demo into InternGPT.
  • [2023/7/26] Release preview code and pretrained models.
  • [2023/6/2] Release paper to arXiv.

DDPS Model

Semantic segmentation performance on ADE20K.

Method Backbone Params (M) mIoU (ss) mIoU (ms) mBIoU Download
DDPS-DeeplabV3+ MobibleNetV2 27.9 38.07 39.07 23.47 TBD
DDPS-Segformer MiT-B0 15.1 41.67 41.95 26.77 TBD
DDPS-DeeplabV3+ ResNet50 54.2 45.26 46.01 30.39 checkpoint
DDPS-Segformer MiT-B2 65.5 49.73 49.96 33.95 checkpoint
DDPS-DeeplabV3+ ResNet101 104.3 46.32 46.99 31.82 checkpoint
DDPS-Segformer MiT-B5 122.8 51.11 51.71 35.66 TBD

Semantic segmentation performance on Cityscapes.

Method Backbone Params (M) mIoU (ss) mIoU (ms) mBIoU Download
DDPS-DeeplabV3+ MobibleNetV2 27.9 78.11 79.73 61.14 TBD
DDPS-Segformer MiT-B0 15.1 78.19 79.62 64.07 checkpoint
DDPS-DeeplabV3+ ResNet50 54.2 81.39 82.27 65.76 checkpoint
DDPS-Segformer MiT-B2 65.5 81.77 82.22 67.75 checkpoint
DDPS-DeeplabV3+ ResNet101 104.3 81.68 82.51 65.96 checkpoint
DDPS-Segformer MiT-B5 122.8 82.42 82.94 66.45 TBD

Installation

See environment.yaml for details.

Usage

Evaluation

sh tools/dist_test_diffusion_origin.sh configs/ade20k/segformer_b2_ade20k_multistep.py checkpoints/ade20k/segformer_b2_multistep/best_mIoU_iter_144000.pth  8 --eval "mIoU"

We are still cleaning up the code and checkpoints, so the results might slightly differ from the reported ones, but you should expect the following results with the above command.

+-------------------+
|    mIoU 0,10,19   |
+-------------------+
| 49.35,49.55,49.66 |
+-------------------+

Visualization

Mask Prior Modeling

prior

Citation

If you find this work helpful, please consider cite our paper.

@misc{lai2023ddps,
  title = {Denoising Diffusion Semantic Segmentation with Mask Prior Modeling},
  author = {Zeqiang Lai and Yuchen duan and Jifeng Dai and Ziheng Li and Ying Fu and Hongsheng Li and Yu Qiao and Wenhai Wang},
  publisher = {arXiv},
  year = {2023},
}

Acknowledgement

MMSegmentation   VQDiffusion   Segformer   Diffusers