Skip to content

Latest commit

 

History

History
439 lines (417 loc) · 19.2 KB

README.md

File metadata and controls

439 lines (417 loc) · 19.2 KB
Raw

PWC

PWC

PWC

SenFormer: Efficient Self-Ensemble Framework for Semantic Segmentation

Efficient Self-Ensemble Framework for Semantic Segmentation by Walid Bousselham, Guillaume Thibault, Lucas Pagano, Archana Machireddy, Joe Gray, Young Hwan Chang and Xubo Song.

This repository contains the official Pytorch implementation of training & evaluation code and the pretrained models for SenFormer.


💾Code Snippet (SenFormer)| ⌨️ Code Snippet (FPNT)| 📜 Paper | 论文 | 🎥 video

🔨 Installation

Conda environment

  • Clone this repository and enter it: git clone [email protected]:WalBouss/SenFormer.git && cd SenFormer.
  • Create a conda environment conda create -n senformer python=3.8, and activate it conda activate senformer.
  • Install Pytorch and torchvision conda install pytorch==1.7.1 torchvision==0.8.2 cudatoolkit=10.2 -c pytorch — (you may also switch to other version by specifying the version number).
  • Install MMCV library pip install mmcv-full==1.4.0
  • Install MMSegmentation library by running pip install -e . in SenFormer directory.
  • Install other requirements pip install timm einops

Here is a full script for setting up a conda environment to use SenFormer (with CUDA 10.2 and pytorch 1.7.1):

conda create -n senformer python=3.8
conda activate senformer
conda install pytorch==1.7.1 torchvision==0.8.2 cudatoolkit=10.2 -c pytorch

git clone [email protected]:WalBouss/SenFormer.git && cd SenFormer
pip install mmcv-full==1.4.0
pip install -e .
pip install timm einops

Datasets

For datasets preparations please refer to MMSegmentation guidelines.

Pretrained weights

ResNet pretrained weights will be automatically downloaded before training.

For Swin Transformer ImageNet pretrained weights, you can either:

  • run bash tools/download_swin_weights.sh in SenFormer project to download all Swin Transformer pretrained weights (it will place weights under pretrain/ folder ).
  • download desired backbone weights here: Swin-T, Swin-S, Swin-B, Swin-L and place them under pretrain/ folder.
  • download weights from official repository then, convert them to mmsegmentation format following mmsegmentation guidelines.

🎯 Model Zoo

SenFormer models with ResNet and Swin's backbones and ADE20K, COCO-Stuff 10K, Pascal Context and Cityscapes.

ADE20K

Backbone weight sharing mIoU mIoU (MS) #params FLOPs Resolution Download
ResNet-50 ✅ 44.38 45.2 55M 179G 512x512 model config
❌ 44.6 45.6 144M model config
ResNet-101 ✅ 46.93 47.9 79M 199G 512x512 model config
❌ 46.5 47.0 163M model config
Swin-Tiny ✅ 46.0 - 59M 179G 512x512 model config
❌ 46.0 46.4 144M model config
Swin-Small ✅ 49.2 - 81M 202G 512x512 model config
❌ 49.2 50.4 165M model config
Swin-Base ✅ 52.2 - 120M 242G 640x640 model config
❌ 51.8 53.2 204M model config
Swin-Large ✅ 53.1 - 233M 546G 640x640 model config
❌ 53.1 54.2 314M model config

COCO-Stuff 10K

Backbone weight sharing mIoU mIoU (MS) #params Resolution Download
ResNet-50 ✅ 40.0 41.3 55M 512x512 model config
❌ 39.0 39.7 144M model config
ResNet-101 ✅ 41.0 42.1 79M 512x512 model config
❌ 39.6 40.6 163M model config
Swin-Large ✅ 49.8 51.5 233M 512x512 model config
❌ 49.1 50.1 314M model config

Pascal Context

Backbone weigth sharing mIoU mIoU (MS) #params Resolution Download
ResNet-50 ✅ 53.2 54.3 55M 480x480 model config
❌ 53.2 54.3 144M model config
ResNet-101 ✅ 54.6 56.6 79M 480x480 model config
❌ 55.1 56.6 163M model config
Swin-Large ✅ 63.1 64.5 233M 480x480 model config
❌ 62.4 64.0 314M model config

Cityscapes

Backbone weight sharing mIoU mIoU (MS) #params Resolution Download
ResNet-50 ✅ 78.8 80.1 55M 512x1024 model config
❌ 78.8 80.1 144M model config
ResNet-101 ✅ 79.9 81.4 79M 512x1024 model config
❌ 80.3 81.4 163M model config
Swin-Large ✅ 82.8 84.0 233M 512x1024 model config
❌ 82.2 83.3 314M model config

🔭 Inference

Download one checkpoint weights from above, for example SenFormer with ResNet-50 backbone on ADE20K:

Inference on a dataset

# Single-gpu testing
python tools/test.py senformer_configs/senformer/ade20k/senformer_fpnt_r50_512x512_160k_ade20k.py /path/to/checkpoint_file

# Multi-gpu testing
./tools/dist_test.sh senformer_configs/senformer/ade20k/senformer_fpnt_r50_512x512_160k_ade20k.py /path/to/checkpoint_file <GPU_NUM>

# Multi-gpu, multi-scale testing
tools/dist_test.sh senformer_configs/senformer/ade20k/senformer_fpnt_r50_512x512_160k_ade20k.py /path/to/checkpoint_file <GPU_NUM> --aug-test

Inference on custom data

To generate segmentation maps for your own data, run the following command:

python demo/image_demo.py ${IMAGE_FILE} ${CONFIG_FILE} ${CHECKPOINT_FILE}

Run python demo/image_demo.py --help for additional options.

🔩 Training

Follow above instructions to download ImageNet pretrained weights for backbones and run one of the following command:

# Single-gpu training
python tools/train.py path/to/model/config 

# Multi-gpu training
./tools/dist_train.sh path/to/model/config <GPU_NUM>

For example to train SenFormer with a ResNet-50 as backbone on ADE20K:

# Single-gpu training
python tools/train.py senformer_configs/senformer/ade20k/senformer_fpnt_r50_512x512_160k_ade20k.py 

# Multi-gpu training
./tools/dist_train.sh senformer_configs/senformer/ade20k/senformer_fpnt_r50_512x512_160k_ade20k.py <GPU_NUM>

Note that the default learning rate and training schedule is for an effective batch size of 16, (e.g. 8 GPUs & 2 imgs/gpu).

⭐ Acknowledgement

This code is build using MMsegmentation library as codebase and uses timm and einops as well.

📚 Citation

If you find this repository useful, please consider citing our work 📝 and giving a star 🌟 :

@article{bousselham2021senformer,
  title={Efficient Self-Ensemble Framework for Semantic Segmentation},
  author={Walid Bousselham, Guillaume Thibault, Lucas Pagano, Archana Machireddy, Joe Gray, Young Hwan Chang, Xubo Song},
  journal={arXiv preprint arXiv:2111.13280},
  year={2021}
}