Threshold Matters in WSSS: Manipulating the Activation for the Robust and Accurate Segmentation Model Against Thresholds (CVPR 2022)

Official pytorch implementation of "Threshold Matters in WSSS: Manipulating the Activation for the Robust and Accurate Segmentation Model Against Thresholds"

Threshold Matters in WSSS: Manipulating the Activation for the Robust and Accurate Segmentation Model Against Thresholds
Minhyun Lee^* , Dongseob Kim^* , Hyunjung Shim
School of Integrated Technology, Yonsei University
_* indicates an equal contribution.

Abstract Weakly-supervised semantic segmentation (WSSS) has recently gained much attention for its promise to train segmentation models only with image-level labels. Existing WSSS methods commonly argue that the sparse coverage of CAM incurs the performance bottleneck of WSSS. This paper provides analytical and empirical evidence that the actual bottleneck may not be sparse coverage but a global thresholding scheme applied after CAM. Then, we show that this issue can be mitigated by satisfying two conditions; 1) reducing the imbalance in the foreground activation and 2) increasing the gap between the foreground and the background activation. Based on these findings, we propose a novel activation manipulation network with a per-pixel classification loss and a label conditioning module. Per-pixel classification naturally induces two-level activation in activation maps, which can penalize the most discriminative parts, promote the less discriminative parts, and deactivate the background regions. Label conditioning imposes that the output label of pseudo-masks should be any of true image-level labels; it penalizes the wrong activation assigned to non-target classes. Based on extensive analysis and evaluations, we demonstrate that each component helps produce accurate pseudo-masks, achieving the robustness against the choice of the global threshold.

Updates

31 Mar, 2022: Initial upload

07 Jul, 2022: updates about ms-coco 2014 dataset

Requirement

• This code is tested on Ubuntu 18.04, with Python 3.6, PyTorch 1.7.1, and CUDA 11.1.

Dataset & pretrained checkpoint

• Download dataset, pretrained checkpoints, and refined seeds for AMN

  • Example directory hierarchy

  AMN
  |--- sess
  |    |--- res50_amn.pth.pth
  |    |--- res50_irn.pth
  |--- result
  |    |--- ir_label
  |    | ...
  | ...
  

• Dataset

  • Pascal VOC 2012

• Pretrained checkpoint

  • AMN
  • Boundary refinement network (IRN) w/ AMN

• Refined seed for AMN (CAM + CRF)

  • Refined seed (ir_label)

Execution

Pseudo-mask generation w/ AMN

• Execute the bash file.

  # Please see these files for the detail of execution.
  bash script/generate_pseudo_mask.sh

Segmentation network

Fort the segmentation network, we experimented with DeepLab-V2 and followed the default training settings of AdvCAM

Acknowledgement

This code is highly borrowed from IRN. Thanks to Jiwoon, Ahn.

Citation

If you find this work useful for your research, please cite our paper:

@InProceedings{Lee2022AMN,
    author    = {Lee, Minhyun, Kim, Dongseob, and Shim, Hyunjung},
    title     = {Threshold Matters in WSSS: Manipulating the Activation for the Robust and Accurate Segmentation Model Against Thresholds},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
    year      = {2022}
}