Skip to content

Latest commit

 

History

History
89 lines (68 loc) · 3.97 KB

README.md

File metadata and controls

89 lines (68 loc) · 3.97 KB
Raw

Progressive Token Length Scaling in Transformer Encoders
for Efficient Universal Segmentation
(official implementation)

Key Features

Efficient encoder design to reduce computational cost of Mask2Former models

  • Simple. Easily pluggable to Mask2Former-style models.
  • Effective. Flexible in reducing encoder cost by 50-75% with little to no segmentation performance degradation.
  • General. Evaluated with various backbones, pre-trained weights as well as DETR model (DINO).

Model Framework

Installation

  • Our project is developed on Mask2Former. Please follow their installation instructions as well as dataset instructions. You may also refer to the requirements.txt.

Download pretrained weights

  • Create a folder named pretrained_ckpt in the root folder using mkdir pretrained_ckpt.
  • Please use the analyze_model code from Mask2Former to compute GFLOPs reported in the paper.
  • Download our pretrained weights and put them in pretrained_ckpt.
    Our pretrained weights includes PRO-SCALE (configuration (3,3,3)) based Mask2Former models with corresponding backbone and trained on corresponding dataset:
Model Name Backbone Dataset Config PQ (%)
r50_proscale_3_3_3_coco Res50 COCO r50_proscale_3_3_3_coco 51.44
swint_proscale_3_3_3_coco SWIN-T COCO swint_proscale_3_3_3_coco 52.88
Example run command (please set the dataset and output directory path): 
DETECTRON2_DATASETS=/path/to/dataset/folder/ python3 test_net.py \
    --num-gpus 1 \
    --eval-only \
    --config-file configs/res50_proscale_3_3_3_coco.yaml \
    MODEL.WEIGHTS pretrained_ckpt/r50_proscale_3_3_3_coco.pth \
    OUTPUT_DIR /path/to/output/folder

Contact

If you have any questions, don't hesitate to contact me at aaich001@ucr.edu.

Acknowledgement

This repository was built on top of Detectron2 and Mask2Former. We thank and acknowledge their effort. Please also see our contemporary work ECO-M2F (paper, code) on efficient transformer encoders.

Citing PRO-SCALE

If you find our work helpful for your research, please consider citing the following BibTeX entry.

@article{aich2024progressive,
  title={Progressive Token Length Scaling in Transformer Encoders for Efficient Universal Segmentation},
  author={Aich, Abhishek and Suh, Yumin and Schulter, Samuel and Chandraker, Manmohan},
  journal={arXiv preprint arXiv:2404.14657},
  year={2024}
}