README.md

Instances as Queries

Abstract

We present QueryInst, a new perspective for instance segmentation. QueryInst is a multi-stage end-to-end system that treats instances of interest as learnable queries, enabling query based object detectors, e.g., Sparse R-CNN, to have strong instance segmentation performance. The attributes of instances such as categories, bounding boxes, instance masks, and instance association embeddings are represented by queries in a unified manner. In QueryInst, a query is shared by both detection and segmentation via dynamic convolutions and driven by parallelly-supervised multi-stage learning. We conduct extensive experiments on three challenging benchmarks, i.e., COCO, CityScapes, and YouTube-VIS to evaluate the effectiveness of QueryInst in object detection, instance segmentation, and video instance segmentation tasks. For the first time, we demonstrate that a simple end-to-end query based framework can achieve the state-of-the-art performance in various instance-level recognition tasks.

Citation

@InProceedings{Fang_2021_ICCV,
    author    = {Fang, Yuxin and Yang, Shusheng and Wang, Xinggang and Li, Yu and Fang, Chen and Shan, Ying and Feng, Bin and Liu, Wenyu},
    title     = {Instances As Queries},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2021},
    pages     = {6910-6919}
}

Results and Models

Model	Backbone	Style	Lr schd	Number of Proposals	Multi-Scale	RandomCrop	box AP	mask AP	Config	Download
QueryInst	R-50-FPN	pytorch	1x	100	False	False	42.0	37.5	config	model | log
QueryInst	R-50-FPN	pytorch	3x	100	True	False	44.8	39.8	config	model | log
QueryInst	R-50-FPN	pytorch	3x	300	True	True	47.5	41.7	config	model | log
QueryInst	R-101-FPN	pytorch	3x	100	True	False	46.4	41.0	config	model | log
QueryInst	R-101-FPN	pytorch	3x	300	True	True	49.0	42.9	config	model | log