train Panoptic Segmentation model on custom dataset

[Cyril9227]

❓ How to train Panoptic Segmentation on a custom dataset ?

Hello everyone,

My question is two-fold :

1. What is the expected Detectron2 format for Panoptic Segmentation ? In the PS paper, they are talking about a pixel-wise annotation (either stuff class or instance ID) but it seems that D2 requires bounding boxes as well.
2. Is it possible to train PS on a custom dataset with only pixel-wise annotations ?

Thanks,

Cyril

[ppwwyyxx]

At the moment we don't have instructions to train panoptic segmentation models on custom dataset. Our code now expects panoptic segmentation data in COCO format, which is unfortunately not well-documented.

[JavierClearImageAI]

At the moment we don't have instructions to train panoptic segmentation models on custom dataset. Our code now expects panoptic segmentation data in COCO format, which is unfortunately not well-documented.

Here the link to the COCO format ... then go to point 4: Panoptic Segmentation. I would also recommend to download the json annotations and mask labels in png format, have a look at them and create yours in the same format.

I am still on my way do it. After converting the annotations into the right format, how to register panoptic annotations for training? is it the same than with instance segmentation and object detection? Since we have now an extra directory with the masks, I am wondering if it will use the same code to register the annotations.

I guess the answer is no, since you created an specific script to do that:
coco_panoptic.py

detectron2.data.datasets.register_coco_panoptic

[mvdelt]

@JavierClearImageAI what is your way to make your custom panoptic segmentation annotations?

[JavierClearImageAI]

@JavierClearImageAI what is your way to make your custom panoptic segmentation annotations?

I ended up using detectron2.data.datasets.register_coco_panoptic_separated

I realized that using detectron2.data.datasets.register_coco_panoptic wasn't working for a custom dataset with new categories, since it registers a “standard” version of COCO panoptic.

To be honest, I don't really like that you have to provide the images, the masks, json for the instances and json for instance segmentation, since the instance segmentation and the semantic segmentation info could be automated by FAIR from the png binary masks... but you will have to do it if you want to register data from the jsons. Otherwise you might need to dive into the code and create your own dataloader, which might take you longer to code.

So, just follow the "coco standard panoptic format" for the panoptic coco-json and the "coco standard instance segmentation format" for the instances.

In my case "sem_seg_root" = "panoptic_root", since I use the same masks for both.

Summarizing:

1. You need to (convert your png image masks to contours)[https://github.com/convert mask binary image to polygon format  cocodataset/cocoapi#131] and create the coco instance segmentation annotations.
2. Then create the json for panoptic (check point 4 from "coco standard panoptic format")
3. Then you need to register your data using the afore mentioned method.
4. Configure the training as you did for custom instance segemtnation training
5. Train

PD: here my dataset registration code:

Method:

detectron2.data.datasets.register_coco_panoptic_separated(name, metadata, 
        image_root, panoptic_root, panoptic_json, sem_seg_root, instances_json).

My_code

register_coco_panoptic_separated(subset+"_"+tr1.backbone_config, {}, 
        join(tr1.dataset_dir,"images/"+subset), 
        join(tr1.dataset_dir,"masks/"+subset), join(anns_semantic_segm_dir, subset+".json"), 
        join(tr1.dataset_dir,"masks/"+subset), join(anns_instance_segm_dir, subset+".json"))

Appendix1: Panoptic annotations format

{
 "info": {...},
 "images": [
    {
      "id": 41,
      "width": 512,
      "height": 384,
      "file_name": "image_253.jpg",
      "date_captured": "2021-02-23 21:14:28.054086"
    }, ...
    ... 
  ],
  "annotations": [
    {
      "image_id": 1,
      "file_name": "image_213.png",
      "segments_info": [
        {
          "id": 1,
          "category_id": 1,
          "bbox": [
            0,
            207,
            511,
            176
          ],
          "iscrowd": 0
        }
      ]
    }, ...
    ...
    "categories": [{...},
      {...},
      ...]
}

[mvdelt]

@JavierClearImageAI Thank you for your detailed answer. So, are you using Panoptic FPN? (not panoptic deeplab?)
I guess register_coco_panoptic_separated is only for Panoptic FPN.

[JavierClearImageAI]

@JavierClearImageAI Thank you for your detailed answer. So, are you using Panoptic FPN? (not panoptic deeplab?)
I guess register_coco_panoptic_separated is only for Panoptic FPN.

I don't think detectron2 has panoptic deeplab backbone network implemented. It seems to be a Google research group backbone net. At least I couldn't see it in configuration options. They have 4 fpn backbone nets for panoptic only. Please let me know if I am wrong

[bowenc0221]

The Panoptic-DeepLab project in Detectron2 might be a good reference. The detectron2.data.datasets.register_coco_panoptic loads the raw panoptic annotation (from both png and json files), and you can process this data depending on the need of your model (example).

[tiusty]

@JavierClearImageAI
Just to clarify a few things:

When you say

I realized that using detectron2.data.datasets.register_coco_panoptic wasn't working for a custom dataset with new categories, since it registers a “standard” version of COCO panoptic.

Standard means that it is follows the same format from the COCO dataset and uses the same category ids that are already defined.

Custom means that is follows the same format from the COCO dataset but uses new category ids. Therefore if I want to train an already pre-trained PanopticFPN model from the model zoo with new categories, I would have to use a custom dataset in COCO format with the new category ids.

It is confusing to me because "standard" version would seem like it is referring just to the format, but it seems like it is also referring to the already defined categories.

Therefore to train a model with new categories we need to use the
register_coco_panoptic_separated
function like you mentioned.

Lastly when you say

So, just follow the "coco standard panoptic format" for the panoptic coco-json and the "coco standard instance segmentation format" for the instances.

I understand the coco standard panoptic format but is the coco standard instance segmentation format the object detection format from https://cocodataset.org/#format-data (i.e # 1 on the list)?

Thanks!

[zhangliyun9120]

@JavierClearImageAI
Hello, thank you for advice! But can you share a whole training panoptic example code for us reference?

And do you know how to use panoptic result to do a evaluation caculate? I just found the PQ metric caculation need the json file and PNG file, don't we use the panoptic result directly? Thank you.

[JavierClearImageAI]

@JavierClearImageAI
Hello, thank you for advice! But can you share a whole training panoptic example code for us reference?

And do you know how to use panoptic result to do a evaluation caculate? I just found the PQ metric caculation need the json file and PNG file, don't we use the panoptic result directly? Thank you.

@zhangliyun9120, I am sorry. My code is property of my company and part of our pipeline. Nevertheless I can do something better, I will create a Pull Request to automate the process of generating the json annotations and data registering so you only need to pass a directory with the png masks. I will integrate it with detectron2 and share it here as well. I will try to do it this weekend. Regards

[zhangliyun9120]

Thank you
But currently I think the whole training code of panoptic FPN model (contain how to register panoptic format dataset and train) is most we wanna.
Can you share it to us like the follow link code framework (he failed to show the normal dataset rigestering although)?
Thank you so much!

https://www.celantur.com/blog/panoptic-segmentation-in-detectron2/

Looking forward to hearing soon.

[zhangliyun9120]

@JavierClearImageAI
Hello, thank you for advice! But can you share a whole training panoptic example code for us reference?
And do you know how to use panoptic result to do a evaluation caculate? I just found the PQ metric caculation need the json file and PNG file, don't we use the panoptic result directly? Thank you.

@zhangliyun9120, I am sorry. My code is property of my company and part of our pipeline. Nevertheless I can do something better, I will create a Pull Request to automate the process of generating the json annotations and data registering so you only need to pass a directory with the png masks. I will integrate it with detectron2 and share it here as well. I will try to do it this weekend. Regards

Can you do that? thanks

[JavierClearImageAI]

@JavierClearImageAI
Hello, thank you for advice! But can you share a whole training panoptic example code for us reference?
And do you know how to use panoptic result to do a evaluation caculate? I just found the PQ metric caculation need the json file and PNG file, don't we use the panoptic result directly? Thank you.

@zhangliyun9120, I am sorry. My code is property of my company and part of our pipeline. Nevertheless I can do something better, I will create a Pull Request to automate the process of generating the json annotations and data registering so you only need to pass a directory with the png masks. I will integrate it with detectron2 and share it here as well. I will try to do it this weekend. Regards

Can you do that? thanks

I just said I will

[JavierClearImageAI]

Thank you
But currently I think the whole training code of panoptic FPN model (contain how to register panoptic format dataset and train) is most we wanna.
Can you share it to us like the follow link code framework (he failed to show the normal dataset rigestering although)?
Thank you so much!

https://www.celantur.com/blog/panoptic-segmentation-in-detectron2/

Looking forward to hearing soon.

I don't have time to write a post for you, I am very sorry.
This should be enough: #1691 (comment) . Otherwise you will have to wait till I create the PR or just find out yourself.

regards

[manchr180]

I am not sure what the "sem_seg_root" should contain. I do not think "sem_seg_root" should be the same as "panoptic_root" which contains the panoptic masks. Should the "sem_seg_root" directory contain greyscale masks for the stuff categories? And if so what values do these masks have? The category ids of the stuff classes as they appear in the instances JSON?

[zhangliyun9120]

I am not sure what the "sem_seg_root" should contain. I do not think "sem_seg_root" should be the same as "panoptic_root" which contains the panoptic masks. Should the "sem_seg_root" directory contain greyscale masks for the stuff categories? And if so what values do these masks have? The category ids of the stuff classes as they appear in the instances JSON?

You are right. Let me conclude:
1, "custom dataset means those dataset are create by people themselves not including new class or not including new class; If you use the dataset you created otherwise coco-official datasets, you will have to register them by yourself;
2, Why they didn't provide a unified panpoptic-model-traning instruction, since the panoptic segmentation approach is "Panoptic FPN" which used instance and semantic method separately and combined their results finally; So you have to code by yourself to combine instance and semantic json files;
3, So the feasible approach is to extract instance and semantic segmentation info separately and then to generate them for json files and save, finally use register_coco_panoptic_separated interface to register them and train. So "sem_seg_root" is semantic sgementation PNG files not the same as "panoptic_root". As follow, shown.
4, Finally, if you wanna solve problem, just go into your codes, no one will help you to coding, just do it.

Good luck!

[manchr180]

Ok, but since register_coco_panoptic_seperated takes as input both the instances JSON for the things categories and the semantic segmentation greyscale masks for the stuff categories why does it need the panoptic masks and the panoptic JSON as well?

[zhangliyun9120]

Ok, but since register_coco_panoptic_seperated takes as input both the instances JSON for the things categories and the semantic segmentation greyscale masks for the stuff categories why does it need the panoptic masks and the panoptic JSON as well?

If you saw the formats of Instance, semantic, and panoptic annotation, you will understand. Because as for seperated the sequence is different.

And I have already finished the panoptic FPN model training codes, considering that I also spent mush time on PanopticFPN model training and no one provide some real-code-help, so in order to provide after-coming people convinence, I will paste my worked codes here to help people who fused, as follow, pls refer to it:

(note that, the things and stuffs are just for visualization, so in in Visualizer.py , draw_panoptic_seg(), you need to align the index, e.g. actually things is 0~ 79, stuff is 80~133, but I setup stuffs as a 54-list, so we have to do:
real_id = category_idx - 80
text = self.metadata.stuff_classes[real_id]
)

Train:

# 1, if your dataset is in COCO format, directlt use this for registering:
root_dir = 'Our'
data_name = 'panoptic-training'
image_root = os.path.join(root_dir, "Panoptic_images")
panoptic_root = os.path.join(root_dir, "Panoptic_annotations")
panoptic_json = os.path.join(root_dir, "Panoptic_annotations.json")
with open(panoptic_json) as panoptic_json_file:
    panoptic_dict = json.load(panoptic_json_file)
sem_seg_root = os.path.join(root_dir, "Semantic_segmentations")
instances_json = os.path.join(root_dir, "Instance_annotations.json")
register_coco_panoptic_separated(data_name, {}, image_root, panoptic_root, panoptic_json, sem_seg_root,
                                 instances_json)

tings = ['person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'traffic light',
         'fire hydrant', 'stop sign', 'parking meter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow',
         'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee',
         'skis', 'snowboard', 'sports ball', 'kite', 'baseball bat', 'baseball glove', 'skateboard', 'surfboard',
         'tennis racket', 'bottle', 'wine glass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple',
         'sandwich', 'orange', 'broccoli', 'carrot', 'hot dog', 'pizza', 'donut', 'cake', 'chair', 'couch',
         'potted plant', 'bed', 'dining table', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard',
         'cell phone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors',
         'teddy bear', 'hair drier', 'toothbrush']

stuffs = ['things', 'banner', 'blanket', 'bridge', 'cardboard', 'counter', 'curtain', 'door-stuff', 'floor-wood',
          'flower', 'fruit', 'gravel', 'house', 'light', 'mirror-stuff', 'net', 'pillow', 'platform',
          'playingfield', 'railroad', 'river', 'road', 'roof', 'sand', 'sea', 'shelf', 'snow', 'stairs', 'tent',
          'towel', 'wall-brick', 'wall-stone', 'wall-tile', 'wall-wood', 'water', 'window-blind', 'window', 'tree',
          'fence', 'ceiling', 'sky', 'cabinet', 'table', 'floor', 'pavement', 'mountain', 'grass', 'dirt', 'paper',
          'food', 'building', 'rock', 'wall', 'rug']

MetadataCatalog.get("panoptic-training_separated").set(thing_classes=tings, stuff_classes=stuffs)

config_file = "COCO-PanopticSegmentation/panoptic_fpn_R_50_3x.yaml"
cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file(config_file))
cfg.DATASETS.TRAIN = ("thermal-panoptic-training_separated",)
cfg.DATASETS.TEST = ()
cfg.DATALOADER.NUM_WORKERS = 4
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url(config_file)  # Let training initialize from model zoo
cfg.SOLVER.IMS_PER_BATCH = 1
cfg.SOLVER.BASE_LR = 0.00025  # pick a good LR
# 300 iterations seems good enough for this toy dataset; you may need to train longer for a practical dataset
cfg.SOLVER.MAX_ITER = 200
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128  # faster, and good enough for this toy dataset (default: 512)
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 134  # 134 classes
cfg.MODEL.SEM_SEG_HEAD.NUM_CLASSES = 134
cfg.INPUT.MASK_FORMAT = "bitmask"

os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
trainer = DefaultTrainer(cfg)
trainer.resume_or_load(resume=False)
trainer.train()

Inference:

# Check result
cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, "model_final.pth")
predictor = DefaultPredictor(cfg)
# for single test image
im = cv2.imread("Our/Panoptic_images/1000.jpg")
panoptic_seg, segments_info = predictor(im)["panoptic_seg"]
v = Visualizer(im[:, :, ::-1], MetadataCatalog.get(cfg.DATASETS.TRAIN[0]), scale=1.2)
v = v.draw_panoptic_seg_predictions(panoptic_seg.to("cpu"), segments_info)
# panoptic segmentation result
plt.imshow(v.get_image())
plt.show()

[manchr180]

Thanks for your engagement! I have managed to train the panoptic model using almost identical code to the one you provided. However, why do you set cfg.MODEL.SEM_SEG_HEAD.NUM_CLASSES to 134 and cfg.MODEL.ROI_HEADS.NUM_CLASSES to 134? From what I understand, ROI_HEADS.NUM_CLASSES is used for the thing classes and SEM_SEG_HEAD.NUM_CLASSES for the stuff classes. So for your case ROI_HEADS.NUM_CLASSES should be 80 and SEM_SEG_HEAD.NUM_CLASSES should be 54. Setting both to 134 won't throw an error but you set redundant heads for both the roi and seg head making the network harder to train. Furthermore, your code does not show how the network uses the panoptic masks and the panoptic json. I tried training setting both the panoptic_json and the panoptic_root to empty strings "" in register_coco_panoptic_separated and it works fine. So I still do not understand the use of these arguments.

[JavierClearImageAI]

@JavierClearImageAI
Just to clarify a few things:

When you say

I realized that using detectron2.data.datasets.register_coco_panoptic wasn't working for a custom dataset with new categories, since it registers a “standard” version of COCO panoptic.

Standard means that it is follows the same format from the COCO dataset and uses the same category ids that are already defined.

Custom means that is follows the same format from the COCO dataset but uses new category ids. Therefore if I want to train an already pre-trained PanopticFPN model from the model zoo with new categories, I would have to use a custom dataset in COCO format with the new category ids.

It is confusing to me because "standard" version would seem like it is referring just to the format, but it seems like it is also referring to the already defined categories.

Therefore to train a model with new categories we need to use the
register_coco_panoptic_separated
function like you mentioned.

Finally, when you say

So, just follow the "coco standard panoptic format" for the panoptic coco-json and the "coco standard instance segmentation format" for the instances.

I understand the coco standard panoptic format but is the coco standard instance segmentation format the object detection format from https://cocodataset.org/#format-data (i.e # 1 on the list)?

Thanks!

When I say standard I mean coco-json format (check Coco dataset webpage or other related sources). When I say custom dataset I mean your own dataset with different classes. Your custom dataset also needs to follow the standard format

[zhangliyun9120]

Thanks for your engagement! I have managed to train the panoptic model using almost identical code to the one you provided. However, why do you set cfg.MODEL.SEM_SEG_HEAD.NUM_CLASSES to 134 and cfg.MODEL.ROI_HEADS.NUM_CLASSES to 134? From what I understand, ROI_HEADS.NUM_CLASSES is used for the thing classes and SEM_SEG_HEAD.NUM_CLASSES for the stuff classes. So for your case ROI_HEADS.NUM_CLASSES should be 80 and SEM_SEG_HEAD.NUM_CLASSES should be 54. Setting both to 134 won't throw an error but you set redundant heads for both the roi and seg head making the network harder to train. Furthermore, your code does not show how the network uses the panoptic masks and the panoptic json. I tried training setting both the panoptic_json and the panoptic_root to empty strings "" in register_coco_panoptic_separated and it works fine. So I still do not understand the use of these arguments.

I think you are right, but why I changed it cannot work, as follow:

#1201 (comment)

How do you setup?

[manchr180]

What error does it throw? That is how I made it work:

with open("categories.json") as json_file: # A json file describing all the categories (including the background) according to the
categories = json.load(json_file) #coco panoptic guidelines

images_dir = "images"
panoptic_json = "panoptic.json"
panoptic_root = "Panoptic_masks"
sem_seg_root_train = "Segmentation_masks"
instances_json = "instances.json"

register_name = "myDataset"
register_coco_panoptic_separated(register_name, {}, images_dir, panoptic_root, panoptic_json,
sem_seg_root, instances_json=instances_json)

dataset_dicts = DatasetCatalog.get(register_name)
metadata = MetadataCatalog.get(register_name)

stuff_names = [f["name"] for f in categories if f["isthing"] == 0]
stuff_ids = [f["id"] for f in categories if f["isthing"] == 0]
stuff_dataset_id_to_contiguous_id = dict(zip(stuff_ids,list(range(0,len(stuff_ids)))))

metadata.dict["stuff_classes"] = stuff_names
metadata.dict["stuff_dataset_id_to_contiguous_id"] = stuff_dataset_id_to_contiguous_id

cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-PanopticSegmentation/panoptic_fpn_R_50_1x.yaml"))
cfg.DATASETS.TRAIN = (register_name)
cfg.DATALOADER.NUM_WORKERS = 2
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-PanopticSegmentation/panoptic_fpn_R_50_1x.yaml")
#cfg.MODEL.DEVICE='cpu'
cfg.SOLVER.IMS_PER_BATCH = 2
cfg.SOLVER.BASE_LR = 0.00025
cfg.SOLVER.MAX_ITER = 20000
cfg.MODEL.SEM_SEG_HEAD.NUM_CLASSES = len(stuff_names)
cfg.MODEL.ROI_HEADS.NUM_CLASSES = len(metadata.dict["thing_dataset_id_to_contiguous_id"])

os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
trainer = DefaultTrainer(cfg)
trainer.resume_or_load(resume=False)
trainer.train()

[zhangliyun9120]

I think

What error does it throw? That is how I made it work:

with open("categories.json") as json_file: # A json file describing all the categories (including the background) according to the
categories = json.load(json_file) #coco panoptic guidelines

images_dir = "images"
panoptic_json = "panoptic.json"
panoptic_root = "Panoptic_masks"
sem_seg_root_train = "Segmentation_masks"
instances_json = "instances.json"

register_name = "myDataset"
register_coco_panoptic_separated(register_name, {}, images_dir, panoptic_root, panoptic_json,
sem_seg_root, instances_json=instances_json)

dataset_dicts = DatasetCatalog.get(register_name)
metadata = MetadataCatalog.get(register_name)

stuff_names = [f["name"] for f in categories if f["isthing"] == 0]
stuff_ids = [f["id"] for f in categories if f["isthing"] == 0]
stuff_dataset_id_to_contiguous_id = dict(zip(stuff_ids,list(range(0,len(stuff_ids)))))

metadata.dict["stuff_classes"] = stuff_names
metadata.dict["stuff_dataset_id_to_contiguous_id"] = stuff_dataset_id_to_contiguous_id

cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-PanopticSegmentation/panoptic_fpn_R_50_1x.yaml"))
cfg.DATASETS.TRAIN = (register_name)
cfg.DATALOADER.NUM_WORKERS = 2
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-PanopticSegmentation/panoptic_fpn_R_50_1x.yaml")
#cfg.MODEL.DEVICE='cpu'
cfg.SOLVER.IMS_PER_BATCH = 2
cfg.SOLVER.BASE_LR = 0.00025
cfg.SOLVER.MAX_ITER = 20000
cfg.MODEL.SEM_SEG_HEAD.NUM_CLASSES = len(stuff_names)
cfg.MODEL.ROI_HEADS.NUM_CLASSES = len(metadata.dict["thing_dataset_id_to_contiguous_id"])

os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
trainer = DefaultTrainer(cfg)
trainer.resume_or_load(resume=False)
trainer.train()

I think your codes have some big problem:

thing_dataset_id_to_contiguous_id you even didn't define, ?
can you just check your final
cfg.MODEL.SEM_SEG_HEAD.NUM_CLASSES and cfg.MODEL.ROI_HEADS.NUM_CLASSES value for me?
are you sure your code work well??

[manchr180]

cfg.MODEL.SEM_SEG_HEAD.NUM_CLASSES = 15
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 15

This is as it should as I have 15 things and 15 stuff classes. I did not have to define thing_dataset_id_to_contiguous_id as it was already in the metadata after I run:

metadata = MetadataCatalog.get(register_name)

metadata.dict["stuff_classes"] and metadata.dict["stuff_dataset_id_to_contiguous_id"] were missing from metadata so I put them in the metadata manually.

[zhangliyun9120]

cfg.MODEL.SEM_SEG_HEAD.NUM_CLASSES = 15
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 15

This is as it should as I have 15 things and 15 stuff classes. I did not have to define thing_dataset_id_to_contiguous_id as it was already in the metadata after I run:

metadata = MetadataCatalog.get(register_name)

metadata.dict["stuff_classes"] and metadata.dict["stuff_dataset_id_to_contiguous_id"] were missing from metadata so I put them in the metadata manually.

Can you show me your categories.json file?
I think maybe my this file is different with coco-official list, since I found the official list is things + stuffs = 200, but actually when we predict by trained panoptic model, in Visualizer.py , draw_panoptic_seg() the self.metadata.thing_classes = 80 and self.metadata.stuff_classes = 54, so finally output category ids, thing_classes : 0-79, stuff_classes: 0-53, they have overlaps.

But my work, I use the pretrained panoptic model to generate my annotation data, so I put thing_classes and stuff_classes in a whole annotation file for a without overlapping, so my catgegories.json: things 0-79 stuffs 80-133, SO I think this is my error condition.

Am I wrong, if wong for my case, how to setup?

[manchr180]

categories.txt

This is my categories.json file. The categories have a hierarchy field that I wrote and use it to handle overlapping polygons when I produce the semantic segmentation grayscale masks. If a polygon with a lower hierarchy overlaps with one with a higher hierarchy, the latter is ignored for the overlapping region. I had to do that to produce the panoptic masks from the coco detection json using the detection2panoptic_coco_format.py from the coco-panoptic API. Otherwise, it throws errors for overlapping polygons.

I am not sure I understand the rest of what you say. Does your model produce overlaps for stuff and things?

[zhangliyun9120]

categories.txt

This is my categories.json file. The categories have a hierarchy field that I wrote and use it to handle overlapping polygons when I produce the semantic segmentation grayscale masks. If a polygon with a lower hierarchy overlaps with one with a higher hierarchy, the latter is ignored for the overlapping region. I had to do that to produce the panoptic masks from the coco detection json using the detection2panoptic_coco_format.py from the coco-panoptic API. Otherwise, it throws errors for overlapping polygons.

I am not sure I understand the rest of what you say. Does your model produce overlaps for stuff and things?

I checked yours and mine, I think categories.json should be no problem.

I got the things after
dataset_dicts = DatasetCatalog.get("panoptic-training_separated")
metadata_panoptic = MetadataCatalog.get("panoptic-training_separated")

But when I setup
cfg.MODEL.SEM_SEG_HEAD.NUM_CLASSES = 54
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 80

I always got this error,
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: cunn_SpatialClassNLLCriterion_updateOutput_kernel: block: [1,0,0], thread: [425,0,0] Assertion t >= 0 && t < n_classes failed.

Do you know why???

And a small details I wanna consult, you use the instance converter command is with things_only, right? :
python converters/panoptic2detection_coco_format.py
--input_json_file sample_data/panoptic_examples.json
--output_json_file converted_data/panoptic_coco_detection_format_things_only.json
--things_only

Also for semantic converter, how did you setup this value? the default of it is OTHER_CLASS_ID = 183

these are my panoptic.json, categories.json, instance.json.:
Categories.txt
Instance.txt
Panoptic.txt

I'm really confused about:
cfg.MODEL.SEM_SEG_HEAD.NUM_CLASSES
cfg.MODEL.ROI_HEADS.NUM_CLASSES
Does anyone can carefully explain these, plzzz? Why setup is ok, but get a Assertion t >= 0 && t < n_classes failed.
error?

[manchr180]

I did not use --things_only argument in converters/panoptic2detection_coco_format.py. That is not so important however because as I mentioned before I do not think regster_coco_panoptic_seperated actually uses the panoptic masks and panoptic jsons and I did not use them either to produce the grayscale semantic segmentation masks. As for your error, I would suggest debugging it using cfg.MODEL.DEVICE='cpu'. This runs the network on CPU instead of GPU and will hopefully produce a better error message. You should also set the number of workers to zero (cfg.DATALOADER.NUM_WORKERS = 0). I suspect the error has to do with wrong input shape. If you get inside the code and reach the lib/python3.8/site-packages/detectron2/modeling/meta_arch/panoptic_fpn.py script you should check this line

sem_seg_results, sem_seg_losses = self.sem_seg_head(features, gt_sem_seg)

Check gt_sem_seg dimensions and see if they are as they should be. You should get inside this line into the semantic_seg.py and reach the point where the loss is calculated. Check if targets and predictions dimensions match.

[zhangliyun9120]

I did not use --things_only argument in converters/panoptic2detection_coco_format.py. That is not so important however because as I mentioned before I do not think regster_coco_panoptic_seperated actually uses the panoptic masks and panoptic jsons and I did not use them either to produce the grayscale semantic segmentation masks. As for your error, I would suggest debugging it using cfg.MODEL.DEVICE='cpu'. This runs the network on CPU instead of GPU and will hopefully produce a better error message. You should also set the number of workers to zero (cfg.DATALOADER.NUM_WORKERS = 0). I suspect the error has to do with wrong input shape. If you get inside the code and reach the lib/python3.8/site-packages/detectron2/modeling/meta_arch/panoptic_fpn.py script you should check this line

sem_seg_results, sem_seg_losses = self.sem_seg_head(features, gt_sem_seg)

Check gt_sem_seg dimensions and see if they are as they should be. You should get inside this line into the semantic_seg.py and reach the point where the loss is calculated. Check if targets and predictions dimensions match.

This is not dimension problem for sure. Did you check this issue:
#1201 (comment)

I'm very confusing about this, that person setup in both thing and stuff as n, n+1 can work well.
But you setup seperately also ok.
I setup both n, also ok. even I setup things = 134, stuff=132,131, 133 ok, but stuff<=130 will be error.....................???

So what's the correct way? @ppwwyyxx
I searched a blog,
https://www.programmersought.com/article/49992368615/
is it reason?

[zhangliyun9120]

I did not use --things_only argument in converters/panoptic2detection_coco_format.py. That is not so important however because as I mentioned before I do not think regster_coco_panoptic_seperated actually uses the panoptic masks and panoptic jsons and I did not use them either to produce the grayscale semantic segmentation masks. As for your error, I would suggest debugging it using cfg.MODEL.DEVICE='cpu'. This runs the network on CPU instead of GPU and will hopefully produce a better error message. You should also set the number of workers to zero (cfg.DATALOADER.NUM_WORKERS = 0). I suspect the error has to do with wrong input shape. If you get inside the code and reach the lib/python3.8/site-packages/detectron2/modeling/meta_arch/panoptic_fpn.py script you should check this line

sem_seg_results, sem_seg_losses = self.sem_seg_head(features, gt_sem_seg)

Check gt_sem_seg dimensions and see if they are as they should be. You should get inside this line into the semantic_seg.py and reach the point where the loss is calculated. Check if targets and predictions dimensions match.

I think you are wrong, I checked the code actually both
cfg.MODEL.SEM_SEG_HEAD.NUM_CLASSES
cfg.MODEL.ROI_HEADS.NUM_CLASSES
should be same to your total categories, in stuff the other-class is should not setup.

[Jiayi-Pan]

Thanks for your engagement! I have managed to train the panoptic model using almost identical code to the one you provided. However, why do you set cfg.MODEL.SEM_SEG_HEAD.NUM_CLASSES to 134 and cfg.MODEL.ROI_HEADS.NUM_CLASSES to 134? From what I understand, ROI_HEADS.NUM_CLASSES is used for the thing classes and SEM_SEG_HEAD.NUM_CLASSES for the stuff classes. So for your case ROI_HEADS.NUM_CLASSES should be 80 and SEM_SEG_HEAD.NUM_CLASSES should be 54. Setting both to 134 won't throw an error but you set redundant heads for both the roi and seg head making the network harder to train. Furthermore, your code does not show how the network uses the panoptic masks and the panoptic json. I tried training setting both the panoptic_json and the panoptic_root to empty strings "" in register_coco_panoptic_separated and it works fine. So I still do not understand the use of these arguments.

I've read the source code, panoptic_json and panoptic_root are only used to register the metadata, however, cite from the document:

• panoptic_root, panoptic_json: Used by COCO-format panoptic evaluation.

So indeed, you can ignore them for the training part

[Phylanxy]

I am trying to implement this approach for a custom dataset. I was able to register my dataset with the nuilt in function --> detectron2.data.datasets.register_coco_panoptic_separated.
In my dataset I have two categories: 1. Room 2. Corridor and I want Room to be thing and Corridor to be stuff. But when registering the dataset, Corridors are assigned to things if I include them in my instances.json (object detection annotation) and if I don't include them, they aren't included as stuff when I use the built in Visualizer to check examples of my dataset. Now, I'm not sure if that affects the training or not but I feel like stuff should to be correctly visualized with the Visualizer.

I included the category IDs in my sem_seg png files in the corresponding pixels for all the Corridors (my only stuff class).

Any leads on what I could have done wrong?

[userwatch]

Hello, I am trying to train a panoptic segmentation model on Detectoron2 with a custom dataset.
I labeled maskesense.ai .This is my polygyon json label file. Do I need to convert it to a binary mask firsly step? Thanks a lot. I would love any advice. I'm a lost :)

{
"info": {
"description": "my-project-name"
},
"images": [
{
"id": 1,
"width": 5663,
"height": 3664,
"file_name": "ex1.JPG"
},
{
"id": 2,
"width": 5716,
"height": 3711,
"file_name": ".ex1.JPG"
}
],
"annotations": [
{
"id": 0,
"iscrowd": 0,
"image_id": 1,
"category_id": 2,
"segmentation": [
[
985.6315432230631,
305.01250000000016,
513.7148765563965,
588.1625000000001,
1794.631543223063,
722.9958333333335,
2981.1648765563964,
722.9958333333335,
3783.4232098897296,
669.0625000000001,
3884.5482098897296,
527.4875000000002,
3776.681543223063,
305.01250000000016,
2940.714876556396,
372.42916666666684,
2165.4232098897296,
392.6541666666668,
1491.256543223063,
358.9458333333335,
924.9565432230631,
345.46250000000015,
945.181543223063,
338.72083333333353
]
],
"bbox": [
513.7148765563965,
305.01250000000016,
3370.833333333333,
417.98333333333335
],
"area": 1048237.6766319443
},
{
"id": 1,
"iscrowd": 0,
"image_id": 1,
"category_id": 2,
"segmentation": [
[
1518.2232098897298,
1417.3875,
1814.856543223063,
1646.6041666666667,
2563.181543223063,
1666.8291666666667,
3507.0148765563963,
1646.6041666666667,
4349.723209889729,
1714.0208333333335,
4936.24820988973,
1720.7625,
5077.82320988973,
1673.5708333333334,
4821.639876556396,
1491.5458333333333,
4686.806543223063,
1424.1291666666668,
4504.781543223063,
1275.8125000000002,
3985.6732098897296,
1316.2625,
3163.1898765563965,
1275.8125000000002,
2603.631543223063,
1269.0708333333334,
2071.0398765563964,
1262.3291666666669,
1774.406543223063,
1262.3291666666669,
1282.2648765563963,
1282.5541666666668,
931.6982098897298,
1275.8125000000002,
763.1565432230631,
1275.8125000000002,
837.3148765563965,
1484.8041666666668,
877.7648765563964,
1612.8958333333335,
1221.5898765563963,
1693.7958333333333,
1538.4482098897297,
1660.0875
]
],
"bbox": [
763.1565432230631,
1262.3291666666669,
4314.666666666667,
458.43333333333317
],
"area": 1518441.3700694435
},
{
"id": 2,
"iscrowd": 0,
"image_id": 2,
"category_id": 2,
"segmentation": [
[
457.27987540108813,
327.8309523809523,
355.2084468296596,
586.4119047619047,
1035.68463730585,
715.7023809523808,
2022.375113496326,
715.7023809523808,
2784.5084468296595,
708.897619047619,
3546.6417801629927,
702.092857142857,
3886.879875401088,
695.2880952380951,
3893.68463730585,
423.09761904761893,
3846.0513039725165,
341.4404761904761,
2975.041780162993,
375.4642857142856,
2621.194161115374,
382.2690476190475,
2206.1036849248976,
321.0261904761904,
1838.6465420677548,
300.61190476190467,
879.1751134963263,
273.392857142857,
538.937018258231,
273.392857142857
]
],
"bbox": [
355.2084468296596,
273.392857142857,
3538.476190476191,
442.30952380952385
],
"area": 1307022.0019614515
},
{
"id": 3,
"iscrowd": 0,
"image_id": 2,
"category_id": 2,
"segmentation": [
[
3206.403684924898,
1423.397619047619,
2587.170351591564,
1566.2976190476188,
2852.5560658772783,
1641.1499999999999,
3703.1513039725164,
1627.540476190476,
4274.751303972516,
1634.3452380952378,
4812.3274944487075,
1647.9547619047617,
5261.4417801629925,
1661.5642857142855,
5288.6608277820405,
1525.4690476190474,
5254.6370182582305,
1334.935714285714,
5207.003684924897,
1328.1309523809523,
4519.722732543945,
1368.9595238095237,
4036.58463730585,
1334.935714285714,
3669.1274944487072,
1321.3261904761903,
3335.694161115374,
1368.9595238095237,
3240.427494448707,
1423.397619047619
]
],
"bbox": [
2587.170351591564,
1321.3261904761903,
2701.4904761904763,
340.2380952380952
],
"area": 681073.9240022673
}
],
"categories": [
{
"id": 1,
"name": "ggg"
},
{
"id": 2,
"name": "ss"
}
]
}

[sila-a0]

Ok, but since register_coco_panoptic_seperated takes as input both the instances JSON for the things categories and the semantic segmentation greyscale masks for the stuff categories why does it need the panoptic masks and the panoptic JSON as well?

If you saw the formats of Instance, semantic, and panoptic annotation, you will understand. Because as for seperated the sequence is different.

And I have already finished the panoptic FPN model training codes, considering that I also spent mush time on PanopticFPN model training and no one provide some real-code-help, so in order to provide after-coming people convinence, I will paste my worked codes here to help people who fused, as follow, pls refer to it:

(note that, the things and stuffs are just for visualization, so in in Visualizer.py , draw_panoptic_seg(), you need to align the index, e.g. actually things is 0~ 79, stuff is 80~133, but I setup stuffs as a 54-list, so we have to do: real_id = category_idx - 80 text = self.metadata.stuff_classes[real_id] )

Train:

# 1, if your dataset is in COCO format, directlt use this for registering:
root_dir = 'Our'
data_name = 'panoptic-training'
image_root = os.path.join(root_dir, "Panoptic_images")
panoptic_root = os.path.join(root_dir, "Panoptic_annotations")
panoptic_json = os.path.join(root_dir, "Panoptic_annotations.json")
with open(panoptic_json) as panoptic_json_file:
    panoptic_dict = json.load(panoptic_json_file)
sem_seg_root = os.path.join(root_dir, "Semantic_segmentations")
instances_json = os.path.join(root_dir, "Instance_annotations.json")
register_coco_panoptic_separated(data_name, {}, image_root, panoptic_root, panoptic_json, sem_seg_root,
                                 instances_json)

tings = ['person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'traffic light',
         'fire hydrant', 'stop sign', 'parking meter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow',
         'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee',
         'skis', 'snowboard', 'sports ball', 'kite', 'baseball bat', 'baseball glove', 'skateboard', 'surfboard',
         'tennis racket', 'bottle', 'wine glass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple',
         'sandwich', 'orange', 'broccoli', 'carrot', 'hot dog', 'pizza', 'donut', 'cake', 'chair', 'couch',
         'potted plant', 'bed', 'dining table', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard',
         'cell phone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors',
         'teddy bear', 'hair drier', 'toothbrush']

stuffs = ['things', 'banner', 'blanket', 'bridge', 'cardboard', 'counter', 'curtain', 'door-stuff', 'floor-wood',
          'flower', 'fruit', 'gravel', 'house', 'light', 'mirror-stuff', 'net', 'pillow', 'platform',
          'playingfield', 'railroad', 'river', 'road', 'roof', 'sand', 'sea', 'shelf', 'snow', 'stairs', 'tent',
          'towel', 'wall-brick', 'wall-stone', 'wall-tile', 'wall-wood', 'water', 'window-blind', 'window', 'tree',
          'fence', 'ceiling', 'sky', 'cabinet', 'table', 'floor', 'pavement', 'mountain', 'grass', 'dirt', 'paper',
          'food', 'building', 'rock', 'wall', 'rug']

MetadataCatalog.get("panoptic-training_separated").set(thing_classes=tings, stuff_classes=stuffs)

config_file = "COCO-PanopticSegmentation/panoptic_fpn_R_50_3x.yaml"
cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file(config_file))
cfg.DATASETS.TRAIN = ("thermal-panoptic-training_separated",)
cfg.DATASETS.TEST = ()
cfg.DATALOADER.NUM_WORKERS = 4
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url(config_file)  # Let training initialize from model zoo
cfg.SOLVER.IMS_PER_BATCH = 1
cfg.SOLVER.BASE_LR = 0.00025  # pick a good LR
# 300 iterations seems good enough for this toy dataset; you may need to train longer for a practical dataset
cfg.SOLVER.MAX_ITER = 200
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128  # faster, and good enough for this toy dataset (default: 512)
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 134  # 134 classes
cfg.MODEL.SEM_SEG_HEAD.NUM_CLASSES = 134
cfg.INPUT.MASK_FORMAT = "bitmask"

os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
trainer = DefaultTrainer(cfg)
trainer.resume_or_load(resume=False)
trainer.train()

Inference:

# Check result
cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, "model_final.pth")
predictor = DefaultPredictor(cfg)
# for single test image
im = cv2.imread("Our/Panoptic_images/1000.jpg")
panoptic_seg, segments_info = predictor(im)["panoptic_seg"]
v = Visualizer(im[:, :, ::-1], MetadataCatalog.get(cfg.DATASETS.TRAIN[0]), scale=1.2)
v = v.draw_panoptic_seg_predictions(panoptic_seg.to("cpu"), segments_info)
# panoptic segmentation result
plt.imshow(v.get_image())
plt.show()

Hello @zhangliyun9120 Can you share these files with me ? https://cocodataset.org/#format-data I can't download from here. I need an example of json files and masks. Best regards,

> panoptic_root = os.path.join(root_dir, "Panoptic_annotations")
> panoptic_json = os.path.join(root_dir, "Panoptic_annotations.json")
> with open(panoptic_json) as panoptic_json_file:
>     panoptic_dict = json.load(panoptic_json_file)
> sem_seg_root = os.path.join(root_dir, "Semantic_segmentations")
> instances_json = os.path.join(root_dir, "Instance_annotations.json")