TypeError: Object of type int64 is not JSON serializable

[VinceKumar]

Running detection2panoptic_coco_format.py first fails on untouched fresh unzipped coco-things

ubuntu@XXXXX:~/GitRepos/panopticapi$ python3 converters/detection2panoptic_coco_format.py --input_json_file /home/ubuntu/coco-things/annotations/instances_val2017.json --output_json_file /home/ubuntu/coco-things/panoptic_things.json
Creating folder /home/ubuntu/coco-things/panoptic_things for panoptic segmentation PNGs
CONVERTING...
COCO detection format:
        JSON file: /home/ubuntu/coco-things/annotations/instances_val2017.json
TO
COCO panoptic format
        Segmentation folder: /home/ubuntu/coco-things/panoptic_things
        JSON file: /home/ubuntu/coco-things/panoptic_things.json


loading annotations into memory...
Done (t=0.47s)
creating index...
index created!
Number of cores: 6, images per core: 834
Core: 0, 0 from 834 images processed
Caught exception in worker thread:
Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/panopticapi/utils.py", line 16, in wrapper
    return f(*args, **kwargs)
  File "/home/ubuntu/GitRepos/panopticapi/converters/detection2panoptic_coco_format.py", line 69, in convert_detection_to_panoptic_coco_format_single_core
    raise Exception("Segments for image {} overlap each other.".format(img_id))
Exception: Segments for image 397133 overlap each other.
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/panopticapi/utils.py", line 20, in wrapper
    raise e
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/panopticapi/utils.py", line 16, in wrapper
    return f(*args, **kwargs)
  File "/home/ubuntu/GitRepos/panopticapi/converters/detection2panoptic_coco_format.py", line 69, in convert_detection_to_panoptic_coco_format_single_core
    raise Exception("Segments for image {} overlap each other.".format(img_id))
Exception: Segments for image 397133 overlap each other.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/GitRepos/panopticapi/converters/detection2panoptic_coco_format.py", line 148, in <module>
    convert_detection_to_panoptic_coco_format(args.input_json_file,
  File "/home/ubuntu/GitRepos/panopticapi/converters/detection2panoptic_coco_format.py", line 118, in convert_detection_to_panoptic_coco_format
    annotations_coco_panoptic.extend(p.get())
  File "/home/ubuntu/miniconda3/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
Exception: Segments for image 397133 overlap each other.
Core: 1, 0 from 834 images processed

I've modified to code to skip over anything with an overlap and then the png maps load but the JSON conversion fails at saving

# modification to skip overlaps
        if np.sum(overlaps_map > 1) != 0:
            bad_img_ids.append(img_id) #init this list as global
            continue

JSON Serialization Error. Something about d_coco['annotations'] fails even when using python ints

Traceback (most recent call last):
  File "/home/ubuntu/GitRepos/panopticapi/converters/detection2panoptic_coco_format.py", line 151, in <module>
    convert_detection_to_panoptic_coco_format(args.input_json_file,
  File "/home/ubuntu/GitRepos/panopticapi/converters/detection2panoptic_coco_format.py", line 127, in convert_detection_to_panoptic_coco_format
    save_json(d_coco, output_json_file)
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/panopticapi/utils.py", line 99, in save_json
    json.dump(d, f)
  File "/home/ubuntu/miniconda3/lib/python3.9/json/__init__.py", line 179, in dump
    for chunk in iterable:
  File "/home/ubuntu/miniconda3/lib/python3.9/json/encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/home/ubuntu/miniconda3/lib/python3.9/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/home/ubuntu/miniconda3/lib/python3.9/json/encoder.py", line 325, in _iterencode_list
    yield from chunks
  File "/home/ubuntu/miniconda3/lib/python3.9/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/home/ubuntu/miniconda3/lib/python3.9/json/encoder.py", line 438, in _iterencode
    o = _default(o)
  File "/home/ubuntu/miniconda3/lib/python3.9/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type int64 is not JSON serializable

    with open(input_json_file, 'r') as f:
        d_coco = json.load(f)
    d_coco['annotations'] = annotations_coco_panoptic
    d_coco['categories'] = categories_list
# converting bounding boxes & area to python integers
    for item in d_coco['annotations']:
        for item2 in item.get('segments_info'):
            if item2.get('bbox'):
                temp_bboxes = [int(num) for num in item2.get('bbox')]
                item2['bbox'] = temp_bboxes
                item2['area'] = int(item2['area'])
    save_json(d_coco, output_json_file) 

Even when I convert both the bounding boxes and area to python integers I still get the error. If I try to copy a sample to a jupyter notebook it has no problem saving to JSON

test1 = {'image_id': 403385, 'file_name': '000000403385.png', 'segments_info': [{'area': 16555, 'iscrowd': 0, 'bbox': [411, 237, 93, 242], 'category_id': 70, 'id': 7906560}, {'area': 7561, 'iscrowd': 0, 'bbox': [9, 313, 142, 78], 'category_id': 81, 'id': 6538619}]}
save_json(test1,output_json_file) # success 

[dokunima]

I have the same problem. I annotated the images using COCO Annotator.
@alexander-kirillov is there a workaround to this issue?

Thank you.

[Chris1nexus]

The fix is simple, you have to typecast img_id -> int(img_id) in line 64 of detection2panoptic_coco_format.py
panoptic_record['image_id'] = INT(img_id)

This is because the code splits the list of image ids with numpy api in line 108, hence all these ints are converted to numpy.int64. With casting to python int, it becomes a serializable object and can be properly saved as json

[dokunima]

Thank you so much @Chris1nexus! I will try that.

[dokunima]

I tried it and it worked! Many thanks @Chris1nexus!!

[Chris1nexus]

no problem m8 @dokunima

[dokunima]

Hi @Chris1nexus, sorry to bother you. I tried using the panoptic .json created for transfer learning on Coco panoptic segmentation (Detectron 2) but I get a KeyError id. I have looked at the .json but there is an "id" field in images. Please see the error below:

extra_annotation_keys)
67 json_file = PathManager.get_local_path(json_file)
68 with contextlib.redirect_stdout(io.StringIO()):
---> 69 coco_api = COCO(json_file)
70 if timer.seconds() > 1:
71 logger.info("Loading {} takes {:.2f} seconds.".format(json_file, timer.seconds()))

/usr/local/lib/python3.7/dist-packages/pycocotools/coco.py in init(self, annotation_file)
84 print('Done (t={:0.2f}s)'.format(time.time()- tic))
85 self.dataset = dataset
---> 86 self.createIndex()
87
88 def createIndex(self):

/usr/local/lib/python3.7/dist-packages/pycocotools/coco.py in createIndex(self)
94 for ann in self.dataset['annotations']:
95 imgToAnns[ann['image_id']].append(ann)
---> 96 anns[ann['id']] = ann
97
98 if 'images' in self.dataset:

KeyError: 'id'

[Chris1nexus]

ok that is about the pycocotools api though, not panopticapi,
notice that the key error happens in ann['id'], that is because the standard for any coco dataset base on object detection requires to set a unique id for each annotation in the dataset (see https://cocodataset.org/#format-data).
Since when you do this :
coco_api = COCO(json_file)
the object indexes the entire dataset, the annotation ID must be unique throughout the dataset, e.g. you can not take annotations of each image, enumerate them and assign them an arbitrary number id.
When using a custom dataset in coco format, you have to take the care to take all annotations, of all images in one big list and start giving them unique ids s.t. anns[your_id] returns one unique annotation associated with your created ids.
So basically do the indexing on annotations only after you know how many of them exist and then do (pseudocode):
annotations =[]
for image in images
anns = getAnnotationsForImage(image)
annotations.extend(anns)

for new_id, ann in enumerate(annotations):
ann['id'] = new_id

then you take the json and set its annotations to the ones that you just modified
coco_json['annotations'] = annotations

with coco_json i am referring to this as specified in the coco standard
{
"info": info, "images": [image], "annotations": [annotation], "licenses": [license],
}

[dokunima]

@Chris1nexus many thanks for the link and the suggestion. I will have a look at the link and try that.

[Chris1nexus]

let me know if it worked also for you @dokunima

[dokunima]

@Chris1nexus. Sure thing. I have been very busy on another project. I will try it when I have some time. Thank you.

[Astrostellar]

The fix is simple, you have to typecast img_id -> int(img_id) in line 64 of detection2panoptic_coco_format.py panoptic_record['image_id'] = INT(img_id)

This is because the code splits the list of image ids with numpy api in line 108, hence all these ints are converted to numpy.int64. With casting to python int, it becomes a serializable object and can be properly saved as json

it works, thanks a lot!!!

[mqbin]

Hi @Chris1nexus, sorry to bother you. I have a custom dataset and want to convert its instance segmentation format to panoptic segmentation format, but I get the following error.Do you know how to fix this error? Please see the error below:

line 68, in convert_detection_to_panoptic_coco_format_single_core raise Exception("Segments for image {} overlap each other.".format(img_id)) Exception: Segments for image 1 overlap each other.

[Chris1nexus]

Hi @Chris1nexus, sorry to bother you. I have a custom dataset and want to convert its instance segmentation format to panoptic segmentation format, but I get the following error.Do you know how to fix this error? Please see the error below:

line 68, in convert_detection_to_panoptic_coco_format_single_core raise Exception("Segments for image {} overlap each other.".format(img_id)) Exception: Segments for image 1 overlap each other.

Hello! Sorry for the late answer, anyway that happens because for a given image, one or more of the associated instance masks overlap(check line 54 to 71 of that script).
I will describe the why of the issue and provide a fix at the end of this comment

The why:
The set intersection of all instance masks associated with a given image is not-empty: this means that a pixel is associated to multiple masks in the current state.
This is not permitted in panoptic segmentation, as is shown in the original panoptic seg paper.
Basically the panoptic segmentation ground truth associated with a given image consists both of instance masks(for things) and semantic masks(for stuff categories).
As all these objects are within the same ground truth format, NO overlap must occurr(e.g. this is to say that you cannot assign to a given pixel of the ground truth two or more categories, like as if a pixel can be both a car and a person)

The fix:
To fix it you can simply modify the conversion code so that for each conflicting pixel only one of the masks is present in that pixel region.
E.g. sort masks by decreasing size, the smaller ones keep only their pixels which do NOT intersect why any of the preceding masks(a similar approach is followed in their original paper).
You SHOULD check what an instance mask is in your custom problem. The above holds for a 2D scene but it might be wrong if 3D structures are considered

[mqbin]

ok that is about the pycocotools api though, not panopticapi, notice that the key error happens in ann['id'], that is because the standard for any coco dataset base on object detection requires to set a unique id for each annotation in the dataset (see https://cocodataset.org/#format-data). Since when you do this : coco_api = COCO(json_file) the object indexes the entire dataset, the annotation ID must be unique throughout the dataset, e.g. you can not take annotations of each image, enumerate them and assign them an arbitrary number id. When using a custom dataset in coco format, you have to take the care to take all annotations, of all images in one big list and start giving them unique ids s.t. anns[your_id] returns one unique annotation associated with your created ids. So basically do the indexing on annotations only after you know how many of them exist and then do (pseudocode): annotations =[] for image in images anns = getAnnotationsForImage(image) annotations.extend(anns)

for new_id, ann in enumerate(annotations): ann['id'] = new_id

then you take the json and set its annotations to the ones that you just modified coco_json['annotations'] = annotations

with coco_json i am referring to this as specified in the coco standard { "info": info, "images": [image], "annotations": [annotation], "licenses": [license], }

Thanks for your suggestion, I will check the code.

[mqbin]

Hi @Chris1nexus , Thank you for your previous reply. I removed this

 if np.sum(overlaps_map > 1) != 0:
     raise Exception("Segments for image {} overlap each other.".format(img_id))

and got the data in panoptic segmentation format, but when I trained with mmdetection's panoptic FPN, I got the following error：

Traceback (most recent call last):
  File "/home/user/mmdetection/tools/train.py", line 242, in <module>
    main()
  File "/home/user/mmdetection/tools/train.py", line 238, in main
    meta=meta)
  File "/home/user/mmdetection/mmdet/apis/train.py", line 244, in train_detector
    runner.run(data_loaders, cfg.workflow)
  File "/home/user/anaconda3/envs/openmmlab/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 136, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/home/user/anaconda3/envs/openmmlab/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 58, in train
    self.call_hook('after_train_epoch')
  File "/home/user/anaconda3/envs/openmmlab/lib/python3.7/site-packages/mmcv/runner/base_runner.py", line 317, in call_hook
    getattr(hook, fn_name)(self)
  File "/home/user/anaconda3/envs/openmmlab/lib/python3.7/site-packages/mmcv/runner/hooks/evaluation.py", line 271, in after_train_epoch
    self._do_evaluate(runner)
  File "/home/user/mmdetection/mmdet/core/evaluation/eval_hooks.py", line 63, in _do_evaluate
    key_score = self.evaluate(runner, results)
  File "/home/user/anaconda3/envs/openmmlab/lib/python3.7/site-packages/mmcv/runner/hooks/evaluation.py", line 368, in evaluate
    results, logger=runner.logger, **self.eval_kwargs)
  File "/home/user/mmdetection/mmdet/datasets/coco_panoptic.py", line 624, in evaluate
    result_files, outfile_prefix, logger, classwise, nproc=nproc)
  File "/home/user/mmdetection/mmdet/datasets/coco_panoptic.py", line 559, in evaluate_pan_json
    self.categories, isthing=isthing)
  File "/home/user/anaconda3/envs/openmmlab/lib/python3.7/site-packages/panopticapi/evaluation.py", line 75, in pq_average
    return {'pq': pq / n, 'sq': sq / n, 'rq': rq / n, 'n': n}, per_class_results

I used a custom dataset, this dataset has coco instance segmentation format, there is only one stuff class is the background class, the other all thing class. I checked the generated panoptic segmentation data and found that each of the images is not labeled with a background class, and I'm not sure if this is the reason for the error.

I want to predict both the instance segmentation mask and the semantic segmentation mask of the image by the panoptic segmentation model, but I am troubled by the panoptic segmentation format conversion for a long time.

[domef]

I have the same error