Instance segmentation: nothing in the prediction

[jeanpat]

Hello,
I try to perform instance segmentation on a small dataset with annotations in coco format. The image loader is the one of the documentation:

datamodule = InstanceSegmentationData.from_coco(train_folder= IMAGE_DIR, train_ann_file = annotation_file,
                                                  transform_kwargs=dict(image_size=(210, 211)), batch_size=2)

It yields:

100%
250/250 [00:00<00:00, 810.88it/s]
INFO     - Autofixing records | icevision.parsers.parser:parse:122
100%
125/125 [00:00<00:00, 1559.83it/s]

The annotations looks like:

print(datamodule.labels)
print(datamodule.num_classes)
['background', 'chrom1', 'chrom2', 'T1', 'T2']

The labels chrom1, chrom2 are polygons and T1 and T2 are dots. The annotation was made with makesense.ai. The annotations were save as two files, a first one in json format for the polygons and a csv for the dots. Polygons annotations look like:

Training seems to go fine (colab+gpu):

trainer = flash.Trainer(max_epochs=5, accelerator="gpu")
INFO:pytorch_lightning.utilities.rank_zero:GPU available: True (cuda), used: True
INFO:pytorch_lightning.utilities.rank_zero:TPU available: False, using: 0 TPU cores
INFO:pytorch_lightning.utilities.rank_zero:IPU available: False, using: 0 IPUs
INFO:pytorch_lightning.utilities.rank_zero:HPU available: False, using: 0 HPUs

with the output:

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/configuration_validator.py:109: PossibleUserWarning: You defined a `validation_step` but have no `val_dataloader`. Skipping val loop.
  category=PossibleUserWarning,
INFO:pytorch_lightning.accelerators.cuda:LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
INFO:pytorch_lightning.callbacks.model_summary:
  | Name          | Type                                 | Params
-----------------------------------------------------------------------
0 | train_metrics | ModuleDict                           | 0     
1 | val_metrics   | ModuleDict                           | 0     
2 | test_metrics  | ModuleDict                           | 0     
3 | adapter       | IceVisionInstanceSegmentationAdapter | 30.9 M
-----------------------------------------------------------------------
16.5 M    Trainable params
14.4 M    Non-trainable params
30.9 M    Total params
123.653   Total estimated model params size (MB)

Epoch 4: 100%
62/62 [00:07<00:00, 8.49it/s, loss=1.28, v_num=4, train_train_loss=1.330]

INFO:pytorch_lightning.utilities.rank_zero:`Trainer.fit` stopped: `max_epochs=5` reached.

But prediction fails:

# 4. Detect objects in a few images!
datamodule = InstanceSegmentationData.from_files(
    predict_files= images,
    batch_size=7,
) #
predictions = trainer.predict(model, datamodule = datamodule)
print(predictions)

It just contains nothing:

INFO:pytorch_lightning.accelerators.cuda:LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

Predicting DataLoader 0: 100%
1/1 [00:00<00:00, 6.65it/s]

[[{'bboxes': [], 'masks': [], 'labels': [], 'scores': []}, {'bboxes': [], 'masks': [], 'labels': [], 'scores': []}, {'bboxes': [], 'masks': [], 'labels': [], 'scores': []}, {'bboxes': [], 'masks': [], 'labels': [], 'scores': []}, {'bboxes': [], 'masks': [], 'labels': [], 'scores': []}, {'bboxes': [], 'masks': [], 'labels': [], 'scores': []}, {'bboxes': [], 'masks': [], 'labels': [], 'scores': []}]]

The thing to remark is that the dots are not visible when an image and its annotations is displayed with pycocotools:

plt.imshow(image, cmap=plt.cm.gray ); plt.axis('off')

annotation_ids = example_coco.getAnnIds(imgIds=image_data['id'], catIds=category_ids, iscrowd=None)
annotations = example_coco.loadAnns(annotation_ids)

example_coco.showAnns(annotations)

Could the dots segmentations be responsible of the failure?
Thanks for your hints