[FR] Support non-binary/grayscale masks in fo.Detection

[Laurent2916]

Proposal Summary

Currently fo.Detection supports:

an instance segmentation mask for the detection within its bounding box, which should be a 2D binary or 0/1 integer numpy array

In practice it is possible to store a [0, 1] float numpy array (i.e. a continuous mask), since there doesn't seem to be checks in place to prevent that. This is very useful when using/training segmentation models that operate on such continuous masks (i.e. non binary/thresholded masks) like MVANet, RMBG-1.4, BiRefNet, etc.

I was wondering if supporting continuous mask would be feasible in Fiftyone, because continuous masks currently are thresholded (at 0) in the UI, which often makes visualizing them difficult. Having the ability to enable, disable and change the threshold value in the UI would be very useful.

Note: Using continuous masks probably breaks some methods such as to_polyline and to_segmentation, though I don't use them often so I don't know.

What areas of FiftyOne does this feature affect?

• App: FiftyOne application
• Core: Core fiftyone Python library
• Server: FiftyOne server

Willingness to contribute

The FiftyOne Community welcomes contributions! Would you or another member of your organization be willing to contribute an implementation of this feature?

• Yes. I can contribute this feature independently
• Yes. I would be willing to contribute this feature with guidance from the FiftyOne community
• No. I cannot contribute this feature at this time

[deltheil]

[...] continuous masks currently are thresholded (at 0) in the UI [...] Having the ability to enable, disable and change the threshold value in the UI would be very useful.

I had a look at the Looker package which takes care of this. So roughly:

• The FiftyOne server graphql API returns masks as they are stored in Mongo, i.e. base64-encoded and zlib compressed numpy format (e.g. see mainSampleQuery)
• They are deserialized client-side part of numpy.ts
• There is a corresponding painter which prepares the overlay pixels drawn in a HTML canvas

FWIW, I have tested this quick-n-dirty patch which is enough in our case (but this is a poor man solution):

--- a/app/packages/looker/src/numpy.ts
+++ b/app/packages/looker/src/numpy.ts
@@ -158,6 +158,13 @@ function parse(array: Uint8Array): OverlayMask {
           rawData.byteLength / ArrayType.BYTES_PER_ELEMENT
         );

+  // Binarize mask with a hard coded threshold (= 127)
+  if (ArrayType === Uint8Array) {
+    for (let i = 0; i < typedData.length; i++) {
+      typedData[i] = typedData[i] >= 127 ? 255 : 0;
+    }
+  }
+
   return {
     arrayType: typedData.constructor.name,
     buffer: typedData.buffer,

Ideally, it would be great if the Segmentation painter (from PainterFactory) could take care of this and thus accept additional arguments to pass an optional threshold and/or adjust mask opacity (alpha matting). Of course, the UI should expose these additional options.

cc @brimoor does that make sense? Could we achieve something similar via the plugins framework?

[brimoor]

cc @benjaminpkane @sashankaryal: I'm onboard with supporting floating point instance masks. What do you think?

[benjaminpkane]

I defer to @sashankaryal

[sashankaryal]

I'm open to it. With the work we did in #5120, it should be an easy lift. I'll own it.

[deltheil]

I'm open to it. With the work we did in #5120, it should be an easy lift. I'll own it.

That's great!

Just to be clear re: continuous masks, and to avoid any ambiguity w.r.t. "floating point" instance masks. In our case, we are currently storing masks with dtype=np.uint8 in the fiftyone.core.labels.Detection.mask field (i.e. with (integer) values between 0 and 255, instead of just 0/1). This works like a charm generally speaking.

As stated above, we would greatly benefit from 1st class citizen support of such masks within the Looker package and on the UI side:

Having the ability to enable, disable and change the threshold value in the UI would be very useful.

Hope that helps.