ValueError: Expected boxes to be a Tensor with a final dimension of 4

[gH2AX]

My setup:

Tensorflow 2.12
Keras_CV 0.4.2

Tried to use the RetinaNet as described here:

https://keras.io/guides/keras_cv/object_detection_keras_cv/

Got the following error:

…/retina_net.py", line 370, in train_step
        y = bounding_box.convert_format(
ValueError: Expected `boxes` to be a Tensor with a final dimension of `4`. Instead, got `boxes.shape=(None, None, None)`.

I have a data generator which returns two things:

1. The image tensor of shape [batch_size, height, width, 3]
2. The labels as a dictionary:

{
'classes': tf.RaggedTensor[batch_size, box_ids],
'boxes': tf.RaggedTensor[batch_size, num_boxes, x, y, w, h]
}

It's even compatible with the visualisation function from the luketils module:

def visualize_dataset(sample, ids):
    visualization.plot_bounding_box_gallery(
        sample[0] * 255,
        value_range = (0, 255),
        bounding_box_format = "xywh",
        y_true = sample[1],
        scale = 4,
        rows = 2,
        cols = 3,
        show = True,
        thickness = 2,
        font_scale = 1,
        class_mapping = ids
    )
visualize_dataset(data.__getitem__(0), {1: "A", 2: "B"})

Please, please help me! I'm a newbie to CV with neural networks and I've been trying to apply object detection to my own dataset using TF/Keras, to no avail, for many weeks now, fighting through so much code that would all break eventually. I was super optimistic when I found the tutorial by Luke Wood, since it had been updated just last month..

[hugopi]

Hi @gH2AX,

I'm not from keras-cv team but i'm also trying to implement a RetinaNet. This is an example of one of my inputs (batch_size=1):

{'images': <tf.Tensor: shape=(1, 512, 512, 3), dtype=float32, numpy=
array([[[[0.4497083, 0.3386866, 0.9999295],
         [0.45919433, 0.34676725, 0.9999304],
         [0.4490056, 0.35414526, 0.99993014],
         ...,
         [0.4071968, 0.36398265, 0.9999284],
         [0.34887525, 0.34571326, 0.9999249],
         [0.39068407, 0.30285043, 0.999925]],
        [[0.46727502, 0.35555062, 0.99993134],
         [0.44514096, 0.36573932, 0.99993056],
         [0.4184395, 0.37311733, 0.99992937],
         ...,
         [0.36573932, 0.3636313, 0.9999264],
         [0.32849786, 0.34957796, 0.99992424],
         [0.36784732, 0.32428184, 0.9999248]],
        [[0.43951958, 0.33587587, 0.9999288],
         [0.4261689, 0.35238862, 0.99992883],
         [0.4114128, 0.36819866, 0.9999288],
         ...,
         [0.35273996, 0.36046928, 0.9999257],
         [0.3299032, 0.3460646, 0.9999241],
         [0.35414526, 0.32147118, 0.9999241]],
        ...,
        [[0.45954564, 0.33657858, 0.9999299],
         [0.45427564, 0.29160774, 0.9999277],
         [0.4760584, 0.25577164, 0.99992734],
         ...,
         [0.4479516, 0.35063195, 0.9999299],
         [0.45322162, 0.36503667, 0.999931],
         [0.45708632, 0.3689013, 0.99993145]],
        [[0.5168132, 0.28739172, 0.9999312],
         [0.5027598, 0.21290879, 0.9999273],
         [0.46973437, 0.23679952, 0.99992627],
         ...,
         [0.3885761, 0.35519928, 0.99992704],
         [0.40087277, 0.37803605, 0.9999288],
         [0.4261689, 0.3819007, 0.99993026]],
        [[0.4978411, 0.32322782, 0.99993163],
         [0.46586967, 0.27966234, 0.9999278],
         [0.4114128, 0.2986344, 0.99992573],
         ...,
         [0.37100932, 0.35238862, 0.9999262],
         [0.38752207, 0.37136066, 0.9999278],
         [0.41914216, 0.37417132, 0.9999295]]]], dtype=float32)>, 'bounding_boxes': {'boxes': <tf.RaggedTensor [[[60.0, 15.0, 10.0, 10.0]]]>, 'classes': <tf.RaggedTensor [[1.0]]>}}

overview of shapes

         test_dataset.take(1)
<TakeDataset element_spec={'images': TensorSpec(shape=(1, 512, 512, 3), dtype=tf.float32, name=None), 'bounding_boxes': {'boxes': RaggedTensorSpec(TensorShape([1, None, 4]), tf.float32, 1, tf.int64), 'classes': RaggedTensorSpec(TensorShape([1, None]), tf.float32, 1, tf.int64)}}>

I don't know if it's the correct input format but it's the only one I succeed to feed the model with.(batch_size>1 also works)

[gH2AX]

Thank you very much for your suggestion; at least, now I get a different error:

ValueError: Exception encountered when calling layer 'RetinaNet' (type RetinaNet).

Missing data for input "input_1". You passed a data dictionary with keys ['images', 'bounding_boxes']. Expected the following keys: ['input_1']

[jbischof]

@gH2AX we aren't usually able to provide 1:1 consulting but if you upload a gist colab with a minimal example to reproduce either ourselves or the community would be happy to take a look. Thanks!

If you identify a bug please file an issue!

[gH2AX]

I think I found the solution!

So, I didn't understand that the inputs need to be strictly formatted as a Tensorflow dataset in a specific way (did I mention that I'm a beginner?). The issue was that in the tutorial you first apply a bunch of confusing transformations on the pascalvoc dataset, but I had my own data, and I tried to filter out only the information which would've been relevant to my case. And since visualizations from the luketils toolkit rendered correctly, I though I was on the right track. So here's my solution to all the newbies out there:

Let's say you have the following data at hand:

• Images: A np.ndarray of shape (NUM_SAMPLES, HEIGHT, WIDTH, CHANNELS)
• Box_Coordinates: A list with np.ndarray elements of shape (NUM_BOXES, X, Y, W, H) for every sample in the image array
• Category_IDs: Another list with np.ndarray elements of shape (NUM_BOXES, ) for every sample in the image array

And here's the magic:

BATCH_SIZE = 6

train_ds = tf.data.Dataset.from_tensor_slices(
   {
       'images': tf.convert_to_tensor(Images),
       'bounding_boxes': {
          'boxes': tf.ragged.constant(Box_Coordinates, dtype = tf.float32, ragged_rank = 1),
          'classes': tf.ragged.constant(Category_IDs, dtype = tf.float32)
       }
   }
).ragged_batch(BATCH_SIZE, drop_remainder = True)

Now feed this into your Keras_CV model, and you're good to go.