Train Model Fail

[saleed]

2020-05-24 22:11:10.842379: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at strided_slice_op.cc:107 : Invalid argument: slice index 25 of dimension 1 out of bounds.
2020-05-24 22:11:10.842005: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at strided_slice_op.cc:107 : Invalid argument: slice index 2 of dimension 1 out of bounds.
2020-05-24 22:11:10.841913: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at strided_slice_op.cc:107 : Invalid argument: slice index 9 of dimension 1 out of bounds.
2020-05-24 22:11:10.842368: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at strided_slice_op.cc:107 : Invalid argument: slice index 30 of dimension 1 out of bounds.
2020-05-24 22:11:10.842348: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at strided_slice_op.cc:107 : Invalid argument: slice index 30 of dimension 1 out of bounds.
2020-05-24 22:11:10.842397: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at strided_slice_op.cc:107 : Invalid argument: slice index 27 of dimension 1 out of bounds.
2020-05-24 22:11:10.842546: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at strided_slice_op.cc:107 : Invalid argument: slice index 17 of dimension 1 out of bounds.
2020-05-24 22:11:10.842558: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at strided_slice_op.cc:107 : Invalid argument: slice index 20 of dimension 1 out of bounds.
2020-05-24 22:11:10.842418: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at strided_slice_op.cc:107 : Invalid argument: slice index 25 of dimension 1 out of bounds.
2020-05-24 22:11:10.842037: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at strided_slice_op.cc:107 : Invalid argument: slice index 2 of dimension 1 out of bounds.
2020-05-24 22:11:10.842358: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at strided_slice_op.cc:107 : Invalid argument: slice index 30 of dimension 1 out of bounds.
2020-05-24 22:11:10.842510: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at strided_slice_op.cc:107 : Invalid argument: slice index 21 of dimension 1 out of bounds.
2020-05-24 22:11:10.841844: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at strided_slice_op.cc:107 : Invalid argument: slice index 9 of dimension 1 out of bounds.
2020-05-24 22:11:10.842509: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at strided_slice_op.cc:107 : Invalid argument: slice index 22 of dimension 1 out of bounds.
2020-05-24 22:11:10.842212: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at strided_slice_op.cc:107 : Invalid argument: slice index 14 of dimension 1 out of bounds.
2020-05-24 22:11:10.842472: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at strided_slice_op.cc:107 : Invalid argument: slice index 21 of dimension 1 out of bounds.
2020-05-24 22:11:10.842292: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at strided_slice_op.cc:107 : Invalid argument: slice index 28 of dimension 1 out of bounds.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1356, in _do_call
return fn(*args)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [2,2] vs. [32,2]
[[{{node gradients_1/mul_12_grad/BroadcastGradientArgs}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train_frcnn.py", line 283, in
loss_class = model_classifier.train_on_batch([X, X2[:, sel_samples, :]], [Y1[:, sel_samples, :], Y2[:, sel_samples, :]])
File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 1621, in train_on_batch
outputs = self.train_function(ins)
File "/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py", line 2103, in call
feed_dict=feed_dict)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 950, in run
run_metadata_ptr)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1350, in _do_run
run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [2,2] vs. [32,2]
[[node gradients_1/mul_12_grad/BroadcastGradientArgs (defined at /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:2138) ]]

Original stack trace for 'gradients_1/mul_12_grad/BroadcastGradientArgs':
File "train_frcnn.py", line 283, in
loss_class = model_classifier.train_on_batch([X, X2[:, sel_samples, :]], [Y1[:, sel_samples, :], Y2[:, sel_samples, :]])
File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 1620, in train_on_batch
self._make_train_function()
File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 1002, in _make_train_function
self.total_loss)
File "/usr/local/lib/python3.6/dist-packages/keras/optimizers.py", line 381, in get_updates
grads = self.get_gradients(loss, params)
File "/usr/local/lib/python3.6/dist-packages/keras/optimizers.py", line 47, in get_gradients
grads = K.gradients(loss, params)
File "/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py", line 2138, in gradients
return tf.gradients(loss, variables, colocate_gradients_with_ops=True)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_impl.py", line 158, in gradients
unconnected_gradients)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_util.py", line 731, in _GradientsHelper
lambda: grad_fn(op, *out_grads))
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_util.py", line 403, in _MaybeCompile
return grad_fn() # Exit early
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_util.py", line 731, in
lambda: grad_fn(op, *out_grads))
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_grad.py", line 1044, in _MulGrad
rx, ry = gen_array_ops.broadcast_gradient_args(sx, sy)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 829, in broadcast_gradient_args
"BroadcastGradientArgs", s0=s0, s1=s1, name=name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 2005, in init
self._traceback = tf_stack.extract_stack()

...which was originally created as op 'mul_12', defined at:
File "train_frcnn.py", line 176, in
model_classifier.compile(optimizer=optimizer_classifier, loss=[losses.class_loss_cls, losses.class_loss_regr(len(classes_count)-1)], metrics={'dense_class_{}'.format(len(classes_count)): 'accuracy'})
File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 899, in compile
sample_weight, mask)
File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 430, in weighted
score_array = fn(y_true, y_pred)
File "/workspace/keras_frcnn/losses.py", line 56, in class_loss_cls
return lambda_cls_class * K.mean(categorical_crossentropy(y_true[0, :, :], y_pred[0, :, :]))
File "/usr/local/lib/python3.6/dist-packages/keras/losses.py", line 37, in categorical_crossentropy
return K.categorical_crossentropy(y_pred, y_true)
File "/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py", line 2582, in categorical_crossentropy
return - tf.reduce_sum(target * tf.log(output),
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py", line 884, in binary_op_wrapper
return func(x, y, name=name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py", line 1180, in _mul_dispatch
return gen_math_ops.mul(x, y, name=name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_math_ops.py", line 6490, in mul
"Mul", x=x, y=y, name=name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 2005, in init
self._traceback = tf_stack.extract_stack()

does any body know the reason of the errors??