ValueError: (F1Score|FBetaScore) expects 2D inputs with shape (batch_size, output_dim).

[msusol]

System information.

== check python ===================================================
python version: 3.11.8
python branch: v3.11.8
python build version: ('v3.11.8:db85d51d3e', 'Feb  6 2024 18:02:37')
python compiler version: Clang 13.0.0 (clang-1300.0.29.30)
python implementation: CPython


== check os platform ===============================================

== are we in docker =============================================
No

== compiler =====================================================
Apple clang version 15.0.0 (clang-1500.1.0.2.5)
Target: arm64-apple-darwin23.4.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

== check pips ===================================================
numpy                        1.26.4
protobuf                     4.25.3
tensorflow                   2.16.1
tensorflow-io-gcs-filesystem 0.36.0
tensorflow-metal             1.1.0

== check for virtualenv =========================================
False

== tensorflow import ============================================
tf.version.VERSION = 2.16.1
tf.version.GIT_VERSION = v2.16.1-0-g5bc9d26649c
tf.version.COMPILER_VERSION = Apple LLVM 14.0.3 (clang-1403.0.22.14.1)

== env ==========================================================
LD_LIBRARY_PATH is unset
DYLD_LIBRARY_PATH is unset

== nvidia-smi ===================================================
tf_env_collect.sh: line 147: nvidia-smi: command not found

== cuda libs  ===================================================

== tensorflow installed from info ==================
Name: tensorflow
Version: 2.16.1
Summary: TensorFlow is an open source machine learning framework for everyone.
Home-page: https://www.tensorflow.org/
Author-email: [email protected]
License: Apache 2.0
Location: /Users/marksusol/.virtualenvs/venv-tf16/lib/python3.11/site-packages
Required-by: tf_keras

== python version  ==============================================
(major, minor, micro, releaselevel, serial)
(3, 11, 8, 'final', 0)

== bazel version  ===============================================
Build label: 6.1.0
Build time: Mon Mar 6 17:10:27 2023 (1678122627)
Build timestamp: 1678122627
Build timestamp as int: 1678122627

Describe the problem/behavior.

fbeta_score = tf.keras.metrics.FBetaScore(average='micro', beta=5.0, threshold=0.6, name='fbeta_score')

model.compile(optimizer=optimizer, metrics=[fbeta_score])

ValueError: FBetaScore expects 2D inputs with shape (batch_size, output_dim). Received input shapes: y_pred.shape=(16, 512, 12) and y_true.shape=(16, 512).

Describe the expected behavior.

Should be able to calculate an FBeta score and compile my model with this metric (discovered in Kaggle competition).

Seemed to work with TFA, similar issue as mine.

https://stackoverflow.com/questions/77236233/f1-score-in-tensorflow

Standalone code to reproduce the issue.

Provide a reproducible test case that is the bare minimum necessary to generate
the problem. If possible, please share a link to Colab/Jupyter/any notebook.

competition.ipynb.zip

Source code / logs.

If including tracebacks, please include the full traceback.

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File <timed exec>:111

File ~/.virtualenvs/venv-tf16/lib/python3.11/site-packages/transformers/modeling_tf_utils.py:1170, in TFPreTrainedModel.fit(self, *args, **kwargs)
   1167 @functools.wraps(keras.Model.fit)
   1168 def fit(self, *args, **kwargs):
   1169     args, kwargs = convert_batch_encoding(*args, **kwargs)
-> 1170     return super().fit(*args, **kwargs)

File ~/.virtualenvs/venv-tf16/lib/python3.11/site-packages/tf_keras/src/utils/traceback_utils.py:70, in filter_traceback.<locals>.error_handler(*args, **kwargs)
     67     filtered_tb = _process_traceback_frames(e.__traceback__)
     68     # To get the full stack trace, call:
     69     # `tf.debugging.disable_traceback_filtering()`
---> 70     raise e.with_traceback(filtered_tb) from None
     71 finally:
     72     del filtered_tb

File /var/folders/c3/gnpb2cfd5znckvplsjxhcy2r0000gn/T/__autograph_generated_filethuyzozh.py:15, in outer_factory.<locals>.inner_factory.<locals>.tf__train_function(iterator)
     13 try:
     14     do_return = True
---> 15     retval_ = ag__.converted_call(ag__.ld(step_function), (ag__.ld(self), ag__.ld(iterator)), None, fscope)
     16 except:
     17     do_return = False

File /var/folders/c3/gnpb2cfd5znckvplsjxhcy2r0000gn/T/__autograph_generated_filehvupbbbe.py:45, in outer_factory.<locals>.inner_factory.<locals>.tf__step_function(model, iterator)
     43 ag__.if_stmt(ag__.ld(self).jit_compile, if_body, else_body, get_state, set_state, ('run_step',), 1)
     44 data = ag__.converted_call(ag__.ld(next), (ag__.ld(iterator),), None, fscope)
---> 45 outputs = ag__.converted_call(ag__.ld(model).distribute_strategy.run, (ag__.ld(run_step),), dict(args=(ag__.ld(data),)), fscope)
     46 outputs = ag__.converted_call(ag__.ld(reduce_per_replica), (ag__.ld(outputs), ag__.ld(self).distribute_strategy), dict(reduction=ag__.ld(self).distribute_reduction_method), fscope)
     47 try:

File /var/folders/c3/gnpb2cfd5znckvplsjxhcy2r0000gn/T/__autograph_generated_filehvupbbbe.py:18, in outer_factory.<locals>.inner_factory.<locals>.tf__step_function.<locals>.run_step(data)
     16 do_return_1 = False
     17 retval__1 = ag__.UndefinedReturnValue()
---> 18 outputs = ag__.converted_call(ag__.ld(model).train_step, (ag__.ld(data),), None, fscope_1)
     19 with ag__.ld(tf).control_dependencies(ag__.ld(_minimum_control_deps)(ag__.ld(outputs))):
     20     ag__.converted_call(ag__.ld(model)._train_counter.assign_add, (1,), None, fscope_1)

File /var/folders/c3/gnpb2cfd5znckvplsjxhcy2r0000gn/T/__autograph_generated_fileqsocv9pw.py:411, in outer_factory.<locals>.inner_factory.<locals>.tf__train_step(self, data)
    409     ag__.if_stmt(ag__.ld(loss) is None, if_body_23, else_body_23, get_state_24, set_state_24, ('loss',), 1)
    410 ag__.converted_call(ag__.ld(self).optimizer.minimize, (ag__.ld(loss), ag__.ld(self).trainable_variables), dict(tape=ag__.ld(tape)), fscope)
--> 411 ag__.converted_call(ag__.ld(self).compiled_metrics.update_state, (ag__.ld(y), ag__.ld(y_pred), ag__.ld(sample_weight)), None, fscope)
    412 return_metrics = {}
    414 def get_state_26():

File /var/folders/c3/gnpb2cfd5znckvplsjxhcy2r0000gn/T/__autograph_generated_filewr15gxbw.py:163, in outer_factory.<locals>.inner_factory.<locals>.tf__update_state(self, y_true, y_pred, sample_weight)
    161 continue__1 = ag__.Undefined('continue__1')
    162 metric_objs = ag__.Undefined('metric_objs')
--> 163 ag__.for_stmt(ag__.converted_call(ag__.ld(zip), tuple(ag__.ld(zip_args)), None, fscope), None, loop_body_2, get_state_9, set_state_9, (), {'iterate_names': '(y_t, y_p, sw, metric_objs, weighted_metric_objs)'})

File /var/folders/c3/gnpb2cfd5znckvplsjxhcy2r0000gn/T/__autograph_generated_filewr15gxbw.py:151, in outer_factory.<locals>.inner_factory.<locals>.tf__update_state.<locals>.loop_body_2(itr_2)
    149     nonlocal sw, y_p, y_t
    150     pass
--> 151 ag__.if_stmt(ag__.not_(continue_), if_body_6, else_body_6, get_state_8, set_state_8, ('sw', 'y_p', 'y_t'), 0)

File /var/folders/c3/gnpb2cfd5znckvplsjxhcy2r0000gn/T/__autograph_generated_filewr15gxbw.py:106, in outer_factory.<locals>.inner_factory.<locals>.tf__update_state.<locals>.loop_body_2.<locals>.if_body_6()
    104         pass
    105     ag__.if_stmt(ag__.not_(continue__1), if_body_3, else_body_3, get_state_3, set_state_3, (), 0)
--> 106 ag__.for_stmt(ag__.ld(metric_objs), None, loop_body, get_state_4, set_state_4, (), {'iterate_names': 'metric_obj'})
    108 def get_state_7():
    109     return ()

File /var/folders/c3/gnpb2cfd5znckvplsjxhcy2r0000gn/T/__autograph_generated_filewr15gxbw.py:105, in outer_factory.<locals>.inner_factory.<locals>.tf__update_state.<locals>.loop_body_2.<locals>.if_body_6.<locals>.loop_body(itr)
    103 def else_body_3():
    104     pass
--> 105 ag__.if_stmt(ag__.not_(continue__1), if_body_3, else_body_3, get_state_3, set_state_3, (), 0)

File /var/folders/c3/gnpb2cfd5znckvplsjxhcy2r0000gn/T/__autograph_generated_filewr15gxbw.py:101, in outer_factory.<locals>.inner_factory.<locals>.tf__update_state.<locals>.loop_body_2.<locals>.if_body_6.<locals>.loop_body.<locals>.if_body_3()
    100 def if_body_3():
--> 101     ag__.converted_call(ag__.ld(metric_obj).update_state, (ag__.ld(y_t), ag__.ld(y_p)), dict(sample_weight=ag__.ld(mask)), fscope)

File /var/folders/c3/gnpb2cfd5znckvplsjxhcy2r0000gn/T/__autograph_generated_fileg8quah8b.py:38, in outer_factory.<locals>.inner_factory.<locals>.tf__decorated(metric_obj, *args, **kwargs)
     36 ag__.for_stmt(ag__.ld(metric_obj).weights, None, loop_body, get_state_1, set_state_1, (), {'iterate_names': 'weight'})
     37 with ag__.ld(tf_utils).graph_context_for_symbolic_tensors(*ag__.ld(args), **ag__.ld(kwargs)):
---> 38     result = ag__.converted_call(ag__.ld(update_state_fn), tuple(ag__.ld(args)), dict(**ag__.ld(kwargs)), fscope)
     40 def get_state_2():
     41     return (result,)

File /var/folders/c3/gnpb2cfd5znckvplsjxhcy2r0000gn/T/__autograph_generated_fileuhto43hd.py:15, in outer_factory.<locals>.inner_factory.<locals>.tf__update_state_fn(*args, **kwargs)
     13 try:
     14     do_return = True
---> 15     retval_ = ag__.converted_call(ag__.ld(ag_update_state), tuple(ag__.ld(args)), dict(**ag__.ld(kwargs)), fscope)
     16 except:
     17     do_return = False

File /var/folders/c3/gnpb2cfd5znckvplsjxhcy2r0000gn/T/__autograph_generated_filezy_cvrig.py:37, in outer_factory.<locals>.inner_factory.<locals>.tf__update_state(self, y_true, y_pred, sample_weight)
     35 y_pred, y_true = ag__.converted_call(ag__.ld(losses_utils).squeeze_or_expand_dimensions, (ag__.ld(y_pred), ag__.ld(y_true)), None, fscope)
     36 ag_fn = ag__.converted_call(ag__.ld(tf).__internal__.autograph.tf_convert, (ag__.ld(self)._fn, ag__.converted_call(ag__.ld(tf).__internal__.autograph.control_status_ctx, (), None, fscope)), None, fscope)
---> 37 matches = ag__.converted_call(ag__.ld(ag_fn), (ag__.ld(y_true), ag__.ld(y_pred)), dict(**ag__.ld(self)._fn_kwargs), fscope)
     38 mask = ag__.converted_call(ag__.ld(losses_utils).get_mask, (ag__.ld(matches),), None, fscope)
     39 sample_weight = ag__.converted_call(ag__.ld(losses_utils).apply_valid_mask, (ag__.ld(matches), ag__.ld(sample_weight), ag__.ld(mask), ag__.ld(self).reduction), None, fscope)

File ~/.virtualenvs/venv-tf16/lib/python3.11/site-packages/keras/src/metrics/metric.py:212, in Metric.__call__(self, *args, **kwargs)
    210 def __call__(self, *args, **kwargs):
    211     self._check_super_called()
--> 212     self.update_state(*args, **kwargs)
    213     return self.result()

File ~/.virtualenvs/venv-tf16/lib/python3.11/site-packages/keras/src/metrics/f_score_metrics.py:162, in FBetaScore.update_state(self, y_true, y_pred, sample_weight)
    160 y_pred = ops.convert_to_tensor(y_pred, dtype=self.dtype)
    161 if not self._built:
--> 162     self._build(y_true.shape, y_pred.shape)
    164 if self.threshold is None:
    165     threshold = ops.max(y_pred, axis=-1, keepdims=True)

File ~/.virtualenvs/venv-tf16/lib/python3.11/site-packages/keras/src/metrics/f_score_metrics.py:124, in FBetaScore._build(self, y_true_shape, y_pred_shape)
    122 def _build(self, y_true_shape, y_pred_shape):
    123     if len(y_pred_shape) != 2 or len(y_true_shape) != 2:
--> 124         raise ValueError(
    125             "FBetaScore expects 2D inputs with shape "
    126             "(batch_size, output_dim). Received input "
    127             f"shapes: y_pred.shape={y_pred_shape} and "
    128             f"y_true.shape={y_true_shape}."
    129         )
    130     if y_pred_shape[-1] is None or y_true_shape[-1] is None:
    131         raise ValueError(
    132             "FBetaScore expects 2D inputs with shape "
    133             "(batch_size, output_dim), with output_dim fully "
   (...)
    136             f"y_true.shape={y_true_shape}."
    137         )

ValueError: in user code:

    File "/Users/marksusol/.virtualenvs/venv-tf16/lib/python3.11/site-packages/tf_keras/src/engine/training.py", line 1398, in train_function  *
        return step_function(self, iterator)
    File "/Users/marksusol/.virtualenvs/venv-tf16/lib/python3.11/site-packages/tf_keras/src/engine/training.py", line 1370, in run_step  *
        outputs = model.train_step(data)
    File "/Users/marksusol/.virtualenvs/venv-tf16/lib/python3.11/site-packages/transformers/modeling_tf_utils.py", line 1652, in train_step  *
        self.compiled_metrics.update_state(y, y_pred, sample_weight)
    File "/Users/marksusol/.virtualenvs/venv-tf16/lib/python3.11/site-packages/tf_keras/src/engine/compile_utils.py", line 620, in update_state  *
        metric_obj.update_state(y_t, y_p, sample_weight=mask)
    File "/Users/marksusol/.virtualenvs/venv-tf16/lib/python3.11/site-packages/tf_keras/src/metrics/base_metric.py", line 153, in decorated  *
        result = update_state_fn(*args, **kwargs)
    File "/Users/marksusol/.virtualenvs/venv-tf16/lib/python3.11/site-packages/tf_keras/src/metrics/base_metric.py", line 140, in update_state_fn  *
        return ag_update_state(*args, **kwargs)
    File "/Users/marksusol/.virtualenvs/venv-tf16/lib/python3.11/site-packages/tf_keras/src/metrics/base_metric.py", line 722, in update_state  *
        matches = ag_fn(y_true, y_pred, **self._fn_kwargs)
    File "/Users/marksusol/.virtualenvs/venv-tf16/lib/python3.11/site-packages/keras/src/metrics/metric.py", line 212, in __call__  **
        self.update_state(*args, **kwargs)
    File "/Users/marksusol/.virtualenvs/venv-tf16/lib/python3.11/site-packages/keras/src/metrics/f_score_metrics.py", line 162, in update_state
        self._build(y_true.shape, y_pred.shape)
    File "/Users/marksusol/.virtualenvs/venv-tf16/lib/python3.11/site-packages/keras/src/metrics/f_score_metrics.py", line 124, in _build
        raise ValueError(

    ValueError: FBetaScore expects 2D inputs with shape (batch_size, output_dim). Received input shapes: y_pred.shape=(16, 512, 12) and y_true.shape=(16, 512).

[msusol]

Same error message using tf.keras.metrics.F1Score BTW

[tilakrayal]

@msusol,
I was facing a different error while executing the mentioned code. Could you please share all the dependencies to replicate the issue. Thank you!

[msusol]

I was facing a different error while executing the mentioned code.

@tilakrayal

I will show the pip list installed so my code would work in TF 2.16.1. I know I was running into an issue with datasets and had to downgrade.

See also https://www.kaggle.com/datasets/gdataranger/tf2-16-requirements-txt

# Following is based on https://github.com/huggingface/datasets/issues/6753#issuecomment-2027906687
!pip install -U datasets==2.16.0
!pip install -U fsspec==2023.10.0
!pip install -U gcsfs==2023.10.0
!pip install -U keras==3.0.5
!pip install -U ml-dtypes==0.3.2
!pip install -U numpy==1.26.4
!pip install -U seqeval
!pip install -U spacy-alignments==0.9.1
!pip install -U tensorflow~=2.16
!pip install -U tensorflow-text~=2.16
!pip install -U tf_keras~=2.16
!pip install -U transformers==4.38.2

Could you please share all the dependencies to replicate the issue.

Please find the zipfile from my 'Colab Notebooks' drive folder + data which I ran on Colab directly.
https://github.com/msusol/kaggle/blob/main/competition-20240410T165614Z-001.zip (28MB > 25MB allowed here)

[msusol]

Any work around for this?

[github-actions]

This issue is stale because it has been open for 14 days with no activity. It will be closed if no further activity occurs. Thank you.

[github-actions]

This issue was closed because it has been inactive for 28 days. Please reopen if you'd like to work on this further.

[msusol]

This is an important issue that needs to be resolved

[msusol]

I was facing a different error while executing the mentioned code.

@tilakrayal

I will show the pip list installed so my code would work in TF 2.16.1. I know I was running into an issue with datasets and had to downgrade.

See also https://www.kaggle.com/datasets/gdataranger/tf2-16-requirements-txt

# Following is based on https://github.com/huggingface/datasets/issues/6753#issuecomment-2027906687
!pip install -U datasets==2.16.0
!pip install -U fsspec==2023.10.0
!pip install -U gcsfs==2023.10.0
!pip install -U keras==3.0.5
!pip install -U ml-dtypes==0.3.2
!pip install -U numpy==1.26.4
!pip install -U seqeval
!pip install -U spacy-alignments==0.9.1
!pip install -U tensorflow~=2.16
!pip install -U tensorflow-text~=2.16
!pip install -U tf_keras~=2.16
!pip install -U transformers==4.38.2

Could you please share all the dependencies to replicate the issue.

Please find the zipfile from my 'Colab Notebooks' drive folder + data which I ran on Colab directly. https://github.com/msusol/kaggle/blob/main/competition-20240410T165614Z-001.zip (28MB > 25MB allowed here)

I provided the zip file to reproduce the issue, but the label "waiting on me" hasn't been changed.