Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modify metadata_register API for support individual metric propagation. #1387

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

yigithanyigit
Copy link
Contributor

In current state of this API it can only propagate explicitly given features, In other words giving an identifier doesn't work.

For example; for propagating PSNR:
psnr_y, psnr_cb, psnr_cr should be given.

With these changes, propagating individual features and propagating with identifier possible.

@yigithanyigit yigithanyigit reopened this Sep 12, 2024
@kylophone
Copy link
Collaborator

I'm not sure if we need an API like this yet (?). I will hold off on the reviewing this PR until we figure that out. In the meantime, I will reply to your patch in ffmpeg-devel.

@yigithanyigit
Copy link
Contributor Author

Thanks for the comment,

I found some problems about this patch and I wanted to add for future reference.

After some tests I realized this patch doesn't work expected in some identifiers (VIF, ADM).
The main reason of the problem those identifiers has some conditional features.

For example:

if (!s->debug) return err;
err |= vmaf_feature_collector_append_with_dict(feature_collector,
s->feature_name_dict, "integer_adm", score, index);
err |= vmaf_feature_collector_append_with_dict(feature_collector,
s->feature_name_dict, "integer_adm_num", score_num, index);
err |= vmaf_feature_collector_append_with_dict(feature_collector,
s->feature_name_dict, "integer_adm_den", score_den, index);
err |= vmaf_feature_collector_append_with_dict(feature_collector,
s->feature_name_dict, "integer_adm_num_scale0", scores[0], index);
err |= vmaf_feature_collector_append_with_dict(feature_collector,
s->feature_name_dict, "integer_adm_den_scale0", scores[1], index);
err |= vmaf_feature_collector_append_with_dict(feature_collector,
s->feature_name_dict, "integer_adm_num_scale1", scores[2], index);
err |= vmaf_feature_collector_append_with_dict(feature_collector,
s->feature_name_dict, "integer_adm_den_scale1", scores[3], index);
err |= vmaf_feature_collector_append_with_dict(feature_collector,
s->feature_name_dict, "integer_adm_num_scale2", scores[4], index);
err |= vmaf_feature_collector_append_with_dict(feature_collector,
s->feature_name_dict, "integer_adm_den_scale2", scores[5], index);
err |= vmaf_feature_collector_append_with_dict(feature_collector,
s->feature_name_dict, "integer_adm_num_scale3", scores[6], index);
err |= vmaf_feature_collector_append_with_dict(feature_collector,
s->feature_name_dict, "integer_adm_den_scale3", scores[7], index);

if (!s->debug) return err;
const double score_num =
(double)vif.scale[0].num + (double)vif.scale[1].num +
(double)vif.scale[2].num + (double)vif.scale[3].num;
const double score_den =
(double)vif.scale[0].den + (double)vif.scale[1].den +
(double)vif.scale[2].den + (double)vif.scale[3].den;
const double score =
score_den == 0.0 ? 1.0f : score_num / score_den;
err |= vmaf_feature_collector_append_with_dict(feature_collector,
s->feature_name_dict, "integer_vif", score, index);
err |= vmaf_feature_collector_append_with_dict(feature_collector,
s->feature_name_dict, "integer_vif_num", score_num, index);
err |= vmaf_feature_collector_append_with_dict(feature_collector,
s->feature_name_dict, "integer_vif_den", score_den, index);
err |= vmaf_feature_collector_append_with_dict(feature_collector,
s->feature_name_dict, "integer_vif_num_scale0", vif.scale[0].num,
index);
err |= vmaf_feature_collector_append_with_dict(feature_collector,
s->feature_name_dict, "integer_vif_den_scale0", vif.scale[0].den,
index);
err |= vmaf_feature_collector_append_with_dict(feature_collector,
s->feature_name_dict, "integer_vif_num_scale1", vif.scale[1].num,
index);
err |= vmaf_feature_collector_append_with_dict(feature_collector,
s->feature_name_dict, "integer_vif_den_scale1", vif.scale[1].den,
index);
err |= vmaf_feature_collector_append_with_dict(feature_collector,
s->feature_name_dict, "integer_vif_num_scale2", vif.scale[2].num,
index);
err |= vmaf_feature_collector_append_with_dict(feature_collector,
s->feature_name_dict, "integer_vif_den_scale2", vif.scale[2].den,
index);
err |= vmaf_feature_collector_append_with_dict(feature_collector,
s->feature_name_dict, "integer_vif_num_scale3", vif.scale[3].num,
index);
err |= vmaf_feature_collector_append_with_dict(feature_collector,
s->feature_name_dict, "integer_vif_den_scale3", vif.scale[3].den,
index);
return err;

As you see those features going to append if debug enabled. However my implementation going to add those features anyway:

static const char *provided_features[] = {
"VMAF_integer_feature_adm2_score", "integer_adm_scale0",
"integer_adm_scale1", "integer_adm_scale2", "integer_adm_scale3",
"integer_adm", "integer_adm_num", "integer_adm_den",
"integer_adm_num_scale0", "integer_adm_den_scale0", "integer_adm_num_scale1",
"integer_adm_den_scale1", "integer_adm_num_scale2", "integer_adm_den_scale2",
"integer_adm_num_scale3", "integer_adm_den_scale3",
NULL
};

We should handle this somehow. Maybe we can add an option to VmafMetadataConfiguration.

Thanks

Yiğithan Yiğit added 2 commits January 29, 2025 04:18
… are ready

The metadata propagation system has been significantly refactored to
reduce complexity and threading issues:

- Changed from per-feature propagation to propagating metadata when VMAF
scores are complete
- Added frame index tracking in metadata handler to ensure ordered
processing
- Added test coverage for non-monotonic frame processing scenarios

This change eliminates the need for complex reordering logic at the
caller level and
reduces potential thread synchronization issues by only propagating
metadata when all
necessary metrics are calculated.
@yigithanyigit
Copy link
Contributor Author

I significantly refactored my patch to escape thread synchronization issue

  • Changed from per-feature propagation to propagating metadata when VMAF
    scores are complete
  • Added frame index tracking in metadata handler to ensure ordered
    processing
  • Added test coverage for non-monotonic frame processing scenarios

This change eliminates the need for complex reordering logic at the caller level and reduces potential thread synchronization issues by only propagating metadata when all necessary metrics are calculated.

@yigithanyigit
Copy link
Contributor Author

@nilfm99 are those failures related to me? It seems like the tox tests are stuck.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants