[Managed Iceberg] Make manifest file writes and commits more efficient

[ahmedabu98]

When writing to Iceberg, we need to write just one manifest file per snapshot.

However, we are currently writing one manifest file per bundle (or one per GIB batch for streaming writes), which is a lot more frequent than needed. In medium/large streaming jobs, we can end up with thousands of extra manifest files. For an Iceberg table, the effect of this inefficiency is felt in two ways:

• Writing more files than necessary
• During query planning, having to open and read more files than necessary

Solution:
Continue writing bundles/batches to data files, but stop writing manifest files at that frequency. Instead, group data files by destination then write and commit just one manifest file per destination. Essentially, the number of manifest files should be 1-1 with snapshots/commits (currently, it's roughly 1-1 with data files).

[github-actions]

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

[ahmedabu98]

Added as a release blocker because these are update-incompatible changes. Streaming writes are going to be officially supported in 2.60.0 so this should get in with it to avoid breaking pipeline update

[ahmedabu98]

assign set of reviewers

[github-actions]

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @kennknowles for label java.
R: @damccorm for label build.
R: @chamikaramj for label io.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[Abacn]

Hi, kindly pin about the status of this PR. Since this is added to 2.60.0, could you please request a expedited review?

[chamikaramj]

Thanks!

[chamikaramj]

Thanks. LGTM.

We can merge after open comments are addressed.

[ahmedabu98]

Will merge when tests go green