[Managed Iceberg] Support writing to partitioned tables

[ahmedabu98]

Fixes #31943

Adds support for writing to partitioned Iceberg tables.

A record writer manager is introduced to open and close writers as necessary. An Iceberg data writer instance is configured to write to only one partition, so multiple writers are needed to write to multiple partitions.

The behavior remains unchanged when writing to unpartitioned tables.

Also some small but key changes:

• Use Iceberg APIs to determine the datafile location (as opposed to hardcoding the location to be at <warehouse>/<table>/<data-file>
• Write manifest files under <table>/metadata/

[github-actions]

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

[ahmedabu98]

assign set of reviewers

[github-actions]

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @m-trieu for label java.
R: @damccorm for label build.
R: @Abacn for label io.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[ahmedabu98]

CC: @arthurpessoa, @chamikaramj

[kennknowles]

Fairly quick pass

[kennknowles]

Based on the documentation, this might be accurately named PartitionedRecordWriter or something. That communicates better than "Manager" which could mean almost anything. And it seems it could be Autocloseable which would enable using it in try-with-resources blocks, no?

[ahmedabu98]

I originally had it as PartitionedRecordWriter but thought it may be misleading since the class is also used for unpartitioned writes

Added Autocloseable implementation

[kennknowles]

Fair enough. A reason I don't like the "Manager" name is that I don't know what it is doing in the managing, and it really doesn't communicate that this is a thing that does the writing. We have so many things called "manager" and none of them have anything in common.

[kennknowles]

LGTM to get it merged for the release.