chore: upgrade to datafusion 43

[ion-elgreco]

Description

Bump kernel

• closes Schema metadata descriptions cause write failures in deltalake >= 0.19.0 #2850

[rtyler]

We need Datafusion to upgrade as well before we can do this. I have some patches to make the latest datafusion work but we're blocked on them to adopt the newest arrow and kernel

[Xuanwo]

Hi, datafusion has been released. Let's rock!

[alamb]

Thank you @ion-elgreco -- let me know if there is anything I can to do help with this PR (e.g. I could work on updating the code to use non deprecated APIs in the core for parquet statistics...)

[ion-elgreco]

Hey @alamb, I am not working on this atm, but @rtyler has picked it up from here

[ion-elgreco]

@alamb there seems to be a regression in DF, see https://github.com/delta-io/delta-rs/actions/runs/10959345721/job/30431504526?pr=2886#step:7:948.

A literal with a list is creating a dtype list with inner type using the name "item", we however have our data internally read according to the parquet spec, so that when we write the parquet files are meeting the field naming of the spec.

I had a quick glance at the Changelog but can't really pinpoint where this change may have occurred, do you have any ideas?

DFSchema { inner: 
Schema { fields: [
Field { name: "items_in_bucket", data_type: List(Field { name: "element", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, 
new schema: 
DFSchema { inner: Schema { fields: [
Field { name: "items_in_bucket", data_type: List(Field { name: "item", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} },

[alamb]

@alamb there seems to be a regression in DF, see https://github.com/delta-io/delta-rs/actions/runs/10959345721/job/30431504526?pr=2886#step:7:948.

A literal with a list is creating a dtype list with inner type using the name "item", we however have our data internally read according to the parquet spec, so that when we write the parquet files are meeting the field naming of the spec.

It sounds somewhat similar to a discussion happening here about nullability (but I think field name is similar in terms of how it is treated): apache/datafusion#11989 (comment)

Update; filed apache/datafusion#12560 to track the regression more

[rtyler]

@alamb thanks for following up! I feel a little sheepish 🐑 insofar that my internal CI showed these errors weeks ago in my integration testing between datafusion main with delta-rs main, I just didn't have the time to triage them like @ion-elgreco has 😄

I've subscribed to the issue you created and can help testing if necessary! Depending on my availability this weekend I can try fixing it in datafusion too 🤔

[matthewmturner]

Hi there - I believe a fix was implemented in datafusion and this PR is not blocked any longer.

[ion-elgreco]

Hi there - I believe a fix was implemented in datafusion and this PR is not blocked any longer.

It's not yet released though

[rtyler]

I can confirm that the latest datafusion main passes tests in CI 🥳

[alamb]

fix

I filed apache/datafusion#12813 to consider releasing a patch version of DataFusion

[ion-elgreco]

@alamb is there any change still required? Because I still see failed tests

[alamb]

@alamb is there any change still required? Because I still see failed tests

I don't know to be honest.

[rtyler]

@ion-elgreco don't worry about this pull request, I will be taking care of it. The 43 release is imminent 🎊

[matthewmturner]

Hi there - just doing a pulse check on this PR. It looks like its been updated to DataFusion v43 but I still see several failing CI checks. Are there new regressions in v43?

[hntd187]

@matthewmturner I just did some more fixes on our side, I noticed we still have what appears to be a regression https://github.com/delta-io/delta-rs/actions/runs/11860379826/job/33055504796?pr=2886#step:4:2436 could someone take a look at this?

[timsaucer]

I think the problem is that the projected schema for array types is changing the field name for the elements of the array from "element" to "item". I have seen this before, but I'm not sure exactly where that is happening. I will try to keep digging in. Which is to say, I think the problem may be upstream in datafusion.

[rtyler]

This last test failure I believe is because of the use of the make_array UDF which expects items in the array to be item rather than element.

I'm not convinced this is a Datafusion issue per se, I'm still digging through some code to see if they've standardized on using "element" vs "item". Either way I don't want to wait for datafusion 44 to merge this change, so I'll be figuring out a workaround shortly 🤗

[timsaucer]

Please let me know what you discover. I do expect we may release a DF 43.1.0 due to a number of changes recently to account for the new default StringView instead of String. If the root problem is on the DF side, we could potentially include it as well.

[rtyler]

@timsaucer My preference would certainly be for make_array to use element for the field name of the elements in the array 😆 We have adopted that because it's consistent with how Apache Parquet does things. In this case, the test itself is using make_array so I'm going to update the test to create the ListArray the Good Ol Fashioned Way ™️ and add a warning for folks sending in ListArrays with item element names.

[hntd187]

Is it perhaps possible to include UDFs that just let you specify an optional field name for the inner field like this? Because I know Map also gets the same treatment of "what are the inner fields called, well we call it this, they call it that." So it might just let more advanced users choose their own adventure instead of having to go back and forth.

[rtyler]

I have rebased on main and consolidated a couple commits. This should be ready to merge if the actions go green @hntd187