Document Arrow <--> Parquet schema conversion better

[alamb]

Which issue does this PR close?

Closes #.

Rationale for this change

There have been several questions / changes related to Arrow and Parquet schema recently for example

• feat: Support round trip reading/writing Arrow type Dictionary(_, FixedSizeBinary(_)) to Parquet #7446 (comment) from @albertlockett
• Support different TimeUnits and timezones when reading Timestamps from INT96 #7220 from @mbutrovich
• write duration to parquet but read as int64 #5625 from @Liyixin95

I realized the high level behavior was not well documented anywhere so I wanted to do that (so I have a place to refer people to mostly)

What changes are included in this PR?

Document the schema conversion process

Are there any user-facing changes?

Documentation only changes, no code or behavior changes

[mbutrovich]

Thanks @alamb. I was just looking at this again because I'd like to see a coercion in DF to keep dictionaries if possible. Currently Comet only knows the primitive datatype and ignores encoding when generating the schema, which results in dictionaries being unpacked in the Parquet reader when we might have preferred to keep them encoded. I'll take another pass through this later to think about what might be missing, but first read was good.

[alamb]

Thanks @alamb. I was just looking at this again because I'd like to see a coercion in DF to keep dictionaries if possible.

I think it is always possible to read Parquet data as dictionaries (at least for strings, we might have to add code / tests for other types).

I think it would make sense to allow writing DictionaryArrays for other types if it isn't already supported.

As I recall one potential complication is that the same column is stored in multiple pages, and each page can have different encodings (e.g. some pages are dictionary encoded and some are plain).

[alamb]

I was reminded on #5626 that this metadata is the same format as used by arrow-cpp, which is an important caveat. I will add to this doc

[tustvold]

This is not true, the schema in the parquet file must be authoritative. The arrow schema is merely a hint - see #1663

Edit: it may take precedence over the embedded arrow schema though, I don't recognise this particular codepath

[alamb]

I am not sure what you mean by "authoritative"

What this method does is override any embedded arrow schema hint

I have reworded it - let me know what you think.

[tustvold]

I think the major confusion, which this PR didn't create, but which it also doesn't really address is that the arrow schema provided may not be what the reader actually uses. If say the arrow schema says TimestampNanoseconds, but the parquet is actually TimestampMilliseconds, IIRC it will return TimestampMilliseconds.

[alamb]

The reason for me writing this PR is that I don't think it is clear how parquet / arrow schema conversions are handled, including the embedded arrow schema hint and then the APIs that let people supply / modify their own hint

I think the major confusion, which this PR didn't create, but which it also doesn't really address is that the arrow schema provided may not be what the reader actually uses. If say the arrow schema says TimestampNanoseconds, but the parquet is actually TimestampMilliseconds, IIRC it will return TimestampMilliseconds.

My experience is that if the hint schema is provided but doesn't match what is read from the file, an error is raised:

https://github.com/apache/arrow-rs/blob/812160005efe3afc63531b8ea051e1fa44a91f67/parquet/src/arrow/arrow_reader/mod.rs#L541-L540

called Result::unwrap() on an Err value: ArrowError("incompatible arrow schema, the following fields could not be cast: [column1]")

The error message is actually pretty bad. I'll make a new PR to improve it.

[alamb]

The error message is actually pretty bad. I'll make a new PR to improve it.

• Made Improve error messages if schema hint mismatches with parquet schema #7481

[tustvold]

My experience is that if the hint schema is provided but doesn't match what is read from the file, an error is raised:

Aah yes, I remember now. If you provide a schema it will use it as a hint for the schema inference process, but if that inference process ignores the hints for any reason it will return an error.

If, however, the schema is embedded in the file, it does not error and behaves as I described above

[alamb]

Since I don't see any reason to rush this PR in I'll plan to leave it open for another day or two to have time to gather comments