feat: Support Arrow type Dictionary(_, FixedSizeBinary(_)) when writing Parquet

[albertlockett]

Which issue does this PR close?

• Closes #7445.

Rationale for this change

Support writing parquet from arrow when the Dictionary type is used and the values in the dictionary are FixedSizeBinary.

What changes are included in this PR?

The type is now supported. We treat the type as a byte array (similar to what we would do the arrow type Dictionary(_, Utf8))

Are there any user-facing changes?

If the user tries to write parquet file from their arrow record batch, and they use the type Dictionary(_, FixedSizeBinary(_)), the write will no longer fail with the error message:

called `Result::unwrap()` on an `Err` value: NYI("Attempting to write an Arrow type that is not yet implemented")

[albertlockett]

Hey @alamb I fixed the linter failures from this run you triggered:
https://github.com/apache/arrow-rs/actions/runs/14695421193

Would you mind triggering the workflow again?

[westonpace]

Just reviewing the changes (no knowledge of the context) this looks good to me. If I understand correctly you are converting from FSB into Binary for the write path and then casting on the read path?

Are there other Parquet implementations that support this type? Is there maybe a test file we could add to an integration test somewhere (e.g. a file of Dictionary(_, FixedSizeBinary(_)) written by parquet-cpp?

I'm not super familiar with the expectations for tests in parquet. I suspect @alamb or @tustvold might know more.

[westonpace]

Just to be thorough can we iterate through the various key types to ensure we got the match statement in byte_array_dictionary correct?

[albertlockett]

Sure, sg thanks @westonpace. I'll make this change

[albertlockett]

@westonpace made this change

[alamb]

I think the idea of this function is to allow round tripping of Dictionary arrow types via parquet.

I am not sure we have documented the behavior of metadata anywhere -- we probably should

Since the arrow type system and the parquet type system are different, to ensure that arrow-rs parquet can recover the original arrow types if data was written to parquet, it adds metadata into the parquet file to help choose which arrow type to use when there are several potential types for a parquet type (e.g. parquet Binary can go to BinaryView or Binary for example)

[albertlockett]

I think the idea of this function is to allow round tripping of Dictionary arrow types via parquet.

I am not sure we have documented the behavior of metadata anywhere -- we probably should

Since the arrow type system and the parquet type system are different, to ensure that arrow-rs parquet can recover the original arrow types if data was written to parquet, it adds metadata into the parquet file to help choose which arrow type to use when there are several potential types for a parquet type (e.g. parquet Binary can go to BinaryView or Binary for example)

@alamb yes, that's the idea with this change. Thanks for the explainer on the metadata

[alamb]

@alamb yes, that's the idea with this change. Thanks for the explainer on the metadata

No problems -- I also made a PR to update the docs to explain it better

• Document Arrow <--> Parquet schema conversion better #7479

[alamb]

Thank you @albertlockett and @westonpace

[alamb]

@albertlockett can you please add one "end to end" roundtrip test, like this:

arrow-rs/parquet/src/arrow/arrow_reader/mod.rs

Lines 1261 to 1319 in 11c99a3

	#[test]
	fn test_float16_roundtrip() -> Result<()> {
	let schema = Arc::new(Schema::new(vec![
	Field::new("float16", ArrowDataType::Float16, false),
	Field::new("float16-nullable", ArrowDataType::Float16, true),
	]));
	let mut buf = Vec::with_capacity(1024);
	let mut writer = ArrowWriter::try_new(&mut buf, schema.clone(), None)?;
	let original = RecordBatch::try_new(
	schema,
	vec![
	Arc::new(Float16Array::from_iter_values([
	f16::EPSILON,
	f16::MIN,
	f16::MAX,
	f16::NAN,
	f16::INFINITY,
	f16::NEG_INFINITY,
	f16::ONE,
	f16::NEG_ONE,
	f16::ZERO,
	f16::NEG_ZERO,
	f16::E,
	f16::PI,
	f16::FRAC_1_PI,
	])),
	Arc::new(Float16Array::from(vec![
	None,
	None,
	None,
	Some(f16::NAN),
	Some(f16::INFINITY),
	Some(f16::NEG_INFINITY),
	None,
	None,
	None,
	None,
	None,
	None,
	Some(f16::FRAC_1_PI),
	])),
	],
	)?;
	writer.write(&original)?;
	writer.close()?;
	let mut reader = ParquetRecordBatchReader::try_new(Bytes::from(buf), 1024)?;
	let ret = reader.next().unwrap()?;
	assert_eq!(ret, original);
	// Ensure can be downcast to the correct type
	ret.column(0).as_primitive::<Float16Type>();
	ret.column(1).as_primitive::<Float16Type>();
	Ok(())
	}

That writes/reads data from an actual parquet file?

Clicked wrong button, needs a round trip test