Allow retrieving Parquet decryption keys using the key metadata

[adamreeve]

Which issue does this PR close?

• Closes Allow retrieving Parquet decryption keys based on the key metadata #7257.

Rationale for this change

This allows Parquet readers to not need to directly specify the data encryption keys required to read an encrypted Parquet file, and instead be able to retrieve them dynamically based on the key metadata.

This is a prerequisite for implementing the higher level Key Management Tools API (#7256)

What changes are included in this PR?

• Adds a new KeyRetriever trait.
• Updates FileDecryptionProperties so that it can store a KeyRetriever rather than explicit decryption keys.
• Adds a new ColumnCryptoMetaData type corresponding to the type of the same name in the Thrift format, so that this can be parsed from the column metadata and used to retrieve the key metadata to pass to the KeyRetriever.
• Changes how we decide if a column is encrypted. We now read this from the file metadata rather than using the FileDecryptionProperties (so FileDecryptor::is_column_encrypted has been removed).

Are there any user-facing changes?

Yes, this adds a new way for users to create FileDecryptionProperties.

[rok]

Thanks for doing this @adamreeve! I did a rough pass and the proposal looks good. I left some comments and will do another pass tomorrow.

[alamb]

Thank you @adamreeve and @rok for the reviews

[alamb]

Thank you @adamreeve and @rok -- this looks good to me

As part of completing the encryption / decryption feature it will be great to add some more end-to-end examples (e.g. using a custom key source). Buy for now this looks good

[alamb]

As potentially follow on item, I think you could avoid the unreachable! code below by changing this to an enum. Something like

enum DecryptionPropertiesBuilder {
  ExplicitKeys {
    footer_key: Vec<u8>,
    column_keys: HashMap<String, Vec<u8>>,
    aad_prefix: Option<Vec<u8>>,
  },
  Retriever {
    key_retriever: Arc<dyn KeyRetriever>
  }
}

The compiler would force you to then explicitly spell out which APIs could be used in what mode.

[adamreeve]

Good point, I did this for FileDecryptionProperties but I'm not sure why I didn't do it for DecryptionPropertiesBuilder too, I'll follow up and tidy this up later.

[adamreeve]

A bit late but I made a PR to follow up on this here: #7477