Add Parquet Modular decryption (read) support + encryption flag (#6637)

* first commit

* Use ParquetMetaDataReader

* Fix CI

* test

* save progress

* work

* Review feedback

* page decompression issue

* add update_aad

* Change encrypt and decrypt to return Results

* Use correct page ordinal and module type in AADs

* Tidy up ordinal types

* Lint

* Fix regular deserialization path

* cleaning

* Update data checks in test

* start non-uniform decryption

* Add missing doc comments

* Make encryption an optional feature

* Handle when a file is encrypted but encryption is disabled or no decryption properties are provided

* Allow for plaintext footer

* work

* Fix method name

* work

* Minor

* work

* work

* work

* Fix reading to end of file

* Refactor tests

* Fix non-uniform encryption configuration

* Don't use footer key for non-encrypted columns

* Rebase and cleanup

* Cleanup

* Cleanup

* Cleanup

* Cleanup

* Cleanup

* Cleanup

* lint

* Remove encryption setup

* Fix building with ring on wasm

* file_decryptor into a seperate module

* lint

* FileDecryptionProperties should have at least one key

* Move cyphertext reading into decryptor

* More tidy up of footer key handling

* Get column decryptors as RingGcmBlockDecryptor

* Use Arc<dyn BlockDecryptor>

* Fix file metadata tests

* Handle reading plaintext footer files without decryption properties

* Split up encryption modules further

* Error instead of panic for AES-GCM-CTR

* load_async

* new_with_options

* Add tests

* get_metadata

* Add CryptoContext to async_reader

* Add row_group_ordinal to InMemoryRowGroup

* Adjust docstrings

* Apply suggestions from code review

Co-authored-by: Adam Reeve <[email protected]>

* Review feedback

* move file_decryption_properties into ArrowReaderOptions

* make create_page_aad method of CryptoContext

* Review feedback

* Infer ModuleType in create_page_aad

* add create_page_header_aad

* Review feedback

* Update parquet/src/arrow/async_reader/store.rs

Co-authored-by: Ed Seidl <[email protected]>

* Review feedback

* Update parquet/src/encryption/ciphers.rs

Co-authored-by: Adam Reeve <[email protected]>

* Review feedback

* WIP: Decryption shouldn't change the API

* WIP: Decryption shouldn't change the API

* WIP: Decryption shouldn't change the API

* WIP: Decryption shouldn't change the API

* WIP: Decryption shouldn't change the API

* WIP: Decryption shouldn't change the API

* Review feedback

* Handle common encryption errors

Co-authored-by: Corwin Joy <[email protected]>
Co-authored-by: Adam Reeve <[email protected]>

* Apply suggestions from code review

Co-authored-by: Adam Reeve <[email protected]>

* Update parquet/src/arrow/async_reader/mod.rs

Co-authored-by: Adam Reeve <[email protected]>

* Fix previous commit

* Add TestReader::new, less pub functions, add test_non_uniform_encryption_disabled_aad_storage

* Review feedback

* Add new CI check

* Rename decryption module to decrypt. This is because we'll introduce encryption module later and we'll have to name it encrypt to not clash with the super module name (encryption). It would be odd to have sub modules called encrypt and decryption.

* Fix failing where encryption is enabled but no decryption properties are provided or where encyption is disabled but file is encrypted.

* Apply suggestions from code review

Co-authored-by: Adam Reeve <[email protected]>

* get_metadata_with_encryption -> get_metadata_with_options

* Use ParquetMetaData instead of RowGroupMetaData in InMemoryRowGroup. Change row_group_ordinal to row_group_idx.

* Review feedback

* Fixes

* Continue refactor away from encryption specific APIs and fix async reader load

* Add default get_metadata_with_options implementation

* Minor tidy ups and test fix

* Update parquet/src/encryption/mod.rs

* Update parquet/src/lib.rs

Co-authored-by: Rok Mihevc <[email protected]>

---------

Co-authored-by: Gidon Gershinsky <[email protected]>
Co-authored-by: Adam Reeve <[email protected]>
Co-authored-by: Ed Seidl <[email protected]>
Co-authored-by: Corwin Joy <[email protected]>
Co-authored-by: Andrew Lamb <[email protected]>

Author: 6 people

.github/workflows/parquet.yml

@@ -111,6 +111,8 @@ jobs:
         run: cargo check -p parquet --all-targets --all-features
       - name: Check compilation  --all-targets --no-default-features --features json
         run: cargo check -p parquet --all-targets --no-default-features --features json
+      - name: Check compilation --no-default-features --features encryption --features async
+        run: cargo check -p parquet --no-default-features --features encryption --features async
 
   # test the parquet crate builds against wasm32 in stable rust
   wasm32-build:

parquet/Cargo.toml

@@ -30,6 +30,8 @@ rust-version = { workspace = true }
 
 [target.'cfg(target_arch = "wasm32")'.dependencies]
 ahash = { version = "0.8", default-features = false, features = ["compile-time-rng"] }
+# See https://github.com/briansmith/ring/issues/918#issuecomment-2077788925
+ring = { version = "0.17", default-features = false, features = ["wasm32_unknown_unknown_js", "std"], optional = true }
 
 [target.'cfg(not(target_arch = "wasm32"))'.dependencies]
 ahash = { version = "0.8", default-features = false, features = ["runtime-rng"] }
@@ -70,6 +72,7 @@ half = { version = "2.1", default-features = false, features = ["num-traits"] }
 sysinfo = { version = "0.33.0", optional = true, default-features = false, features = ["system"] }
 crc32fast = { version = "1.4.2", optional = true, default-features = false }
 simdutf8 = { version = "0.1.5", optional = true, default-features = false }
+ring = { version = "0.17", default-features = false, features = ["std"], optional = true }
 
 [dev-dependencies]
 base64 = { version = "0.22", default-features = false, features = ["std"] }
@@ -117,6 +120,8 @@ sysinfo = ["dep:sysinfo"]
 crc = ["dep:crc32fast"]
 # Enable SIMD UTF-8 validation
 simdutf8 = ["dep:simdutf8"]
+# Enable Parquet modular encryption support
+encryption = ["dep:ring"]
 
 
 [[example]]

parquet/README.md

@@ -63,6 +63,7 @@ The `parquet` crate provides the following features which may be enabled in your
 - `crc` - enables functionality to automatically verify checksums of each page (if present) when decoding
 - `experimental` - Experimental APIs which may change, even between minor releases
 - `simdutf8` (default) - Use the [`simdutf8`] crate for SIMD-accelerated UTF-8 validation
+- `encryption` - support for reading / writing encrypted Parquet files
 
 [`arrow`]: https://crates.io/crates/arrow
 [`simdutf8`]: https://crates.io/crates/simdutf8
@@ -76,12 +77,14 @@ The `parquet` crate provides the following features which may be enabled in your
   - [x] Row record reader
   - [x] Arrow record reader
   - [x] Async support (to Arrow)
+  - [x] Encrypted files
 - [x] Statistics support
 - [x] Write support
   - [x] Primitive column value writers
   - [ ] Row record writer
   - [x] Arrow record writer
   - [x] Async support
+  - [ ] Encrypted files
 - [x] Predicate pushdown
 - [x] Parquet format 4.0.0 support