_internal.DeltaError: Failed to parse parquet: Parquet error: Z-order failed while scanning data

[ZulqarnainB]

Environment

Delta-rs version: 0.25.4

Binding: Python 3.10 (same error occurred using Python 3.12 as well)

Environment:

• Cloud provider: AWS
• OS: Amazon Linux 2, Jupyter Lab 3
• Other: AWS Sagemaker - Instance Type = ml.r6i.8xlarge

---

Bug

What happened: When I try to perform z ordering on a dataset with about 16 million records and 400+ columns, z-ordering fails returning the following error:

_internal.DeltaError: Failed to parse parquet: Parquet error: Z-order failed while scanning data: ResourcesExhausted("Failed to allocate additional 1220294400 bytes for ExternalSorter[15] with 0 bytes already allocated for this reservation - 185759036 bytes remain available for the total pool")

Using following code to z order:

DeltaTable(
table_uri=delta_table_path,
storage_options=storage_options
).optimize.z_order(columns=["customer_seq"],
partition_filters=[("effective_date", "=", run_date.strftime('%Y-%m-%d'))]
)

What you expected to happen: I expected the Z Ordering to work

How to reproduce it:

More details: Data is partitioned, and z-ordering is being performed on a single partition.

[ion-elgreco]

This is an issue upstream in Datafusion: apache/datafusion#12136