Wrong query results for filters that involve partition columns and data file columns and pushdown_filters is enabled

[adriangb]

Describe the bug

Filters such as partition_col = col_from_file are never applied if datafusion.execution.parquet.pushdown_filters = true

To Reproduce

With datafusion-cli:

COPY  (
  SELECT arrow_cast('a', 'Utf8') AS val
)  TO 'test_files/scratch/test/part=a/123.parquet'
STORED AS PARQUET;
COPY  (
  SELECT arrow_cast('b', 'Utf8') AS val
)  TO 'test_files/scratch/test/part=b/123.parquet'
STORED AS PARQUET;
COPY  (
  SELECT arrow_cast('xyz', 'Utf8') AS val
)  TO 'test_files/scratch/test/part=c/123.parquet'
STORED AS PARQUET;

set datafusion.execution.parquet.pushdown_filters = true;

CREATE EXTERNAL TABLE test(part text, val text)
STORED AS PARQUET
PARTITIONED BY (part)
LOCATION 'test_files/scratch/test/';

SELECT * FROM test;

explain analyze
select * from test
where part != val;

> select * from test
where part != val;
+-----+------+
| val | part |
+-----+------+
| a   | a    |
| xyz | c    |
| b   | b    |
+-----+------+
3 row(s) fetched. 

Which is clearly wrong.

Expected behavior

> select * from test
where part != val;
+-----+------+
| val | part |
+-----+------+
| xyz | c    |
+-----+------+

Additional context

No response

[alamb]

sounds serious

[alamb]

This is all the more reason I think to avoid adding more complexity to ListingTable as we are disucssing in

• Support metadata columns (location, size, last_modified) in ListingTableProvider #15173

[adriangb]

I think the fix is relatively simple though: any filters that reference both partition columns and data columns need to be marked as Inexact. I'm traveling so don't know that I can make a PR soon but it's a couple LOC.

[adriangb]

Also although yes this is serious I suspect is pretty rare to have a filter that depends on both a partition column and data column. It hasn't been reported for years...

[alamb]

Also although yes this is serious I suspect is pretty rare to have a filter that depends on both a partition column and data column. It hasn't been reported for years...

I also think it is not that common to turn on filter_pushdown and use it with ListingTable - e.g. InfluxData and Comet both do not use ListingTable 🤔

[adriangb]

Good point

[adriangb]

Looking at the git blame... it looks like I introduced the bug in #15263.
Feeling guilty so here is the fix: #15935