Skip to content

Commit 478672b

Browse files
minor: commit compatibility doc (#1358)
Leftover from #1349
1 parent 95c158b commit 478672b

File tree

1 file changed

+16
-6
lines changed

1 file changed

+16
-6
lines changed

docs/source/user-guide/compatibility.md

+16-6
Original file line numberDiff line numberDiff line change
@@ -32,12 +32,6 @@ be used in production.
3232

3333
There is an [epic](https://github.com/apache/datafusion-comet/issues/313) where we are tracking the work to fully implement ANSI support.
3434

35-
## Regular Expressions
36-
37-
Comet uses the Rust regexp crate for evaluating regular expressions, and this has different behavior from Java's
38-
regular expression engine. Comet will fall back to Spark for patterns that are known to produce different results, but
39-
this can be overridden by setting `spark.comet.regexp.allowIncompatible=true`.
40-
4135
## Floating number comparison
4236

4337
Spark normalizes NaN and zero for floating point numbers for several cases. See `NormalizeFloatingNumbers` optimization rule in Spark.
@@ -46,6 +40,22 @@ because they are handled well in Spark (e.g., `SQLOrderingUtil.compareFloats`).
4640
functions of arrow-rs used by DataFusion do not normalize NaN and zero (e.g., [arrow::compute::kernels::cmp::eq](https://docs.rs/arrow/latest/arrow/compute/kernels/cmp/fn.eq.html#)).
4741
So Comet will add additional normalization expression of NaN and zero for comparison.
4842

43+
## Incompatible Expressions
44+
45+
Some Comet native expressions are not 100% compatible with Spark and are disabled by default. These expressions
46+
will fall back to Spark but can be enabled by setting `spark.comet.expression.allowIncompatible=true`.
47+
48+
## Array Expressions
49+
50+
Comet has experimental support for a number of array expressions. These are experimental and currently marked
51+
as incompatible and can be enabled by setting `spark.comet.expression.allowIncompatible=true`.
52+
53+
## Regular Expressions
54+
55+
Comet uses the Rust regexp crate for evaluating regular expressions, and this has different behavior from Java's
56+
regular expression engine. Comet will fall back to Spark for patterns that are known to produce different results, but
57+
this can be overridden by setting `spark.comet.regexp.allowIncompatible=true`.
58+
4959
## Cast
5060

5161
Cast operations in Comet fall into three levels of support:

0 commit comments

Comments
 (0)