You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardexpand all lines: docs/source/user-guide/compatibility.md
+16-6
Original file line number
Diff line number
Diff line change
@@ -32,12 +32,6 @@ be used in production.
32
32
33
33
There is an [epic](https://github.com/apache/datafusion-comet/issues/313) where we are tracking the work to fully implement ANSI support.
34
34
35
-
## Regular Expressions
36
-
37
-
Comet uses the Rust regexp crate for evaluating regular expressions, and this has different behavior from Java's
38
-
regular expression engine. Comet will fall back to Spark for patterns that are known to produce different results, but
39
-
this can be overridden by setting `spark.comet.regexp.allowIncompatible=true`.
40
-
41
35
## Floating number comparison
42
36
43
37
Spark normalizes NaN and zero for floating point numbers for several cases. See `NormalizeFloatingNumbers` optimization rule in Spark.
@@ -46,6 +40,22 @@ because they are handled well in Spark (e.g., `SQLOrderingUtil.compareFloats`).
46
40
functions of arrow-rs used by DataFusion do not normalize NaN and zero (e.g., [arrow::compute::kernels::cmp::eq](https://docs.rs/arrow/latest/arrow/compute/kernels/cmp/fn.eq.html#)).
47
41
So Comet will add additional normalization expression of NaN and zero for comparison.
48
42
43
+
## Incompatible Expressions
44
+
45
+
Some Comet native expressions are not 100% compatible with Spark and are disabled by default. These expressions
46
+
will fall back to Spark but can be enabled by setting `spark.comet.expression.allowIncompatible=true`.
47
+
48
+
## Array Expressions
49
+
50
+
Comet has experimental support for a number of array expressions. These are experimental and currently marked
51
+
as incompatible and can be enabled by setting `spark.comet.expression.allowIncompatible=true`.
52
+
53
+
## Regular Expressions
54
+
55
+
Comet uses the Rust regexp crate for evaluating regular expressions, and this has different behavior from Java's
56
+
regular expression engine. Comet will fall back to Spark for patterns that are known to produce different results, but
57
+
this can be overridden by setting `spark.comet.regexp.allowIncompatible=true`.
58
+
49
59
## Cast
50
60
51
61
Cast operations in Comet fall into three levels of support:
0 commit comments