Skip to content

Commit 0915342

Browse files
authored
doc: Add initial doc how to expand Comet exceptions (#170)
* doc: Add initial doc how to expand Comet exceptions * doc: Add initial doc how to expand Comet exceptions. refmt * doc: Add initial doc how to expand Comet exceptions. refmt
1 parent ccc4a95 commit 0915342

File tree

1 file changed

+60
-0
lines changed

1 file changed

+60
-0
lines changed

DEBUGGING.md

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,3 +94,63 @@ https://mail.openjdk.org/pipermail/hotspot-dev/2019-September/039429.html
9494
9595
Detecting the debugger
9696
https://stackoverflow.com/questions/5393403/can-a-java-application-detect-that-a-debugger-is-attached#:~:text=No.,to%20let%20your%20app%20continue.&text=I%20know%20that%20those%20are,meant%20with%20my%20first%20phrase).
97+
98+
# Verbose debug
99+
By default, Comet outputs the exception details specific for Comet.
100+
101+
```scala
102+
scala> spark.sql("my_failing_query").show(false)
103+
104+
24/03/05 17:00:07 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)/ 1]
105+
org.apache.comet.CometNativeException: Internal error: MIN/MAX is not expected to receive scalars of incompatible types (Date32("NULL"), Int32(15901)).
106+
This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker
107+
at org.apache.comet.Native.executePlan(Native Method)
108+
at org.apache.comet.CometExecIterator.executeNative(CometExecIterator.scala:65)
109+
at org.apache.comet.CometExecIterator.getNextBatch(CometExecIterator.scala:111)
110+
at org.apache.comet.CometExecIterator.hasNext(CometExecIterator.scala:126)
111+
112+
```
113+
114+
There is a verbose exception option by leveraging DataFusion [backtraces](https://arrow.apache.org/datafusion/user-guide/example-usage.html#enable-backtraces)
115+
This option allows to append native DataFusion stacktrace to the original error message.
116+
To enable this option with Comet it is needed to include `backtrace` feature in [Cargo.toml](https://github.com/apache/arrow-datafusion-comet/blob/main/core/Cargo.toml) for DataFusion dependencies
117+
118+
```
119+
datafusion-common = { version = "36.0.0", features = ["backtrace"] }
120+
datafusion = { default-features = false, version = "36.0.0", features = ["unicode_expressions", "backtrace"] }
121+
```
122+
123+
Then build the Comet as [described](https://github.com/apache/arrow-datafusion-comet/blob/main/README.md#getting-started)
124+
125+
Start Comet with `RUST_BACKTRACE=1`
126+
127+
```commandline
128+
RUST_BACKTRACE=1 $SPARK_HOME/spark-shell --jars spark/target/comet-spark-spark3.4_2.12-0.1.0-SNAPSHOT.jar --conf spark.sql.extensions=org.apache.comet.CometSparkSessionExtensions --conf spark.comet.enabled=true --conf spark.comet.exec.enabled=true --conf spark.comet.exec.all.enabled=true
129+
```
130+
131+
Get the expanded exception details
132+
```scala
133+
scala> spark.sql("my_failing_query").show(false)
134+
24/03/05 17:00:49 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
135+
org.apache.comet.CometNativeException: Internal error: MIN/MAX is not expected to receive scalars of incompatible types (Date32("NULL"), Int32(15901))
136+
137+
backtrace:
138+
0: std::backtrace::Backtrace::create
139+
1: datafusion_physical_expr::aggregate::min_max::min
140+
2: <datafusion_physical_expr::aggregate::min_max::MinAccumulator as datafusion_expr::accumulator::Accumulator>::update_batch
141+
3: <futures_util::stream::stream::fuse::Fuse<S> as futures_core::stream::Stream>::poll_next
142+
4: comet::execution::jni_api::Java_org_apache_comet_Native_executePlan::{{closure}}
143+
5: _Java_org_apache_comet_Native_executePlan
144+
(reduced)
145+
146+
This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker
147+
at org.apache.comet.Native.executePlan(Native Method)
148+
at org.apache.comet.CometExecIterator.executeNative(CometExecIterator.scala:65)
149+
at org.apache.comet.CometExecIterator.getNextBatch(CometExecIterator.scala:111)
150+
at org.apache.comet.CometExecIterator.hasNext(CometExecIterator.scala:126)
151+
(reduced)
152+
153+
```
154+
Note:
155+
- The backtrace coverage in DataFusion is still improving. So there is a chance the error still not covered, if so feel free to file a [ticket](https://github.com/apache/arrow-datafusion/issues)
156+
- The backtrace evaluation comes with performance cost and intended mostly for debugging purposes

0 commit comments

Comments
 (0)