You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have searched the existing issues, and I could not find an existing issue for this bug
Current Behavior
dbt throws error when spark thrift uses GPU (Rapids)
Expected Behavior
When attempting to use dbt run with Spark Thrift that utilizes the Rapids library, the following error occurs:
org.apache.hive.service.cli.HiveSQLException: Error running query: java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: com/nvidia/spark/rapids/RuleNotFoundExprMeta
at org.apache.spark.sql.hive.thriftserver.HiveThriftServerErrors$.runningQueryError(HiveThriftServerErrors.scala:46)
Steps To Reproduce
My Environment Details:
DBT version : 1.8.3
Spark version : 3.5.0
Rapids : rapids-4-spark_2.12-23.12.0.jar
When i make dbt model and run in spark thrift with rapids library, such error occurred
amychen1776
changed the title
[Bug] Support for Rapids (GPU) with Spark Thrift in DBT
[Feature] Support for Rapids (GPU) with Spark Thrift in DBT
Oct 1, 2024
Is this a new bug in dbt-core?
Current Behavior
dbt throws error when spark thrift uses GPU (Rapids)
Expected Behavior
When attempting to use dbt run with Spark Thrift that utilizes the Rapids library, the following error occurs:
org.apache.hive.service.cli.HiveSQLException: Error running query: java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: com/nvidia/spark/rapids/RuleNotFoundExprMeta
at org.apache.spark.sql.hive.thriftserver.HiveThriftServerErrors$.runningQueryError(HiveThriftServerErrors.scala:46)
Steps To Reproduce
My Environment Details:
DBT version : 1.8.3
Spark version : 3.5.0
Rapids : rapids-4-spark_2.12-23.12.0.jar
When i make dbt model and run in spark thrift with rapids library, such error occurred
Relevant log output
Environment
Which database adapter are you using with dbt?
spark
Additional Context
I posted the issue somewhere else by mistake before, so I'm attaching it again.
[dbt not support rapids spark] (dbt-labs/docs.getdbt.com#6102 (comment))
The text was updated successfully, but these errors were encountered: