Skip to content

Commit

Permalink
remove cuDF jar dependency (NVIDIA#71)
Browse files Browse the repository at this point in the history
* remove cuDF jar dependency

Signed-off-by: Allen Xu <[email protected]>

* update doc after remove cudf dependency

Signed-off-by: Allen Xu <[email protected]>
  • Loading branch information
wjxiz1992 authored May 31, 2022
1 parent 8ec1939 commit cde3d2e
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 9 deletions.
5 changes: 2 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,20 +73,19 @@ the way. Make sure the prerequisites are all met, or the build will fail with er
accordingly such as "cmake not found" or "ninja not found" etc.
## How to use
When building the jar, cudf jar and spark-rapids plugin jar will be downloaded to your local maven
When building the jar, spark-rapids plugin jar will be downloaded to your local maven
repository, usually in your `~/.m2/repository`.
Add the artifact jar to the Spark, for example:
```bash
ML_JAR="target/rapids-4-spark-ml_2.12-22.06.0-SNAPSHOT.jar"
CUDF_JAR="~/.m2/repository/ai/rapids/cudf/22.06.0-SNAPSHOT/cudf-22.06.0-SNAPSHOT.jar"
PLUGIN_JAR="~/.m2/repository/com/nvidia/rapids-4-spark_2.12/22.06.0-SNAPSHOT/rapids-4-spark_2.12-22.06.0-SNAPSHOT.jar"
$SPARK_HOME/bin/spark-shell --master $SPARK_MASTER \
--driver-memory 20G \
--executor-memory 30G \
--conf spark.driver.maxResultSize=8G \
--jars ${ML_JAR},${CUDF_JAR},${PLUGIN_JAR} \
--jars ${ML_JAR},${PLUGIN_JAR} \
--conf spark.plugins=com.nvidia.spark.SQLPlugin \
--conf spark.rapids.sql.enabled=true \
--conf spark.task.resource.gpu.amount=0.08 \
Expand Down
6 changes: 0 additions & 6 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -95,12 +95,6 @@
<artifactId>rapids-4-spark_2.12</artifactId>
<version>22.06.0-SNAPSHOT</version>
</dependency>
<!-- https://mvnrepository.com/artifact/ai.rapids/cudf -->
<dependency>
<groupId>ai.rapids</groupId>
<artifactId>cudf</artifactId>
<version>22.06.0-SNAPSHOT</version>
</dependency>


<dependency>
Expand Down

0 comments on commit cde3d2e

Please sign in to comment.