Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] BasePythonRunner has a new parameter metrics in Spark 4.0 #11895

Open
hyperbolic2346 opened this issue Dec 19, 2024 · 1 comment · May be fixed by #11905
Open

[BUG] BasePythonRunner has a new parameter metrics in Spark 4.0 #11895

hyperbolic2346 opened this issue Dec 19, 2024 · 1 comment · May be fixed by #11905
Assignees
@firestarman
Labels
bug Something isn't working Spark 4.0+ Spark 4.0+ issues

Comments

@hyperbolic2346
Copy link
Collaborator

hyperbolic2346 commented Dec 19, 2024 •
edited
Loading

Describe the bug
CI found an issue with the GPUBasePythonRunner where it is not passing the metrics parameter to the parent class BasePythonRunner.

Steps/Code to reproduce bug
Build the image for spark 4.0

[2024-12-19T16:52:14.524Z] [INFO] --- scala-maven-plugin:4.9.1:compile (scala-compile-first) @ rapids-4-spark-sql_2.13 ---

[2024-12-19T16:52:14.779Z] [INFO] Compiler bridge file: /home/jenkins/agent/workspace/jenkins-rapids_scala213_nightly-dev-github-345/scala2.13/target/spark400/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.13-1.10.0-bin_2.13.14__61.0-1.10.0_20240505T232140.jar

[2024-12-19T16:52:41.249Z] [INFO] compiling 479 Scala sources and 58 Java sources to /home/jenkins/agent/workspace/jenkins-rapids_scala213_nightly-dev-github-345/scala2.13/sql-plugin/target/spark400/classes ...

[2024-12-19T16:52:41.250Z] [ERROR] [Error] /home/jenkins/agent/workspace/jenkins-rapids_scala213_nightly-dev-github-345/sql-plugin/src/main/spark341db/scala/org/apache/spark/sql/rapids/execution/python/shims/GpuBasePythonRunner.scala:36: not enough arguments for constructor BasePythonRunner: (funcs: Seq[org.apache.spark.api.python.ChainedPythonFunctions], evalType: Int, argOffsets: Array[Array[Int]], jobArtifactUUID: Option[String], metrics: Map[String,org.apache.spark.util.AccumulatorV2[Long,Long]]): org.apache.spark.api.python.BasePythonRunner[IN,org.apache.spark.sql.vectorized.ColumnarBatch].

[2024-12-19T16:52:41.250Z] Unspecified value parameter metrics.
@hyperbolic2346 hyperbolic2346 added ? - Needs Triage Need team to review and classify bug Something isn't working labels Dec 19, 2024
@gerashegalov
Copy link
Collaborator

It will come up in the audit for apache/spark@2b1369b

@sameerz sameerz added the Spark 4.0+ Spark 4.0+ issues label Dec 24, 2024
@firestarman firestarman self-assigned this Dec 25, 2024
@GaryShen2008 GaryShen2008 removed the ? - Needs Triage Need team to review and classify label Dec 25, 2024
@firestarman firestarman linked a pull request Dec 25, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@firestarman firestarman

Labels
bug Something isn't working Spark 4.0+ Spark 4.0+ issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix the build error for Spark 400 firestarman/spark-rapids
5 participants
@gerashegalov @hyperbolic2346 @sameerz @firestarman @GaryShen2008