User-Defined UDF failed after upgrading from Spark Rapids 23.10.0 to 24.08.1 #433
Comments
|
The CMakeLists you link to is compiled against libcudf 24.08 (GIT_TAG branch-24.08), but it sounds like you're running against 24.10.0-SNAPSHOT. It's important to compile the RAPIDS Accelerated UDFs against the same libcudf version used at runtime, as there's no guarantee of backwards compatibility across versions. I'm not 100% convinced that is the root cause here, but mixing versions between compile and runtime can cause very odd issues. Since this is a SNAPSHOT build it will be a bit trickier to get the correct version since it doesn't have a nice, stable tag to reference. If you cannot use 24.08 instead and need 24.10.0-SNAPSHOT, you can examine the contents of the cudf-java-version-info.properties file within the rapids4spark jar you are using to see what git hash was used for cudf. You can then update the GIT_TAG in your CMakeLists for the cudf dependency to pick up that same version you're running with. If that does not resolve the issue, it would help if we had more visibility into what type the native UDF did receive. Maybe that will give us a clue as to what's going wrong. For example we could change the error code to something like this: if (v->type().id() != cudf::type_id::LIST) {
std::stringstream ss;
ss << "expected LIST found type ID" << static_cast<int>(v->type().id());
throw_java_exception(env, ILLEGAL_ARG_CLASS, ss.str().c_str());
return 0;
} |
|
@jlowe Thanks for your comments. After updating CMakeLists.txt with and GIT_REPOSITORY https://github.com/rapidsai/cudf.git I am able to compile our .so file. Thanks so much for your supports. |
I followed the tutorial to compile my own libatgxnativeudfjni.so which contains two UDFs, retrieve_rs_id and retrieve_positive_genotype defined by myself. With this .so file, I can execute my UDFs with Spark Rapids 23.10.0. However, after recently I upgraded my Spark Rapids to 24.10.0-SNAPSHOT build locally, my UDFs can not be executed anymore.
It starts to throw this ERROR message when I execute the following command.
where I have confirmed that csq.Existing_variation has type LIST of STRING.
and my RetrieveRsIDJni.cpp is started with
which I mainly take CosineSimilarityJni.cpp as a reference.
What could be the root cause that make my function no more able to be executed ?
I only modified CMakeLists.txt file with newer
https://github.com/NVIDIA/spark-rapids-examples/blob/main/examples/UDF-Examples/RAPIDS-accelerated-UDFs/src/main/cpp/CMakeLists.txt
The text was updated successfully, but these errors were encountered: