Evaluetion Error

[MarioUncle-Wang]

我用nuscenes mini训练一个模型 要测试的时候出现如下报错，请问什么原因？如何处理？2024-08-22 15:56:50,509 - WARNING - No custom op centerpoint_postprocess found, try JIT build
2024-08-22 15:56:50,509 - WARNING - No custom op centerpoint_postprocess found, try JIT build
Compiling user custom op, it will cost a few seconds.....
/home/higer/anaconda3/envs/paddle_env/lib/python3.8/site-packages/setuptools/sandbox.py:15: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
import pkg_resources
/home/higer/anaconda3/envs/paddle_env/lib/python3.8/site-packages/pkg_resources/init.py:3144: DeprecationWarning: Deprecated call to pkg_resources.declare_namespace('mpl_toolkits').
Implementing implicit namespace packages (as specified in PEP 420) is preferred to pkg_resources.declare_namespace. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(pkg)
/home/higer/anaconda3/envs/paddle_env/lib/python3.8/site-packages/pkg_resources/init.py:3144: DeprecationWarning: Deprecated call to pkg_resources.declare_namespace('google').
Implementing implicit namespace packages (as specified in PEP 420) is preferred to pkg_resources.declare_namespace. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(pkg)
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
/home/higer/code/Paddle3D/paddle3d/ops/centerpoint_postprocess/postprocess.cu(147): error: too few arguments in function call

1 error detected in the compilation of "/home/higer/code/Paddle3D/paddle3d/ops/centerpoint_postprocess/postprocess.cu".
error: command '/home/higer/anaconda3/envs/paddle_env/bin/nvcc' failed with exit code 1
2024-08-22 15:56:54,392 - ERROR - centerpoint_postprocess builded fail!
2024-08-22 15:56:54,392 - ERROR - centerpoint_postprocess builded fail!
Traceback (most recent call last):
File "/home/higer/code/Paddle3D/paddle3d/ops/init.py", line 168, in _load_module
self.module = importlib.import_module(self.modulename)
File "/home/higer/anaconda3/envs/paddle_env/lib/python3.8/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1014, in _gcd_import
File "", line 991, in _find_and_load
File "", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'centerpoint_postprocess'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/higer/anaconda3/envs/paddle_env/lib/python3.8/site-packages/paddle/utils/cpp_extension/extension_utils.py", line 1152, in run_cmd
return subprocess.check_call(command, shell=True, stdout=DEVNULL)
File "/home/higer/anaconda3/envs/paddle_env/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'cd /home/higer/.cache/paddle_extensions && /home/higer/anaconda3/envs/paddle_env/bin/python centerpoint_postprocess_setup.py build' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "tools/evaluate.py", line 115, in
main(args)
File "tools/evaluate.py", line 110, in main
trainer.evaluate()
File "/home/higer/code/Paddle3D/paddle3d/apis/trainer.py", line 497, in evaluate
result = validation_step(self.model, sample)
File "/home/higer/code/Paddle3D/paddle3d/apis/pipeline.py", line 122, in validation_step
outputs = model(sample)
File "/home/higer/anaconda3/envs/paddle_env/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py", line 948, in call
return self.forward(*inputs, **kwargs)
File "/home/higer/code/Paddle3D/paddle3d/models/base/base_model.py", line 72, in forward
return self.test_forward(samples, *args, **kwargs)
File "/home/higer/code/Paddle3D/paddle3d/models/detection/centerpoint/centerpoint.py", line 163, in test_forward
preds = self.bbox_head.predict_by_custom_op(samples, preds,
File "/home/higer/anaconda3/envs/paddle_env/lib/python3.8/site-packages/decorator.py", line 232, in fun
return caller(func, *(extras + args), **kw)
File "/home/higer/anaconda3/envs/paddle_env/lib/python3.8/site-packages/paddle/fluid/dygraph/base.py", line 375, in _decorate_function
return func(*args, **kwargs)
File "/home/higer/code/Paddle3D/paddle3d/models/detection/centerpoint/center_head.py", line 320, in predict_by_custom_op
bboxes, scores, labels = centerpoint_postprocess.centerpoint_postprocess(
File "/home/higer/code/Paddle3D/paddle3d/ops/init.py", line 186, in getattr
module = self._load_module()
File "/home/higer/code/Paddle3D/paddle3d/ops/init.py", line 172, in _load_module
self.module = self.jit_build()
File "/home/higer/code/Paddle3D/paddle3d/ops/init.py", line 159, in jit_build
return paddle_jit_load(
File "/home/higer/anaconda3/envs/paddle_env/lib/python3.8/site-packages/paddle/utils/cpp_extension/cpp_extension.py", line 944, in load
_jit_compile(file_path, verbose)
File "/home/higer/anaconda3/envs/paddle_env/lib/python3.8/site-packages/paddle/utils/cpp_extension/extension_utils.py", line 1113, in _jit_compile
run_cmd(compile_cmd, verbose)
File "/home/higer/anaconda3/envs/paddle_env/lib/python3.8/site-packages/paddle/utils/cpp_extension/extension_utils.py", line 1155, in run_cmd
raise RuntimeError("Failed to run command: {}, errors: {}".format(
RuntimeError: Failed to run command: , errors: Command 'cd /home/higer/.cache/paddle_extensions && /home/higer/anaconda3/envs/paddle_env/bin/python centerpoint_postprocess_setup.py build' returned non-zero exit status 1.

[will-jl944]

您使用的模型需要自定义算子centerpoint_postprocess，在自动编译自定义算子的时候失败了。

请提供一下使用的paddlepaddle-gpu版本，本地的CUDA版本信息。