Config file needs modification- not starting training #74
Comments
|
Any ideas for this below mentioned error. |
|
when run setup.py find ValueError: './utils/box_overlaps.pyx' doesn't match any files could you answer this question? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
(math) kd636624@login-g:~/demo/math/VoxelNet-tensorflow[1053]$ python3 train.py --max-epoch 20
2021-03-25 14:21:03.448203: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/intel/impi/2018.4.274/compilers_and_libraries/linux/mpi/lib64:/opt/intel/impi/2018.4.274/compilers_and_libraries/linux/mpi/lib:/opt/intel/Compiler/19.0/1.144/rwthlnk/daal/lib/intel64_lin:/opt/intel/Compiler/19.0/1.144/rwthlnk/daal/lib/ia32_lin:/opt/intel/Compiler/19.0/1.144/rwthlnk/ipp/lib/intel64_lin:/opt/intel/Compiler/19.0/1.144/rwthlnk/ipp/lib/ia32_lin:/opt/intel/Compiler/19.0/1.144/rwthlnk/mkl/lib/intel64_lin:/opt/intel/Compiler/19.0/1.144/rwthlnk/mkl/lib/ia32_lin:/opt/intel/Compiler/19.0/1.144/rwthlnk/tbb/lib/intel64_lin/gcc4.7:/opt/intel/Compiler/19.0/1.144/rwthlnk/tbb/lib/ia32_lin/gcc4.7:/opt/intel/Compiler/19.0/1.144/rwthlnk/compiler/lib/intel64_lin:/opt/intel/Compiler/19.0/1.144/rwthlnk/compiler/lib/ia32_lin
2021-03-25 14:21:03.448243: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
GPU Available from Config 0,1,2,3
GPU count from Config 4
split:
object dir: ./data/object/training
data_tag: 7481
f_rgb: 7481
f_lidar: 7481
Dataset total length: 7481
split:
object dir: ./data/object/testing
data_tag: 7518
f_rgb: 7518
f_lidar: 7518
Dataset total length: 7518
Process Process-9:
Process Process-10:
Process Process-11:
Process Process-12:
Process Process-13:
Process Process-14:
Process Process-15:
2021-03-25 14:21:19.517590: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-03-25 14:21:19.519417: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-03-25 14:21:19.520750: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
Process Process-16:
Traceback (most recent call last):
File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 241, in loader_worker_main
self.fill_queue(batch_size)
File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 167, in fill_queue
voxel[idx * single_batch_size:(idx + 1) * single_batch_size])
File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 274, in build_input
feature = np.concatenate(feature_list)
File "<array_function internals>", line 6, in concatenate
ValueError: need at least one array to concatenate
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 241, in loader_worker_main
self.fill_queue(batch_size)
File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 241, in loader_worker_main
self.fill_queue(batch_size)
File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 167, in fill_queue
voxel[idx * single_batch_size:(idx + 1) * single_batch_size])
File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 167, in fill_queue
voxel[idx * single_batch_size:(idx + 1) * single_batch_size])
File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 241, in loader_worker_main
self.fill_queue(batch_size)
File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 274, in build_input
feature = np.concatenate(feature_list)
File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 241, in loader_worker_main
self.fill_queue(batch_size)
File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 241, in loader_worker_main
self.fill_queue(batch_size)
File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 274, in build_input
feature = np.concatenate(feature_list)
File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 167, in fill_queue
voxel[idx * single_batch_size:(idx + 1) * single_batch_size])
File "<array_function internals>", line 6, in concatenate
File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 167, in fill_queue
voxel[idx * single_batch_size:(idx + 1) * single_batch_size])
File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 167, in fill_queue
voxel[idx * single_batch_size:(idx + 1) * single_batch_size])
File "<array_function internals>", line 6, in concatenate
File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 274, in build_input
feature = np.concatenate(feature_list)
File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 274, in build_input
feature = np.concatenate(feature_list)
File "<array_function internals>", line 6, in concatenate
File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 274, in build_input
feature = np.concatenate(feature_list)
File "<array_function internals>", line 6, in concatenate
ValueError: need at least one array to concatenate
File "<array_function internals>", line 6, in concatenate
ValueError: need at least one array to concatenate
ValueError: need at least one array to concatenate
ValueError: need at least one array to concatenate
ValueError: need at least one array to concatenate
Traceback (most recent call last):
Traceback (most recent call last):
File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 241, in loader_worker_main
self.fill_queue(batch_size)
File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 167, in fill_queue
voxel[idx * single_batch_size:(idx + 1) * single_batch_size])
File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 274, in build_input
feature = np.concatenate(feature_list)
File "<array_function internals>", line 6, in concatenate
File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
ValueError: need at least one array to concatenate
File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 241, in loader_worker_main
self.fill_queue(batch_size)
File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 167, in fill_queue
voxel[idx * single_batch_size:(idx + 1) * single_batch_size])
File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 274, in build_input
feature = np.concatenate(feature_list)
File "<array_function internals>", line 6, in concatenate
ValueError: need at least one array to concatenate
2021-03-25 14:21:20.790883: E tensorflow/core/common_runtime/session.cc:91] Failed to create session: Invalid argument: 'visible_device_list' listed an invalid GPU id '2' but visible device count is 2
2021-03-25 14:21:20.790922: E tensorflow/c/c_api.cc:2184] Invalid argument: 'visible_device_list' listed an invalid GPU id '2' but visible device count is 2
Traceback (most recent call last):
File "train.py", line 142, in
tf.app.run(main)
File "/rwthfs/rz/cluster/home/kd636624/demo/math/lib64/python3.6/site-packages/tensorflow/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/rwthfs/rz/cluster/home/kd636624/demo/math/lib64/python3.6/site-packages/absl/app.py", line 303, in run
_run_main(main, args)
File "/rwthfs/rz/cluster/home/kd636624/demo/math/lib64/python3.6/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "train.py", line 60, in main
with tf.Session(config=config) as sess:
File "/rwthfs/rz/cluster/home/kd636624/demo/math/lib64/python3.6/site-packages/tensorflow/python/client/session.py", line 1596, in init
super(Session, self).init(target, graph, config=config)
File "/rwthfs/rz/cluster/home/kd636624/demo/math/lib64/python3.6/site-packages/tensorflow/python/client/session.py", line 711, in init
self._session = tf_session.TF_NewSessionRef(self._graph._c_graph, opts)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 'visible_device_list' listed an invalid GPU id '2' but visible device count is 2
Problems: Config file doesn't get imported in Kitti_loader, Kitti_loader fails to load complete dataset and training is not starting at all. Any help would be greatly appreciated. Thanks
The text was updated successfully, but these errors were encountered: