Error when running main.py

[tszngai]

Hi Fabiocarrara, thank you for your efforts in this incredible project. As I am following with the instructions, I have encountered some errors when running the main.py.

According to the other two posts, I have changed the caffe path to the correct one and remove the GPU 0 flags already. When I run main.py, I got the following errors:

py_lab@Pie:~/deep-parking$ python main.py
INFO 2018-07-09 17:16:03: Setting up mAlexNet trained on PKLot_train, validated on PKLot_val, CNRPark-EXT_val, tested on PKLot_test_CNRPark-EXT_test ...
INFO 2018-07-09 17:16:04: Training on PKLot_train while validating on PKLot_val, CNRPark-EXT_val ...
E0709 17:16:04.984817 4622 io.cpp:80] Could not open or find file /home/py_lab/deep-parking/PKLot/PKLotUFPR04/Sunny/2012-12-08/Empty/2012-12-08_14_25_10#010.jpg
F0709 17:16:04.985540 4622 image_data_layer.cpp:77] Check failed: cv_img.data Could not load UFPR04/Sunny/2012-12-08/Empty/2012-12-08_14_25_10#010.jpg
*** Check failure stack trace: ***
@ 0x7eff5db9f5cd google::LogMessage::Fail()
@ 0x7eff5dba1433 google::LogMessage::SendToLog()
@ 0x7eff5db9f15b google::LogMessage::Flush()
@ 0x7eff5dba1e1e google::LogMessageFatal::~LogMessageFatal()
@ 0x7eff5e07b358 caffe::ImageDataLayer<>::DataLayerSetUp()
@ 0x7eff5dfe598e caffe::BasePrefetchingDataLayer<>::LayerSetUp()
@ 0x7eff5df47d07 caffe::Net<>::Init()
@ 0x7eff5df4a44e caffe::Net<>::Net()
@ 0x7eff5e0d44fa caffe::Solver<>::InitTrainNet()
@ 0x7eff5e0d59c5 caffe::Solver<>::Init()
@ 0x7eff5e0d5cdf caffe::Solver<>::Solver()
@ 0x7eff5e0cd801 caffe::Creator_SGDSolver<>()
@ 0x40a288 train()
@ 0x407208 main
@ 0x7eff5c335830 __libc_start_main
@ 0x407ad9 _start
@ (nil) (unknown)

It says that it could not find the file at /home/py_lab/deep-parking/PKLot/PKLotUFPR04/Sunny/2012-12-08/Empty/2012-12-08_14_25_10#010.jpg
The problem is that a '/' is missing before UFPR04. UFPR04 is actually a folder within PKLot so it should instead be home/py_lab/deep-parking/PKLot/PKLot/UFPR04/Sunny/2012-12-08/Empty/2012-12-08_14_25_10#010.jpg

Could you please advice how to fix this issue?

[tszngai]

I figured it out... It was because I did not end the path with '/' in the config.py files. Now I fixed it and it still could find the path...

py_lab@Pie:~/deep-parking$ python main.py
INFO 2018-07-09 18:57:21: Setting up mAlexNet trained on PKLot_train, validated on PKLot_val, CNRPark-EXT_val, tested on PKLot_test_CNRPark-EXT_test ...
INFO 2018-07-09 18:57:22: Training on PKLot_train while validating on PKLot_val, CNRPark-EXT_val ...
E0709 18:57:23.132761 5623 io.cpp:80] Could not open or find file /home/py_lab/deep-parking/PKLot/PKLot/UFPR04/Sunny/2012-12-08/Empty/2012-12-08_14_25_10#010.jpg
F0709 18:57:23.132954 5623 image_data_layer.cpp:77] Check failed: cv_img.data Could not load UFPR04/Sunny/2012-12-08/Empty/2012-12-08_14_25_10#010.jpg
*** Check failure stack trace: ***
@ 0x7f4d63fde5cd google::LogMessage::Fail()
@ 0x7f4d63fe0433 google::LogMessage::SendToLog()
@ 0x7f4d63fde15b google::LogMessage::Flush()
@ 0x7f4d63fe0e1e google::LogMessageFatal::~LogMessageFatal()
@ 0x7f4d644ba358 caffe::ImageDataLayer<>::DataLayerSetUp()
@ 0x7f4d6442498e caffe::BasePrefetchingDataLayer<>::LayerSetUp()
@ 0x7f4d64386d07 caffe::Net<>::Init()
@ 0x7f4d6438944e caffe::Net<>::Net()
@ 0x7f4d645134fa caffe::Solver<>::InitTrainNet()
@ 0x7f4d645149c5 caffe::Solver<>::Init()
@ 0x7f4d64514cdf caffe::Solver<>::Solver()
@ 0x7f4d6450c801 caffe::Creator_SGDSolver<>()
@ 0x40a288 train()
@ 0x407208 main
@ 0x7f4d62774830 __libc_start_main
@ 0x407ad9 _start
@ (nil) (unknown)
INFO 2018-07-09 18:57:23: Setting up AlexNet trained on PKLot_train, validated on PKLot_val, CNRPark-EXT_val, tested on PKLot_test_CNRPark-EXT_test ...
INFO 2018-07-09 18:57:24: Training on PKLot_train while validating on PKLot_val, CNRPark-EXT_val ...
E0709 18:57:24.787290 5637 io.cpp:80] Could not open or find file /home/py_lab/deep-parking/PKLot/PKLot/UFPR04/Sunny/2012-12-08/Empty/2012-12-08_14_25_10#010.jpg
F0709 18:57:24.787415 5637 image_data_layer.cpp:77] Check failed: cv_img.data Could not load UFPR04/Sunny/2012-12-08/Empty/2012-12-08_14_25_10#010.jpg
*** Check failure stack trace: ***
@ 0x7f8efe7d45cd google::LogMessage::Fail()
@ 0x7f8efe7d6433 google::LogMessage::SendToLog()
@ 0x7f8efe7d415b google::LogMessage::Flush()
@ 0x7f8efe7d6e1e google::LogMessageFatal::~LogMessageFatal()
@ 0x7f8efecb0358 caffe::ImageDataLayer<>::DataLayerSetUp()
@ 0x7f8efec1a98e caffe::BasePrefetchingDataLayer<>::LayerSetUp()
@ 0x7f8efeb7cd07 caffe::Net<>::Init()
@ 0x7f8efeb7f44e caffe::Net<>::Net()
@ 0x7f8efed094fa caffe::Solver<>::InitTrainNet()
@ 0x7f8efed0a9c5 caffe::Solver<>::Init()
@ 0x7f8efed0acdf caffe::Solver<>::Solver()
@ 0x7f8efed02801 caffe::Creator_SGDSolver<>()
@ 0x40a288 train()
@ 0x407208 main
@ 0x7f8efcf6a830 __libc_start_main
@ 0x407ad9 _start
@ (nil) (unknown)

It is looking for an 'empty' folder within '2012-12-08', but there is no such folder.. Any idea?

[fabiocarrara]

Hi @tszngai ,
we used the already segmented patches for PKLot, that are in the folder PKLotSegmented under the dataset root folder. I think you are pointing to the full images instead.
Try replacing

/home/py_lab/deep-parking/PKLot/PKLot/

with

/home/py_lab/deep-parking/PKLot/PKLotSegmented/

as root folder in config.py.

[tszngai]

Thank you for the reply @fabiocarrara !
I replaced it with the PKLotSegmented and after 20 hours of training, new errors came up..

I0712 06:15:46.586170 9696 solver.cpp:414] Test net output #1: loss = 0.335053 (* 1 = 0.335053 loss)
I0712 06:15:46.586190 9696 solver.cpp:347] Iteration 444, Testing net (#3)
I0712 06:17:01.682238 9696 blocking_queue.cpp:49] Waiting for data
DEBUG 2018-07-12 06:18:04: Summarizing mAlexNet-tr_PKLot_train-vl_PKLot_val_CNRPark-EXT_val-ts_PKLot_test_CNRPark-EXT_test ...
Traceback (most recent call last):
File "main.py", line 104, in
pyffe.summarize(exps).to_csv('results.csv')
File "/home/py_lab/deep-parking/pyffe/experiment.py", line 84, in summarize
r = e.summarize()
File "/home/py_lab/deep-parking/pyffe/experiment.py", line 42, in decorator
return function(*args, **kwargs)
File "/home/py_lab/deep-parking/pyffe/experiment.py", line 548, in summarize
last_iter = log_data['train']['iteration'][-1]
IndexError: list index out of range

[fabiocarrara]

It seems that one of the log files of the experiments is not properly formatted. It may indicate there was a problem with a specific experiment. Please, manually check the log files in the runs/ subfolders for errors. You may then rerun only the failed experiments by commenting the finished ones in main.py.

If there are no failed experiments, there is probably a bug in the summarize() function.
However, you could always process the results of the finished experiments by parsing the logs in other ways, e.g by grepping or using external code.

[tszngai]

Thanks again for your response @fabiocarrara . Now that I have my own training models, I would like to use them to write a function that can detect vehicles from any input images. Could you advice the approaches or if there are any useful tutorials you know? Thank you!