We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
我使用mini数据集进行了训练,在第22个epoch的时候,程序卡死,然后我对最后的两个epoch进行了验证,结果map出奇的低。 ================epoch 20================= AP: 0.0003 mATE: 1.0000 mASE: 1.0000 mAOE: 1.0000 mAVE: 1.0000 mAAE: 1.0000 NDS: 0.0001 Eval time: 0.4s
Per-class results: Object Class AP ATE ASE AOE AVE AAE car 0.003 1.000 1.000 1.000 1.000 1.000 truck 0.000 1.000 1.000 1.000 1.000 1.000 bus 0.000 1.000 1.000 1.000 1.000 1.000 trailer 0.000 1.000 1.000 1.000 1.000 1.000 construction_vehicle 0.000 1.000 1.000 1.000 1.000 1.000 pedestrian 0.000 1.000 1.000 1.000 1.000 1.000 motorcycle 0.000 1.000 1.000 1.000 1.000 1.000 bicycle 0.000 1.000 1.000 1.000 1.000 1.000 traffic_cone 0.000 1.000 1.000 nan nan nan barrier 0.000 1.000 1.000 1.000 nan nan ================epoch 21================= AP: 0.0003 mATE: 1.0000 mASE: 1.0000 mAOE: 1.0000 mAVE: 1.0000 mAAE: 1.0000 NDS: 0.0002 Eval time: 0.4s
Per-class results: Object Class AP ATE ASE AOE AVE AAE car 0.003 1.000 1.000 1.000 1.000 1.000 truck 0.000 1.000 1.000 1.000 1.000 1.000 bus 0.000 1.000 1.000 1.000 1.000 1.000 trailer 0.000 1.000 1.000 1.000 1.000 1.000 construction_vehicle 0.000 1.000 1.000 1.000 1.000 1.000 pedestrian 0.000 1.000 1.000 1.000 1.000 1.000 motorcycle 0.000 1.000 1.000 1.000 1.000 1.000 bicycle 0.000 1.000 1.000 1.000 1.000 1.000 traffic_cone 0.000 1.000 1.000 nan nan nan barrier 0.000 1.000 1.000 1.000 nan nan Testing DataLoader 0: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 162/162 [00:29<00:00, 5.46it/s] TEST Profiler Report
The text was updated successfully, but these errors were encountered:
运行python bevdepth/exps/nuscenes/mv/bev_depth_lss_r50_256x704_128x128_24e_ema.py -b 1 --gpus 1 进行训练 运行python bevdepth/exps/nuscenes/mv/bev_depth_lss_r50_256x704_128x128_24e_ema.py --ckpt_path outputs/bev_depth_lss_r50_256x704_128x128_24e_ema/lightning_logs/version_2/21.pth -e -b 1 --gpus 1 进行测试
Sorry, something went wrong.
No branches or pull requests
我使用mini数据集进行了训练,在第22个epoch的时候,程序卡死,然后我对最后的两个epoch进行了验证,结果map出奇的低。
================epoch 20=================
AP: 0.0003
mATE: 1.0000
mASE: 1.0000
mAOE: 1.0000
mAVE: 1.0000
mAAE: 1.0000
NDS: 0.0001
Eval time: 0.4s
Per-class results:
Object Class AP ATE ASE AOE AVE AAE
car 0.003 1.000 1.000 1.000 1.000 1.000
truck 0.000 1.000 1.000 1.000 1.000 1.000
bus 0.000 1.000 1.000 1.000 1.000 1.000
trailer 0.000 1.000 1.000 1.000 1.000 1.000
construction_vehicle 0.000 1.000 1.000 1.000 1.000 1.000
pedestrian 0.000 1.000 1.000 1.000 1.000 1.000
motorcycle 0.000 1.000 1.000 1.000 1.000 1.000
bicycle 0.000 1.000 1.000 1.000 1.000 1.000
traffic_cone 0.000 1.000 1.000 nan nan nan
barrier 0.000 1.000 1.000 1.000 nan nan
================epoch 21=================
AP: 0.0003
mATE: 1.0000
mASE: 1.0000
mAOE: 1.0000
mAVE: 1.0000
mAAE: 1.0000
NDS: 0.0002
Eval time: 0.4s
Per-class results:
Object Class AP ATE ASE AOE AVE AAE
car 0.003 1.000 1.000 1.000 1.000 1.000
truck 0.000 1.000 1.000 1.000 1.000 1.000
bus 0.000 1.000 1.000 1.000 1.000 1.000
trailer 0.000 1.000 1.000 1.000 1.000 1.000
construction_vehicle 0.000 1.000 1.000 1.000 1.000 1.000
pedestrian 0.000 1.000 1.000 1.000 1.000 1.000
motorcycle 0.000 1.000 1.000 1.000 1.000 1.000
bicycle 0.000 1.000 1.000 1.000 1.000 1.000
traffic_cone 0.000 1.000 1.000 nan nan nan
barrier 0.000 1.000 1.000 1.000 nan nan
Testing DataLoader 0: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 162/162 [00:29<00:00, 5.46it/s]
TEST Profiler Report
| Action | Mean duration (s) | Num calls | Total time (s) | Percentage % |
| Total | - | 2823 | 87.467 | 100 % |
| run_test_evaluation | 31.038 | 1 | 31.038 | 35.486 |
| [Strategy]SingleDeviceStrategy.test_step | 0.15896 | 162 | 25.751 | 29.441 |
| [LightningModule]BEVDepthLightningModel.test_epoch_end | 2.6193 | 1 | 2.6193 | 2.9946 |
| [EvaluationEpochLoop].test_dataloader_idx_0_next | 0.010395 | 162 | 1.6839 | 1.9252 |
| [Strategy]SingleDeviceStrategy.batch_to_device | 0.003483 | 162 | 0.56424 | 0.64509 |
| [LightningModule]BEVDepthLightningModel.test_dataloader | 0.099374 | 1 | 0.099374 | 0.11361 |
| [Callback]TQDMProgressBar.on_test_batch_end | 0.00025195 | 162 | 0.040816 | 0.046664 |
| [Callback]ModelSummary.on_test_batch_end | 9.2947e-06 | 162 | 0.0015057 | 0.0017215 |
| [Callback]TQDMProgressBar.on_test_batch_start | 8.4548e-06 | 162 | 0.0013697 | 0.0015659 |
| [LightningModule]BEVDepthLightningModel.on_test_model_train | 0.0010831 | 1 | 0.0010831 | 0.0012383 |
| [LightningModule]BEVDepthLightningModel.on_test_model_eval | 0.00094745 | 1 | 0.00094745 | 0.0010832 |
| [Callback]EMACallback.on_test_batch_start | 4.0783e-06 | 162 | 0.00066068 | 0.00075535 |
| [Callback]TQDMProgressBar.on_test_start | 0.00061182 | 1 | 0.00061182 | 0.00069948 |
| [Callback]EMACallback.on_test_batch_end | 2.4168e-06 | 162 | 0.00039152 | 0.00044762 |
| [LightningModule]BEVDepthLightningModel.test_step_end | 1.8431e-06 | 162 | 0.00029859 | 0.00034137 |
| [Callback]TQDMProgressBar.on_test_end | 0.00028655 | 1 | 0.00028655 | 0.00032761 |
| [Callback]GradientAccumulationScheduler.on_test_batch_end | 1.4585e-06 | 162 | 0.00023627 | 0.00027012 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_test_batch_end | 1.4072e-06 | 162 | 0.00022797 | 0.00026063 |
| [Callback]ModelSummary.on_test_batch_start | 1.3618e-06 | 162 | 0.00022062 | 0.00025223 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_test_batch_start | 1.3274e-06 | 162 | 0.00021504 | 0.00024585 |
| [Callback]GradientAccumulationScheduler.on_test_batch_start | 1.1475e-06 | 162 | 0.0001859 | 0.00021253 |
| [LightningModule]BEVDepthLightningModel.on_test_batch_end | 1.0331e-06 | 162 | 0.00016737 | 0.00019135 |
| [LightningModule]BEVDepthLightningModel.on_test_batch_start | 9.7408e-07 | 162 | 0.0001578 | 0.00018041 |
| [Strategy]SingleDeviceStrategy.test_step_end | 8.8581e-07 | 162 | 0.0001435 | 0.00016406 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': None}.setup | 8.6112e-05 | 1 | 8.6112e-05 | 9.845e-05 |
| [Callback]TQDMProgressBar.on_test_epoch_start | 1.1989e-05 | 1 | 1.1989e-05 | 1.3707e-05 |
| [Callback]EMACallback.on_test_epoch_end | 9.393e-06 | 1 | 9.393e-06 | 1.0739e-05 |
| [Callback]ModelSummary.on_test_end | 8.157e-06 | 1 | 8.157e-06 | 9.3258e-06 |
| [Callback]ModelSummary.on_test_start | 6.458e-06 | 1 | 6.458e-06 | 7.3833e-06 |
| [Callback]TQDMProgressBar.setup | 5.747e-06 | 1 | 5.747e-06 | 6.5704e-06 |
| [Callback]EMACallback.teardown | 5.065e-06 | 1 | 5.065e-06 | 5.7907e-06 |
| [LightningModule]BEVDepthLightningModel.on_load_checkpoint | 4.902e-06 | 1 | 4.902e-06 | 5.6044e-06 |
| [Callback]EMACallback.on_test_start | 4.671e-06 | 1 | 4.671e-06 | 5.3403e-06 |
| [LightningModule]BEVDepthLightningModel.on_test_epoch_end | 4.038e-06 | 1 | 4.038e-06 | 4.6166e-06 |
| [LightningModule]BEVDepthLightningModel.configure_callbacks | 3.997e-06 | 1 | 3.997e-06 | 4.5697e-06 |
| [Callback]EMACallback.setup | 3.806e-06 | 1 | 3.806e-06 | 4.3513e-06 |
| [LightningModule]BEVDepthLightningModel.configure_sharded_model | 3.26e-06 | 1 | 3.26e-06 | 3.7271e-06 |
| [Callback]EMACallback.on_epoch_end | 3.089e-06 | 1 | 3.089e-06 | 3.5316e-06 |
| [Callback]EMACallback.on_configure_sharded_model | 2.58e-06 | 1 | 2.58e-06 | 2.9497e-06 |
| [Callback]EMACallback.on_epoch_start | 2.052e-06 | 1 | 2.052e-06 | 2.346e-06 |
| [Callback]TQDMProgressBar.on_test_epoch_end | 2.044e-06 | 1 | 2.044e-06 | 2.3369e-06 |
| [Callback]EMACallback.on_test_epoch_start | 2.032e-06 | 1 | 2.032e-06 | 2.3232e-06 |
| [LightningModule]BEVDepthLightningModel.prepare_data | 1.966e-06 | 1 | 1.966e-06 | 2.2477e-06 |
| [Callback]ModelSummary.on_test_epoch_start | 1.854e-06 | 1 | 1.854e-06 | 2.1196e-06 |
| [Callback]EMACallback.on_before_accelerator_backend_setup | 1.802e-06 | 1 | 1.802e-06 | 2.0602e-06 |
| [Callback]EMACallback.on_test_end | 1.636e-06 | 1 | 1.636e-06 | 1.8704e-06 |
| [Callback]TQDMProgressBar.on_epoch_end | 1.597e-06 | 1 | 1.597e-06 | 1.8258e-06 |
| [LightningModule]BEVDepthLightningModel.on_test_start | 1.552e-06 | 1 | 1.552e-06 | 1.7744e-06 |
| [Callback]TQDMProgressBar.teardown | 1.542e-06 | 1 | 1.542e-06 | 1.7629e-06 |
| [Callback]GradientAccumulationScheduler.on_test_start | 1.4e-06 | 1 | 1.4e-06 | 1.6006e-06 |
| [Callback]GradientAccumulationScheduler.on_test_epoch_start | 1.349e-06 | 1 | 1.349e-06 | 1.5423e-06 |
| [Callback]TQDMProgressBar.on_configure_sharded_model | 1.318e-06 | 1 | 1.318e-06 | 1.5068e-06 |
| [Callback]ModelSummary.setup | 1.296e-06 | 1 | 1.296e-06 | 1.4817e-06 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_test_start | 1.245e-06 | 1 | 1.245e-06 | 1.4234e-06 |
| [Callback]ModelSummary.on_configure_sharded_model | 1.23e-06 | 1 | 1.23e-06 | 1.4062e-06 |
| [Callback]TQDMProgressBar.on_before_accelerator_backend_setup | 1.215e-06 | 1 | 1.215e-06 | 1.3891e-06 |
| [Callback]GradientAccumulationScheduler.setup | 1.214e-06 | 1 | 1.214e-06 | 1.3879e-06 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown | 1.163e-06 | 1 | 1.163e-06 | 1.3296e-06 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_test_epoch_end | 1.148e-06 | 1 | 1.148e-06 | 1.3125e-06 |
| [LightningModule]BEVDepthLightningModel.teardown | 1.147e-06 | 1 | 1.147e-06 | 1.3113e-06 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model | 1.131e-06 | 1 | 1.131e-06 | 1.2931e-06 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': None}.on_before_accelerator_backend_setup| 1.12e-06 | 1 | 1.12e-06 | 1.2805e-06 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_test_end | 1.11e-06 | 1 | 1.11e-06 | 1.269e-06 |
| [Callback]GradientAccumulationScheduler.on_test_end | 1.107e-06 | 1 | 1.107e-06 | 1.2656e-06 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start | 1.08e-06 | 1 | 1.08e-06 | 1.2347e-06 |
| [Callback]GradientAccumulationScheduler.on_configure_sharded_model | 1.069e-06 | 1 | 1.069e-06 | 1.2222e-06 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_test_epoch_start | 1.054e-06 | 1 | 1.054e-06 | 1.205e-06 |
| [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup | 1.037e-06 | 1 | 1.037e-06 | 1.1856e-06 |
| [Callback]ModelSummary.on_test_epoch_end | 1.032e-06 | 1 | 1.032e-06 | 1.1799e-06 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end | 1.026e-06 | 1 | 1.026e-06 | 1.173e-06 |
| [Callback]ModelSummary.teardown | 1.023e-06 | 1 | 1.023e-06 | 1.1696e-06 |
| [Callback]ModelSummary.on_before_accelerator_backend_setup | 9.86e-07 | 1 | 9.86e-07 | 1.1273e-06 |
| [Callback]TQDMProgressBar.on_epoch_start | 9.76e-07 | 1 | 9.76e-07 | 1.1158e-06 |
| [LightningModule]BEVDepthLightningModel.setup | 9.64e-07 | 1 | 9.64e-07 | 1.1021e-06 |
| [Callback]GradientAccumulationScheduler.on_epoch_start | 9.51e-07 | 1 | 9.51e-07 | 1.0873e-06 |
| [Callback]GradientAccumulationScheduler.teardown | 9.42e-07 | 1 | 9.42e-07 | 1.077e-06 |
| [LightningModule]BEVDepthLightningModel.on_test_end | 8.99e-07 | 1 | 8.99e-07 | 1.0278e-06 |
| [Callback]ModelSummary.on_epoch_end | 8.9e-07 | 1 | 8.9e-07 | 1.0175e-06 |
| [Callback]ModelSummary.on_epoch_start | 8.88e-07 | 1 | 8.88e-07 | 1.0152e-06 |
| [Callback]GradientAccumulationScheduler.on_test_epoch_end | 8.81e-07 | 1 | 8.81e-07 | 1.0072e-06 |
| [Callback]GradientAccumulationScheduler.on_epoch_end | 8.39e-07 | 1 | 8.39e-07 | 9.5922e-07 |
| [LightningModule]BEVDepthLightningModel.on_test_epoch_start | 7.55e-07 | 1 | 7.55e-07 | 8.6318e-07 |
| [Strategy]SingleDeviceStrategy.on_test_start | 7.51e-07 | 1 | 7.51e-07 | 8.586e-07 |
| [Strategy]SingleDeviceStrategy.on_test_end | 7.13e-07 | 1 | 7.13e-07 | 8.1516e-07 |
| [LightningModule]BEVDepthLightningModel.on_epoch_start | 6.39e-07 | 1 | 6.39e-07 | 7.3056e-07 |
| [LightningModule]BEVDepthLightningModel.on_epoch_end | 5.36e-07 | 1 | 5.36e-07 | 6.128e-07 |
The text was updated successfully, but these errors were encountered: