Evaluation with model_main.py not using old checkpoints/don't evaluate choosen dataset

[Petros626]

Hello,

I want to evaluate my training and test data and I found several approaches do to that with eval.py and model_main.py.

Now my problem is, that the script begins to train from the beginning and evaluates the test data, even if the the flag --eval_training_data is True. One guess is to change the eval_input_reader in my config file to train.record, but not sure.

Could please someone help me with this, don't want to start from the beginning, if i already have checkpoints...

set CONFIG_FILE=C:\Users\petros.katsoulakos\models2\models-master\research\object_detection\training\ssd_mobilenet_v2_quantized_300x300_coco_custom_aspect_ratios.config

set OUTPUT_DIR=C:\Users\petros.katsoulakos\models2\models-master\research\object_detection\tensorboard_outputs\after_training\eval_train_data

set CHECKPOINT_PATH= C:\Users\petros.katsoulakos\models2\models-master\research\object_detection\training\model.ckpt-200000

python model_main.py --pipeline_config_path=%CONFIG_FILE% --model_dir=%OUTPUT_DIR% --eval_training_data=True --checkpoint_dir=%CHECKPOINT_PATH% --run_once=True

[gadagashwini]

Hi @Petros626,
You can try,
python model_main.py
--pipeline_config_path=%CONFIG_FILE%
--model_dir=%OUTPUT_DIR%
--eval_training_data=True
--checkpoint_dir=%CHECKPOINT_PATH%
--run_once=True

pipeline.config

eval_config: {
  metrics_set: "coco_detection_metrics"
  num_examples:  # no of test images
  num_visualizations:  # no of visualizations for tensorboard
}

eval_input_reader: {
  tf_record_input_reader {
    input_path: "/path/to/test-data.record"
  }
  label_map_path: "/path/to/label_map.pbtxt"
  shuffle: true
  num_readers: 1
}

Thank you!

[Petros626]

Hi @Petros626, You can try, python model_main.py --pipeline_config_path=%CONFIG_FILE% --model_dir=%OUTPUT_DIR% --eval_training_data=True --checkpoint_dir=%CHECKPOINT_PATH% --run_once=True

pipeline.config

eval_config: {
  metrics_set: "coco_detection_metrics"
  num_examples:  # no of test images
  num_visualizations:  # no of visualizations for tensorboard
}

eval_input_reader: {
  tf_record_input_reader {
    input_path: "/path/to/test-data.record"
  }
  label_map_path: "/path/to/label_map.pbtxt"
  shuffle: true
  num_readers: 1
}

Thank you!

Hello @gadagashwini the suggested command is the same like mine, so the problem is the last saved checkpoint is not readed, TF will start a new training and the second thing is that i have to force in config file by changing the eval_input_reader to train.record to evaluate on training data, so the flag doesn't working. I expect a right behaviour, by using the model_main.py, but seems, that you have to configure the job, if you don't want to evaluate the test.record (default).

Your changing in the config file is to evaluate of COCO, which makes no sense in my case, because I have 3D-LiDAR data, so I evaluated on my training and test data.

Here are my summaries to the behaviour:
model_main.py:
->only new training and evaluation possible, pure evaluation on train o. test only with new unnecessary(have already checkpoints) possible
->eval_training_data=True doesn't work, must force via config file to eval on train o. test.

eval.py:
->evaluation possible without training (meant for this)
->loads the last checkpoint from training (hoped mode_main.py will too)
-> also here forcing via config file
-> flag doesn't working too

[Petros626]

@gadagashwini any tips?