Error while training on SageMaker #2408
Unanswered
ChikkathotadappaAshwin-zz
asked this question in
Help
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
During training on Sagemaker , the training was stopped with displaying below error, but with same code I was able to succeed running earlier, due to unsatisfactory results, I increased my training data and started to train once again
After making several searches, I found which is answered for similar type of issue, please let me know, if this is the same error
UnexpectedStatusException: Error for Training job train-job-fp-od-initial-fp-aug-20210525-exp-3: Failed. Reason: AlgorithmError: ExecuteUserScriptError:
Command "/usr/bin/python3 train_sm_toolkit.py
message from log:
Traceback (most recent call last):
File "train_sm_toolkit.py", line 432, in
train(args)
File "train_sm_toolkit.py", line 288, in train
train_generator, validation_generator = create_generators(args, backbone.preprocess_image)
File "train_sm_toolkit.py", line 256, in create_generators
**common_args
File "/opt/ml/code/keras_retinanet/preprocessing/coco.py", line 46, in init
super(CocoGenerator, self).init(**kwargs)
File "/opt/ml/code/keras_retinanet/preprocessing/generator.py", line 89, in init
self.group_images()
File "/opt/ml/code/keras_retinanet/preprocessing/generator.py", line 291, in group_images
order.sort(key=lambda x: self.image_aspect_ratio(x))
File "/opt/ml/code/keras_retinanet/preprocessing/generator.py", line 291, in
order.sort(key=lambda x: self.image_aspect_ratio(x))
File "/opt/ml/code/keras_retinanet/preprocessing/coco.py", line 124, in image_aspect_ratio
image = self.coco.loadImgs(self.image_ids[image_index])[0]
File "/usr/local/lib/python3.6/dist-packages/pycocotools/coco.py", line 229, in loadImgs
return [self.imgs[id] for id in ids]
File "/usr/local/lib/python3.6/dist-packages/pycocotools/coco.py", line 229, in
return [self.imgs[id] for id in ids]
Beta Was this translation helpful? Give feedback.
All reactions