Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trainning with my own images got "cost is nan" #336

Closed
ghost opened this issue Jan 4, 2020 · 1 comment
Closed

Trainning with my own images got "cost is nan" #336

ghost opened this issue Jan 4, 2020 · 1 comment

Comments

@ghost
Copy link

ghost commented Jan 4, 2020

I have been continously training the lanenet with my own images. I first trained the network with some images, and then ran inference on a larger set. If I did not get good inference results on some of the images, I added them to the training set and train the network on top of the trained model. Keep doing this, I have trained the network for about 300K epoch. Now, I found if I added more images and trained again on top of the existing model, I kept getting the following error message:

cost is: nan
binary cost is: 1.8876
instance cost is: inf

Why? How can I fix this?

@MaybeShewill-CV
Copy link
Owner

@deeprun refer to #116

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant