-
Notifications
You must be signed in to change notification settings - Fork 174
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Loss becomes nan after a while #7
Comments
Try using the Keras optimizer rather than Lasagne's. |
What would be the exact changes in the code? |
Same here, I have edited model.py : |
I'm seeing this issue too. I switched to using the Keras optimizer instead of Lasagne's, making the same changes that @aglotero cited above. For the first 8990 (out of 12188) iterations the loss function was working properly. Then it looks like starting at iteration 9000 I started seeing the
Interestingly, the loss spiked at iteration 8960. Here is the plot for the first 9000 iterations. Some notes: I am using dropout on the RNN layers hence the plot, and I increased the data size being trained on by upping the max duration to 15.0 seconds. My mini batch size is 24. |
Using the SGM optimizer with the |
FWIW I fixed this by dropping the learning rate and removing the dropout layers I added. I left the clipnorm value at 100. |
Hi! |
@a00achild1 I don't think you want clipnorm set to 1. Were you getting NaNs before with the clipnorm set to a higher val (~100)? |
@dylanbfox thanks for quick response! What do you mean you don't think setting clipnorm to 1 is a good idea? Is the performance affected by the small clipnorm value? |
What is your learning rate? Trying dropping that and keeping the clipnorm higher. |
@dylanbfox my learning rate is 2e-4, the default value. |
I have a similar problem as the other threads described. But my model has NaN value after 1st iteration. Ex. Keras , learning_rate=2e-4, clipnorm = 1 |
I just found a solution! We should use Keras-1.1.0 or above version for Keras package. |
@a00achild1 Hey. Did you find out why your graph turns into that? I'm currently at that stage. |
Why is this happening? And how to solve it?
The text was updated successfully, but these errors were encountered: