Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Numerical instability (NaN's) during training with power-spherical-vae #41

Open
BerndDoser opened this issue Nov 7, 2023 · 0 comments

Comments

@BerndDoser
Copy link
Member

The training of the power-spherical-vae crashes in epoch 378 with NaN values within the beta function:

File "/local_data/doserbd/git/Spherinator/models/../external/power_spherical/power_spherical/distributions.py", line 162, in __init__
243     MarginalTDistribution(
244   File "/local_data/doserbd/git/Spherinator/models/../external/power_spherical/power_spherical/distributions.py", line 102, in __init__
245     torch.distributions.Beta(
246   File "/local_data/doserbd/miniconda3/envs/spherinator/lib/python3.10/site-packages/torch/distributions/beta.py", line 38, in __init__
247     self._dirichlet = Dirichlet(concentration1_concentration0, validate_args=validate_args)
248   File "/local_data/doserbd/miniconda3/envs/spherinator/lib/python3.10/site-packages/torch/distributions/dirichlet.py", line 54, in __init__
249     super().__init__(batch_shape, event_shape, validate_args=validate_args)
250   File "/local_data/doserbd/miniconda3/envs/spherinator/lib/python3.10/site-packages/torch/distributions/distribution.py", line 62, in __init__
251     raise ValueError(
252 ValueError: Expected parameter concentration (Tensor of shape (64, 2)) of distribution Dirichlet(concentration: torch.Size([64, 2])) to satisfy the constraint IndependentConstraint(GreaterThan(lower_bound=0.0), 1), but found invalid values:
253 tensor([[nan, 1.],
254         [nan, 1.],
255         [nan, 1.],
256         [nan, 1.],
257         [nan, 1.],
258         [nan, 1.],
...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant