CUDA error: uncorrectable ECC error encountered

[tienmng]

Hello,

I have a dataset of 50 data points in xyz and npz format. When I used nequip-train command to train the model, I got "RuntimeError: CUDA error: uncorrectable ECC error encountered" error.

I very much appreciate it if anyone could take a look at my input files and help me with this problem. Files are attached.

input-allegro.zip

[simonbatzner]

Hi @tienmng, thanks for your questions, a few points below:

• if you're sure this should be a periodic structure, then keep the cell and pbc as true
• in general, the extxyz option is better to work with

According to stackoverflow and this github discussion the error you're seeing is related to a hardware issue. On the github discussion they said reinstalling helps fix it. Can you share the full stacktrace as well as your install (pytorch version etc.)?

The reason I could see potentially for why you're not seeing it for the non-pbc version is that it may be out-of-memory related potentially? In such a case you will need much less memory in non-pbc since you're not computing edges across the pbc. But please share the full stacktrace first and try reinstalling this. Based on googleing, this sounds hardware related.

One other thing to try to find out if it's memory-related: you can run the same setup but with a 5 Angstrom r_max? 8A will require quite a bit of memory and you have a large structure + a semi-large network.

[tienmng]

Hi @simonbatzner, thank you very much for your help! I reinstalled the package and the model was able to be trained.
Besides, I have a large dataset of 20K data points (300 atoms each), besides batch size, l_max, num_layers, and r_max, what parameters need to be reduced to avoid CUDA out of memory?

[simonbatzner]

That's great to hear. By far the most important one is r_max, since number of edges grows cubically with the cutoff radius. Often 5-7 is enough, I would definitely scan that as my first hyperparameter. Usually also 2 layers are enough. If you want to push less, you can reduce the env_embed_multiplicity or the MLP size but I would only touch those if you have no other options, large MLPs are particularly important, and I wouldn't recommend a env_embed_multiplicity below 8 or 16 or so. Let me know if you have any issues going forward.