Not all energy calculated in energy.raw during 02.fp step

[anseongpark]

Summary

Sometimes, energy.raw file at iteration.0000x/02.fp/ doesn't have all the task's energy.
For example, when implementing 02.fp step with 30 tasks, other files such as coord.raw, box.raw have all 30 task's data. However, energy.raw sometimes has less than 30, which abrupt the process.
From the raw code, I think I found the way that dpgen reads the energy data. After the first principle calculation of a task is finished, DPGEN searches for a specific key word (a specific sentence that has energy value in it) and save the energy data. However, I found out that some of my first principle calculation ended without such specific key word (sentence), so that the DPGEN could not find the task's energy value.
Repeating the job several times, it rarely successfully calculated all the tasks so that it proceeds the iteration.
I wonder why this happen, really appreciate your help.
Thank you.

DPGEN Version and Platform
14 DeepModeling
15 ------------
16 Version: 0.10.6
17 Date: Nov-01-2022
18 Path: /home/Software/python-3.8.3/lib/python3.8/site-packages/dpgen
19
20 Dependency
21 ------------
22 numpy 1.23.3 /home/Software/python-3.8.3/lib/python3.8/site-packages/numpy
23 dpdata 0.2.10 /home/Software/python-3.8.3/lib/python3.8/site-packages/dpdata
24 pymatgen 2019.6.5 /home/Software/python-3.8.3/lib/python3.8/site-packages/pymatgen
25 monty 2022.9.9 /home/Software/python-3.8.3/lib/python3.8/site-packages/monty
26 ase 3.22.1 /home/Software/python-3.8.3/lib/python3.8/site-packages/ase
27 paramiko 2.11.0 /home/Software/python-3.8.3/lib/python3.8/site-packages/paramiko
28 custodian 2022.5.26 /home/Software/python-3.8.3/lib/python3.8/site-packages/custodian

error
127 Traceback (most recent call last):
128 File "/home/Software/python-3.8.3/lib/python3.8/site-packages/numpy/lib/npyio.py", line 1565, in savetxt
129 v = format % tuple(row) + newline
130 TypeError: must be real number, not NoneType
131
132 The above exception was the direct cause of the following exception:
133
134 Traceback (most recent call last):
135 File "/home/Software/python-3.8.3/bin/dpgen", line 8, in
136 sys.exit(main())
137 File "/home/Software/python-3.8.3/lib/python3.8/site-packages/dpgen/main.py", line 185, in main
138 args.func(args)
139 File "/home/Software/python-3.8.3/lib/python3.8/site-packages/dpgen/generator/run.py", line 3642, in gen_run
140 run_iter (args.PARAM, args.MACHINE)
141 File "/home/Software/python-3.8.3/lib/python3.8/site-packages/dpgen/generator/run.py", line 3631, in run_iter
142 post_fp (ii, jdata)
143 File "/home/Software/python-3.8.3/lib/python3.8/site-packages/dpgen/generator/run.py", line 3508, in post_fp
144 post_fp_vasp(iter_index, jdata)
145 File "/home/Software/python-3.8.3/lib/python3.8/site-packages/dpgen/generator/run.py", line 3151, in post_fp_vasp
146 all_sys.to_deepmd_raw(sys_data_path)
147 File "/home/Software/python-3.8.3/lib/python3.8/site-packages/dpdata/system.py", line 1476, in to_format
148 return self.to_fmt_obj(ff(), *args, **kwargs)
149 File "/home/Software/python-3.8.3/lib/python3.8/site-packages/dpdata/system.py", line 1087, in to_fmt_obj
150 return fmtobj.to_labeled_system(self.data, *args, **kwargs)
151 File "/home/Software/python-3.8.3/lib/python3.8/site-packages/dpdata/format.py", line 77, in to_labeled_system
152 return self.to_system(data, *args, **kwargs)
153 File "/home/Software/python-3.8.3/lib/python3.8/site-packages/dpdata/plugins/deepmd.py", line 22, in to_system
154 dpdata.deepmd.raw.dump(file_name, data)
155 File "/home/Software/python-3.8.3/lib/python3.8/site-packages/dpdata/deepmd/raw.py", line 76, in dump
156 np.savetxt(os.path.join(folder, 'energy.raw'), np.reshape(data['energies'], [nframes, 1]))
157 File "<array_function internals>", line 180, in savetxt
158 File "/home/Software/python-3.8.3/lib/python3.8/site-packages/numpy/lib/npyio.py", line 1567, in savetxt
159 raise TypeError("Mismatch between array dtype ('%s') and "
160 TypeError: Mismatch between array dtype ('object') and format specifier ('%.18e')

[wanghan-iapcm]

Could you please check the configuration of the failed task? and repeat manually the fp calculation to check what happened.

[njzjz]

I transfer the issue to dpdata. It looks to me that there is at least a bug in dpdata. When the energy is not found, there should be an error, or dpdata should ignore the frame. None should not be appended, and I remember NumPy has disallowed such usage.