【Force Error Initially Decreasing then Increasing During Training】

[hellolori]

Dear Developers,

I am currently learning how to use deepMD to train potential functions. I have successfully reproduced the CH4 example from the official website. However, I encountered a problem when applying the training method to another test system. The figure below shows the changes in energy and force errors with training steps from the "lcurve.out" file.

It can be seen that as the training steps increase, the force error initially decreases but then starts to increase.

Here are the selected data from the "lcurve.out" file:

My "input.json" parameters are set as follows:

{
    "_comment": " model parameters",
    "model": {
        "type_map":     ["Nb", "Se"],
        "descriptor" :{
            "type":             "se_e2_a",
            "sel":              "auto",
            "rcut_smth":        0.50,
            "rcut":             6.00,
            "neuron":           [25, 50, 100],
            "resnet_dt":        false,
            "axis_neuron":      16,
            "seed":             1,
            "_comment":         " that's all"
        },
        "fitting_net" : {
            "neuron":           [240, 240, 240],
            "resnet_dt":        true,
            "seed":             1,
            "_comment":         " that's all"
        },
        "_comment":     " that's all"
    },

    "learning_rate" :{
        "type":         "exp",
        "decay_steps":  10000,
        "start_lr":     0.001,
        "stop_lr":      3.51e-8,
        "_comment":     "that's all"
    },

    "loss" :{
        "type":         "ener",
        "start_pref_e": 0.02,
        "limit_pref_e": 1,
        "start_pref_f": 1000,
        "limit_pref_f": 1,
        "start_pref_v": 1,
        "limit_pref_v": 1,
        "_comment":     " that's all"
    },

    "training" : {
        "training_data": {
            "systems":     ["../00.data/training_data"],
            "batch_size":  "auto",
            "_comment":    "that's all"
        },
        "validation_data":{
            "systems":     ["../00.data/validation_data"],
            "batch_size":  "auto",
            "numb_btch":   1,
            "_comment":    "that's all"
        },
        "numb_steps":   2000000,
        "seed":         10,
        "disp_file":    "lcurve.out",
        "disp_freq":    500,
        "save_freq":    200000,
        "_comment":     "that's all"
    },

    "_comment":         "that's all"
}

The training and validation sets contain the following information.

The information of training_systems:

Data Summary
Labeled System
-------------------
Frame Numbers      : 720
Atom Numbers       : 27
Including Virials  : Yes
Element List       :
-------------------
Nb  Se
9  18

The information of validation_systems:

Data Summary
Labeled System
-------------------
Frame Numbers      : 180
Atom Numbers       : 27
Including Virials  : Yes
Element List       :
-------------------
Nb  Se
9  18

I have the following questions:

(1) Why does the force error increase after initially decreasing as the number of steps increases?
(2) How can I reduce the force error?

Thank you for your assistance.

[njzjz]

I have the same suggestion as that in #2278.