Run manager not receiving completed runs from agents during Jacobian filling with super parameters

[ckikuchi]

Hello,
I am using pestpp-glm in parallel, with the following settings:

++ hotstart_resfile(.rei)
++ base_jacobian(.jcb)
++ n_iter_base(1)
++ n_iter_super(10)
++ max_n_super(20)
++ lambdas(1e6, 1e7, 5.0e7, 1e8, 5.0e8, 1.0e9)
++ upgrade_augment(true)
++ jac_scale(true)
++ svd_pack(eigen)
++ panther_agent_freeze_on_fail(true)
++ der_forgive(true)

I am encountering an issue (this has happened both times I tried) where pestpp completes the parameter upgrades without problem and moves on to filling the Jacobian. When each agent completes a run, it says that the results have been sent to the run manager. However, the run manager does not show the completed runs, and the .rmr file does not show that the manager received the completed runs. Both times, one or more runs failed due to computer issues (one didn't have the right software installed, things like that), but my understanding is that the ++der_forgive() option should allow the run to continue?

I would appreciate any suggestions. I am wondering if one of the pestpp settings I have chosen is causing a problem with communications between the manager and agents?

Thank you!

-Colin

[jtwhite79]

@ckikuchi hard to say what is happening. Can you email me the rec and rmr file from this run (dont post the rmr file b/c it has some detailed machine info)?

[ckikuchi]

Hi Jeremy,
Thanks for taking a look at this! What is the best email to use? I only have your USGS email - is that still active?
-Colin