Better error reporting of model crashes

[aidanheerdegen]

@blimlim response to a question on the ACCESS-Hive forum

The error “payu: Model exited with error code 1; aborting” can be a difficult one to find the cause of. My understanding is that whenever any of ESM’s submodels (e.g. atmosphere, ocean, ice) crash during a simulation, that crash will be picked up by Payu, which then reports the "payu: Model exited with error code 1; aborting” error to the user. As a result there could be a large range of different causes.

got me thinking. Can payu do better with error reporting of model crashes? Can payu identify which model crashed? I guess probably only through inspection of the logs, but are there any options in mpirun that could be used to report more fine-grained error codes?