Skip to content

Commit

Permalink
Update rerunning_cycles.md
Browse files Browse the repository at this point in the history
  • Loading branch information
aosprey authored Sep 2, 2024
1 parent d205ddb commit c931620
Showing 1 changed file with 8 additions and 5 deletions.
13 changes: 8 additions & 5 deletions docs/rerunning_cycles.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,15 +35,18 @@ Go to `share/data/History_data/CICEhist`, and edit `ice.restart_file` to point t

You will need to re-run all tasks, including post-processing for the cycle you are re-running (*N-1*)
and these need to run before the tasks in cycle *N*.

Postproc and the ozone redistribution scheme can complicate things a bit, and ideally postproc should run straight after the coupled task.
Follow the instructions carefully to ensure this is done correctly.
If in doubt ask Annette for guidance.

1. Retrigger the `coupled` task for cycle *N-1*
2. Pause `postproc` for cycle *N*.
3. Reset all post-processing tasks in cycle *N-1* to waiting.
3. Reset all post-processing tasks in cycle *N-1* to waiting (so it's easier to see when tasks have been re-run).
4. If the JDMA task for *N-1* had completed, you will need to delete the data from JDMA.
It needs to finish uploading before it can be deleted.
If, for example, JDMA is down and the upload hasn't started you can get JASMIN to cancel the request from the queue.
5. Once the coupled task for *N-1* has successfully completed, trigger coupled for cycle *N*.
6. You may need to manually re-trigger the post-processing tasks in cycle *N-1*.
These need to be done in order, check the graph view in cylc if in doubt.
7. Once all tasks in cycle *N-1* have completed, un-pause `postproc` in cycle *N*.
5. Once coupled *N-1* has finished, manually trigger i) postproc for cycle *N-1* and ii) coupled for cycle *N*.
6. Postproc may fail with an error like `ValueError: Incorrect size for fixed length header; given 0 words but should be 256.`. Have a look at the `*po` files in `History_Data`. Is there a zero-length `po` file from the month before (so cycle *N-2*)? If so delete this file and re-try. Do not delete any other files.
7. After these two tasks (postproc *N-1* and coupled *N*) have successfully completed, re-trigger the rest of the post-processing tasks for cycle *N-1* in order. You can check the graph view in cylc if in doubt. The `compress_netcdf` task may not show up, in which case you need to insert it first.
8. Once all tasks in cycle *N-1* have completed, un-pause `postproc` in cycle *N*, and the suite should continue on as normal.

0 comments on commit c931620

Please sign in to comment.