Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

backport fix from #5712 #5927

Closed
wants to merge 4 commits into from
+56 −5
Closed

backport fix from #5712 #5927

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
67e417b
backport two fixes from #5712
wxtim Jan 16, 2024
79712dc
tests
wxtim Jan 16, 2024
3bae035
Response to review.
wxtim Jan 17, 2024
f363b7f
Update changes.d/fix.5712.md
wxtim Jan 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changes.d/fix.5712.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fixed a bug preventing simulation of task retries if job submission unset.
1 change: 1 addition & 0 deletions cylc/flow/task_events_mgr.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -778,6 +778,7 @@ def _process_message_check(

if (
itask.state(TASK_STATUS_WAITING)
and itask.tdef.run_mode == 'live'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Disables logging?

Copy link
Member Author

@wxtim wxtim Jan 17, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code within this section is designed to avoid problems with polling - but there will be no polling in simulation mode, and it causes simulated tasks to remain in waiting mode forever. #5712 has this change.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But this is the process message check run on incomming messages, it doesn't trigger polling?!

Copy link
Member Author

@wxtim wxtim Jan 17, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Disables logging?

And returns False, rather than True.

No it doesn't trigger polling.

Comment at line 799

        # Ignore messages if task has a retry lined up
        # (caused by polling overlapping with task failure)

In simulation mode this check weeds out the submitted message. Alternatively, I think that the following diff would be more reasonable for the same effect:

-
-            else:
+                return False
+            elif flag == self.FLAG_POLLED_IGNORED:
                 LOG.warning(
                     f"[{itask}] "
                     f"{self.FLAG_POLLED_IGNORED}{message}{timestamp}"
                 )
-            return False
+                return False
 
         severity = cast(int, LOG_LEVELS.get(severity, INFO))

IMO I think it would make sense to make a change which matches master, then change both later if you think this or some other approach is more sensible.

Copy link
Member Author

@wxtim wxtim Jan 17, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've pushed up this change to see if it breaks owt on CI

and
(
(
Expand Down
5 changes: 4 additions & 1 deletion cylc/flow/task_job_mgr.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1000,7 +1000,10 @@ def _simulation_submit_task_jobs(self, itasks, workflow):
itask.waiting_on_job_prep = False
itask.submit_num += 1
self._set_retry_timers(itask)
itask.platform = {'name': 'SIMULATION'}
itask.platform = {
'name': 'SIMULATION',
'submission retry delays': [1]
}
itask.summary['job_runner_name'] = 'SIMULATION'
itask.summary[self.KEY_EXECUTE_TIME_LIMIT] = (
itask.tdef.rtconfig['job']['simulated run length']
Expand Down
32 changes: 32 additions & 0 deletions tests/functional/modes/03-simulation.t
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
#!/usr/bin/env bash
# THIS FILE IS PART OF THE CYLC WORKFLOW ENGINE.
# Copyright (C) NIWA & British Crown (Met Office) & Contributors.
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.

# Test that simulation mode runs, and reruns a failed task successfully
# when execution retry delays is configured.

. "$(dirname "$0")/test_header"
set_test_number 2

install_workflow "${TEST_NAME_BASE}" "${TEST_NAME_BASE}"
run_ok "${TEST_NAME_BASE}-validate" cylc validate "${WORKFLOW_NAME}"
workflow_run_ok "${TEST_NAME_BASE}-run" \
cylc play \
--no-detach \
--mode=simulation \
--reference-test "${WORKFLOW_NAME}"
purge
exit
16 changes: 16 additions & 0 deletions tests/functional/modes/03-simulation/flow.cylc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
[scheduler]
[[events]]
workflow timeout = PT30S

[scheduling]
initial cycle point = 2359
[[graph]]
R1 = get_observations

[runtime]
[[get_observations]]
execution retry delays = PT2S
[[[simulation]]]
Comment on lines +12 to +13
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
execution retry delays = PT2S
[[[simulation]]]
execution retry delays = PT1S
[[[simulation]]]
default run length = PT0S

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The point of this test was, at least partly to test the interaction of this setting.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, what do you mean? I'm just suggesting this to speed up the test

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't want the default run length.

fail cycle points = all
fail try 1 only = True

2 changes: 2 additions & 0 deletions tests/functional/modes/03-simulation/reference.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
23590101T0000Z/get_observations -triggered off [] in flow 1
23590101T0000Z/get_observations -triggered off [] in flow 1
Loading