Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set memory for ignored tasks (Express Merge) #12085

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

LinaresToine
Copy link

@LinaresToine LinaresToine commented Aug 27, 2024

Fixes #12086

Status

In development

Description

Currently, "Merge" tasks, among others, have their job resource information hardcoded in setRuntimeMonitors. For these tasks, the functions that set a different memory value do not work, as they are set to ignore such tasks:

The problem with this is that Express Merge jobs in particular have shown to need more memory than the default value of 2.3 GiB. Since the memory setting functions ignore merge tasks, these jobs cant be given more memory without changing the current code.

This patch allows to avoid the tasks from being ignored in case a task is specified
Adds support for setting the memory of specified taskNames, and if such taskName is of a "Merge" task it would not be ignored.

Is it backward compatible (if not, which system it affects?)

MAYBE

Related PRs

NONE

External dependencies / deployment changes

NONE

@@ -2002,6 +2009,10 @@ def updateArguments(self, kwargs):

if kwargs.get("Memory") is not None:
self.setMemory(kwargs.get("Memory"))
if kwargs.get("TaskMemory") is not None:
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The configuration used from T0 side would have a feature named "TaskMemory"

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 4 new failures
  • Python3 Pylint check: failed
    • 11 warnings and errors that must be fixed
    • 1 warnings
    • 153 comments to review
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded
    • 26 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15172/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Can one of the admins verify this patch?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support for setting maxPSS in specified tasks (Merge tasks)
2 participants