Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding all the rule states in getTapRecallUsage and a bug fix #8741

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

aspiringmind-code
Copy link
Contributor

Fix #8736

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Pylint check: succeeded
    • 1 warnings
    • 8 comments to review
  • Pycodestyle check: succeeded
    • 17 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-CRABServer-PR-test/2200/artifact/artifacts/PullRequestReport.html

@belforte
Copy link
Member

Aside the minor comment above, there is a fundamental problem here.

  • for tape recall we do not want to count rules in OK status, since we want to know what is being asked, not what is on disk. True that now we remove OK rules after 4 days, but still it is better to avoid it
  • for input lock almost everything should be OK

I am still puzzled that code could find a sensible value for input lock while ignoring OK rules. That needs to be understood.
Can you look at it, or do I need to do it ?

@@ -124,5 +124,5 @@ def getTapeRecallUsage(rucioClient=None, account=None):

rules = rucioClient.list_replication_rules(filters=filters)
usage = sum(getRuleQuota(rucioClient, rule['id']) for rule in rules\
if rule['state'] in ['REPLICATING', 'STUCK', 'SUSPENDED']) # in Bytes
if rule['state'] in ['OK', 'REPLICATING', 'STUCK', 'SUSPENDED', 'INJECT', 'WAITING_APPROVAL']) # in Bytes
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you want to select all possible states, why not simply remove this line ?
In any case, although we use automatic approve here, rules in WAITING_APPROVAL should not be counted as "using space". I am not sure about rules in INJECTING, i.e. whether it comes before or after APPROVAL.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't remove the line as i still wanted to test that only one of these 6 states are being used and not any random value. Acc to Hasan, INJECT is the first step and it comes before WAITING_APPROVAL, followed by REPLICATING.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we should not count INJECT either. Or account separately for "pendind rules". But let's try not to make it too complicated

@aspiringmind-code
Copy link
Contributor Author

I will look at it.

@aspiringmind-code
Copy link
Contributor Author

@belforte The use of getTapeRecallUsage was merged by me on 17th Sept in #8697 and the release we are running is that of v3.240904 (4th September). Hence the changes are not yet in production and therefore we have the correct thing from using rucioClient.get_local_account_usage in the 4th Sept version.

@belforte
Copy link
Member

thanks. So may be we stick with get_local_account_usage for crab_input ? Why bother to sum all rules when we have the total with a single call ?
Maybe I forgetting something ?

@aspiringmind-code
Copy link
Contributor Author

We did the change to address #8661 as we wanted to filter by activity=AnalysisTapeRecall instead of user, and use user argument only if provided.

@belforte
Copy link
Member

thanks for fixing my confusion ! and sorry for noise.

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Pylint check: succeeded
    • 1 warnings
    • 8 comments to review
  • Pycodestyle check: succeeded
    • 17 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-CRABServer-PR-test/2204/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Pylint check: succeeded
    • 6 comments to review
  • Pycodestyle check: succeeded
    • 33 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-CRABServer-PR-test/2205/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Pylint check: failed
    • 2 warnings and errors that must be fixed
    • 12 comments to review
  • Pycodestyle check: succeeded
    • 33 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-CRABServer-PR-test/2206/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Pylint check: succeeded
    • 12 comments to review
  • Pycodestyle check: succeeded
    • 31 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-CRABServer-PR-test/2207/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Pylint check: succeeded
    • 12 comments to review
  • Pycodestyle check: succeeded
    • 31 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-CRABServer-PR-test/2208/artifact/artifacts/PullRequestReport.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

report of quota used by crab_input is wrong
3 participants