Skip to content

Releases: dmwm/WMCore

WMCore 1.4.4 services production release

12 Jan 14:51
Compare
Choose a tag to compare

This release moves the workflow archival from ReqMgr2 to a newer microservice, MSRuleCleaner.

Changes since release: 1.4.3.
Release date: 2 Dec 2020.

Central services

Features and/or feature changes

  • No longer resolve StepChain parentage for workflows in normal-archived #10201
  • MSRuleCleaner: Add workflow archival logic to MSRuleCleaner #10153
  • Add all Rucio lfns2pfns args. to getPFN. Fix #10056 #10122
  • Add requiresApproval method to Rucio class. Manage RSE rule approval requirements in insertContainerRules #10105 #10130

Bug Fixes

  • MSTransferor: do not consider Rucio blocks with 0 or None size #10205
  • Fix pileup location with multi-RSE rules #10194
  • Fix GlobalWorkQueue CherryPy threads initialization and logger #10099
  • Fix regexp for enable cherrypy threads && Fix file permissions for dmwm-service-key.pem #10155

Enhancements

  • Reuse Rucio object in WorkQueue start policy modules #10099
  • Mock Rucio APIs #10099
  • Remove DDM service wrapper module #10190

WMAgent

Features

  • Add requiresApproval method to Rucio class. Manage RSE rule approval requirements in insertContainerRules #10105 #10130

Bug Fixes

  • Fix servicekey permission in the agent deployment; remove central services url option #10163
  • Remove badly encoded single quotes from Rucio module #10124
  • Add only merged parents to the ACDC docs #9997
  • Fix JobAccountant missing parent dbsbuffer file #9997
  • Catch UnicodeDecodeError in JobAccountant #10152

Enhancements

  • Reuse Rucio object in WorkQueue start policy modules #10099
  • Minor updates to some json test templates #10113

Python3 related changes

  • [py2py3] WMCore/WMException.py - issue 10173 - src #10174
  • [py2py3] src/python/A/B/C slice 2 - issue 10098 - src #10103

WMCore 1.4.3 services and WMAgent production release

02 Dec 13:26
Compare
Choose a tag to compare

This release brings in a new MicroService called MSRuleCleaner (still in beta version), which is responsible for removing the no longer output data rules created by WMAgent (under wma_prod account).
It also comes with an up-to-date rucio-clients version - 1.23.11 - which has a more restrictive certificate permission checks.

Changes since release: 1.4.2.
Release date: 2 Dez 2020.

Central services

Features and/or feature changes

  • Deprecate usage of useRucio parameter within MicroServices #10094
  • Remove PhEDEx logic from MSOutput/MSMonitor #10094
  • Added PyPI support for reqmon, reqmgr2ms, global-workqueue, acdcserver #10059
  • Throttling re-factoring and added timerange for user activities under throttling #9211
  • Removed CherryPy WMDataMining thread/code #10065
  • Added new exception to the Rucio wrapper (WMRucioDIDNotFoundException) #10023
  • MSRuleClener logic for output data rule cleanup #10023 #10079 #10080 #10100

Bug Fixes

Enhancements

  • Switched validation script to Rucio (including rollback of python3 changes) #10045
  • Removed DataStructs/DASDocument module #10096
  • Update setup dependencies for CRAB services #10084
  • Only inject test templates that can have MSTransferor making data placement #9784
  • Update workflow json templates #9784
  • Script to recursively parse cms-sw/cmsdist spec files #10036

WMAgent

Features and/or feature changes

  • Removing file status condition from GetUnsubscribedDatasets query #10067

Bug Fixes

  • Evaluate all WQEs when local resources are available #10072 #10102

Enhancements

Python3 related changes

  • WMCore/DataStructs - futurize #10012
  • WMCore/Algorithms - futurize #10012

WMCore 1.4.2 services production release

05 Nov 08:30
Compare
Choose a tag to compare

This release no longer uses any of the PhEDEx functionality and Rucio is enabled as the standard Data Management system. It also brings an alpha version of a new microservice (MSRuleCleaner), which will be responsible for cleaning unneeded input and output data rules. Last but not least, RucioInjector component now supports block deletion and output container data placement, meaning it now provides 100% of the expected functionality.

Changes since release: 1.4.1.
Release date: 5 Nov 2020.

Central services

Features and/or feature changes

  • MSOutput: Add getTransferInfo to MSOutput and info request to MSManager. #9978
  • MSOutput: treat NANOAOD/NANOAODSIM as any other datatier in the output data placement logic #9988
  • MSOutput: create Disk rules with an weight attr, default to ddm_quota #9993
  • MSOutput: use different lifetime configuration for RelVal Disk rules #10046
  • If listDataRules API fail, raise a WMRucioException #9999
  • MSTransferor: remove PhEDEx functionality #10002
  • MSRulecleaner initial commit. #10003

Bug Fixes

  • MSOutput: Skip Tape (too) data placement for Resubmission workflows #9983
  • Properly validate RequestPriority parameter #9994
  • MSTransferor: resolve RSE expression for pileup location #10041
  • Fix pileup data location in global workqueue #10042

Enhancements

  • Use list_replication_rules Rucio API to list rules #9963
  • Add CMSMonitoring dependency for ReqMgr2 #10004
  • Introduce deploy-centralvm.sh #10014
  • List of json templates that can undergo input data placement #10034

WMAgent

Features

  • Disable PhEDExInjector; Enable all tiers to be injected in Rucio #9980
  • Treat NANOAOD/NANOAODSIM as any other datatier in the output data placement logic #9988
  • Skip Custodial and make generic Disk NonCustodial data placement in RucioInjector #9989
  • If listDataRules API fail, raise a WMRucioException #9999
  • Implementation for the Rucio getPFN method #10029
  • Increase polling cycle for WorkQueueManagerLocationPoller to 1h #9981
  • Block deletion logic implemented in RucioInjector #9959 #10044
  • Deal with T0 output data subscription in RucioInjector #10006

Bug Fixes

  • Only add replicas if parent block has been properly added/attached into Rucio #10024
  • Ignore availability for block level rules against the origin site/RSE #10031

Enhancements

Python3 related changes

  • Revert stage2 fixers on DQMUpload module #9945
  • [py2py3] WMCore/WMFactory - futurize #9971
  • [py2py3] WMCore/Credential - futurize #10008

WMCore 1.4.1 services and WMAgent production release

06 Oct 07:30
Compare
Choose a tag to compare

This release completes all the MicroServices and Global WorkQueue work required to support Rucio as a Data Management system in production.

Changes since release: 1.4.0.
Release date: 6 Oct 2020.

Central services

Features and/or feature changes

  • MSOutput: make RSE expression configurable when picking a Tape RSE #9957
  • MSOutput: skip tape data placement for RelVals #9920
  • MSOutput: flag HIRelVal workflows as RelVal in the mongodb records #9919
  • MSOutput: new DatasetSize param for the mongo record; fetch container size from DM systems #9916
  • MSOutput: make alert notification and RSE attribute configurable #9915
  • Avoid resolving container location with the dataset_locks, whenever possible #9955
  • Different algorithm to run the block/container data location #9936
  • Add listParentDIDs functionality to Rucio service #9935
  • Drop Tape RSEs from the pileup location #9930
  • Decouple PhEDEx from the DBS3Reader wrapper module #9927
  • PyPI packaging for ReqMgr2 #9874
  • MSTransferor: fix DQMHarvest input data placement; print exception error #9949

Bug Fixes

  • MSMonitor: deal with ZeroDivision exception for large rules in INJECT state #9951
  • Drop Tape RSEs from the MSTransferor input location #9944
  • Fix Rucio data location method; support dropping Tape RSEs #9936

Enhancements

  • Update python requirements #9941
  • Remove wmarchive dependency #9921

WMAgent

Features

  • Flag to disable PhEDExInjector; setup to enable all datatiers in RucioInjector #9947
  • Construct the PU location via rucio considering MSTransferor locks #9930
  • Drop Tape RSEs from the pileup location #9930
  • Decouple PhEDEx from the DBS3Reader wrapper module #9927

Bug Fixes

  • Avoid passing unicode strings to ParameterSet function #9924
  • Fix checkStuckLGE script referencing unitialized variable #9922

Enhancements

Python3 related changes

WMCore 1.4.0 services production release

05 Oct 09:56
Compare
Choose a tag to compare

In short, this release brings a substantial amount of changes to support Python3 within WMCore, depending on the python future package. The majority of the work to integrate Rucio within WMCore MicroServices has been completed as well.

Changes since release: 1.3.6.
Release date: 15 Sept 2020.

Central services

Features

  • A substantial portion of the code now depends on the future python package, to support python 3 in parallel with python 2.
  • Rely on twine to upload pypi packages #9780
  • Bump rucio-clients from 1.22.2 to .1.23.0 #9846
  • Added getAccountLimits method to the Rucio wrapper #9857
  • MS: Track Rucio rule progress in MSMonitor #9881 #9899
  • Use python and python libraries from CVMFS COMP under slc7_amd64_gcc630 #9819
  • MSTransferor: support input data placement with Rucio #9759
  • Default rucio rules to asynchronous #9759
  • Rucio wrapper API to find where data is locked and available #9889
  • Single node dataset-level subscription for DQMHarvest workflows #9889
  • MS: Update MSOutput record schema #9901
  • MS: Update schema for MSOutput records with OutputMap; Updated MSOutput logic #9904
  • Added TiersToDM parameter to the campaign configuration #9906
  • MSOutput: skip output data placement for Resubmission request type #9910
  • MSOutput: support custodial/tape data placement #9911
  • Added Rucio pick site function #9884
  • MSOutput: PhEDEx/Rucio logic to decide which RSE to use for Tape transfers #9912
  • Get block replicas info using the Rucio bulk API #9900

Bug Fixes

  • Update Rucio get_account_usage to get_local_account_usage API #9850
  • Update ParentageResolved flag for StepChains without parent as well #9865
  • Do not send parent/child data to DBS if there is nothing to be fixed #9853
  • Fix HEAD request via pycurl_manager; added a few extra getters #9855
  • Improve py2 compatibility of WMException init #9868
  • Fix Rucio rule creation when there are duplicate rules #9897
  • Avoid failing MSTransferor cycle if Campaigns are misconfigured #9909

Enhancements

  • Add MSOutput to WMCore diagram #9804
  • Add MSOutput to Micro Services diagram. #9802
  • Fix missing job report error codes #9781
  • Update WMAgent and WMCore python requirements #9825
  • Script to print a summary of LQEs stuck in Available for a given workflow #9863
  • Support caching of RSE expressions in the Rucio wrapper #9892

WMAgent

Features

  • Use python and python libraries from CVMFS COMP under slc7_amd64_gcc630 #9819
  • Use PSN as PNN for opportunistic diskless resources #9693
  • Force HEPCloud resources to be down in the CERN agents #9844

Bug Fixes

  • Add leading zeros to adler32 to ensure 8 chars length #9809

Enhancements

  • Notify Todor upon component automatic restarts #9866

Python3 related changes

  • PY3: PSetTweaks - compatible with both python2 and python3 #9740
  • PY3: Utils - compatible with both python2 and python3 #9731
  • PY3: Configuration - sys.version_info compatible with py2.6 #9839
  • PY3: TestInit - Remove dependency on MySQL-python #9848
  • PY3: update stdlib - not pickle #9762 #9875
  • PY3: WMCore.Cache - compatible with both python2 and python3 #9818
  • PY3: Properly deal with strings in REST/Main #9871

WMCore 1.3.6 services production release

07 Jul 07:54
Compare
Choose a tag to compare

In short, this release brings a fully functional MSOutput microservice; a remake of the MSTransferor logic; and the first steps towards supporting python3 in WMCore.

Changes since release: 1.3.5.
Release date: 7 July 2020.

Central services

Features

  • MS: Add support for Rucio in MSOutput Consumer thread && Handle DuplicateRule error from Rucio client.#9772
  • Validate/fail workflow assignment for improper use of TrustSitelists/TrustPUSitelists (skip ACDCs). #9770 #9793
  • Remove status 'completed' from msOutput. #9773

Bug Fixes

  • MS: Couple primary/parent block distribution to where the PU dataset is. #9764
  • MS: In case the pileup needs to be transferred, transfer all the input blocks to the same location. #9800
  • MS: consider datatiers to be excluded in MSOutput; properly deal with DQMHarvest and skipped output datasets. #9755
  • MS: Properly handle exceptions inside docTransformer for MSOutput Producer thread. #9750
  • MS: Slice large list of runs when calling DBS blocks REST API. #9747
  • Fix RequestPriority update during assignment. #9748
  • Update old integration Rucio URLs. #9775

Enhancements

  • Add MSOutput to WMCore diagram. #9804
  • Add MSOutput to Micro Services diagram. #9802
  • Reduce default number of pycurl connections to 50. #9785
  • Replace StringIO by the python3 io module. #8733

WMAgent

Features

  • Add guid consistency check support for secondary input. #9716

Bug Fixes

  • Fix python version evaluation in Configuration module. #9766

Enhancements

  • Remove duplicated REQUIRED_OS logic, as this has been integrated in T2_CH_CERN_HLT already. #9767
  • Add nose2 to the wmagent requirements file. #9778
  • Only fail cleanup jobs if they fail to delete all input files. #9754
  • Add CreationFailure exit code. #9533
  • Print WQ queue url when fetching available elements; break long line down. #9742
  • Replace StringIO by the python3 io module. #8733
  • Enable RucioInjector component for NANO tiers. #9733

1.0.14 for CRAB - #1

18 Mar 18:47
Compare
Choose a tag to compare
Pre-release
Merge pull request #6718 from emaszs/1.0.14_crab

Allow pycurl_manager.py to accept encoded string as data. Fixes #6467

1.0.12 CRAB 2

12 Feb 14:21
Compare
Choose a tag to compare

Next release for CRAB

Tag 1 for CRAB based on WMCore 1.0.12

21 Jan 17:37
Compare
Choose a tag to compare
Merge pull request #6527 from mmascher/1.0.12_crab

Remove line that becames problematic in building packages