-
Notifications
You must be signed in to change notification settings - Fork 107
Data Locking in WMCore
Datasets and files that are in use by active workflows need to be excluded from the automatic data cleanup mechanisms in Dynamic Data Management (DDM/Dynamo). This "data locking" feature was previously provided by Unified. Now the functionality is provided by WMCore.
- Data locking is done at the dataset level. Locking at the block level is not implemented.
- Data locking is global. Site specific locks are not implemented.
- Datasets associated with workflows in the following states need to be locked.
['assignment-approved', 'assigned', 'staging', 'staged', 'acquired',
'running-open', 'running-closed', 'force-complete', 'completed']
-
Any parent datasets of datasets used by workflows with the property:
"IncludeParents": True -
Transient output and unmerged LFNs Generated by retrieving the workflow property:
OutputModulesLFNBases -
"Ad hoc" locks A set of manually configurable things to lock. Provided by "adhoc_lock.json" in Unified.
A combination of the globallocks, parentlocks and protectedlfns APIs is used to determine the set of global datasets and unmerged files that are in use and should not be removed. API details are at https://github.com/dmwm/WMCore/wiki/wmstatsserver-api
DBS - must be queried to discover parent datasets
New requests could include a property for any required parent datasets. This would eliminate DBS dependency and potential issues arising from failed DBS queries. Any request already in the system would not have this property, some could still be active for ~6 months.