-
Notifications
You must be signed in to change notification settings - Fork 54
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
doc: added deletion overview with very high level chart. #288 #295
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
@@ -0,0 +1,38 @@ | ||||||
--- | ||||||
id: deletion-overview | ||||||
title: Deletion Overview | ||||||
--- | ||||||
|
||||||
Deletion in rucio can be performed in broadly two ways. | ||||||
|
||||||
- Rule based deletion: | ||||||
|
||||||
Each rule has a corresponding lifetime associated with it (default is None). The attribute `expires at` of the rule is the time when the lifetime is set plus the lifetime associated with rule. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
Additionally each rule can be `locked` for additional protection. If a rule is locked the rule is protected from expiration. | ||||||
The `Judge-cleaner` daemon is responsible for handling expired rules. If an expired rule is not locked it sets `tombstones` on all replicas not covered by the rule anymore (or any other rule). These replicas then become eligible for deletion. | ||||||
|
||||||
- DID based deletion: | ||||||
|
||||||
Additionally, each did also has a lifetime associated with it (default is None). The metadata `expires_at` of the did is calculated similarly to a rule. | ||||||
The `Undertaker` daemon is responsible for handling expired dids. The daemon checks if none of the associated rules to the did are locked, if not it removes all rules of the did (see section above). The did itself is then removed from the catalog. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
**The did expiration overrules the rule expiration. But the locked rules are protected.** | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
After the tombstone is set for replicas, the actual data deletion is done by the `Reaper` daemon. The reaper physically deletes the tombstoned replicas from storage. | ||||||
The deletion service supports two different modes: greedy and non-greedy. | ||||||
|
||||||
- Greedy | ||||||
|
||||||
The reaper daemon gets all the replicas with tombstone in the RSE and immediately delete all replicas. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
- Non-greedy | ||||||
|
||||||
The reaper daemon first check if the free space is needed in the RSE. The needed free space is the difference of minimum free space (attrribute set for RSE) and actual free space in RSE. Deletion only occurs once free space is needed. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
Deletions are processed by Least Recently Used (LRU) algorithm, thus oldest accessed (tombstoned) replicas are deleted first. | ||||||
|
||||||
|
||||||
![Judge-cleaner chart](/img/Judge-cleaner.png) | ||||||
|
||||||
![Undertaker chart](/img/undertaker.png) | ||||||
|
||||||
![Reaper chart](/img/reaper.png) |
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit: all other filenames in this asset folder are lowercase; for consistency, this could be renamed to |
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "get list of all replica with tombstone" -> "get list of all replicas with tombstone" There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There are two identical "get list of all replicas with tombstone" blocks; the chart could be simplified so that there is only one block, to which you can either get via "Is greedy? YES" or "needed_free_space >= 0" |
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "Are rule locked?" should be "Is rule locked?" There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. also - "child(s)" should likely be "child(ren)" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.