Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc: added deletion overview with very high level chart. #288 #295

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions docs/started/concepts/deletion_overview.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
---
id: deletion-overview
title: Deletion Overview
---

Deletion in rucio can be performed in broadly two ways.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Deletion in rucio can be performed in broadly two ways.
Deletion in Rucio can be performed in broadly two ways.


- Rule based deletion:

Each rule has a corresponding lifetime associated with it (default is None). The attribute `expires at` of the rule is the time when the lifetime is set plus the lifetime associated with rule.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Each rule has a corresponding lifetime associated with it (default is None). The attribute `expires at` of the rule is the time when the lifetime is set plus the lifetime associated with rule.
Each rule has a corresponding lifetime associated with it (default is None). The attribute `expires_at` of the rule is the time when the lifetime is set plus the lifetime associated with the rule.

Additionally each rule can be `locked` for additional protection. If a rule is locked the rule is protected from expiration.
The `Judge-cleaner` daemon is responsible for handling expired rules. If an expired rule is not locked it sets `tombstones` on all replicas not covered by the rule anymore (or any other rule). These replicas then become eligible for deletion.

- DID based deletion:

Additionally, each did also has a lifetime associated with it (default is None). The metadata `expires_at` of the did is calculated similarly to a rule.
The `Undertaker` daemon is responsible for handling expired dids. The daemon checks if none of the associated rules to the did are locked, if not it removes all rules of the did (see section above). The did itself is then removed from the catalog.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The `Undertaker` daemon is responsible for handling expired dids. The daemon checks if none of the associated rules to the did are locked, if not it removes all rules of the did (see section above). The did itself is then removed from the catalog.
The `Undertaker` daemon is responsible for handling expired DIDs. The daemon checks if none of the associated rules to the DID are locked, if not it removes all rules of the DID (see section above). The DID itself is then removed from the catalog.


**The did expiration overrules the rule expiration. But the locked rules are protected.**
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
**The did expiration overrules the rule expiration. But the locked rules are protected.**
**The DID expiration overrules the rule expiration. But the locked rules are protected.**


After the tombstone is set for replicas, the actual data deletion is done by the `Repear` daemon. The reaper physically deletes the tombstoned replicas from storage.
panta-123 marked this conversation as resolved.
Show resolved Hide resolved
The deletion service supports two different modes: greedy and non-greedy.

- Greedy

The repear daemon gets all the replicas with tombstone in the RSE and immediately delete all replicas.

- Non-greedy

The repear daemon first check if the free space is needed in the RSE. The needed free space is the difference of minimum free space (attrribute set for RSE) and actual free space in RSE. Deletion only occurs once free space is needed.
Deletions are processed by Least Recently Used (LRU) algorithm, thus oldest accessed (tombstoned) replicas are deleted first.


![Judge-cleaner chart](/img/Judge-cleaner.png)

![Undertaker chart](/img/undertaker.png)

![Repear chart](/img/repear.png)

Binary file added website/static/img/Judge-cleaner.png
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: all other filenames in this asset folder are lowercase; for consistency, this could be renamed to judge-cleaner.png

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added website/static/img/repear.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added website/static/img/undertaker.png
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Are rule locked?" should be "Is rule locked?"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also - "child(s)" should likely be "child(ren)"

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.