Add `Bucket Lock` immutable object support for Google Cloud Storage. #776

renormalize · 2024-09-10T04:45:39Z

What this PR does / why we need it:

Enables etcd-backup-restore to handle immutable objects as provided by bucket lock in Google Cloud Storage.

A new method IsDeletable() is added to the Snapshot struct. This method returns a bool which informs the caller whether the Snapshot instance can be deleted from the object store. It internally checks the immutability expiration time of the object and returns true if it has expired.
Custom metadata can be added to objects in Google Cloud Storage. In case any particular object (snapshot) is to be excluded during restoration, custom metadata can be added to that object with the key x-etcd-snapshot-exclude and value true.
The List() method of SnapStore interface is now enhanced. It takes a bool.
- false: List() will only return the objects in the object store that do not have the custom metadata x-etcd-snapshot-exclude added to it.
- true: List() will return all objects present in the object store. This variant is used in garbage collection.

Which issue(s) this PR fixes:
Fixes # #777 only for GCS

Special notes for your reviewer:

This PR only adds support for GCS. Identical PRs adding support for other providers will be raised separately.

Release note:

etcd-backup-restore now supports immutable objects for storage provider: Google Cloud Storage, provided by the [Bucket Lock](https://cloud.google.com/storage/docs/bucket-lock) feature.

Snapshots garbage collection performed by etcd-backup-restore (if enabled) for GCS is performed only when the objects' immutability period expires.

Support for Bucket lock (GCS) in etcd-backup-restore is backward compatible. For more info please refer to this doc: https://github.com/gardener/etcd-backup-restore/blob/master/docs/usage/immutable_snapshots.md

- Added `RetentionExpiry` field to Snapshot struct to store retention expiry time. - Added `IsDeletable` method to `Snapshot` struct to determine if a snapshot is deletable based on `RetentionExpiry`. - Introduced `ExcludeSnapshotMetadataKey` constant for marking snapshots to be ignored. - Modified garbage collection logic to skip immutable snapshots. - Updated GCS snapstore list function to exclude objects with `x-etcd-snapshot-exclude` metadata.

…a `bool` argument.

…napshots.

…e tagged snapshots' behavior.

pkg/types/snapstore.go

* This rename makes the distinction clearer between retention expiry which is used for delta snapshots typically, and the immutability expiry time of a snapshot.

renormalize · 2024-09-23T06:03:13Z

@ishan16696 @seshachalam-yv I've added docs/usage/immutable_snapshots.md. Please take a look.

ishan16696

Thanks @renormalize for making changes and adding a doc. I have added few nits:

docs/usage/immutable_snapshots.md

anveshreddy18

Overall looks good to me, just couple of nits.

pkg/snapstore/snapshot_test.go

pkg/snapstore/gcs_snapstore.go

…pshots.md`

anveshreddy18

Thanks for the PR.
LGTM

ishan16696 · 2024-09-25T06:39:02Z

@renormalize can you please update the release notes as requested here: #776 (comment)

etcd-backup-restore now supports immutable objects for storage provider: Google Cloud Storage, provided by the [Bucket Lock](https://cloud.google.com/storage/docs/bucket-lock) feature.

Snapshots garbage collection performed by etcd-backup-restore(if enabled) is performed only when the objects immutability period expires.

Support for Bucket lock in etcd-backup-restore is backward compatible. For more info please refer to this doc: https://github.com/gardener/etcd-backup-restore/blob/master/docs/usage/immutable_snapshots.md

ishan16696

Thanks @renormalize for PR and addressing the review comments.
LGMT!!

seshachalam-yv and others added 4 commits September 10, 2024 10:09

Call the correct method for GC.

46c9d69

Check immutability for chunk deletion in GarbageCollectChunks().

8eb63e1

Enable List calls to return untagged and tagged snapshots based on …

8d7acb3

…a `bool` argument.

renormalize added platform/gcp Google cloud platform/infrastructure area/robustness Robustness, reliability, resilience related area/backup Backup related area/storage Storage related labels Sep 10, 2024

renormalize added this to the 2024-Q3 milestone Sep 10, 2024

renormalize self-assigned this Sep 10, 2024

renormalize requested a review from a team as a code owner September 10, 2024 04:45

gardener-robot added the needs/review Needs review label Sep 10, 2024

gardener-robot-ci-1 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Sep 10, 2024

gardener-robot added the size/m Size of pull request is medium (see gardener-robot robot/bots/size.py) label Sep 10, 2024

gardener-robot-ci-2 added needs/ok-to-test Needs approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Sep 10, 2024

renormalize added 2 commits September 10, 2024 12:34

Added unit tests for IsDeletable().

7c097f3

Adapt unit tests for List() to handle listing tagged and untagged s…

d7be197

…napshots.

gardener-robot-ci-2 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Sep 10, 2024

gardener-robot-ci-1 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Sep 10, 2024

Improve comments as suggested by @seshachalam-yv

3545d03

gardener-robot-ci-1 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Sep 10, 2024

Defined an interface which mock snapstores should implement to emulat…

f40496b

…e tagged snapshots' behavior.

gardener-robot-ci-3 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Sep 11, 2024

ishan16696 self-assigned this Sep 12, 2024

gardener-robot-ci-3 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Sep 12, 2024

gardener-robot-ci-1 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Sep 12, 2024

renormalize force-pushed the immutable-gcs branch from bceba62 to a592264 Compare September 12, 2024 09:31

gardener-robot added size/l Size of pull request is large (see gardener-robot robot/bots/size.py) needs/second-opinion Needs second review by someone else and removed size/m Size of pull request is medium (see gardener-robot robot/bots/size.py) labels Sep 18, 2024

gardener-robot-ci-2 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Sep 18, 2024

ishan16696 reviewed Sep 19, 2024

View reviewed changes

pkg/types/snapstore.go Outdated Show resolved Hide resolved

pkg/types/snapstore.go Outdated Show resolved Hide resolved

Rename RetentionExpiry to ImmutabilityExpiryTime

a6680d4

* This rename makes the distinction clearer between retention expiry which is used for delta snapshots typically, and the immutability expiry time of a snapshot.

gardener-robot-ci-3 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Sep 20, 2024

gardener-robot-ci-1 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Sep 20, 2024

Add docs/usage/immutable_snapshots.md.

f227198

gardener-robot-ci-3 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Sep 23, 2024

gardener-robot-ci-1 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Sep 23, 2024

ishan16696 assigned anveshreddy18 Sep 23, 2024

ishan16696 requested changes Sep 23, 2024

View reviewed changes

docs/usage/immutable_snapshots.md Outdated Show resolved Hide resolved

docs/usage/immutable_snapshots.md Outdated Show resolved Hide resolved

docs/usage/immutable_snapshots.md Outdated Show resolved Hide resolved

anveshreddy18 reviewed Sep 24, 2024

View reviewed changes

pkg/snapstore/snapshot_test.go Outdated Show resolved Hide resolved

pkg/snapstore/gcs_snapstore.go Outdated Show resolved Hide resolved

renormalize added 2 commits September 24, 2024 12:49

Address review comments from @ishan16696 on `docs/usage/immutable_sna…

c0ea11f

…pshots.md`

Address review comments from @anveshreddy18

040b232

gardener-robot-ci-1 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Sep 24, 2024

anveshreddy18 approved these changes Sep 24, 2024

View reviewed changes

ishan16696 added the merge/squash Should be merged via 'Squash and merge' label Sep 25, 2024

ishan16696 approved these changes Sep 25, 2024

View reviewed changes

renormalize mentioned this pull request Sep 27, 2024

DEP-06: Immutable ETCD Backups seshachalam-yv/etcd-druid#1

Closed

renormalize merged commit a754925 into gardener:master Sep 30, 2024
9 checks passed

gardener-robot added the status/closed Issue is closed (either delivered or triaged) label Sep 30, 2024

renormalize mentioned this pull request Sep 30, 2024

Add Container level WORM immutable object support for Azure Blob Storage. #781

Open

shreyas-s-rao modified the milestones: 2024-Q3, v0.31.0 Oct 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `Bucket Lock` immutable object support for Google Cloud Storage. #776

Add `Bucket Lock` immutable object support for Google Cloud Storage. #776

renormalize commented Sep 10, 2024 •

edited

Loading

renormalize commented Sep 23, 2024

ishan16696 left a comment

anveshreddy18 left a comment •

edited

Loading

anveshreddy18 left a comment

ishan16696 commented Sep 25, 2024 •

edited

Loading

ishan16696 left a comment

Add Bucket Lock immutable object support for Google Cloud Storage. #776

Add Bucket Lock immutable object support for Google Cloud Storage. #776

Conversation

renormalize commented Sep 10, 2024 • edited Loading

renormalize commented Sep 23, 2024

ishan16696 left a comment

Choose a reason for hiding this comment

anveshreddy18 left a comment • edited Loading

Choose a reason for hiding this comment

anveshreddy18 left a comment

Choose a reason for hiding this comment

ishan16696 commented Sep 25, 2024 • edited Loading

ishan16696 left a comment

Choose a reason for hiding this comment

Add `Bucket Lock` immutable object support for Google Cloud Storage. #776

Add `Bucket Lock` immutable object support for Google Cloud Storage. #776

renormalize commented Sep 10, 2024 •

edited

Loading

anveshreddy18 left a comment •

edited

Loading

ishan16696 commented Sep 25, 2024 •

edited

Loading