Backup PV (Longhorn based) to external S3 Backup Location #7193

jakoberpf · 2023-12-10T13:06:58Z

jakoberpf
Dec 10, 2023

Hello people,

am trying to make a new backup process for my k3s cluster and longhorn as storage provider, but I fail to get the PVs backup up on my external S3 bucket. Scratching my head I am wondering what I am doing wrong or if its even supposed to work this way.

The current setup:

K3S v1.25.15-rc2+k3s1 Kubernetes Cluster
Longhorn v1.4.4 for replicated Storage with CSI support and external-snapshotter deployed
Velero Helm Chart v5.1.2

initContainers:
  - name: velero-plugin-for-aws
    image: velero/velero-plugin-for-aws:v1.8.0
    imagePullPolicy: IfNotPresent
    volumeMounts:
      - mountPath: /target
        name: plugins
  - name: velero-plugin-for-csi
    image: velero/velero-plugin-for-csi:v0.6.0
    imagePullPolicy: IfNotPresent
    volumeMounts:
      - mountPath: /target
        name: plugins      

configuration:
  volumeSnapshotLocation:
    - name: default
      provider: aws
      config:
        region: storj
  backupStorageLocation:
    - name: default
      provider: aws
      bucket: velero-backup
      config:
        region: storj
        s3ForcePathStyle: true
        s3Url: https://gateway.storjshare.io
        insecureSkipTLSVerify: true
  # Enable CSI snapshot support
  features: EnableCSI
snapshotsEnabled: true

PostgreSQL Cluster deployed in default namespace

The aspired process

What I assume or aspire is on command velero backup create default-zelos --include-namespaces default --wait or appropriate CRD/Schedule velero creates a backup of manifest and PVs in the S3 Backup Location.

What is actually happening

On the command velero backup create default-zelos --include-namespaces default --wait I see that a backup of the manifest is created on S3, but there is no backup of the PVs on S3. On Longhorn a Snapshot and Backup is created.

velero backup describe default-zelos --details

Name:         default-zelos
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.25.15-rc2+k3s1
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=25

Phase:  Completed


Namespaces:
  Included:  default
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Or label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto
Snapshot Move Data:          false
Data Mover:                  velero

TTL:  720h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  4h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2023-12-10 11:47:10 +0100 CET
Completed:  2023-12-10 11:47:53 +0100 CET

Expiration:  2024-01-09 11:47:10 +0100 CET

Total items to be backed up:  46
Items backed up:              46

Backup Item Operations:
  Operation for volumesnapshots.snapshot.storage.k8s.io default/velero-data-postgresql-ha-postgresql-0-hkmbn:
    Backup Item Action Plugin:  velero.io/csi-volumesnapshot-backupper
    Operation ID:               default/velero-data-postgresql-ha-postgresql-0-hkmbn/2023-12-10T10:47:19Z
    Items to Update:
              volumesnapshots.snapshot.storage.k8s.io default/velero-data-postgresql-ha-postgresql-0-hkmbn
    Phase:    Completed
    Created:  2023-12-10 11:47:19 +0100 CET
    Started:  2023-12-10 11:47:19 +0100 CET
  Operation for volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-cd8bd97d-9c9c-4a6d-8e73-1e31b5cf8cfa:
    Backup Item Action Plugin:  velero.io/csi-volumesnapshotcontent-backupper
    Operation ID:               snapcontent-cd8bd97d-9c9c-4a6d-8e73-1e31b5cf8cfa/2023-12-10T10:47:19Z
    Items to Update:
              volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-cd8bd97d-9c9c-4a6d-8e73-1e31b5cf8cfa
    Phase:    Completed
    Created:  2023-12-10 11:47:19 +0100 CET
    Started:  2023-12-10 11:47:19 +0100 CET
  Operation for volumesnapshots.snapshot.storage.k8s.io default/velero-data-postgresql-ha-postgresql-1-hcsrx:
    Backup Item Action Plugin:  velero.io/csi-volumesnapshot-backupper
    Operation ID:               default/velero-data-postgresql-ha-postgresql-1-hcsrx/2023-12-10T10:47:29Z
    Items to Update:
              volumesnapshots.snapshot.storage.k8s.io default/velero-data-postgresql-ha-postgresql-1-hcsrx
    Phase:    Completed
    Created:  2023-12-10 11:47:29 +0100 CET
    Started:  2023-12-10 11:47:29 +0100 CET
  Operation for volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-f9a711a7-8827-44a1-b653-5c2ea3d2c519:
    Backup Item Action Plugin:  velero.io/csi-volumesnapshotcontent-backupper
    Operation ID:               snapcontent-f9a711a7-8827-44a1-b653-5c2ea3d2c519/2023-12-10T10:47:29Z
    Items to Update:
              volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-f9a711a7-8827-44a1-b653-5c2ea3d2c519
    Phase:    Completed
    Created:  2023-12-10 11:47:29 +0100 CET
    Started:  2023-12-10 11:47:29 +0100 CET
  Operation for volumesnapshots.snapshot.storage.k8s.io default/velero-data-postgresql-ha-postgresql-2-hqxdv:
    Backup Item Action Plugin:  velero.io/csi-volumesnapshot-backupper
    Operation ID:               default/velero-data-postgresql-ha-postgresql-2-hqxdv/2023-12-10T10:47:39Z
    Items to Update:
              volumesnapshots.snapshot.storage.k8s.io default/velero-data-postgresql-ha-postgresql-2-hqxdv
    Phase:    Completed
    Created:  2023-12-10 11:47:39 +0100 CET
    Started:  2023-12-10 11:47:39 +0100 CET
  Operation for volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-6b97b31f-20e7-43d1-96a0-dca1ec0f6c95:
    Backup Item Action Plugin:  velero.io/csi-volumesnapshotcontent-backupper
    Operation ID:               snapcontent-6b97b31f-20e7-43d1-96a0-dca1ec0f6c95/2023-12-10T10:47:39Z
    Items to Update:
              volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-6b97b31f-20e7-43d1-96a0-dca1ec0f6c95
    Phase:    Completed
    Created:  2023-12-10 11:47:39 +0100 CET
    Started:  2023-12-10 11:47:39 +0100 CET
Resource List:
  apps/v1/ControllerRevision:
    - default/postgresql-ha-postgresql-5b8bd596ff
  apps/v1/Deployment:
    - default/postgresql-ha-pgpool
  apps/v1/ReplicaSet:
    - default/postgresql-ha-pgpool-8694f4686c
  apps/v1/StatefulSet:
    - default/postgresql-ha-postgresql
  discovery.k8s.io/v1/EndpointSlice:
    - default/csi-snapshotter-jjmlb
    - default/kubernetes
    - default/postgresql-ha-pgpool-vxscc
    - default/postgresql-ha-postgresql-headless-97xwr
    - default/postgresql-ha-postgresql-pjmcd
  snapshot.storage.k8s.io/v1/VolumeSnapshot:
    - default/velero-data-postgresql-ha-postgresql-0-hkmbn
    - default/velero-data-postgresql-ha-postgresql-1-hcsrx
    - default/velero-data-postgresql-ha-postgresql-2-hqxdv
  snapshot.storage.k8s.io/v1/VolumeSnapshotClass:
    - longhorn-velero
  snapshot.storage.k8s.io/v1/VolumeSnapshotContent:
    - snapcontent-6b97b31f-20e7-43d1-96a0-dca1ec0f6c95
    - snapcontent-cd8bd97d-9c9c-4a6d-8e73-1e31b5cf8cfa
    - snapcontent-f9a711a7-8827-44a1-b653-5c2ea3d2c519
  v1/ConfigMap:
    - default/istio-ca-root-cert
    - default/kube-root-ca.crt
    - default/postgresql-ha-postgresql-hooks-scripts
  v1/Endpoints:
    - default/csi-snapshotter
    - default/kubernetes
    - default/postgresql-ha-pgpool
    - default/postgresql-ha-postgresql
    - default/postgresql-ha-postgresql-headless
  v1/Event:
    - default/vault-backend.179b8b3c15839849
  v1/Namespace:
    - default
  v1/PersistentVolume:
    - pvc-0e720644-fdc9-49aa-91a8-29b6575bef59
    - pvc-ce334095-d28b-4133-b8f7-7a61a6642d39
    - pvc-ebd5492c-f636-4296-aa49-ce9288101eec
  v1/PersistentVolumeClaim:
    - default/data-postgresql-ha-postgresql-0
    - default/data-postgresql-ha-postgresql-1
    - default/data-postgresql-ha-postgresql-2
  v1/Pod:
    - default/postgresql-ha-pgpool-8694f4686c-qxkdw
    - default/postgresql-ha-postgresql-0
    - default/postgresql-ha-postgresql-1
    - default/postgresql-ha-postgresql-2
  v1/Secret:
    - default/postgresql-ha-pgpool
    - default/postgresql-ha-postgresql
    - default/sh.helm.release.v1.postgresql-ha.v1
  v1/Service:
    - default/csi-snapshotter
    - default/kubernetes
    - default/postgresql-ha-pgpool
    - default/postgresql-ha-postgresql
    - default/postgresql-ha-postgresql-headless
  v1/ServiceAccount:
    - default/csi-snapshotter
    - default/default

Velero-Native Snapshots: <none included>

So my question is, how do I get velero to also backup the PVs into the S3 Backup Location? I would like to deprecate my use of the Longhorn Backup Features as looks me into Longhorn.

draghuram · 2023-12-12T19:00:34Z

draghuram
Dec 12, 2023

You need to do either file system backup or snapshot data mover backups. See details here:

https://velero.io/docs/v1.12/file-system-backup/
https://velero.io/docs/v1.12/csi-snapshot-data-movement/

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Backup PV (Longhorn based) to external S3 Backup Location #7193

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment

{{title}}

Select a reply

Backup PV (Longhorn based) to external S3 Backup Location #7193

jakoberpf Dec 10, 2023

The current setup:

The aspired process

What is actually happening

Replies: 1 comment

draghuram Dec 12, 2023

jakoberpf
Dec 10, 2023

draghuram
Dec 12, 2023