Velero backup takes nearly 15 hours for 300 namespaces and 3500 volumes.. How to reduce the backup time ?

[Naveen-Kamagani]

I am using velero to backup the namespaces and PV's without restic. I have nearly 300 namespaces and 3500 volumes. Using scheduler I trigger backups daily at 12 AM UTC time.

We are using backup storage location as AWS S3 bucket and created volume snapshot location to store volume snapshots and configured in scheduler. We have enabled include cluster resources also..
But the problem is daily the backup is taking nearly 14 to 15 hours of time.

Is there a way to reduce the backup time ?
Also velero backups are not incremental, because daily backups are taking more time compared to older backups ?

Could some one help us to reduce the backup time to 2 to 3 hours ?.

[sseago]

It sounds like you're using native snapshots (since you're configuring the VSL) -- native snapshots should be incremental. What VolumeSnapshotter are you using? The AWS/EBS? GCP? or Azure?

The velero backup itself (kubernetes data) is not incremental, though. But for backups with large amounts of volume data, volume backup is usually the biggest component of the time needed for backup. You should be able to determine from the backup logs where most of the time is being spent -- if it's in volume snapshot backup, then subsequent backups should be much faster. If kubernetes data is the bulk of your backup time, then incremental volume backup won't speed things up as much.

[Lyndon-Li]

I see you only mentioned volumes here, not sure how many resources are there, but let's talk about volumes only for now.

If you are using Velero's native snapshot backup method, it calls storage API to take snapshot without further actions. A snapshot creation process from the storage side takes time, i.e., several or tens of seconds.
However, Velero's native snapshots are done in sequence, so let say each volume creation takes 20s, you will get a total time ~= 20 hours.

On the other hand, from 1.12, Velero has changed the CSI snapshot backup method to make the most time consuming part as asynchronous, so you can have a try with the CSI snapshot backup.
Most popular storage providers have supported CSI snapshot, if yours doesn't, let us know the provider name and what snapshot plugin you are using.