Rustic for file integrity checking / bitrot protection with cold storage backups?

[masda70]

I plan to install Rustic to backup a system that has no underlying file system integrity checking mechanism, and no possibility to migrate to one in the near future (e.g., ZFS).

The plan is backup to cold storage, for cost-efficiency reasons (multi TB data). I understand that rustic is the only tool with efficient deduplicated backups that supports cold storage.

I have also read that rustic diff can check the local filesystem against a snapshot. How feasible and efficient would it be to to periodically run rustic diff (but also rustic check and other backup checking mechanisms) in order to protect against data degradation (e.g., bitrot etc) ? Any recommandations or things I should know?

[aawsome]

Thanks @masda70 for opening this discussion!

A general disclaimer: Note that rustic is in a beta phase, so be sure to judge the risk yourself. If you have ultra-critical data, you shouldn't rely on rustic as single backup solution (but, of course, in that case you should not rely on any single backup solution). In general we do encourage people to double-check their rustic repository using restic check, but this obviously doesn't work well with cold storage which is not supported by restic...

About rustic diff: running that only needs to read metadata from the repository which is always stored in hot repository part and usually also cached. It basically just uses computes SHA256 hashes of file/file parts and compares them with the hashes saved in the repository metadata. So, this is a good way to detect bitrot or undesired changes of your local filesystem.

If you need to check your backup storage, you have to run rustic check. But note that this only checks metadata until you specify --read-data. If you use that option on the cold storage, it will read all data which means that all data needs to be warmed-up and read. This can be a problem with cold storage.

An alternative check which maybe can be performed directly on the storage is checking all repository files against their SHA256 hash - the filename should always be identical to its checksum.

Moreover, in near future with #1230 there will be the possibility to only check parts of the data, e.g. only a random subset. This may be a good compromise between security and handling costs for cold storage.

Also, remember to always (and regularly) do restore tests to ensure that data really is restorable. Note that rustic restore needs to warm up only repository files needed for the requested restore.

[aawsome]

Assuming a periodic set of backup tasks and checks, at what stage would rustic diff against the local filesystem make sense for my objective? I'm expecting that running such diff would be able to detect bitrot in files that had already been backed up by a previous job if their modification dates were not changed. However, it wouldn't be able to differentiate files which could have disappeared due to file system corruption vs normal deletion of files.

Is my understanding correct? Are there any built-in mechanisms in rustic to differentiate between normal deletions and potential filesystem corruption? Such differentiability could be made possible if one had a log of normal file system deletions maintained using a separate audit tool. Otherwise, is my understanding correct that intelligent log analysis (e.g., using folder-based rules) would be needed, with difficult cases ultimately forwarded to a human? Are there any built-in mechanisms in rustic to differentiate between normal deletions and potential filesystem corruption?

Actually this is a good point. For complete deletions, rustic has no way to figure out if that was by user's request or if something went badly wrong. However, rustic could use the metadata information to check only files which have not changed w.r.t. the snapshot. I'll try to make a draft PR for this...

[aawsome]

I'm not sure I fully understand this part. Am I correct in understanding that rustic check would primarily verify the existence of files in cold storage rather than detecting bitrot, since cold storage providers (with S3-like interfaces) should not typically update checksums without user intervention? If we only check metadata, I don't expect cold storage providers to update checksums in case of bitrot on their side, or be much more durable that my local storage, so it would be mostly file existence, I think.

rustic will be able to detect bitrot, but it needs to download the data from the cold storage to do so. But rustic has no mechanism to repair a bitrot

[aawsome]

If a part goes missing from cold storage, is it correct to consider that only the snapshots and files pointing to that part would be unrecoverable?

If you are talking about a data part, this is correct. If your keyfile is damaged for some reason (in hot and cold part and you don't have multiple keyfiles), you loose access to your whole repository...

[aawsome]

see #1250

[masda70]

see #1250

Thanks, that's awesome!

[aawsome]

Ah, and a general remark: Depending on your cold-storage solution you may want you storage provider to be able and guarantee integrity of you repository. In that case rustic check can be used to double-check (better safe than sorry) .

If you have to ensure integrity yourself, rustic only allows to check but does not offer error-correction codes or something like this. So, in this case you'll need an extra solution for that.

[simonsan]

@masda70 Would you be able to write a tiny article for our docs for this use case/scenario? I added rustic-rs/docs#91 as a tracking issue for it, where we can discuss this.