Could rustic copy prioritize newest snapshots?

[mprasil]

Right now the copy operation first copies all data and then creates snapshots as last step. For huge repositories this could be days of copying before there are any usable snapshots.

One can somewhat work around that by copying few useful snapshots first and then running broad copy that includes all of the snapshots, but I wonder if something like that couldn't be the default?

It could go something like this:

1. start from the newest snapshot and copy all objects for that
2. store snapshot
3. go back to step 1 until all snapshots are copied

The idea is that the destination would contain some usable snapshots ASAP rather than at the very end of the process it would also presumably copy data that's more likely to be useful first. Seeing that I'm able to replicate this process by copying individual snapshots by running rustic copy for each snapshot makes me thing it's possible, is there anything preventing rustic from doing exactly the same?

[aawsome]

Thanks for the question @mprasil.
The answer is: It is not that easy. The reason is, snapshots reference blobs. Those are accessed using the index and finally the pack files they are stored in. So we have a dependency chain:

snapshots -> index -> packs

In order to not save invalid snapshots, rustic must be sure that all dependent pack files (i.e. all packs which contain dependent blobs) and all index files which reference the used blobs are really present in the repository. Therefore we have to wait until they are really saved.

If you just copy one snapshot, rustic will therefore copy all blobs and create index entries, but before saving the snapshot, it will finalize the last already started pack and the last already started index file and save it to the storage before finally saving the snapshot to the storage.

So, in your example copying snapshots one-by-one and always finalizing pack and index files may result in lot of small pack and/or index files compared to the current processing.
(And yes, in theory rustic could still try to prioritize blobs and to detect when all needed blobs and index entries are really written. Then the snapshot could be safely written, too. But this would require a big redesing in how rustic currently works and will take some time to get. It would e.g. also allow to delete pack files earlier during a prune run).

Another issue is the priority of copying. You prefer to have newest snapshots first, but maybe someone else would want to have oldest snapshots first...

However, we could make it optional to process snapshots one-by-one, but then users should be able to also choose their own priority (e.g. old vs new first).