Restore/Rollback Snapshot from Remote Server

[norsemangrey]

This question is probably not directly related to zfs_autobackup.

I'm trying to use zfs_autobackup to pull snapshots taken on a main server to a backup server. This seems to be working, however I'm not quite sure about how the restore/rollback procedure would work. The plan is to keep more snapshots on the backup server than the main server. As I see it there are two different scenarios:

1. I need to roll back to a snapshot that no longer exists on the main server (only on the backup server).

2. I need to restore the entire dataset on the main server (because it is deleted or corrupt, etc.)

I have understood that I need to use the send/receive commands, but I am not sure what should/is happening. I'm apparently missing some understanding about rollback from a snapshot. Do I transfer transfer the desired snapshot and then do a rollback on the main server afterwards or does this happen as part of the receive command?

I'm trying with something like this:

sudo zfs send backup-pool-1/main-server/fast-pool-1/testset@2023-01-01T00:00:01 | ssh [email protected] zfs receive fast-pool-1/testset

Which results in the following error message. However I don't want to overwrite the entire dataset, I just want to do a rollback as I would from a local snapshot. Can a snapshot even overwrite the entire dataset? Would I not need to transfer the whole dataset from the backup server in that case?

cannot receive new filesystem stream: destination exists must specify -f to overwrite it

I have also tried something like this (adding the -i option):

sudo zfs send -i backup-pool-1/main-server/fast-pool-1/testset@2023-01-01T00:00:01 backup-pool-1/main-server/fast-pool-1/testset | ssh [email protected] zfs receive fast-pool-1/testset

But I get a permission denied. I have added rollback and receive permissions on the dataset on the main server. Does it need additional permissions or do I need to SSH as root (something I'm trying to avoid)?

cannot receive incremental stream: permission denied

[Scrin]

You cannot insert "old" snapshots into a dataset, so when you want to rollback to a snapshot that no longer exists on the host (but does on the backup), you have to "overwrite" or "recreate" the entire dataset on the host. For this you have two options:

• forcefully overwrite the entire dataset with the -f (which is what the error tells you), giving you just that snapshot and nothing else (this is fastest, and indeed transfers the snapshot in full, and only that snapshot)
• "recreate" the history by sending the oldest snapshot you want to have available first, and then all subsequent snapshots you want in chronological order from there (using the incremental flag to transfer only the deltas). This obviously requires using the same -f flag (or destroying the original dataset first)

In case you need or want to have the "existing" dataset available during this, you can either rename the existing dataset to something else, or receive from the backup into a different dataset and then after all transfers are complete, swap these by renaming them. This also eliminates the need of using the -f since you're not overwriting anything. (you might need to adjust mountpoints or other properties during this, depending on your setup/configuration)

It's also worth keeping in mind that rolling back to a snapshot older than the latest snapshot causes "branching" in the "history", which causes issues in future backups if no other actions are made. In such case if you want to preserve backups that were taken after the restored snapshot but before the rollback, one solution would be creating a zfs clone of that in the backups, and then destroying the "conflicting" snapshots from the original backup dataset to allow future backups to arrive

[psy0rz]

Yes, in short: You need a common snapshot as starting point if you want to restore a snapshot from the backup server to the server.

So you could:

• first roll back the server to the common snapshot (zfs rollback)
• then do a incremental zfs send of the data between common-snapshot and snapshot you want: zfs send -i "dataset@commonsnapshot" "dataset@desiredsnapshot"

if there is no common snapshot you need to destroy the dataset on the server and do a full send: zfs send dataset@desiredsnapshot

[norsemangrey]

@Scrin @psy0rz thank you for the feedback and explanations. I have tried experimenting a bit further based on your responses.

• Starting from scratch I created a dataset fast-pool-1/testset on the main server and created a file with some text.
• I made a snapshot, fast-pool-1/testset@2023-01-01T00:00:01, of this dataset on the main server and did a replication of that dataset from the backup server to backup-pool-1/main-server/fast-pool-1.
• I'm now assuming that @2023-01-01T00:00:01 is the common snapshot existing on both the main server and the backup server.
• I then make some changes to the text file and take a new snapshot of the dataset on the main server, fast-pool-1/testset@2023-01-01T00:00:02.
• I pull this snapshot to the backup server and delete the snapshot from the main server so now only the backup server has this snapshot, but both still have the first/common one.
• I do another change to the text file. Now I want to roll back to the snapshot before this change, which is only on the backup server.
• I roll the dataset on the main server back to the common snapshot zfs rollback fast-pool-1/testset@2023-01-01T00:00:01.
• Then I try to send the second snapshot back from the backup server using the following command: sudo zfs send -i backup-pool-1/main-server/fast-pool-1/testset@2023-01-01T00:00:01 backup-pool-1/main-server/fast-pool-1/testset@2023-01-01T00:00:02 | ssh [email protected] zfs receive fastpool-1/testset.

However, I get the error message: cannot receive incremental stream: destination fast-pool-1/testset has been modified.

Am I still missing some steps here?

[psy0rz]

try to add -F to the zfs recieve

[Scrin]

Indeed that error (in this case) means that the dataset has been modified after your rollback, which can happen pretty easily especially if the dataset is mounted and anything happens to access it and you have atime enabled (typically it is by default). As mentioned, the -F flag to zfs receive forces a rollback before receiving the stream, discarding any such (unintentional in this case) modifications to the dataset. Using -F can lead to data loss if not used carefully, but in this particular case it's fine.