Support for zfs clones

[pol66]

Hi,

I am using zfs a lot and one of the typical use-case is to server as the backend for lxc containers.
It is very practical to have a container as a base template and create new one as a clone of this base. E.g. you would have something like:

tank/lxc/debian-base
tank/lxc/nginx

where nginx would be debian with nginx installed.
The dataset tank/lxc/nginx is clone of tank/lxc/debian-base.

Now I want to backup all my lxc containers. The problem is, that both datasets will be send and received full. So it will take much more space on backup server than on the source!

For now, I have to count with this situation, so I make the full backup of the base dataset and then I make "replication" send of the cloned dataset. It will create the cloned dataset as clone also on the backup server. Afterwards I can use zfs-autobackup as usual.

Now I was thinking, zfs-autobackup could recognise such a situation and do the replication itself.
It would be probably necessary to check "origin" property. If "origin" property would be set, it should use "zfs send -R ..." for replication instead of full transfer.

What do you think about it?

[psy0rz]

This is a feature we should definitely have. I have to think about how to implement it in a clean way. Only a zfs send -R is not enough: We also need to make sure the origin is send first and we need some extra parameters to zfs send/recv i think?

[pol66]

You are definitely right about checking, if the origin is allready present. The question is what to do, if the origin is not present? Probably to synchronize the origin first, what should be easy if both datasets (origin and clone) have the same zfs_autobackup property. But what should happen, if the origin has different zfs_autobackup property or none?

Concerning the parameters to zfs send/recv I allways used only "zfs send -R ..." and "zfs recv -F". Not sure, if it covers all possible situations...

[psy0rz]

In our case we have to specify the origin on zfs recv with -o origin=... i think? We cant use the -R option for various reasons.

[pol66]

You are probably right. Using zfs recv with -o origin=... should be enough.

[wxcafe]

👍 that would be great

[psy0rz]

This also is neccesary to correctly support ZFS rootfilesystems on Ubuntu 20 via zsys (https://github.com/ubuntu/zsys)

zsys heavily relies on clones, so we should support for clones and promoting/demoting will be added in 3.1.

[crispyduck00]

Hi! First thanks for this great tool!

I am just switching from sanoid/syncoid to zfs-autobackup.

So far perfect, But now I realized exactly this behavior. I am doing nearly the same as pol66, but in my case with dozens of Windows VMs, compared to a small lxc we have here several GB we waste for each clone on the backup destination.

Any idea when this will be implemented?

I also don't only want to use it just as backup, in my case it is for a hot standby NFS server. I am afraid I will loose my clones this way completely.

br,
cd

[psy0rz]

its next on the list of big changes, since i need it myself as well. (for a docker swarm)

[crispyduck00]

its next on the list of big changes, since i need it myself as well. (for a docker swarm)

Great, thanks! any idea when? As it is on the 3.3 milestone I don't expect it will be in near future, or?

I am actually testing many other tools, but seems zfs-autobackup is so far the best one. At the moment I only miss this feature.

br
cd

[psy0rz]

in de next couple of months

[crispyduck00]

Great!

[crispyduck00]

Hi, one more question? pol66 means in his workaround doing send/recv via zfs, not autobackup?

I am testing and playing around at the moment with so many different zfs replication tools.

None is really fulfilling all my needs, several are doing the replication also of clones right, but will not be able to delete missing on the target, or they will also delete on the target by using -R and -F, but I am afraid that this could lead to data loss by mistake ore failure.
zfs-autobackup has the destroy-missing, for this and in combination with holds this is for sure much saver.

But I really need this clone feature, so I am thinking if I should use one of the other tools and add some additional snapshots with holds to prevent data loss, or if I should already use zfs-autobackup, but then I need to find a way to handle a huge number and often changing clones till this is supported.

Br
cd

[psy0rz]

Since this seems to be the one missing key-feature, i'll start with it in the next couple of days. Expect a test versions with preliminary support soon. There are a lot of edge cases where supporting clones will need some attention, so i will create a bunch of test scenarios first.

[crispyduck00]

Great. I will be there for testing.

[crispyduck00]

Hi. Would really like to switch to zfs_autobackup, but seems clones are still not supported, or Iam wrong?
I really need the clones in my case.
Br, cd

[tuffnatty]

@crispyduck00 There is limited support for clones, for details read the discussion in #82 .

[crispyduck00]

Thanks. Will check it and see if it fits my needs.

[knuuuut]

It looks like this hasn't been implemented in 3.2.2. Any news on it?