Feature Request: A more configurable snapshot retention schedule

[mlan]

Motivation

In some use-cases it might be desirable to keep snapshots for a longer time, even a few years. Users might ask to resurrect very old files, for example. Yet, keeping all snapshots for longer times might not be practical. A more configurable snapshot retention schedule might be useful in such scenarios.

Defining the snapshot retention schedule using dataset properties makes individual control easy. Perhaps you want to keep snapshots for users’ home directory around for longer than others for instance.

Synchronizing snapshots, including the ones kept for archiving purposes, is still desirable so that they are available should a fail-over event occur, for example.

Proposal

This proposal came up during some back and forth with Philip. A two level archive scheme was deemed sufficient in most cases. The archive scheme is activated by setting additional dataset properties:

zreparc1_timeperiod=timeperiod
zreparc2_timeperiod=timeperiod

When set, the expire process will check to see how old the most recent "zreparc1” snapshot is. If it is older than the timeperiod, it will rotate he oldest snapshot to be renamed zreparc1_something, instead of deletion. If it is already satisfied, but zreparc2_timeperiod is set, then same logic is used for the "zreparc2" snapshot.

The number of archive snapshots to keep is also defined by dataset properties:

zreparc1_savecount=savecount
zreparc2_savecount=savecount

Alternatives

There already exists third party software that provides zrep with a more configurable retention schedule. Namely zrep-expire. However, deploying it might have some undesired side effect. When the zrep synchronization, between master and backup server, fails for a while, due to network outage for example, resumption of snapshot synchronization might be impossible. This can happen since, during the network outage, zrep-expire running on the backup server will continue expire snapshots including the last successfully synced one.

[darkpixel]

One minor point:

This can happen since, during the network outage, zrep-expire running on the backup server will continue expire snapshots including the last successfully synced one.

That sounds like a zrep-expire problem. Maybe they can look at the zrep:savecount property (I think it defaults to 5) and not remove the last X number of snapshots.

[ppbrown]

just FYI:
I DEFINITELY want to code this feature.
However, it is significant code change, and thus requires significant mental peace time to make it.
So... it may be a while.

[ppbrown]

Note to self: this property should ideally be set LOCALLY, not inherited, on the SNAPSHOT, not the filesystem.
That way, it is possible to use multiple ZREPTAGs, and have individual retention periods for them.

Although if someone with an older system wanted it, I guess they could give up differentiation, and just set it at the top level.
So, zrep should just check for the property on the snapshot. It shouldnt enforce "local".

[JeffSaxeVA]

Just chiming in on this conversation: I believe I have a different method of achieving the simultaneous goals of "use ZREP to perform server-to-server replication" and "retain medium-term and long-term snapshots independently on each side on different schedules". I have decided to use ZREP to transmit the snapshots, but then combine it with another open-source package (zfs-auto-snapshot) to actually create and later destroy the retained snapshots. So ZREP itself is creating the "@zrep_000554" snapshots for the purpose of delivering the current contents to the far side, but as a side effect (because of the default "-i" in the zfs send command) it is delivering the "@zrep" and also the "@routinesnap_hourly-2019-12-05-1017" and "@routinesnap_daily-2019-12-05-0000" snaps. Then because my "/etc/cron.hourly/zfs-auto-snapshot" file has a different value for "--keep=nnn" on one side versus the other, I can have hourly snaps deleted after 96 hours on one side but after 168 hours on the other. This is not quite in production for my end users, but almost, and it's working very smoothly!

In order to make this all work, I needed to have the currently-Master side of the ZREP create zfs-auto snaps, but the Non-Master side specifically not create them (because "zfs send -i" will refuse to apply if other snapshots pre-exist on the receive side). To be clear, zfs-auto-snapshot is creating them on only the Master (sending) side, then ZREP is making them (also) appear on the Non-Master side as well, and then zfs-auto-snapshot is destroying the oldest ones on each side, independently. So I had to lightly modify the zrep script and then lightly modify the zfs-auto-snapshot script to get this to happen. I pestered Philip in a direct email yesterday about this, but I guess I should do the right thing and just post my suggestion as an Issue and Pull Request here on GitHub. But if this all works and satisfies your ( @mlan ) goal, then it might not be necessary for you ( @ppbrown ) to actually implement the fancy retention schedule in ZREP at all — since the zfs-auto-snapshot authors and the cron.d / cron.hourly / cron.daily implementors have already basically accomplished this collection of logic.

Concrete example: I have ZREP_SAVE_COUNT at its default of just 5, and it's running every minute, so (assuming no _unsent disruptions have happened recently) only the last 5 "@zrep" snaps exist on both sides. But going backward in time, we have "frequent" auto snaps happening every 10 minutes, saving 12 of them, and "hourly" auto snaps happening obviously every hour, saving 96 or 168 of them. If a network disruption occurs, then the sending side builds up a moderate number of "_unsent" snaps, and when the network is healed, then all those "_unsent" and all the intervening "frequent" and "hourly" snaps are all delivered at once. ZREP cleans up its own "@zrep" snaps back down to 5, and zfs-auto-snapshot cleans up its own via its own prefixes and labels, and they stay out of each other's way.

(in one datacenter)

root@datacenter-1:~# cat /etc/cron.d/zfs-auto-snapshot 
PATH="/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin"

*/10 * * * * root which zfs-auto-snapshot > /dev/null || exit 0 ; zfs-auto-snapshot --quiet --syslog --label=frequent --keep=12  --prefix=routinesnap   source5/source5

root@datacenter-1:~# cat /etc/cron.hourly/zfs-auto-snapshot 
#!/bin/sh

# Only call zfs-auto-snapshot if it's available
which zfs-auto-snapshot > /dev/null || exit 0

exec zfs-auto-snapshot --quiet --syslog --label=hourly --keep=96   --prefix=routinesnap  source5/source5

(the other datacenter, with a very similar cron job)

root@datacenter-2:~# cat /etc/cron.hourly/zfs-auto-snapshot 
#!/bin/sh

# Only call zfs-auto-snapshot if it's available
which zfs-auto-snapshot > /dev/null || exit 0

exec zfs-auto-snapshot --quiet --syslog --label=hourly --keep=168   --prefix=routinesnap  source5/source5

(result, the last several hours' worth)

source5/source5@routinesnap_hourly-2019-12-05-0717       0B      -       24K  -
source5/source5@routinesnap_hourly-2019-12-05-0817       0B      -       24K  -
source5/source5@routinesnap_hourly-2019-12-05-0917       0B      -       24K  -
source5/source5@routinesnap_hourly-2019-12-05-1017       0B      -       24K  -
source5/source5@routinesnap_hourly-2019-12-05-1117       0B      -       24K  -
source5/source5@routinesnap_hourly-2019-12-05-1217       0B      -       24K  -
source5/source5@routinesnap_hourly-2019-12-05-1317       0B      -       24K  -
source5/source5@routinesnap_frequent-2019-12-05-1410     0B      -       24K  -
source5/source5@routinesnap_hourly-2019-12-05-1417       0B      -       24K  -
source5/source5@routinesnap_frequent-2019-12-05-1420     0B      -       24K  -
source5/source5@routinesnap_frequent-2019-12-05-1430     0B      -       24K  -
source5/source5@routinesnap_frequent-2019-12-05-1440     0B      -       24K  -
source5/source5@routinesnap_frequent-2019-12-05-1450     0B      -       24K  -
source5/source5@routinesnap_frequent-2019-12-05-1500     0B      -       24K  -
source5/source5@routinesnap_frequent-2019-12-05-1510     0B      -       24K  -
source5/source5@routinesnap_hourly-2019-12-05-1517       0B      -       24K  -
source5/source5@routinesnap_frequent-2019-12-05-1520     0B      -       24K  -
source5/source5@routinesnap_frequent-2019-12-05-1530     0B      -       24K  -
source5/source5@routinesnap_frequent-2019-12-05-1540     0B      -       24K  -
source5/source5@routinesnap_frequent-2019-12-05-1550     0B      -       24K  -
source5/source5@zrep_007388                              0B      -       24K  -
source5/source5@zrep_007389                              0B      -       24K  -
source5/source5@zrep_00738a                              0B      -       24K  -
source5/source5@routinesnap_frequent-2019-12-05-1600     0B      -       24K  -
source5/source5@zrep_00738b                              0B      -       24K  -
source5/source5@zrep_00738c                              0B      -       24K  -

Please let me know your thoughts. Thanks!

[ppbrown]

hi Jeff, it seems to me that your setup is quite complicated, and if i implement this existing ticket correctly, you would then not have any need to run your additional layer on top of step any more.

[ppbrown]

note to self: implement expire differently on master vs slave. only rotate to arc copies on master. but expire arc copies on both sides, respecting local expire count values.

[JeffSaxeVA]

Oops, I'm sorry, I realize that in my above reply, I actually reversed the sense of lowercase "-i" versus uppercase "-I" on the zfs send. I think it was clear what I meant, that the ZREP script doing its regular zfs send's is also implicitly, as a side-effect, sending all the zfs-auto-snaps intermingled, so the whole stream of snaps all show up on both sides.

Thank you for your consideration of my approach, Philip. I agree it was kind of a lot of experimentation to get it all to work, but now that I've done it, in retrospect it doesn't seem complicated to me. :-) Searching for a tool that anyone else wrote to do regular, routine ZFS snapshots with retention, I found zfs-auto-snapshot and it seemed very obvious to use it; it seems mature. I was dismayed when I discovered that "zfs send -I" balked when the receiving side already had any snapshots present in a read-only filesystem. I changed it to "zfs send -i", but then it still balked when it was in the process of delivering a zrep snap and a zfs-auto-snap happened in the middle of it. This seems (to me) like a bug in ZFS, and I was very close to reporting it to the OpenZFS developers as an Issue. But when I realized that I could make a few tweaks and have just the receiving side not create the routine snaps, then it all made sense to me, and I think it's a cohesive whole system.

Regardless, I respect your opinions; I don't know how you will approach programming the multiple retention schedules for the "@zrep" snap names, but if you can get it working, it will be very useful to users of ZREP with no other tools. I will still cobble together my patch against 1.8.0 for setting zrep:master to "no" instead of to "nothing" (inherit), and I respectfully ask that you consider merging that in, even if it doesn't directly lead to any functional benefits for you. It doesn't hurt the logic, either, and it makes it possible for me (and others) to "sense" when one server is currently on the read-only receiving side, to program my other dependent logic. Although, now that I think about it one more time before commenting... it might be that I can just change my logic to sense whether the zfs "readonly" flag is set on, because that also happens when ZREP is failing over! So perhaps it's not important for you to merge that in. Silly me.

[JeffSaxeVA]

By the way, this change of strategy worked; I completely reverted my edits to the zrep script, using the "stock" 1.8.0 version of your work, Philip, and then I just made a couple-line edit in the package version of /sbin/zfs-auto-snapshot to sense for the ZFS filesystem being readonly.

for ii in $TARGETS
do
    checkreadonly=`zfs get -H -o value readonly $ii`
    if [ -n "$opt_do_snapshots" -a "$checkreadonly" != "on" ]
    then

This works great, and I don't even need to know, or pass to zfs-auto-snapshot, or hard-code, the specific name of the $ZREP:master tag — "readonly" is always the same property. So now whichever side is master creates the auto-snaps, whichever side is non-master has the filesystem readonly and so bypasses creating them, but then both sides end up having them present because "zfs send -I" makes it so, and then the "keep=" setting on each side destroys them. I'm very pleased and so are my colleagues.

@mlan, I think this discussion probably has enough info for you to implement what I did if you think it's useful to you. Remember if you "apt-get install zfs-auto-snapshot", their package immediately creates cron.d / cron.hourly / cron.daily / etc. files with the default string "//" on the end, so it will soon eagerly start making snapshots of all filesystems, and that may disrupt your existing ZREP. So be sure to edit all those files first thing and comment out the lines until you decide on the rest of your configuration. I'm happy to answer questions offline.