"zfs send -t" fails with "snapshot not found"

[asomers]

A very frequent problem I have is this:

1. zfs send -I pool/fs@most_recent_snap | something
2. Transfer gets interrupted for some reason
3. zfs send -t token | something fails with snapshot pool/fs@some_intermediate_snap not found

The problem is that something deleted some_intermediate_snap on the source side. It may have happened during the original transfer, but that's supposed to be ok because zfs send puts a hold on all snapshots. But once the transfer got interrupted, the snapshots would get automatically deleted. Or, it may have happened after the original transfer got interrupted. Either way, the receive_resume_token is now useless and I need to start from scratch.

What would be the best solution to this problem? My only ideas are:

1. Abandon -I. Instead, use -i, placing a user hold on each incremental snapshot. But this would make my wrapper script much more complicated.
2. Add an option to zfs send that would cause it to leave its holds in place after a transfer failure, rather than cleaning them up.

Would option 2 be acceptable? Are there any other possibilities?

[rincebrain]

zfs send -I foo@snap2 foo@snap5, if interrupted in the middle of snap4, you could then delete the incomplete state on the receiver (which would still have up to snap3 complete), and then rollback and resume from whichever snapshot you have in common, with -I or -i as you like.

If you lack a snapshot in common between src and dst, you get to start over.

Personally, I'd probably add bookmarks for the intervening snapshots on the source so that if this case arose, I could just do zfs send -i [bookmark of most recent snapshot the destination received completely] [first snapshot on the source after that] && zfs send -I [first snapshot after that] [latest snapshot];.

[asomers]

Yeah, a rollback is my only recourse. It's a PITA though, because I haven't figured out how to automate it. Worse, this probably most commonly occurs when the oldest snapshot gets deleted, and I'm doing the initial transfer. When that happens, rollback isn't an option and I have to start from scratch. I've thought about using bookmarks, but that only half solves the problem. A bookmark can be used as the incremental source for a send, but not the destination. So far I don't see a way to fully solve the problem without placing holds on the destination snapshots.

[rincebrain]

Yeah, if you're worried about them being deleted before an initial snapshot arrives, then holds are your only option; you can also use send -h to make holds get passed to the destination as well if that helps any.