Dedup log status?

[DemiMarie]

What is the status of dedup logs? This forum post suggests that Intel’s all-Optane SSDs (now discontinued outside of the enterprise market) are the only ones capable of handling the dedup table workload. Has deduplication logging been implemented?

[rincebrain]

The DDT log implementation, as far as I know, never went beyond the prototype stage in a branch by mahrens.

Dedup tables in general can be stored on normal SSDs perfectly well now (as that post indicates, it was written mostly about before special type vdevs, which can offload metadata permanently onto flash, were available). Optane is fantastic for performance in a number of use cases, but is hardly a requirement. I'm sure it'll perform better on Optane drives in most cases, just like it will perform better on normal SSDs than it would if I stored the dedup tables on spinning drives that think of random IO as the devil incarnate, but as long as the SSD has a reasonable endurance rating (by which I mean, "not a random USB flash drive"), it should be reasonable.

So if you really, really need to use dedup, you could use a special vdev of sufficient redundancy of SSDs to keep the metadata in general on, or a DDT-only special vdev if you really need to.

But I would still advise against using dedup unless you are very sure you need to for your workload. It's hard to get rid of once you start using it, it has significant performance penalties even with the nicest configuration you can design, and a lot of people who think they have very deduplicatable workloads find that, for one reason or another, they were mistaken.

So I would be very sure before committing to using it, but no, I don't think Optane is a requirement.

(You could also track #13392 as a much nicer alternative to dedup for many use cases, though it's not yet merged.)

[DemiMarie]

But I would still advise against using dedup unless you are very sure you need to for your workload. It's hard to get rid of once you start using it, it has significant performance penalties even with the nicest configuration you can design, and a lot of people who think they have very deduplicatable workloads find that, for one reason or another, they were mistaken.

The workload that motivated me to want dedup is that of Qubes OS. Qubes OS users often have a large number of very similar virtual machines. To oversimplify matters, root volumes include the operating system and most installed software, while private volumes store user data. Users often find themselves having to have several template VMs (VMs with their own root volume, as opposed to VMs that use a snapshot of a template VM's private volume) that are very similar, due to slight differences in installed software or configuration.
Cloning a VM is CoW, but as both VMs are updated, the CoW gets broken. Deduplication would make identical blocks on both sides identical again. @marmarek and @tlaurion might be able to go into more details.