Deduplication technique could be used in design

[yuanjingsong]

What would you like to be added:
Data deduplication is a technique that reuses the repeated data chunk (block in this project) among different files, and it has been used in Ceph project.
There are several file systems use the deduplication to improve read performance and save storage cost. ATC 19', TOS 19'

Why is this needed:

1. The design of splitting files into chunks and managing metadata by database is similar to the deduplication design.
2. Deduplication could save disk cost and improve read performance.

[davies]

Data dedup is usually expensive to use, have few value to common use cases.

For some cases that have lots of duplicated data, may be we can use hard link, copy_file_range(), snapshot (planned) to avoid the duplicated data, also much faster.

[yuanjingsong]

Okay, thanks for reply.

[vipsland]

i think u need to think blake3 instead of sha256. blake3 will be very cheap

[flibustenet]

Dedup can reduce network and number of blocks (on cache and on storage).
When juicefs already use block and index where do you see an expensive downside ?

[vipsland]

Feature request : pls have a plugin whereby we can write our own interfacing for dedup etc, change our own hash to blake3 etc

Data dedup is usually expensive to use... i dont agree.

it's expensive not to deploy

For some cases that have lots of duplicated data, may be we can use hard link, copy_file_range(), snapshot (planned) to avoid the duplicated data, also much faster.

actually if zfs is compatible will have built in dedup without this additional part BUT still we shld code our own dedup too.