Add a component for image based deduplication

[satishjasthi]

This issue is to create a new component for image based deduplication for the Fondant-cc-25m data preprocessing pipeline using imagededup library

[geroldmeisinger]

I'm watching your project with interest and recently had to deduplicate an image dataset on my own for controlnet training. Out of interest, I wonder why you choose imagededup specificly and if you have a comparison of different approaches? (btw: In my research I stumbled upon fastdup by visual-layer, which was easy to use and fast, and fiftyone by voxel51, which seems more sophisticated (and also includes a image suite).)

[satishjasthi]

Hi @geroldmeisinger, Thanks for your suggestions, Indeed Fastdup and Fiftyone's image uniqueness component seems much better than imagededup lib. I'll be performing test between Fastdup and Fiftyone to select the best one for image deduplication component. And Ill share the comparison results

[geroldmeisinger]

1. img2dataset, the download tool from LAION, mentions..
2. DataComp: In search of the next generation of multimodal datasets a super-phat 12.8B dataset and collab of some 30 authors, which mentions...
3. "F Deduplication against evaluation sets" -> Contrastive Learning with Large Memory Bank and Negative Embedding Subtraction for Accurate Copy Detection as "We employ the deduplication model proposed by Yokoo [145], which earned 1st place in the Facebook AI Image Similarity Challenge (ISC) [ 40].", which is
4. The 2021 Image Similarity Dataset and Challenge