can not dowload?

[sri-hk]

hi，nice wrok ! but the data can not be downloaded.

this dataset/
laion2b-en-vit-h-14-embeddings became disabled，any other soultion to get your deduplicated laion-2b-en data?

look forward to your reply，3ks!

[Jesse-XIE]

+1

[ryanwebster90]

as sri-hk said, LAION is no longer distributing laion2b and its variants. So, as I only provided a filtering of that data it won't be available anymore. I'll remove that code soon and maybe add functionality to deduplicate your own dataset / another dataset.

[ppwwyyxx]

@ryanwebster90 Can you provide deduped results based on urls?

This way it does not require users to download the now-deleted dataset on huggingface. With urls, labs that have downloaded laion2b (but in a different format/order from the huggingface dataset) will be able to leverage your deduped results.

[ryanwebster90]

@ppwwyyxx I can not, as the dataset is facing ethical issues, and don't plan to. For now, I'd suggest to check out DataComp-1B, and perhaps I'll plan to deduplicate that dataset.

[sri-hk]

as sri-hk said, LAION is no longer distributing laion2b and its variants. So, as I only provided a filtering of that data it won't be available anymore. I'll remove that code soon and maybe add functionality to deduplicate your own dataset / another dataset.

Thank you for your reply.
I find another way to deduplicate. So far so good. Phash is highly efficient and also fast, but you may need many cpus and memories.