Add minhash deduplicator based on RAY and Redis #489
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yxdyc
reviewed
Nov 18, 2024
| @@ -172,6 +172,7 @@ Data-Juicer 中的算子分为以下 5 种类型。 | |||
| | document_simhash_deduplicator |     | 使用 SimHash 在文档级别对样本去重 | [code](../data_juicer/ops/deduplicator/document_simhash_deduplicator.py) | [tests](../tests/ops/deduplicator/test_document_simhash_deduplicator.py) | | |||
| | image_deduplicator |  | 使用文档之间图像的精确匹配在文档级别删除重复样本 | [code](../data_juicer/ops/deduplicator/image_deduplicator.py) | [tests](../tests/ops/deduplicator/test_image_deduplicator.py) | | |||
| | video_deduplicator |  | 使用文档之间视频的精确匹配在文档级别删除重复样本 | [code](../data_juicer/ops/deduplicator/video_deduplicator.py) | [tests](../tests/ops/deduplicator/test_video_deduplicator.py) | | |||
| | ray_redis_minhash_deduplicator |     | 使用 MinHashLSH 在文档级别对样本去重，面向 RAY 分布式模式 | [code](../data_juicer/ops/deduplicator/ray_redis_minhash_deduplicator.py) | - | | |||
There was a problem hiding this comment.
Also correct the number of OPs +=1. Later, I will try to implement an auto-complement template for OP-developer
| @@ -643,6 +643,17 @@ process: | |||
| redis_port: 6380 # the port of redis instance, please note that the default port of redis is 6379 which is the same as default port for ray, so we need to modify the default redis config to use it in other port | |||
| lowercase: false # whether to convert text to lower case | |||
| ignore_non_character: false # whether to ignore non-alphabet characters, including whitespaces, digits, and punctuations | |||
| - ray_redis_minhash_deduplicator: # the document deduplicator that can run on multi-nodes using minhashLSH algorithm | |||
| redis_address: 'redis://localhost:6379' # the address of the redis instance | |||
There was a problem hiding this comment.
The former three OPs (ray_video_deduplicator, ray_image_deduplicator, ray_document_deduplicator) also use redis, but the default args is redis_port: 6380 and here is redis_address: 'redis://localhost:6379'. We may consider unify them?
| @OPERATORS.register_module(OP_NAME) | ||
| class RayRedisMinhashDeduplicator(Deduplicator): | ||
| """ | ||
| A basic exact matching deduplicator for RAY. |
| add_uid_column, batch_format='pyarrow').materialize() | ||
| dataset_with_id.map_batches(calculate_minhash, | ||
| batch_format='pyarrow').groupby( | ||
| HashKeys.minhash).aggregate( |
There was a problem hiding this comment.
Although we are (ideally) optimizing this via multiple distributed UnionSets, it is worth profiling this line and (then co-optimizing with Ray-Team)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As the title says