Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deterministic replication of bloom object creations #32

Merged
merged 1 commit into from
Dec 12, 2024

Conversation

KarthikSubbarao
Copy link
Member

@KarthikSubbarao KarthikSubbarao commented Dec 10, 2024

With this PR, the following changes are made:

  1. Every bloom object creation is replicated with a custom created BF.INSERT command that includes every bloom property that can be a result of global default configs on the primary. With this, the exact same bloom object created on the primary will be created on the replica. Replicated properties include: Capacity, False Positive Rate, Tightening Ratio, Seed (used by hash functions), Items (incase of a object creation + item add operation).
  2. Refactored BloomFilterType and BloomFilter structures to not have public exposed structure members. This is accessible from getter methods now.
  3. Updated the DIGEST logic to include the is_seed_random bool property which indicates whether the bloom object is using a fixed seed or random seed.
  4. Added python integration testing for validation of deterministic replication creation behavior wise. Also, when the primary node and replica nodes have different global configs, tests were added to validate the digest of the bloom object created from replicated command matches the object on the primary node

@KarthikSubbarao KarthikSubbarao marked this pull request as draft December 10, 2024 06:13
@KarthikSubbarao KarthikSubbarao force-pushed the replicate branch 3 times, most recently from e32fd32 to 592d622 Compare December 10, 2024 11:10
@KarthikSubbarao KarthikSubbarao marked this pull request as ready for review December 10, 2024 11:20
@KarthikSubbarao KarthikSubbarao self-assigned this Dec 10, 2024
src/wrapper/bloom_callback.rs Outdated Show resolved Hide resolved
src/bloom/command_handler.rs Outdated Show resolved Hide resolved
src/bloom/data_type.rs Show resolved Hide resolved
@KarthikSubbarao KarthikSubbarao force-pushed the replicate branch 3 times, most recently from 56fbff4 to 5582277 Compare December 11, 2024 05:32
…e properties on replica nodes to match the object on the primary

Signed-off-by: Karthik Subbarao <[email protected]>
Copy link
Contributor

@zackcam zackcam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

@KarthikSubbarao KarthikSubbarao merged commit 391d947 into valkey-io:unstable Dec 12, 2024
6 checks passed
KarthikSubbarao added a commit that referenced this pull request Dec 12, 2024
…e properties on replica nodes to match the object on the primary (#32)

Signed-off-by: Karthik Subbarao <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants