Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: implement mini-batch & continue-as-new for replicator workflow #94

Merged

Conversation

bestmike007
Copy link
Contributor

What changed? Why?

  • Implement mini batches, which can be distributed to different workers
  • Continue workflow as new when checkpoint size reached, in order to prevent workflow timeout

How did you test the change?

  • GitHub Actions
  • go run ./cmd/admin workflow start --workflow replicator --input '{"StartHeight": 1, "EndHeight": 10001}' --blockchain ethereum --network mainnet --env local

@bestmike007 bestmike007 requested a review from a team as a code owner February 23, 2024 17:33
@bestmike007 bestmike007 force-pushed the feat/replicator-workflow-mini-batch branch from 80f810f to 7a47879 Compare February 25, 2024 06:19
BlockHeight uint64
Tag uint32
BlockHeight uint64
ValidateSince uint64
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: maybe validateStart would be a better name. Please also add a comment on whether this is inclusive or exclusive index.

Signed-off-by: bestmike007 <[email protected]>
@HenryYYang HenryYYang merged commit 57b2d41 into coinbase:master Feb 25, 2024
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants