Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

polygon/sync: retry ufc on busy response #13640

Merged
merged 4 commits into from
Jan 31, 2025
Merged

Conversation

taratorio
Copy link
Member

@taratorio taratorio commented Jan 31, 2025

a user on discord flagged the following issue (discord link):

[INFO] [01-31|10:44:40.015] [sync] update fork choice done           in=807.13624ms
[INFO] [01-31|10:44:40.022] [sync] update fork choice                block=67365147 hash=0xb697ef9de67d61f300fda27367617ec270d9bcd15e9a7c73facdc253b6fc0273 age=3s
[EROR] [01-31|10:44:40.022] failed to update fork choice             latestValidHash=0x0000000000000000000000000000000000000000000000000000000000000000 err="fork choice update failure: status=5, validationErr=''"
[EROR] [01-31|10:44:40.038] runPostForkchoiceInBackground            error="[1/6 OtterSync] db closed"
[EROR] [01-31|10:44:43.029] polygon sync crashed - stopping node     err="pos sync failed: fork choice update failure: status=5, validationErr=''"

Note: status=5 means ExecutionStatus_Busy

This can happen because there is a possibility that runPostForkchoiceInBackground acquires the execution engine semaphore after a successful UFC (line which says update fork choice done in=807.13624ms) to do pruning. But very shortly after that there is another call to UFC (in this case 7ms after - line which says update fork choice block=67365147) which then fails with a Busy response since it can't acquire the semaphore (take by runPostForkchoiceInBackground for pruning).

This PR fixes this situation by adding retries on Busy response when calling UFC. Such a retry already existed when calling InsertBlocks so this is something that just got missed.

@taratorio taratorio added this to the 3.0.0-beta2 milestone Jan 31, 2025
@taratorio taratorio enabled auto-merge (squash) January 31, 2025 14:10
@taratorio taratorio merged commit dcb4ec2 into main Jan 31, 2025
12 of 13 checks passed
@taratorio taratorio deleted the astrid-retry-ufc-on-busy branch January 31, 2025 14:36
taratorio added a commit that referenced this pull request Jan 31, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants