Support reconnection/resync with Typha #10306
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
marvin-tigera
added
release-note-required
fasaxc
force-pushed
the
typha-reconnection
branch
from
ca8aba2 to
a25c0f5
Compare
fasaxc
force-pushed
the
typha-reconnection
branch
4 times, most recently
from
0106904 to
d599369
Compare
fasaxc
added
docs-not-required
- Add optional callback for reconnection-aware clients.
- Adjust Typha discovery to reset after all Typhas have been tried.
- Make the dedupe buffer reconnection-aware. It now
- Stores off the keys that it had previously seen when it gets the
OnTyphaConnectionRestarted() call.
- Discards those seen keys as the resync progresses.
- Synthesises deletions for KVs that weren't seen during the resync.
- Recalculates the UpdateType when sending keys downstream so that
the calculation graph sees a resync as a sequence of updates for
exisitng keys.
- Refactor the client so that it
- Does one connecction synchronously (including connection attempts
to mutliple Typha instances as before)
- Reconnects in the background after a failure.
- Sends WaitForDatastore/ResyncInProgress messages when it's doing
a reconnection.
- Re-uses a single connection attempt tracker so that we cycle through
Typha instances on reconnect.
- Varous minor changes:
- Add "done" channels to various components to avoid "log to testing.T
after test finished" errors.
- Add 32 bit random value to connection ID. Makes it a lot more
greppable in logs.
fasaxc
force-pushed
the
typha-reconnection
branch
from
d599369 to
8a00314
Compare
aaaaaaaalex
reviewed
Apr 30, 2025
| log.Debugf("Typha supports node resource updates: %v", supportsNodeResourceUpdates) | ||
| configParams.SetUseNodeResourceUpdates(supportsNodeResourceUpdates) | ||
| // Up-to-date Typha client will refuse to connect unless Typha signals | ||
| // that it supports node resource updates. |
There was a problem hiding this comment.
Any issue with version skew on-upgrade?
There was a problem hiding this comment.
I think the feature was added in 2019 so, yes, but only if you're skipping a dozen versions. Even in that case, it will only block new felix from talking to old typha, felix will just keep trying untill it connects to an up-level typha.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add optional callback for reconnection-aware clients.
Adjust Typha discovery to reset after all Typhas have been tried.
Make the dedupe buffer reconnection-aware. It now
Refactor the client so that it
felix_resyncs_startedandfelix_resync_statePrometheus metrics useful again.Varous minor changes:
Related issues/PRs
CORE-11348
Todos
The client will still bail out if it can't connect to any Typha. Not sure if that's desirable or not; we could keep retrying but issues like running out of file handles might be better handled with a restart.
Release Note
Reminder for the reviewer
Make sure that this PR has the correct labels and milestone set.
Every PR needs one
docs-*label.docs-pr-required: This change requires a change to the documentation that has not been completed yet.docs-completed: This change has all necessary documentation completed.docs-not-required: This change has no user-facing impact and requires no docs.Every PR needs one
release-note-*label.release-note-required: This PR has user-facing changes. Most PRs should have this label.release-note-not-required: This PR has no user-facing changes.Other optional labels:
cherry-pick-candidate: This PR should be cherry-picked to an earlier release. For bug fixes only.needs-operator-pr: This PR is related to install and requires a corresponding change to the operator.