Add "2ha.sh" script, managing 2-node Canonical K8s HA AA clusters #692
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Scenario overview:
* Canonical K8s cluster containing 2 nodes
* Dqlite data store (unable to obtain quorum)
* Primary node dqlite files stored on DRBD
* sync block-level replication between the two nodes
* cluster monitoring and failover handled through Pacemaker
Script functionality:
* boostrap the service
* wait for a DRBD primary to be elected
* detect the node role based on the DRBD status and Dqlite state
* have the replica wait for the primary to be ready before continuing
* recover Dqlite after failovers
* transfer and apply recovery files to secondary nodes
* transfer Dqlite files to DRBD and other backup locations, creating
necessary symlinks
* install required packages
* purge all K8s data
* clear Pacemaker taints
* remove recovery data
"2ha.sh start_service" is intended to be used as part of a systemd
unit that bootstraps the k8s services, coordinating with the other
node and taking any necessary steps to recover Dqlite.
We're adding a guide that covers the 2-node A-A HA scenario.
bschimke95
reviewed
Sep 25, 2024
|
Thanks for reviewing this PR! I'll address the comments right away.
I admit that Openstack Devstack changed my perception of what a large bash script means but I see your point. |
petrutlucian94
force-pushed
the
KU-1606/2ha_script
branch
3 times, most recently
from
57bd703
to
aa01e0e
Compare
petrutlucian94
force-pushed
the
KU-1606/2ha_script
branch
from
aa01e0e
to
33ed437
Compare
louiseschmidtgen
requested changes
Sep 26, 2024
There was a problem hiding this comment.
Amazing work on the 2-node HA set-up @petrutlucian94!
Please iterate over my polishing comments.
I am requesting changes because I would like to discuss the alternative solution with PostgreSQL.
petrutlucian94
force-pushed
the
KU-1606/2ha_script
branch
from
30d0367
to
8d690c3
Compare
|
@louiseschmidtgen Thanks for reviewing the docs! I've addressed most comments and left a few questions. |
louiseschmidtgen
approved these changes
Sep 30, 2024
There was a problem hiding this comment.
Couple more small comments that need to be addressed, afterwards we are good to go. Thank you @petrutlucian94
|
Great work @petrutlucian94! |
evilnick
pushed a commit
to evilnick/k8s-snap
that referenced
this pull request
Nov 14, 2024
…nonical#692) Scenario overview: * Canonical K8s cluster containing 2 nodes * Dqlite data store (unable to obtain quorum) * Primary node dqlite files stored on DRBD * sync block-level replication between the two nodes * cluster monitoring and failover handled through Pacemaker Script functionality: * boostrap the service * wait for a DRBD primary to be elected * detect the node role based on the DRBD status and Dqlite state * have the replica wait for the primary to be ready before continuing * recover Dqlite after failovers * transfer and apply recovery files to secondary nodes * transfer Dqlite files to DRBD and other backup locations, creating necessary symlinks * install required packages * purge all K8s data * clear Pacemaker taints * remove recovery data "2ha.sh start_service" is intended to be used as part of a systemd unit that bootstraps the k8s services, coordinating with the other node and taking any necessary steps to recover Dqlite.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Scenario overview:
Script functionality:
"2ha.sh start_service" is intended to be used as part of a systemd unit that bootstraps the k8s services, coordinating with the other node and taking any necessary steps to recover Dqlite.
This PR also adds a "how-to" guide for the 2-node A-A HA setup.