From b6f94a2d1904d5053b56181870b8ccbe2b2185f9 Mon Sep 17 00:00:00 2001 From: Michael Demmer Date: Fri, 30 Aug 2024 10:39:43 -0700 Subject: [PATCH] Tabletbalancer docs (#1804) * add command line flags for tabletbalancer Signed-off-by: Michael Demmer * add a feature description of the tablet balancer Signed-off-by: Michael Demmer * remove aliases Signed-off-by: Michael Demmer * update enable-balancer flag to reflect changes Signed-off-by: Michael Demmer * one more reference Signed-off-by: Michael Demmer --------- Signed-off-by: Michael Demmer --- .../reference/features/tablet-balancer.md | 89 +++++++++++++++++++ .../21.0/reference/programs/vtgate/_index.md | 3 + 2 files changed, 92 insertions(+) create mode 100644 content/en/docs/21.0/reference/features/tablet-balancer.md diff --git a/content/en/docs/21.0/reference/features/tablet-balancer.md b/content/en/docs/21.0/reference/features/tablet-balancer.md new file mode 100644 index 000000000..23a46e0a0 --- /dev/null +++ b/content/en/docs/21.0/reference/features/tablet-balancer.md @@ -0,0 +1,89 @@ +--- +title: VTGate Tablet Balancer +--- + +# VTGate Tablet Balancer + +When a VTGate routes a query and has multiple available tablets for a given shard / tablet type (e.g. REPLICA), +it implements a balancing mechainsm to pick a candidate tablet to route to. At a high level, this process aims to +maintain an even distribution of query load to each tablet, while preferentially routing to tablets in the same cell +as the VTGate to reduce latency. + +In other words, the balancer attempts to achieve two objectives: + +1. Balance the load across the available tablets +2. Prefer a tablet in the same cell as the vtgate if possible + +## Default Policy + +The default behavior is a local cell affinity round robin policy. + +This means that when routing a given query, all the available tablets are randomly shuffled while preferring tablets +in the same cell as the VTGate. So if there is one or more available tablets in the same cell as the VTGate, the query +will be routed to one of those tablet(s), otherwise it will randomly pick a tablet in another cell. + +In many cases this approach suffices, since if there are a proportional number of local tablets in each cell to +satisfy the inbound traffic to the vtgates in that cell, then in general the queries will be distributed evenly to +each tablet. + +## Balancer Motivation + +However, in some topologies, a simple affinity algorithm does not effectively balance the load. + +As a simple example: + +Given three cells with vtgates, four replicas spread into those cells, where each vtgate +receives an equal query share. If each routes only to its local cell, the tablets will be +unbalanced since two of them receive 1/3 of the queries, but the two replicas in the same +cell will only receive 1/6 of the queries. + +``` + Cell A: 1/3 --> vtgate --> 1/3 => vttablet + + Cell B: 1/3 --> vtgate --> 1/3 => vttablet + + Cell C: 1/3 --> vtgate --> 1/6 => vttablet + \-> 1/6 => vttablet +``` + +Other topologies that can cause similar pathologies include cases where there may be cells +containing replicas but no local vtgates, and/or cells that have only vtgates but no replicas. + +For these topologies, the tabletBalancer proportionally assigns the output flow to each tablet, +preferring the local cell where possible, but only as long as the global query balance is +maintained. + +## Algorithm + +To accomplish this goal, the balancer is given: + +* The list of cells that receive inbound traffic to vtgates (from configuration) +* The local cell where the vtgate exists (from configuration) +* The set of tablets and their cells (learned from discovery) + +The model assumes there is an equal probablility of a query coming from each vtgate cell, i.e. +traffic is effectively load balanced between the cells with vtgates. + +Given that information, the balancer builds a simple model to determine how much query load +would go to each tablet if vtgate only routed to its local cell. Then if any tablets are +unbalanced, it shifts the desired allocation away from the local cell preference in order to +even out the query load. + +Based on this global model, the vtgate then probabalistically picks a destination for each +query to be sent and uses these weights to order the available tablets accordingly. + +Assuming each vtgate is configured with and discovers the same information about the topology, +and the input flow is balanced across the vtgate cells (as mentioned above), then each vtgate +should come the the same conclusion about the global flows, and cooperatively should +converge on the desired balanced query load. + +## Configuration + +To enable the balancer requires the following configuration: + + * `--enable-balancer`: Enables the balancer. **Not enabled by default** + * `--balancer-vtgate-cells`: Specifies the set of cells that contain vtgates + +Optionally this behavior can be restricted only when routing to certain keyspaces as a means of controlling rollout: + + * `--balancer-keyspaces`: Specifies the set of keyspaces for which the balancer should be enabled. diff --git a/content/en/docs/21.0/reference/programs/vtgate/_index.md b/content/en/docs/21.0/reference/programs/vtgate/_index.md index 2a5b680d1..183d7028f 100644 --- a/content/en/docs/21.0/reference/programs/vtgate/_index.md +++ b/content/en/docs/21.0/reference/programs/vtgate/_index.md @@ -45,6 +45,8 @@ vtgate \ --allow-kill-statement Allows the execution of kill statement --allowed_tablet_types strings Specifies the tablet types this vtgate is allowed to route queries to. Should be provided as a comma-separated set of tablet types. --alsologtostderr log to standard error as well as files + --balancer-keyspaces strings When in balanced mode, a comma-separated list of keyspaces for which to use the balancer (optional) + --balancer-vtgate-cells strings When in balanced mode, a comma-separated list of cells that contain vtgates (required) --bind-address string Bind address for the server. If empty, the server will listen on all available unicast and anycast IP addresses of the local system. --buffer_drain_concurrency int Maximum number of requests retried simultaneously. More concurrency will increase the load on the PRIMARY vttablet when draining the buffer. (default 1) --buffer_keyspace_shards string If not empty, limit buffering to these entries (comma separated). Entry format: keyspace or keyspace/shard. Requires --enable_buffer=true. @@ -71,6 +73,7 @@ vtgate \ --discovery_high_replication_lag_minimum_serving duration Threshold above which replication lag is considered too high when applying the min_number_serving_vttablets flag. (default 2h0m0s) --discovery_low_replication_lag duration Threshold below which replication lag is considered low enough to be healthy. (default 30s) --emit_stats If set, emit stats to push-based monitoring and stats backends + --enable-balancer Enable the tablet balancer to evenly spread query load for a given tablet type --enable-partial-keyspace-migration (Experimental) Follow shard routing rules: enable only while migrating a keyspace shard by shard. See documentation on Partial MoveTables for more. (default false) --enable-views Enable views support in vtgate. --enable_buffer Enable buffering (stalling) of primary traffic during failovers.