Switch to `ReplicaOrdering.RANDOM` for select LBPs #32

Bouncheck · 2024-11-19T09:40:28Z

This setting has the benefit of evenly distributing the load across replicas. Using round robin policies with NEUTRAL ordering can easily lead to spikes in load on singular nodes during cluster grow and uneven workload afterwards when using tablets.

The reason for not switching to RANDOM for rack aware LBP right now is that it is slightly broken in that configuration.
See java-driver/369.

This setting has the benefit of evenly distributing the load across replicas. Using round robin policies with `NEUTRAL` ordering can easily lead to spikes in load on singular nodes during cluster grow and uneven workload afterwards when using tablets. The reason for not switching to `RANDOM` for rack aware LBP right now is that it is slightly broken in that configuration. See java-driver/369.

Bouncheck · 2024-11-19T09:48:20Z

Previous switch to ReplicaOrdering.NEUTRAL was done in response to this issue scylladb/java-driver#255
While that resolved that issue it silently introduced another. I think the underlying cause for 255 could be the problem I'm describing in scylladb/java-driver#369 . It would match why the driver distributed the load across local DC instead of local rack.

fruch · 2024-11-19T10:51:42Z

tools/stress/src/org/apache/cassandra/stress/util/JavaDriverClient.java


        if (settings.node.rack != null) {
            RackAwareRoundRobinPolicy.Builder policyBuilder = RackAwareRoundRobinPolicy.builder();
            if (settings.node.datacenter != null)
                policyBuilder.withLocalDc(settings.node.datacenter);
            policyBuilder = policyBuilder.withLocalRack(settings.node.rack);
            ret = policyBuilder.build();
+            replicaOrdering = ReplicaOrdering.NEUTRAL;


so this mean that a mix of using Rackaware and tablets would be imbalanced ?

and would be needed to be fix on the driver end ?

All round robin policies (rack,dc) used with TokenAwarePolicy can be imbalanced with tablets when using neutral ordering. I think this combination should not be used if we want load to be as balanced as possible. Long story short let's say we have RF=3 and 6 nodes [A,B,C,D,E,F] and tablets are spread evenly but only on ABC (this can happen when growing the cluster). If round robin happens to point to either D,E,F,A then that request will hit replica A first. This results in A getting 4/6 of the load, B 1/6, and C 1/6. However if RF=3 and cluster has only A,B,C then all will be nearly perfectly balanced.

I'll try to make a comment with broader explanation what happens in scylladb/scylladb#19107 to better illustrate this issue with neutral ordering.

The rack aware one just does not work correctly (will ignore rack awareness in favor of local dc) with random ordering so it has to stay neutral for now. This is the part that needs fixing on driver's end.

fruch

LGTM

Main reason for version change: Using cassandra-stress 3.17 to mittigate - Switch to ReplicaOrdering.RANDOM for select LBPs [32](scylladb/cassandra-stress#32) Other Noticable Changes since the last version used in SCT: - Add support for hostname verification [31](scylladb/cassandra-stress#31) - Print thread dump on specific signals [27](scylladb/cassandra-stress#27) - Replace uninterruptible wait [26](scylladb/cassandra-stress#26) - Make it use DCAwareRoundRobinPolicy unless rack is provided [21](scylladb/cassandra-stress#21) - feature(docker): adding support for dependabot [19](scylladb/cassandra-stress#19) Signed-off-by: Dusan Malusev <[email protected]>

Main reason for version change: Using cassandra-stress 3.17 to mittigate - Switch to ReplicaOrdering.RANDOM for select LBPs [32](scylladb/cassandra-stress#32) Other Noticable Changes since the last version used in SCT: - Add support for hostname verification [31](scylladb/cassandra-stress#31) - Print thread dump on specific signals [27](scylladb/cassandra-stress#27) - Replace uninterruptible wait [26](scylladb/cassandra-stress#26) - Make it use DCAwareRoundRobinPolicy unless rack is provided [21](scylladb/cassandra-stress#21) - feature(docker): adding support for dependabot [19](scylladb/cassandra-stress#19) Signed-off-by: Dusan Malusev <[email protected]> (cherry picked from commit 02997a6) # Conflicts: # defaults/docker_images/cassandra-stress/values_cassandra-stress.yaml

Main reason for version change: Using cassandra-stress 3.17 to mittigate - Switch to ReplicaOrdering.RANDOM for select LBPs [32](scylladb/cassandra-stress#32) Other Noticable Changes since the last version used in SCT: - Add support for hostname verification [31](scylladb/cassandra-stress#31) - Print thread dump on specific signals [27](scylladb/cassandra-stress#27) - Replace uninterruptible wait [26](scylladb/cassandra-stress#26) - Make it use DCAwareRoundRobinPolicy unless rack is provided [21](scylladb/cassandra-stress#21) - feature(docker): adding support for dependabot [19](scylladb/cassandra-stress#19) Signed-off-by: Dusan Malusev <[email protected]> (cherry picked from commit 02997a6)

Main reason for version change: Using cassandra-stress 3.17 to mittigate - Switch to ReplicaOrdering.RANDOM for select LBPs [32](scylladb/cassandra-stress#32) Other Noticable Changes since the last version used in SCT: - Add support for hostname verification [31](scylladb/cassandra-stress#31) - Print thread dump on specific signals [27](scylladb/cassandra-stress#27) - Replace uninterruptible wait [26](scylladb/cassandra-stress#26) - Make it use DCAwareRoundRobinPolicy unless rack is provided [21](scylladb/cassandra-stress#21) - feature(docker): adding support for dependabot [19](scylladb/cassandra-stress#19) Signed-off-by: Dusan Malusev <[email protected]> (cherry picked from commit 02997a6) # Conflicts: # defaults/docker_images/cassandra-stress/values_cassandra-stress.yaml

Main reason for version change: Using cassandra-stress 3.17 to mittigate - Switch to ReplicaOrdering.RANDOM for select LBPs [32](scylladb/cassandra-stress#32) Other Noticable Changes since the last version used in SCT: - Add support for hostname verification [31](scylladb/cassandra-stress#31) - Print thread dump on specific signals [27](scylladb/cassandra-stress#27) - Replace uninterruptible wait [26](scylladb/cassandra-stress#26) - Make it use DCAwareRoundRobinPolicy unless rack is provided [21](scylladb/cassandra-stress#21) - feature(docker): adding support for dependabot [19](scylladb/cassandra-stress#19) Signed-off-by: Dusan Malusev <[email protected]> (cherry picked from commit 02997a6)

Main reason for version change: Using cassandra-stress 3.17 to mittigate - Switch to ReplicaOrdering.RANDOM for select LBPs [32](scylladb/cassandra-stress#32) Other Noticable Changes since the last version used in SCT: - Add support for hostname verification [31](scylladb/cassandra-stress#31) - Print thread dump on specific signals [27](scylladb/cassandra-stress#27) - Replace uninterruptible wait [26](scylladb/cassandra-stress#26) - Make it use DCAwareRoundRobinPolicy unless rack is provided [21](scylladb/cassandra-stress#21) - feature(docker): adding support for dependabot [19](scylladb/cassandra-stress#19) Signed-off-by: Dusan Malusev <[email protected]> (cherry picked from commit 02997a6) # Conflicts: # defaults/docker_images/cassandra-stress/values_cassandra-stress.yaml

Bouncheck requested review from CodeLieutenant and dkropachev November 19, 2024 10:06

Bouncheck mentioned this pull request Nov 19, 2024

[Tablets] Requests served imbalance after adding nodes to cluster scylladb/scylladb#19107

Closed

2 tasks

fruch reviewed Nov 19, 2024

View reviewed changes

fruch approved these changes Nov 19, 2024

View reviewed changes

CodeLieutenant approved these changes Nov 19, 2024

View reviewed changes

CodeLieutenant merged commit 58ff646 into scylladb:master Nov 19, 2024
2 checks passed

swasik mentioned this pull request Nov 19, 2024

Streaming after adding nodes causes to existing (not new) nodes run with 100% CPU on all shards scylladb/scylladb#20921

Closed

2 tasks

CodeLieutenant mentioned this pull request Nov 20, 2024

fix(cassandra-stress): bump c-s version to 3.17.0 scylladb/scylla-cluster-tests#9296

Merged

2 tasks

Bouncheck self-assigned this Nov 20, 2024

mergify bot mentioned this pull request Nov 26, 2024

[Backport 6.0] fix(cassandra-stress): bump c-s version to 3.17.0 scylladb/scylla-cluster-tests#9376

Merged

2 tasks

This was referenced Nov 26, 2024

[Backport 6.1] fix(cassandra-stress): bump c-s version to 3.17.0 scylladb/scylla-cluster-tests#9377

Merged

[Backport 6.2] fix(cassandra-stress): bump c-s version to 3.17.0 scylladb/scylla-cluster-tests#9378

Merged

mergify bot mentioned this pull request Nov 26, 2024

[Backport perf-v16] fix(cassandra-stress): bump c-s version to 3.17.0 scylladb/scylla-cluster-tests#9382

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch to `ReplicaOrdering.RANDOM` for select LBPs #32

Switch to `ReplicaOrdering.RANDOM` for select LBPs #32

Bouncheck commented Nov 19, 2024

Bouncheck commented Nov 19, 2024

fruch Nov 19, 2024

Bouncheck Nov 19, 2024 •

edited

Loading

fruch left a comment

Switch to ReplicaOrdering.RANDOM for select LBPs #32

Switch to ReplicaOrdering.RANDOM for select LBPs #32

Conversation

Bouncheck commented Nov 19, 2024

Bouncheck commented Nov 19, 2024

fruch Nov 19, 2024

Choose a reason for hiding this comment

Bouncheck Nov 19, 2024 • edited Loading

Choose a reason for hiding this comment

fruch left a comment

Choose a reason for hiding this comment

Switch to `ReplicaOrdering.RANDOM` for select LBPs #32

Switch to `ReplicaOrdering.RANDOM` for select LBPs #32

Bouncheck Nov 19, 2024 •

edited

Loading