Skip to content

Commit

Permalink
HA - Smart Routing doc updates (#1139)
Browse files Browse the repository at this point in the history
I think I have caught everything.

The one thing that is missing is the tutorial we discussed; with Hendy
moving into the team, she might pick that up. Either way, a separate PR
will be raised for that work.

I've also made the changes to the Phone Homes, which Gareth had
previously updated for before the name change was agreed.

---------

Co-authored-by: James Holgate <[email protected]>
Co-authored-by: James Holgate <[email protected]>
  • Loading branch information
3 people authored Jul 4, 2024
1 parent 2b1b075 commit aaefef4
Show file tree
Hide file tree
Showing 14 changed files with 312 additions and 75 deletions.
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
package com.hazelcast.client.config;

import com.hazelcast.client.HazelcastClient;
import com.hazelcast.client.config.ClusterRoutingConfig;
import com.hazelcast.client.impl.connection.tcp.RoutingMode;
import com.hazelcast.config.AwsConfig;
import com.hazelcast.core.HazelcastInstance;

Expand All @@ -12,7 +14,7 @@ public static void main(String[] args) throws Exception {
clientConfig.getConnectionStrategyConfig().getConnectionRetryConfig().setMaxBackoffMillis(5000);
ClientNetworkConfig networkConfig = clientConfig.getNetworkConfig();
networkConfig.addAddress("10.1.1.21", "10.1.1.22:5703")
.setSmartRouting(true)
.setClusterRoutingConfig(new ClusterRoutingConfig().setRoutingMode(RoutingMode.ALL_MEMBERS))
.addOutboundPortDefinition("34700-34710")
.setRedoOperation(true)
.setConnectionTimeout(5000);
Expand Down
Binary file added docs/modules/ROOT/images/client-routing.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
11 changes: 6 additions & 5 deletions docs/modules/ROOT/pages/capacity-planning.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -219,13 +219,14 @@ defined for each version of the software. We recommend
staying below these numbers. Please contact Hazelcast if you plan to
use higher limits.

* Maximum 100 multi-socket clients per member
* Maximum 1,000 unisocket clients per member
* Maximum 100 clients using the `ALL_MEMBERS` cluster routing mode per member
* Maximum 1,000 clients using the `SINGLE_MEMBER` cluster routing mode per member
* Maximum of 200GB xref:storage:high-density-memory.adoc[High-Density Memory Store] per member

Multi-socket clients are the smart clients which maintain a connection to each member.
Unisocket clients have a single connection to the entire cluster.
You can find more information about these two modes here: xref:clients:java.adoc#java-client-operation-modes[Java Client Operation Modes].
Clients that use the `ALL_MEMBERS` cluster routing mode maintain a connection to each member.
Clients that use the `SINGLE_MEMBER` cluster routing mode have a single connection to the entire cluster.
You can also choose to connect to a partition group which provides direct connections to members in that group with those members acting as a gateway to the other members in the cluster.
You can find more information about the cluster routing modes here: xref:clients:java.adoc#java-cluster-routing-modes[Java Cluster Routing Modes].

== Size for Failures

Expand Down
2 changes: 1 addition & 1 deletion docs/modules/ROOT/pages/faq.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -237,7 +237,7 @@ Yes. Hazelcast performed a successful test on Amazon EC2 with 200 members.

Yes. However, there are some points you should consider.
The environment should be LAN with a high stability and the network speed should be 10 Gbps or higher.
If the number of members is high, the client type should be selected as Unisocket, not Smart Client.
If the number of members is high, you can select the client type as `SINGLE_MEMBER` or `MULTI_MEMBER`, but not `ALL_MEMBERS`.
In the case of Smart Clients, since each client opens a connection to the members,
these members should be powerful enough (for example, more cores) to handle hundreds or thousands of connections and client requests.
Also, you should consider using Near Caches in clients to lower the network traffic.
Expand Down
4 changes: 2 additions & 2 deletions docs/modules/ROOT/pages/phone-homes.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ The following information is sent in a phone home:
* Number of active client connections during the phone home ping; bands of 5, 10, 20, 40, 60, 100, 150, 300, 600 and > 600
* Number of clients connected since the last phone home ping
* Number of clients disconnected since the last phone home ping
* Number of active client connections with routing mode SMART, UNISOCKET and SUBSET
* Number of active client connections with routing mode ALL_MEMBERS, SINGLE_MEMBER, and MULTI_MEMBER
* Total duration of client connections since the last phone home ping
* Versions of the connected, disconnected or active clients since the last phone home ping
* Cluster uptime
Expand Down Expand Up @@ -78,7 +78,7 @@ The following information is sent in a phone home:
** Whether HD memory is enabled
** Whether Tiered Storage is enabled
** Whether User Code Namespaces is enabled; if so, count of registered user code namespaces
** Count of submitted placement controled jobs
** Count of submitted placement controlled jobs
**Disabling Phone Homes**

Expand Down
336 changes: 285 additions & 51 deletions docs/modules/clients/pages/java.adoc

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -443,9 +443,9 @@ block if there is a shortage of entropy and the returned random values could
theoretically be vulnerable to a cryptographic attack.
If this is a concern in your application, use `/dev/random` instead.

Hazelcast's Java smart client automatically makes use of extra I/O threads
for encryption/decryption and this have a significant impact on the performance.
This can be changed using the `hazelcast.client.io.input.thread.count` and
Clients using Hazelcast's Java `ALL_MEMBERS` and `MULTI_MEMBER` cluster routing modes automatically make use of extra I/O threads
for encryption/decryption and this has a significant impact on the performance.
Thee number of threads used can be changed using the `hazelcast.client.io.input.thread.count` and
`hazelcast.client.io.output.thread.count` client system properties.
By default it is 1 input thread and 1 output thread. If TLS/SSL is enabled,
it defaults to 3 input threads and 3 output threads.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,9 @@ SEDA can lead to throughput and latency issues on machines with limited cores (f

A Thread-Per-Core (TPC) design uses one thread for every CPU core and every thread does everything. With each TPC thread able to do networking/storage/compute, scaling is mostly a case of increasing the number of threads to spread the work. This means the system can more easily scale up and down based on the number of available cores, and make more efficient use of resources on machines with either a low or high number of cores. Every task is non-blocking, which makes it easier to scale vertically, and results in high throughput, low latency and significantly improved system performance.

When TPC is enabled, smart clients will connect to the number of cores specified in the client config. If TPC is enabled on both client and server, clients connect directly to one of the TPC threads instead of using the legacy network threads. TPC-enabled servers continue to use the same ports for discovery, which means there's no difference in how the cluster member list is created and TPC-aware servers are backward compatible with clients that don't use TPC.
When TPC is enabled, clients using the `ALL_MEMBERS` cluster routing mode connect to the number of cores specified in the client config. If TPC is enabled on both client and server, clients connect directly to one of the TPC threads instead of using the legacy network threads. TPC-enabled servers continue to use the same ports for discovery, which means there's no difference in how the cluster member list is created and TPC-aware servers are backward compatible with clients that don't use TPC.

NOTE: Unisocket clients will ignore TPC and connect to a random member in the cluster. You must use smart clients with TPC. For further information on the `ClientTpcConfig` class used to specify the number of cores to use for connections, refer to https://docs.hazelcast.org/docs/{full-version}/javadoc/com/hazelcast/client/config/ClientTpcConfig.html[Class ClientTpcConfig] in the Java API documentation.
NOTE: Clients using the `SINGLE_MEMBER` cluster routing mode ignore TPC and connect to a random member in the cluster. Your clients must use the `ALL_MEMBERS` cluster routing mode with TPC. For further information on the `ClientTpcConfig` class used to specify the number of cores to use for connections, refer to https://docs.hazelcast.org/docs/{full-version}/javadoc/com/hazelcast/client/config/ClientTpcConfig.html[Class ClientTpcConfig] in the Java API documentation.

[[tpc-config]]
== Configuration Options
Expand Down
4 changes: 2 additions & 2 deletions docs/modules/clusters/pages/network-configuration.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -797,7 +797,7 @@ You cannot define both elements in the declarative configuration, i.e., the `net
and `advanced-network` elements cannot be configured at the same time. In the
programmatic configuration, an enabled `AdvancedNetworkConfig` takes precedence over
the `NetworkConfig`. `AdvancedNetworkConfig` is disabled by default, therefore the
unisocket member configuration under `NetworkConfig` is used in the default case.
single socket member configuration under `NetworkConfig` is used in the default case.

When using the advanced network configuration, the following configurations are defined
member-wide:
Expand Down Expand Up @@ -1118,7 +1118,7 @@ replication. See the xref:wan:advanced-features.adoc#securing-wan-connections.ad
[qanda]
Can I multiplex protocols on a single advanced network endpoint? For example, can I use a single server socket to listen for `MEMBER` and `CLIENT` protocols?::
No, each endpoint configuration that defines a server socket must bind to a different socket address.
Can I mix unisocket and advanced network members in the same cluster?::
Can I mix single socket and advanced network members in the same cluster?::
No, the results will be undefined.
Can I configure multiple server socket endpoints for the same protocol?::
You can only configure multiple server socket endpoints for `WAN` protocol. For other protocols (`MEMBER`, `CLIENT`, `REST`, `MEMCACHE`), a single server socket can be configured.
4 changes: 2 additions & 2 deletions docs/modules/data-structures/pages/replicated-map.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@ eventually being replicated to all other members.
Replicated Map is suitable for objects, catalog data, or idempotent calculable data, such as HTML pages. It fully implements the `java.util.Map` interface, but it lacks the methods from `java.util.concurrent.ConcurrentMap` since
there are no atomic guarantees to writes or reads.

NOTE: If a Replicated Map is used from a unisocket client, and this
unisocket client is connected to a lite member, the entry listeners cannot be registered or de-registered.
NOTE: If a Replicated Map is used from a client using the `SINGLE_MEMBER` cluster routing mode, and this
client is connected to a lite member, the entry listeners cannot be registered or de-registered.

NOTE: You cannot use Replicated Map from a lite member.
A `com.hazelcast.replicatedmap.ReplicatedMapCantBeCreatedOnLiteMemberException`
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ NOTE: If you don't specify `service-name` and fall back to a `namespace` only di
* `expose-externally`: if set to `true`, it fails fast if an external address cannot be found for each member; if set to `false`, it does not check for external member addresses; by default it tries to resolve external addresses but fails silently
+
NOTE: If your service type is `NodePort`, there might be connectivity problems from outside to the Kubernetes cluster. Therefore, it's crucial to verify that NodePort IPs are externally accessible.
* `service-per-pod-label-name`, `service-per-pod-label-value`: service label and value used to tag services that expose each Hazelcast member with a separate Kubernetes service (for connecting Hazelcast Smart Client running outside the Kubernetes cluster)
* `service-per-pod-label-name`, `service-per-pod-label-value`: service label and value used to tag services that expose each Hazelcast member with a separate Kubernetes service (for connecting a Hazelcast client using the `ALL_MEMBERS` cluster routing mode and running outside the Kubernetes cluster)
* `use-node-name-as-external-address`: if set to `true`, uses the node name to connect to a `NodePort` service instead of looking up the external IP using the API; `false` by default
* `kubernetes-api-retries`: number of retries in case of issues while connecting to Kubernetes API; defaults to `3`
* `kubernetes-master`: URL of Kubernetes Master; `https://kubernetes.default.svc` by default
Expand Down
4 changes: 2 additions & 2 deletions docs/modules/migrate/pages/upgrading-from-imdg-3.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -60,10 +60,10 @@ having some extra cores available for I/O can make a significant
difference. If less than 20 cores are detected, 3+3 IO threads are used
and the behavior remains the same as Hazelcast IMDG 3.x series.

A smart client, by default, gets 3+3 (3 input and 3 output) I/O threads to
A client using either the `ALL_MEMBERS` or `MULTI_MEMBER` cluster routing mode, by default, gets 3+3 (3 input and 3 output) I/O threads to
speed up the performance. Previously, this was
1+1. However, the client I/O can become a bottleneck with too few threads.
If TLS/SSL is enabled, then by default a smart client
If TLS/SSL is enabled, then by default a client using either the `ALL_MEMBERS` or `MULTI_MEMBER` cluster routing mode
makes use of 3+3 I/O threads which was already the case with previous versions.

There has been a new performance feature since IMDG 4.0 called
Expand Down
2 changes: 1 addition & 1 deletion docs/modules/security/pages/native-client-security.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -1568,7 +1568,7 @@ is suitable for the scenarios where you need to elevate privileges temporarily
on a single member (preferably a xref:management:cluster-utilities.adoc#enabling-lite-members[lite member]) for a
limited time period. The clients which want to use these temporary permissions
have to access the cluster through this single new member, meaning that you need
to disable xref:clients:java.adoc#setting-smart-routing[smart routing] for such clients.
to configure the xref:clients:java.adoc#configure-cluster-routing-mode[SINGLE_MEMBER] cluster routing mode for such clients.
+
Note that, the `create` and `destroy` permissions will not work when using
the `NONE` option, since the distributed objects need to be created/destroyed on all the members.
Expand Down
4 changes: 2 additions & 2 deletions docs/modules/spring/pages/configuration.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -186,8 +186,8 @@ While configuring Hazelcast within the Spring context, you can also pass values
<hz:client id="client">
<hz:cluster-name name="${cluster.name}"/>
<hz:network connection-timeout="1000"
redo-operation="true"
smart-routing="true">
redo-operation="true">
<hz:cluster-routing mode="ALL_MEMBERS"/>
<hz:member>10.10.1.2:5701</hz:member>
<hz:member>10.10.1.3:5701</hz:member>
</hz:network>
Expand Down

0 comments on commit aaefef4

Please sign in to comment.