From 45d1da52e2457e3016fb1d26e1cdedf3ada4727a Mon Sep 17 00:00:00 2001 From: Rossco99 Date: Mon, 10 Feb 2025 21:02:52 +0800 Subject: [PATCH] Updated all WAX Hyperion Technical How To guides to v3.3.10-1 (#192) * Update wax-intro-to-hyperion-full-history.md v3.3.10-1 * Update wax-build-hyperion-software-components.md v3.3.10-1 * Update wax-config-hyperion-software-components.md v.3.3.10-1 * Update wax-running-hyperion.md v3.3.10-1 * Update wax-clustering-elasticsearch-for-hyperion.md v3.3.10-1 * Update wax-rectify-missing-blocks-in-hyperion.md v3.3.10-1 * Update wax-hyperion-indexer-scaling.md v3.3.10-1 * Update wax-hyperion-streaming-load-balancer.md v3.3.10-1 * Update wax-hyperion-repair-indexed-data.md v3.3.10-1 * Update wax-hyperion-data-tiers.md - Fixed Screen Snips --- .../wax-build-hyperion-software-components.md | 27 ++++++++++++------- ...x-clustering-elasticsearch-for-hyperion.md | 2 +- ...wax-config-hyperion-software-components.md | 22 +++++++-------- .../wax-hyperion/wax-hyperion-data-tiers.md | 11 ++++---- .../wax-hyperion-indexer-scaling.md | 4 +-- .../wax-hyperion-repair-indexed-data.md | 2 +- .../wax-hyperion-streaming-load-balancer.md | 2 +- .../wax-intro-to-hyperion-full-history.md | 7 ++--- .../wax-rectify-missing-blocks-in-hyperion.md | 4 +-- .../wax-hyperion/wax-running-hyperion.md | 9 ++++--- 10 files changed, 51 insertions(+), 39 deletions(-) diff --git a/docs/en/operate/wax-hyperion/wax-build-hyperion-software-components.md b/docs/en/operate/wax-hyperion/wax-build-hyperion-software-components.md index 5538b474..466f7419 100644 --- a/docs/en/operate/wax-hyperion/wax-build-hyperion-software-components.md +++ b/docs/en/operate/wax-hyperion/wax-build-hyperion-software-components.md @@ -12,7 +12,7 @@ Once again this Technical How To series will cover some of the EOS RIO same cont ![](https://miro.medium.com/v2/resize:fit:598/0*QSOXBoNEcm0pWwSl.png) -_This article has been updated to reflect the current Hyperion deployment in September 2023._ +_This article has been updated to reflect the current Hyperion deployment in December 2024._ # Build WAX Hyperion Software Components @@ -24,11 +24,11 @@ The process for building each of these primary building blocks is covered below: ## **WAX Software State-History (SHIP) Node** -The WAX Hyperion deployment requires access to a fully sync’d WAX State-History Node, the current recommend version is `v4.0.4wax02`. This build process is extensively covered in [WAX Technical How To #7](https://medium.com/eosphere/wax-technical-how-to-7-9ccc102efd9d). +The WAX Hyperion deployment requires access to a fully sync’d WAX State-History Node, the current recommend version is `v5.0.1wax01`. This build process is extensively covered in [WAX Technical How To #7](https://medium.com/eosphere/wax-technical-how-to-7-9ccc102efd9d). ## RabbitMQ -To install the latest RabbitMQ currently `3.12.6` be sure to check their latest [Cloudsmith Quick Start Script](https://www.rabbitmq.com/install-debian.html), this in our experience is the simplest way to ensure you are current and correctly built. +To install the latest RabbitMQ currently `4.0.5` and Erlang `27.2` be sure to check their latest [Cloudsmith Quick Start Script](https://www.rabbitmq.com/install-debian.html), this in our experience is the simplest way to ensure you are current and correctly built. The summary process is below: @@ -75,9 +75,16 @@ EOF **Check Version** > sudo rabbitmqctl version ``` + +If you performed an upgrade to RabbitMQ version 4 you may need to enable all stable feature flags as below: + +``` +> sudo rabbitmqctl enable_feature_flag all +``` + ## Redis -Our current WAX Hyperion deployment is running on the latest Redis stable version `v7.2.1` which is built as below: +Our current WAX Hyperion deployment is running on the latest Redis stable version `v7.3.1` which is built as below: ``` > sudo apt install lsb-release curl gpg @@ -97,7 +104,7 @@ Our current WAX Hyperion deployment is running on the latest Redis stable versio ## Node.js -Hyperion requires Node.js v18 , our WAX Hyperion deployment is running the current LTS `v18.18.0` which is built below: +Hyperion requires Node.js v22 , our WAX Hyperion deployment is running the current LTS `v22.12.0` which is built below: ``` #Download and import the Nodesource GPG key# @@ -110,7 +117,7 @@ Hyperion requires Node.js v18 , our WAX Hyperion deployment is running the curre > curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | sudo gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg #Create .deb repository# -> NODE_MAJOR=18 +> NODE_MAJOR=22 > echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_$NODE_MAJOR.x nodistro main" | sudo tee /etc/apt/sources.list.d/nodesource.list @@ -125,7 +132,7 @@ Hyperion requires Node.js v18 , our WAX Hyperion deployment is running the curre ## PM2 -The latest public version is `5.3.0` and is built as below: +The latest public version is `5.4.3` and is built as below: ``` > sudo apt update @@ -139,7 +146,7 @@ The latest public version is `5.3.0` and is built as below: ## Elasticsearch -Currently our WAX Hyperion is using Elasticsearch `8.5.2` with great results, however the current recommended Elasticsearch version is `8.9.0` which I expect will work just as well or better. Build the latest Elasticsearch `8.x` as below: +Currently our WAX Hyperion is using Elasticsearch `8.13.2` with great results, however the current recommended Elasticsearch version is `8.17.0` which I expect will work just as well or better. Build the latest Elasticsearch `8.x` as below: ``` > sudo apt update @@ -183,7 +190,7 @@ The utilised Kibana version should be paired with the installed Elasticsearch ve ## **EOS RIO Hyperion Indexer and API** -Currently (September 2023) the most robust and production ready version of Hyperion from our experience is `3.3.9–8` and is used in our WAX Hyperion Fully History Service. The EOS RIO Team are constantly developing and improving their code, the best way to stay on top of the current recommend version is to join the [Hyperion Telegram Group](https://t.me/EOSHyperion). Build Hyperion from `main` as below: +Currently (December 2024) the most robust and production ready version of Hyperion from our experience is `3.3.10-1` and is used in our WAX Hyperion Fully History Service. The EOS RIO Team are constantly developing and improving their code, the best way to stay on top of the current recommend version is to join the [Hyperion Telegram Group](https://t.me/EOSHyperion). Build Hyperion from `main` as below: ``` > git clone https://github.com/eosrio/hyperion-history-api.git @@ -197,6 +204,8 @@ Currently (September 2023) the most robust and production ready version of Hyper > npm audit fix ``` +_Hyperion version '3.5.0' is expected to release in the next few weeks, this guide will be updated accordingly._ + After all Hyperion Software Components are built and provisioned you can now proceed to configuration. The next **WAX Hyperion Full History** sub article will walk through the technical configuration of each component. diff --git a/docs/en/operate/wax-hyperion/wax-clustering-elasticsearch-for-hyperion.md b/docs/en/operate/wax-hyperion/wax-clustering-elasticsearch-for-hyperion.md index 5fe3e70c..d6e02a7c 100644 --- a/docs/en/operate/wax-hyperion/wax-clustering-elasticsearch-for-hyperion.md +++ b/docs/en/operate/wax-hyperion/wax-clustering-elasticsearch-for-hyperion.md @@ -16,7 +16,7 @@ Once again this Technical How To series may cover some of EOS RIO’s same conte ![](https://miro.medium.com/v2/resize:fit:598/0*SMEjl3XTe2BtL1W3.png) -_This article has been updated to reflect the current Hyperion deployment in September 2023._ +_This article has been updated to reflect the current Hyperion deployment in December 2024._ ## Clustering Elasticsearch for WAX Hyperion Full History diff --git a/docs/en/operate/wax-hyperion/wax-config-hyperion-software-components.md b/docs/en/operate/wax-hyperion/wax-config-hyperion-software-components.md index 87667695..7a9d534a 100644 --- a/docs/en/operate/wax-hyperion/wax-config-hyperion-software-components.md +++ b/docs/en/operate/wax-hyperion/wax-config-hyperion-software-components.md @@ -12,7 +12,7 @@ Once again this Technical How To series will cover some of EOS RIO’s same cont ![](https://miro.medium.com/v2/resize:fit:598/0*e9la30mQaOGWxfmM.png) -_This article has been updated to reflect the current Hyperion deployment in September 2023._ +_This article has been updated to reflect the current Hyperion deployment in December 2024._ # Configure WAX Hyperion Software Components @@ -95,20 +95,20 @@ Debug Redis issues in the logs: ## Node.js -Nothing to configure here, however ensure you are running Node.js v16 +Nothing to configure here, however ensure you are running Node.js v22 ``` > node -v -v18.x.x +v22.x.x ``` ## PM2 -Nothing to configure here, check that you are running the latest PM2 version`5.3.0` +Nothing to configure here, check that you are running the latest PM2 version`5.4.3` ``` > pm2 --version -5.3.0 +5.4.3 ``` ## Elasticsearch @@ -194,15 +194,15 @@ Enter host password for user 'elastic': { "name" : "", "cluster_name" : "", - "cluster_uuid" : "exucKwVpRJubHGq5Jwu1_Q", + "cluster_uuid" : "36T00kPBT6qq8Tb1JYUgCw", "version" : { - "number" : "8.10.1", + "number" : "8.13.2", "build_flavor" : "default", "build_type" : "deb", - "build_hash" : "a94744f97522b2b7ee8b5dc13be7ee11082b8d6b", - "build_date" : "2023-09-14T20:16:27.027355296Z", + "build_hash" : "16cc90cd2d08a3147ce02b07e50894bc060a4cbf", + "build_date" : "2024-04-05T14:45:26.420424304Z", "build_snapshot" : false, - "lucene_version" : "9.7.0", + "lucene_version" : "9.10.0", "minimum_wire_compatibility_version" : "7.17.0", "minimum_index_compatibility_version" : "7.0.0" }, @@ -483,7 +483,7 @@ Configure as below, take note of the **#UPDATE#** parameters "root_only": false }, "scaling": { - "readers": 2, #INCREASE READERS# + "readers": 1, "ds_queues": 1, "ds_threads": 1, "ds_pool_size": 1, diff --git a/docs/en/operate/wax-hyperion/wax-hyperion-data-tiers.md b/docs/en/operate/wax-hyperion/wax-hyperion-data-tiers.md index 2617595f..c7072789 100644 --- a/docs/en/operate/wax-hyperion/wax-hyperion-data-tiers.md +++ b/docs/en/operate/wax-hyperion/wax-hyperion-data-tiers.md @@ -12,7 +12,7 @@ Once again this Technical How To series will cover some of EOS RIO’s same cont [Learn more about EOS RIO Hyperion](https://eosrio.io/hyperion/) -![image](https://github.com/user-attachments/assets/2e4b17c5-8d89-49f1-bf97-76e39748ab91) +![image](https://github.com/user-attachments/assets/2ce31a7d-8a87-48a4-b85a-b5bb1e0685a8) # Using Data Tiers with WAX Hyperion Full History @@ -52,7 +52,8 @@ The indices we are most interested in are the `wax-action` and `wax-delta` i > GET /_cat/indices/wax* ``` -![image](https://github.com/user-attachments/assets/e5391d4a-0129-4348-a4a4-d96a04189cc1) +![image](https://github.com/user-attachments/assets/8c46dd0d-34d4-4a9c-a058-d09b8a071b07) + The specific settings below show the existing data tier, as expected `data_content` by default. @@ -60,7 +61,7 @@ The specific settings below show the existing data tier, as expected `data_cont > GET /wax-action-v1-000023/_settings ``` -![image](https://github.com/user-attachments/assets/684b0011-bc93-40fa-a772-80f7f637622f) +![image](https://github.com/user-attachments/assets/82830550-359f-4129-94a3-403ff4803d1b) ## Set Index Tier @@ -73,7 +74,7 @@ Set the desired older index to `data_warm`. } ``` -![image](https://github.com/user-attachments/assets/a57ceafd-8036-4853-935d-10f00622fb01) +![image](https://github.com/user-attachments/assets/c9ced417-bb61-4e37-9a66-27c623c36f47) ## Observe Index Allocation @@ -88,7 +89,7 @@ wax-action-v1-000023 wax-es-node-1 -> 10.125.0.89 fQeNr0UkSXuAnofq-rceoA wax-es- wax-action-v1-000023 wax-es-node-1 -> 10.125.0.89 fQeNr0UkSXuAnofq-rceoA wax-es-node-3 ``` -![image](https://github.com/user-attachments/assets/a50ee046-b3a7-4956-882f-776b4d4e409b) +![image](https://github.com/user-attachments/assets/e68130ee-7215-4272-8451-83331242f6e7) The move process may take some time and appears to use very little resources, which is obviously great for an in production cluster. diff --git a/docs/en/operate/wax-hyperion/wax-hyperion-indexer-scaling.md b/docs/en/operate/wax-hyperion/wax-hyperion-indexer-scaling.md index d43eec8b..8b48f229 100644 --- a/docs/en/operate/wax-hyperion/wax-hyperion-indexer-scaling.md +++ b/docs/en/operate/wax-hyperion/wax-hyperion-indexer-scaling.md @@ -16,7 +16,7 @@ Once again this Technical How To series will cover some of EOS RIO’s same cont ![](https://miro.medium.com/v2/resize:fit:598/0*KXa0o0Fa-YGqItzy.png) -_This article has been updated to reflect the current Hyperion deployment in September 2023._ +_This article has been updated to reflect the current Hyperion deployment in December 2024._ # WAX Hyperion Full History Indexer Scaling @@ -141,7 +141,7 @@ Here are a few suggestions on starting your indexer for the first time and then ``` "scaling": { - "readers": 2, + "readers": 1, "ds_queues": 2, "ds_threads": 2, "ds_pool_size": 6, diff --git a/docs/en/operate/wax-hyperion/wax-hyperion-repair-indexed-data.md b/docs/en/operate/wax-hyperion/wax-hyperion-repair-indexed-data.md index 7c2db0e2..0f5f9168 100644 --- a/docs/en/operate/wax-hyperion/wax-hyperion-repair-indexed-data.md +++ b/docs/en/operate/wax-hyperion/wax-hyperion-repair-indexed-data.md @@ -18,7 +18,7 @@ Once again this Technical How To series will cover some of EOS RIO’s same cont ![](https://miro.medium.com/v2/resize:fit:598/0*GqDtvMh8QunDs6Mc.png) -_This article has been updated to reflect the current Hyperion deployment in September 2023._ +_This article has been updated to reflect the current Hyperion deployment in December 2024._ # Repair WAX Hyperion Indexed Data diff --git a/docs/en/operate/wax-hyperion/wax-hyperion-streaming-load-balancer.md b/docs/en/operate/wax-hyperion/wax-hyperion-streaming-load-balancer.md index d9955db9..82736b9b 100644 --- a/docs/en/operate/wax-hyperion/wax-hyperion-streaming-load-balancer.md +++ b/docs/en/operate/wax-hyperion/wax-hyperion-streaming-load-balancer.md @@ -17,7 +17,7 @@ Once again this Technical How To series will cover some of EOS RIO’s same cont ![](https://miro.medium.com/v2/resize:fit:598/0*VJM9YUUmYPzi9zEc.png) -_This article has been updated to reflect the current Hyperion deployment in September 2023._ +_This article has been updated to reflect the current Hyperion deployment in December 2024._ # WAX Hyperion Streaming with a WAX Load Balancer diff --git a/docs/en/operate/wax-hyperion/wax-intro-to-hyperion-full-history.md b/docs/en/operate/wax-hyperion/wax-intro-to-hyperion-full-history.md index f54c46ff..09934cbc 100644 --- a/docs/en/operate/wax-hyperion/wax-intro-to-hyperion-full-history.md +++ b/docs/en/operate/wax-hyperion/wax-intro-to-hyperion-full-history.md @@ -14,7 +14,7 @@ EOSphere have extensive production experience with providing Full History servic ![](https://miro.medium.com/v2/resize:fit:598/1*ATnQKunF5rVtPEsL-sP0JA.png) -_This article has been updated to reflect the current Hyperion deployment in September 2023._ +_This article has been updated to reflect the current Hyperion deployment in December 2024._ # Introduction to WAX Hyperion Full History @@ -131,13 +131,14 @@ The recommendations below are for WAX Mainnet and WAX Testnet with Full History, * Modern CPU, 3Ghz+, 8 Cores+ * 64GB RAM * Enterprise Grade SSD/NVMe - _The current (September 2023) Elasticsearch Database is 26TB, I suggest provisioning 35–40TB across the cluster for Full History service longevity_ + _The current (December 2024) Elasticsearch Database is 32TB, I suggest provisioning 40–45TB across the cluster for Full History service longevity_ * 1Gb/s+ LAN * **WAX State-History** * Modern CPU, 4Ghz+, 4 Cores * 128GB RAM * 256GB DISK 1 _(Enterprise Grade SSD/NVMe)_ - * 16TB DISK 2 _(SAS or SATA are OK)_ + * 18TB+ DISK 2 _(SAS or SATA are OK)_ + * 1Gb/s+ LAN **WAX Testnet** diff --git a/docs/en/operate/wax-hyperion/wax-rectify-missing-blocks-in-hyperion.md b/docs/en/operate/wax-hyperion/wax-rectify-missing-blocks-in-hyperion.md index 6863884a..64d249b9 100644 --- a/docs/en/operate/wax-hyperion/wax-rectify-missing-blocks-in-hyperion.md +++ b/docs/en/operate/wax-hyperion/wax-rectify-missing-blocks-in-hyperion.md @@ -3,7 +3,7 @@ title: Rectify Missing Blocks in WAX Hyperion Full History --- Initially indexing the complete WAX Mainnet from genesis will take time and it can be quite heavy on hardware resources causing pauses or stops in indexing, furthermore if you have taken our suggestion on batching block ranges to index you may encounter a scenario where you have gaps in your full history indexed blocks set. -This next sub-article in the series will go through the process of identifying gaps in your full history blocks set and how to manually correct them. +This next sub-article in the series will go through the process of identifying gaps in your full history blocks set and how to *manually* correct them. EOS RIO have an excellent [Hyperion Documentation Repository](https://hyperion.docs.eosrio.io/) including details on how to run their Hyperion Full History product, however this article expands on their documentation specifically in relation to operating Hyperion using Elasticsearch 8.x @@ -13,7 +13,7 @@ Once again this Technical How To series will cover some of EOS RIO’s same cont ![](https://miro.medium.com/v2/resize:fit:449/0*pY_GWzqlqGfAO7Kf.png) -_This article has been updated to reflect the current Hyperion deployment in September 2023._ +_This article has been updated to reflect the current Hyperion deployment in December 2024._ ## Rectify Missing Blocks in WAX Hyperion Full History diff --git a/docs/en/operate/wax-hyperion/wax-running-hyperion.md b/docs/en/operate/wax-hyperion/wax-running-hyperion.md index 9a4f1816..32ae7035 100644 --- a/docs/en/operate/wax-hyperion/wax-running-hyperion.md +++ b/docs/en/operate/wax-hyperion/wax-running-hyperion.md @@ -12,7 +12,7 @@ Once again this Technical How To series will cover some of EOS RIO’s same cont ![](https://miro.medium.com/v2/resize:fit:598/0*K8f_xpc-l5NSUL7H.png) -_This article has been updated to reflect the current Hyperion deployment in September 2023._ +_This article has been updated to reflect the current Hyperion deployment in December 2024._ # Running WAX Hyperion Full History @@ -170,7 +170,7 @@ Below is the initial configuration used for `wax.config.json` : "root_only": false }, "scaling": { - "readers": 2, #INCREASE READERS# + "readers": 1, "ds_queues": 1, "ds_threads": 1, "ds_pool_size": 1, @@ -252,7 +252,7 @@ Make sure the following is configured or amended in the `wax.config.json` : "live_reader": false, #DISABLED FOR BULK INDEXING# "abi_scan_mode": false, #SET FOR INDEXING PHASE# "scaling": { #CONSERVATIVE SETTINGS# - "readers": 2, + "readers": 1, "ds_queues": 1, "ds_threads": 2, "ds_pool_size": 2, @@ -366,8 +366,9 @@ In particular make sure `last_indexed_block` is equal to `total_indexed_block ``` > curl [http://:7000/v2/health](https://wax-testnet.eosphere.io/v2/health) -{"version":"3.3.9-8","version_hash":"b94f99d552a8fe85a3ab2c1cb5b84ccd6ded6af4","host":"wax-testnet.eosphere.io","health":[{"service":"RabbitMq","status":"OK","time":1695700845755},{"service":"NodeosRPC","status":"OK","service_data":{"head_block_num":268459315,"head_block_time":"2023-09-26T04:00:45.500","time_offset":210,"last_irreversible_block":268458983,"chain_id":"1064487b3cd1a897ce03ae5b6a865651747e2e152090f99c1d19d44e01aea5a4"},"time":1695700845710},{"service":"Elasticsearch","status":"OK","service_data":{"active_shards":"100.0%","head_offset":2,"first_indexed_block":2,"last_indexed_block":268459313,"total_indexed_blocks":268459311,"missing_blocks":0,"missing_pct":"0.00%"},"time":1695700845712}],"features":{"streaming":{"enable":true,"traces":true,"deltas":true},"tables":{"proposals":true,"accounts":true,"voters":true},"index_deltas":true,"index_transfer_memo":true,"index_all_deltas":true,"deferred_trx":false,"failed_trx":false,"resource_limits":false,"resource_usage":false},"cached":true,"query_time_ms":0.158,"last_indexed_block":268459318,"last_indexed_block_time":"2023-09-26T04:00:47.000"} +{"version":"3.3.10","version_hash":"3c24fea14d424eeef5fe2719238ef4e2ee65dc4e","host":"wax-testnet.eosphere.io","health":[{"service":"RabbitMq","status":"OK","time":1735014759377},{"service":"NodeosRPC","status":"OK","service_data":{"head_block_num":347074519,"head_block_time":"2024-12-24T04:32:39.500","time_offset":-165,"last_irreversible_block":347074187,"chain_id":"f16b1833c747c43682f4386fca9cbb327929334a762755ebec17f6f23c9b8a12"},"time":1735014759335},{"service":"Elasticsearch","status":"OK","service_data":{"active_shards":"100.0%","head_offset":3,"first_indexed_block":2,"last_indexed_block":347074516,"total_indexed_blocks":347074514,"missing_blocks":0,"missing_pct":"0.00%"},"time":1735014759338}],"features":{"streaming":{"enable":true,"traces":true,"deltas":true},"tables":{"proposals":true,"accounts":true,"voters":true},"index_deltas":true,"index_transfer_memo":true,"index_all_deltas":true,"deferred_trx":false,"failed_trx":false,"resource_limits":false,"resource_usage":false},"cached":true,"query_time_ms":0.323,"last_indexed_block":347074522,"last_indexed_block_time":"2024-12-24T04:32:41.000"} ``` + Congratulations you have now successfully built, configured and are running a WAX Hyperion Full History Service, ready to be made publicly available from behind a SSL offloading Load Balancer such as HAProxy. The next **WAX Hyperion Full History** sub article with cover missing blocks troubleshooting, Elasticsearch monitoring as well as ensuring Hyperion web services are available.