allora-network · xmariachi · Jan 28, 2025 · Jan 28, 2025 · Jan 28, 2025 · Jan 30, 2025
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -56,6 +56,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Security
 
 
+## v0.9.0
+
+### Added
+
+* [#108](https://github.com/allora-network/allora-offchain-node/pull/108) Removed ignite, GRPC/RPC, data model restructure, refactor and cleanup
+* [#98](https://github.com/allora-network/allora-offchain-node/pull/98) Multiple RPC nodes, with rpc-looping management
+
+### Changed
+
+### Fixed
+
+### Removed
+
+### Fixed
+
+### Security
+
+
 ## v0.8.1
 
 ### Added
@@ -64,7 +82,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 * [#104](https://github.com/allora-network/allora-offchain-node/pull/104) Customize rpc client configurations
 * [#105](https://github.com/allora-network/allora-offchain-node/pull/105) Make HTTP timeout configurable
 * [#106](https://github.com/allora-network/allora-offchain-node/pull/106) Improve log context by adding topic id and actor type to logger
-* [#98](https://github.com/allora-network/allora-offchain-node/pull/98) RPC nodes as an array, with rpc-looping management
 
 ### Changed
 

diff --git a/README.md b/README.md
@@ -58,17 +58,32 @@ chmod +x start.local
 ```
 
 ## Prometheus Metrics
-Some metrics has been provided for in the node. You can access them with port `:2112/metrics`. Here are the following list of existing metrics: 
+
+Some metrics has been provided for in the node. You can access them with port `:2112/metrics`. Here are the following list of existing metrics:
+
 - `allora_worker_inference_request_count`: The total number of times worker requests inference from source
 - `allora_worker_forecast_request_count`: The total number of times worker requests forecast from source
 - `allora_reputer_truth_request_count`: The total number of times reputer requests truth from source
 - `allora_worker_data_build_count`: The total number of times worker built data successfully
 - `allora_reputer_data_build_count`: The total number of times reputer built data successfully
 - `allora_worker_chain_submission_count`: The total number of worker commits to the chain
 - `allora_reputer_chain_submission_count`: The total number of reputer commits to the chain
+- `allora_application_finished_count`: The total number of application runs finished
+- `allora_worker_process_finished_count`: The total number of worker processes finished
+- `allora_reputer_process_finished_count`: The total number of reputer processes finished
 
 > Please note that we will keep updating the list as more metrics are being added
 
+## Architecture
+
+![Architecture](./assets/architecture.png)
+
+The Offchain Node is currently separated into two main packages:
+- __lib__: the Allora Client - this may be eventually separated into the Allora Go SDK to use the client separately
+- __usecase__: the particular usecase for the Offchain Node, ie. launching the app, read user config, and launching routines for workers and reputers as configured per user.
+
+Each routine uses the chain interactions to communicate with the chain, by use of RPC (txs) and GRPC (queries) in each case.
+
 ## Cycle Example
 
 Visualization of timeline and submission windows for an actor, in this case, a worker. 
@@ -160,28 +175,32 @@ The node will use the following timeouts:
 * `timeoutRPCSecondsRegistration`: Timeout for whole RPC registration process in seconds, including retries.
 * `timeoutHTTPConnection`: Timeout for HTTP connection (underlying to the RPC client) in seconds.
 
-### RPC nodes
+### GRPC / RPC connections
+
+From v0.9.0, the node supports multiple GRPC / RPC connections.
+GRPC is used for queries, while RPC is used for transactions. At least one GRPC and one RPC node must be provided.
+GRPC nodes are configured in the `nodeGrpcs` array in the config.json file.
+RPC nodes are configured in the `nodeRpcs` array in the config.json file.
 
-From v0.7.1, the RPC nodes to connect to are configured as an array `nodeRpcs` in the config.json file, instead of a single string.
-The offchain node will try to connect to the RPC nodes in order of appearance in the array  .
-If the node is unable to connect to an RPC node, or the node has some particular type of error(e.g. a full mempool, or a `429 Too Many Requests` error), it will switch to the next one in the array.
-If the node has tried all the RPC nodes in the array, and there was an error, it will log an error for that submission and will not try to submit that payload again.
+The offchain node (via the `ConnectionManager`) will try to connect to the GRPC/RPC nodes in order of appearance in the array and then go switching when appropriate for error handling and spreading the load.
 
 ### Error handling
 
 Error handling is done differently for different types of errors.
 Note: when an account sequence mismatch is detected, the node will attempt to set the expected sequence number and retry the transaction before broadcasting.
 Note: the node will check if the actor is whitelisted on worker setup and before submitting a payload.
 
-#### Retries 
-- `accounSequenceRetryDelay`: For the "account sequence mismatch" error. 
-- `retryDelay`: For all other errors that need retry delays.
+#### Retries and Delays
 
+- `accounSequenceRetryDelay`: For the "account sequence mismatch" error.
+- `retryDelay`: For all other errors that need retry delays.
+- `launchRoutineDelay`: The start of the routine performs many chain interactions. This often triggers "429 Too Many Requests" error. This delay is used to avoid this error, and is set to 10 seconds by default.
 
 ### Smart Window Detection
 
 The node will automatically detect the submission window length for each topic on each actor type.
 This can be configured by the following settings in the config.json:
+
 * `blockDurationEstimated`: Estimated network block time in seconds. Minimum is 1.
 * `windowCorrectionFactor`: Correction factor to fine-tune the submission window length. Higher values optimize the number of calls for window checking. Minimum is 0.5.
 

diff --git a/assets/architecture.png b/assets/architecture.png
diff --git a/config.example.json b/config.example.json
@@ -1,17 +1,22 @@
 {
     "wallet": {
+      "chainId": "allora-testnet-1",
+      "keyringBackend": "test",
+      "keyringPassphrase": "",
       "addressKeyName": "testkey",
       "addressRestoreMnemonic": "your mnemonic here",
       "alloraHomeDir": "",
-      "gas": "auto",
-      "gasAdjustment": 1.2,
       "gasPrices": "auto",
-      "gasPriceUpdateInterval": 60,
-      "maxFees": 500000,
+      "gasPriceUpdateInterval": 10,
+      "maxFees": 50000000,
+      "BaseGas": 200000,
+      "GasPerByte": 1,
       "nodeRpcs": ["https://allora-rpc.testnet.allora.network"],
+      "nodegRpcs": ["allora-grpc.testnet.allora.network:443"],
       "maxRetries": 5,
       "retryDelay": 3,
       "accountSequenceRetryDelay": 5,
+      "launchRoutineDelay": 5,
       "submitTx": true,
       "blockDurationEstimated": 10,
       "windowCorrectionFactor": 0.8,

diff --git a/gas_and_fees.md b/gas_and_fees.md
@@ -11,8 +11,25 @@ Allora Network implements the [Feemarket](https://github.com/skip-mev/feemarket)
 
 ## Settings
 
-- `gas` (string): can be set to `auto` or a specific gas value. If set to `auto`, the node will automatically calculate the gas limit based on the estimated gas used by the transactions. Recommended: `auto`.
-- `gasAdjustment` (float): is used to adjust the gas limit. Recommended: `1.0-1.2`.
-- `gasPrices` (string): can be set to `auto` or a specific gas price. If set to `auto`, the node will automatically calculate the gas price based on chain's feemarket-based gas prices. This is the recommended setting, since feemarket introduces variability in the gas price in the chain. Recommended: `auto`.
-- `gasPriceUpdateInterval` (int): is the interval in seconds at which the node will update the gas price from the network. This is only relevant when `gasPrices` is set to `auto`. Recommended: `60`.
+### Fee and gas calculation
+- `BaseGas` (int): set the base gas for a transaction. This is the minimum gas that will be used for a transaction. Recommended: `200000`.
+- `GasPerByte` (int): set the gas price per byte for a transaction. This is the price of the transaction in `uallo` per byte. Recommended: `1`.
 - `maxFees` (float): set the max fees that can be paid for a transaction. They are expressed numerically in `uallo`. It is recommended to adjust this value based on experience to optimize results, although a conservative value of `500000` could be a good starting point.
+
+### Gas prices
+The chain's feemarket module specifies the variable gas price. The offchain node may configure a fixed price or set it to auto, in which case the node will automatically calculate the gas price based on chain's feemarket-based gas prices. This is the recommended setting.
+- `gasPrices` (string): can be set to `auto` or a specific gas price. If set to `auto`, the node will automatically calculate the gas price based on chain's feemarket-based gas prices. This is the recommended setting, since feemarket introduces variability in the gas price in the chain. Recommended: `auto`.
+- `gasPriceUpdateInterval` (int): is the interval in seconds at which the node will update the gas price from the network. This is only relevant when `gasPrices` is set to `auto`. Recommended: similar to estimated block duration. It can vary greatly, so it is recommended to set it to a conservative value and adjust accordingly.
+
+
+## Adjustments
+
+### Out of gas
+A transaction must define its gas limit. In case of a failure, the node will parse the `used` amount expressed by the chain plus a small gas excess times number of iterations to help retrials.
+
+### Insufficient fees
+If gas is OK, the transaction may incur in an insufficient fees error. This means the gas price at which gas is paid is not enough to pay for the transaction as per current gas prices in the chain.
+The node will retry with the amount expressed by the chain, provided it is lower than the `maxFees` set by the user.
+
+
+
diff --git a/go.mod b/go.mod
@@ -14,13 +14,14 @@ require (
 	github.com/allora-network/allora-chain v0.8.1
 	github.com/cometbft/cometbft v0.38.15
 	github.com/cosmos/cosmos-sdk v0.50.11
-	github.com/ignite/cli/v28 v28.6.1
+	github.com/cosmos/gogoproto v1.7.0
 	github.com/joho/godotenv v1.5.1
 	github.com/prometheus/client_golang v1.20.5
 	github.com/rs/zerolog v1.33.0
 	github.com/skip-mev/feemarket v1.1.1
 	github.com/stretchr/testify v1.10.0
 	golang.org/x/exp v0.0.0-20250106191152-7588d65b2ba8
+	google.golang.org/grpc v1.67.1
 )
 
 require (
@@ -39,8 +40,7 @@ require (
 	github.com/Microsoft/go-winio v0.6.1 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/bgentry/speakeasy v0.1.1-0.20220910012023-760eaf8b6816 // indirect
-	github.com/blang/semver/v4 v4.0.0 // indirect
-	github.com/cenkalti/backoff v2.2.1+incompatible // indirect
+	github.com/bufbuild/protocompile v0.14.0 // indirect
 	github.com/cenkalti/backoff/v4 v4.3.0 // indirect
 	github.com/cespare/xxhash/v2 v2.3.0 // indirect
 	github.com/cockroachdb/apd/v3 v3.2.1 // indirect
@@ -56,7 +56,6 @@ require (
 	github.com/cosmos/cosmos-proto v1.0.0-beta.5 // indirect
 	github.com/cosmos/go-bip39 v1.0.0 // indirect
 	github.com/cosmos/gogogateway v1.2.0 // indirect
-	github.com/cosmos/gogoproto v1.7.0 // indirect
 	github.com/cosmos/iavl v1.2.2 // indirect
 	github.com/cosmos/ics23/go v0.11.0 // indirect
 	github.com/cosmos/ledger-cosmos-go v0.13.3 // indirect
@@ -96,6 +95,7 @@ require (
 	github.com/hashicorp/go-immutable-radix v1.3.1 // indirect
 	github.com/hashicorp/go-metrics v0.5.3 // indirect
 	github.com/hashicorp/go-plugin v1.6.0 // indirect
+	github.com/hashicorp/go-uuid v1.0.2 // indirect
 	github.com/hashicorp/golang-lru v1.0.2 // indirect
 	github.com/hashicorp/hcl v1.0.0 // indirect
 	github.com/hashicorp/yamux v0.1.1 // indirect
@@ -142,6 +142,7 @@ require (
 	github.com/syndtr/goleveldb v1.0.1-0.20220721030215-126854af5e6d // indirect
 	github.com/tendermint/go-amino v0.16.0 // indirect
 	github.com/tidwall/btree v1.7.0 // indirect
+	github.com/ugorji/go/codec v1.2.11 // indirect
 	github.com/zondax/hid v0.9.2 // indirect
 	github.com/zondax/ledger-go v0.14.3 // indirect
 	go.etcd.io/bbolt v1.4.0-alpha.0.0.20240404170359-43604f3112c5 // indirect
@@ -158,7 +159,6 @@ require (
 	google.golang.org/genproto v0.0.0-20240701130421-f6361c86f094 // indirect
 	google.golang.org/genproto/googleapis/api v0.0.0-20241015192408-796eee8c2d53 // indirect
 	google.golang.org/genproto/googleapis/rpc v0.0.0-20241007155032-5fefd90f89a9 // indirect
-	google.golang.org/grpc v1.67.1 // indirect
 	google.golang.org/protobuf v1.35.1 // indirect
 	gopkg.in/ini.v1 v1.67.0 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect

diff --git a/go.sum b/go.sum
@@ -65,18 +65,13 @@ github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6r
 github.com/bgentry/speakeasy v0.1.0/go.mod h1:+zsyZBPWlz7T6j88CTgSN5bM796AkVf0kBD4zp0CCIs=
 github.com/bgentry/speakeasy v0.1.1-0.20220910012023-760eaf8b6816 h1:41iFGWnSlI2gVpmOtVTJZNodLdLQLn/KsJqFvXwnd/s=
 github.com/bgentry/speakeasy v0.1.1-0.20220910012023-760eaf8b6816/go.mod h1:+zsyZBPWlz7T6j88CTgSN5bM796AkVf0kBD4zp0CCIs=
-github.com/bits-and-blooms/bitset v1.8.0 h1:FD+XqgOZDUxxZ8hzoBFuV9+cGWY9CslN6d5MS5JVb4c=
-github.com/bits-and-blooms/bitset v1.8.0/go.mod h1:7hO7Gc7Pp1vODcmWvKMRA9BNmbv6a/7QIWpPxHddWR8=
-github.com/blang/semver/v4 v4.0.0 h1:1PFHFE6yCCTv8C1TeyNNarDzntLi7wMI5i/pzqYIsAM=
-github.com/blang/semver/v4 v4.0.0/go.mod h1:IbckMUScFkM3pff0VJDNKRiT6TG/YpiHIM2yvyW5YoQ=
 github.com/btcsuite/btcd/btcec/v2 v2.3.4 h1:3EJjcN70HCu/mwqlUsGK8GcNVyLVxFDlWurTXGPFfiQ=
 github.com/btcsuite/btcd/btcec/v2 v2.3.4/go.mod h1:zYzJ8etWJQIv1Ogk7OzpWjowwOdXY1W/17j2MW85J04=
 github.com/btcsuite/btcd/btcutil v1.1.6 h1:zFL2+c3Lb9gEgqKNzowKUPQNb8jV7v5Oaodi/AYFd6c=
 github.com/btcsuite/btcd/btcutil v1.1.6/go.mod h1:9dFymx8HpuLqBnsPELrImQeTQfKBQqzqGbbV3jK55aE=
 github.com/bufbuild/protocompile v0.14.0 h1:z3DW4IvXE5G/uTOnSQn+qwQQxvhckkTWLS/0No/o7KU=
 github.com/bufbuild/protocompile v0.14.0/go.mod h1:N6J1NYzkspJo3ZwyL4Xjvli86XOj1xq4qAasUFxGups=
 github.com/casbin/casbin/v2 v2.1.2/go.mod h1:YcPU1XXisHhLzuxH9coDNf2FbKpjGlbCg3n9yuLkIJQ=
-github.com/cenkalti/backoff v2.2.1+incompatible h1:tNowT99t7UNflLxfYYSlKYsBpXdEet03Pg2g16Swow4=
 github.com/cenkalti/backoff v2.2.1+incompatible/go.mod h1:90ReRw6GdpyfrHakVjL/QHaoyV4aDUVVkXQJJJ3NXXM=
 github.com/cenkalti/backoff/v4 v4.1.1/go.mod h1:scbssz8iZGpm3xbr14ovlUdkxfGXNInqkPWOWmG2CLw=
 github.com/cenkalti/backoff/v4 v4.3.0 h1:MyRJ/UdXutAwSAT+s3wNd7MfTIcy71VQueUuFK343L8=
@@ -399,8 +394,6 @@ github.com/hudl/fargo v1.3.0/go.mod h1:y3CKSmjA+wD2gak7sUSXTAoopbhU08POFhmITJgmK
 github.com/iancoleman/strcase v0.3.0 h1:nTXanmYxhfFAMjZL34Ov6gkzEsSJZ5DbhxWjvSASxEI=
 github.com/iancoleman/strcase v0.3.0/go.mod h1:iwCmte+B7n89clKwxIoIXy/HfoL7AsD47ZCWhYzw7ho=
 github.com/ianlancetaylor/demangle v0.0.0-20200824232613-28f6c0f3b639/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
-github.com/ignite/cli/v28 v28.6.1 h1:PmlZ7ulD/Y9yw3QZ2EN0joMBEyGIrJbEyVSxNywEeMI=
-github.com/ignite/cli/v28 v28.6.1/go.mod h1:HzT0fYcTX1q8rbgm/OTxTwHl0rsJSOxJ9O3urSNu/6Y=
 github.com/improbable-eng/grpc-web v0.15.0 h1:BN+7z6uNXZ1tQGcNAuaU1YjsLTApzkjt2tzCixLaUPQ=
 github.com/improbable-eng/grpc-web v0.15.0/go.mod h1:1sy9HKV4Jt9aEs9JSnkWlRJPuPtwNr0l57L4f878wP8=
 github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
@@ -683,7 +676,6 @@ github.com/tidwall/btree v1.7.0 h1:L1fkJH/AuEh5zBnnBbmTwQ5Lt+bRJ5A8EWecslvo9iI=
 github.com/tidwall/btree v1.7.0/go.mod h1:twD9XRA5jj9VUQGELzDO4HPQTNJsoWWfYEL+EUQ2cKY=
 github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
 github.com/tv42/httpunix v0.0.0-20150427012821-b75d8614f926/go.mod h1:9ESjWnEqriFuLhtthL60Sar/7RFoluCcXsuvEwTV5KM=
-github.com/ugorji/go v1.1.7 h1:/68gy2h+1mWMrwZFeD1kQialdSzAb432dtpeJ42ovdo=
 github.com/ugorji/go v1.1.7/go.mod h1:kZn38zHttfInRq0xu/PH0az30d+z6vm202qpg1oXVMw=
 github.com/ugorji/go/codec v1.1.7/go.mod h1:Ax+UKWsSmolVDwsd+7N3ZtXu+yMGCf907BLYF3GoBXY=
 github.com/ugorji/go/codec v1.2.11 h1:BMaWp1Bb6fHwEtbplGBGJ498wD+LKlNSl25MjdZY4dU=

diff --git a/lib/README.md b/lib/README.md
@@ -0,0 +1,65 @@
+
+# `lib` Package Documentation
+
+## Overview
+
+The `lib` package provides core functionality for the Allora off-chain node, handling configurations, RPC/GRPC connections, wallet management, and metrics collection. It serves as the foundation layer for the application's infrastructure.
+
+## Core Components
+
+### Configuration
+
+- `WalletConfig`: Manages wallet-related configurations including:
+  - Address management (key name, mnemonic)
+  - Gas settings
+  - Chain configuration
+  - RPC endpoints
+- `WorkerConfig`: Handles worker-specific settings, including:
+  - Inference endpoints
+  - Forecast configurations
+  - Topic management
+- `ReputerConfig`: Handles reputer-specific settings, including:
+  - Ground Truth endpoint
+  - IsNegative loss function endpoint
+  - Topic management
+- `ChainConfig`: Handles chain-specific settings, including:
+  - Chain ID
+  - Chain endpoints
+  - Chain gas prices
+
+
+### Connection Management
+
+The lib package performs queries through GRPC and transactions through RPC. 
+
+- `ConnectionManager`: Thread-safe manager for multiple RPC/GRPC connections
+  - Handles both query and transaction nodes separately
+  - Provides automatic failover and switching between nodes
+  - Manages connection lifecycle
+  - A ConnectionManager maintains a Wallet instance and a WalletConfig.
+- `NodeConfig`: Individual node configuration and connection management. The same NodeConfig is used for both query and transaction nodes for simplicity, although they use different clients and their interfaces are different.
+
+- GRPC/RPC operations include retry mechanisms, both intra-node and inter-node(switching between nodes). Different types of errors are handled differently to optimize retry usage via a comprehensive error handling mechanism via ABCI and HTTP error codes capturing and falling back to string matching.
+
+### Wallet Management
+
+- Thread-safe wallet implementation.
+- Once initialized, the wallet is expected to only change its sequence number, which is done in a thread-safe manner.
+
+### Metrics
+
+- Singleton metrics collector for Prometheus integration
+- Thread-safe counter management
+- Custom metric registration
+
+### Subpackages
+
+Subpackages are used to encapsulate functionality that is not directly Allora. Eg. rpcclient, grpcclient, transaction boilerplate code, etc.
+
+## Architecture Decisions
+
+1. Singleton pattern for metrics to ensure single source of truth
+2. Separation of query and transaction nodes for better reliability
+3. Thread-safe implementations for concurrent operations
+4. Clear separation between configuration and runtime components
+
diff --git a/lib/auth/codec_keyring.go b/lib/auth/codec_keyring.go
@@ -0,0 +1,24 @@
+package auth
+
+import (
+	"sync"
+
+	"github.com/cosmos/cosmos-sdk/codec"
+	codectypes "github.com/cosmos/cosmos-sdk/codec/types"
+	cryptocodec "github.com/cosmos/cosmos-sdk/crypto/codec"
+)
+
+var (
+	keyringCodec     codec.Codec
+	keyringCodecOnce sync.Once
+)
+
+// GetKeyringCodec returns a singleton instance of the keyring codec
+func GetKeyringCodec() codec.Codec {
+	keyringCodecOnce.Do(func() {
+		registry := codectypes.NewInterfaceRegistry()
+		cryptocodec.RegisterInterfaces(registry)
+		keyringCodec = codec.NewProtoCodec(registry)
+	})
+	return keyringCodec
+}