Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PROTO-3189 Removed ignite, GRPC/RPC, data model restructure, refactor and cleanup / PROTO-3037 #108

Open
wants to merge 21 commits into
base: dev
Choose a base branch
from

Conversation

xmariachi
Copy link
Collaborator

@xmariachi xmariachi commented Jan 28, 2025

Purpose of Changes and their Description

  • Removing ignite
  • Using rpc and grpc as tx/query servers
  • Restructuring for a better data model and to avoid innecessary duplications. Among others
    • 1 wallet instead of N wallets, one per topic, which did not make sense. Definition of new Wallet to keep all data and functions re. wallet.
    • Duplication of Worker and Reputer in UserConfig and ChainConfig, where ultimately were not used
    • Node maintains node-related data only, cleaned from other data.
    • Centralization of RPC manager for wallet and wallet configs, since only one is needed.
  • Realignment for usecase/lib.

TESTING

Testing has been launching it manually via multi-topic, with 1 woker and reputer per topic, configuration. Using testnet endpoints from Foundation, L5 and Polkachu.
The only errors that have been observed are the expected account sequence mismatch which are working fine on retry.
One time it has been observed that
No other errors observed during several hours, which adds to the previous #98 testing (which provides the connection multi-connection backbone) which was run over a weekend with the same results, so confidence is high.

Also of course existent unit tests and mocks have been modified accordingly.

Are these changes tested and documented?

  • If tested, please describe how. If not, why tests are not needed. -- See Testing section above
  • If documented, please describe where. If not, describe why docs are not needed. -- Modified README and added lib/README
  • Added to Unreleased section of CHANGELOG.md?

@xmariachi xmariachi changed the title WIP removing ignite, grpc/rpc, TODO restructuring 1 wallet WIP removing ignite, grpc/rpc, TODO restructuring 1 wallet, clean up, and more Jan 28, 2025
@xmariachi xmariachi marked this pull request as ready for review January 30, 2025 18:47
@xmariachi xmariachi requested a review from amimart January 30, 2025 18:48
@felipead
Copy link
Collaborator

It would be nice to see some automated unit or integration tests, or at the very least document how you tested things manually, including testing scenarios, etc...

These are very profound changes...

@xmariachi xmariachi changed the title WIP removing ignite, grpc/rpc, TODO restructuring 1 wallet, clean up, and more Removed ignite, GRPC/RPC, big data restructure, refactor and cleanup Jan 31, 2025
@xmariachi xmariachi changed the title Removed ignite, GRPC/RPC, big data restructure, refactor and cleanup Removed ignite, GRPC/RPC, data model restructure, refactor and cleanup Jan 31, 2025
@xmariachi
Copy link
Collaborator Author

It would be nice to see some automated unit or integration tests, or at the very least document how you tested things manually, including testing scenarios, etc...

These are very profound changes...

Update descrption with more details about the testing done.
Agree on the integration testing, this has been an ongoing thing missing in this repo. Because of its reliance on the chain responses and data, this repo's integration tests could be very beneficial.

lib/rpcclient/keys.go Outdated Show resolved Hide resolved
lib/metrics_config.go Outdated Show resolved Hide resolved
Comment on lines +47 to +80
// TODO: Investigate and ecide if we want to keep this
// spin up goroutine for monitoring and reconnect purposes - TODO test and configure
// go func() {
// for {
// state := chain.Client.GRPCClient.GetState()
// if state == connectivity.TransientFailure || state == connectivity.Shutdown {
// fmt.Println("GRPC Connection lost, attempting to reconnect...")
// for {
// if chain.Client.GRPCClient.WaitForStateChange(context.Background(), state) {
// break
// }
// time.Sleep(10 * time.Second)
// }
// }
// time.Sleep(10 * time.Second)
// }
// }()
return grpcConnection, nil
}

// An interceptor that logs the gRPC request, for debugging purposes
// func loggerHeaderInterceptor() grpc.UnaryClientInterceptor {
// return func(
// ctx context.Context,
// method string,
// req, reply interface{},
// cc *grpc.ClientConn,
// invoker grpc.UnaryInvoker,
// opts ...grpc.CallOption,
// ) error {
// log.Info().Str("method", method).Msg("Intercepting gRPC request")
// return invoker(ctx, method, req, reply, cc, opts...)
// }
// }
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what are pros/cons for including?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left code in for ease of use on debugging. But there is no need to set up a headerInterceptor on prod code if nothing is done with it. It could just be deleted though, since it is already in the codebase history if needed :D

Re. the commented goroutine for the grpc conn health, it is something I've seen in other places. Works with and without it for what I've seen, so I'm reluctant to add it without fully getting the pro/cons of it.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@felipead @spooktheducks @fernandofcampos you may have opinions here

lib/errors.go Show resolved Hide resolved
lib/wallet.go Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probs should also be tied to the wallet. perhaps there should be a distinct wallet package in our golang allora sdk that groups all the functionality i'm calling out. then this repo and others can utilize it. if you agree, please add these notes to the go-sdk project

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I called it auth and the Wallet and WalletConfig may be in. But yeah, its name may also be wallet if so, for clarity, since it's the major guest there.

@xmariachi xmariachi changed the title Removed ignite, GRPC/RPC, data model restructure, refactor and cleanup PROTO-3189 Removed ignite, GRPC/RPC, data model restructure, refactor and cleanup / PROTO-3037 Feb 4, 2025
@xmariachi
Copy link
Collaborator Author

xmariachi commented Feb 4, 2025

Pushed new changes:

  • Multi-backend configuration, tested with os/file/test, with Passphrase. This is relevant to configs, but default is still test with empty passphrase.
  • Added Gas parsing. Calculations to estimate BaseGas using "used" and "wanted" to guess the tx-related gas, with this to get the expected BaseGas, and use it with small iterative correction.
  • Fees retrial implementing PROTO-3037 (check what fee is expected instead of iterate with small increases).
  • Fixing context cancellation to use signals correctly.

cc @kpeluso @spooktheducks @felipead @clementupshot

@@ -160,8 +78,30 @@ func (node *NodeConfig) SendDataWithRetry(ctx context.Context, req sdktypes.Msg,
case ErrorProcessingFees:
// Error has not been handled, just mark as recalculate fees on this iteration
log.Info().Msg("Insufficient fees, marking fee recalculation on tx broadcasting for retrial")
recalculateFees += 1
// TODO Handle fee and "out of gas" error differently
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how "differently"?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants