Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(config)_: integrate new rpc providers configs #6178

Open
wants to merge 6 commits into
base: develop
Choose a base branch
from

Conversation

friofry
Copy link
Contributor

@friofry friofry commented Dec 6, 2024

This is the follow-up PR for the improved RPC configuration. See description: #6151

@status-im-auto
Copy link
Member

status-im-auto commented Dec 6, 2024

Jenkins Builds

Click to see older builds (56)
Commit #️⃣ Finished (UTC) Duration Platform Result
✔️ dc9415f #1 2024-12-06 13:16:17 ~4 min macos 📦zip
✔️ dc9415f #1 2024-12-06 13:16:28 ~4 min linux 📦zip
✔️ dc9415f #1 2024-12-06 13:16:30 ~4 min ios 📦zip
✖️ dc9415f #1 2024-12-06 13:16:35 ~4 min tests 📄log
✔️ dc9415f #1 2024-12-06 13:16:36 ~4 min windows 📦zip
✔️ dc9415f #1 2024-12-06 13:17:45 ~6 min android 📦aar
✖️ dc9415f #1 2024-12-06 13:18:36 ~6 min tests-rpc 📄log
✔️ dc9415f #1 2024-12-06 13:22:02 ~10 min macos 📦zip
✖️ e0e484f #2 2024-12-06 13:20:51 ~1 min tests 📄log
✔️ e0e484f #2 2024-12-06 13:22:37 ~3 min windows 📦zip
✔️ e0e484f #2 2024-12-06 13:23:45 ~4 min ios 📦zip
✔️ e0e484f #2 2024-12-06 13:25:25 ~5 min android 📦aar
✖️ e0e484f #2 2024-12-06 13:25:38 ~6 min tests-rpc 📄log
✔️ e0e484f #2 2024-12-06 13:25:45 ~6 min linux 📦zip
✔️ e0e484f #2 2024-12-06 13:25:54 ~6 min macos 📦zip
✔️ e0e484f #2 2024-12-06 13:30:46 ~8 min macos 📦zip
✖️ d2795bc #3 2024-12-06 13:25:08 ~1 min tests 📄log
✔️ d2795bc #3 2024-12-06 13:26:35 ~3 min windows 📦zip
✔️ d2795bc #3 2024-12-06 13:27:46 ~3 min ios 📦zip
✔️ d2795bc #3 2024-12-06 13:30:06 ~4 min macos 📦zip
✔️ d2795bc #3 2024-12-06 13:31:13 ~5 min android 📦aar
✔️ d2795bc #3 2024-12-06 13:31:24 ~5 min linux 📦zip
✖️ d2795bc #3 2024-12-06 13:31:35 ~5 min tests-rpc 📄log
✔️ d2795bc #3 2024-12-06 13:39:04 ~8 min macos 📦zip
526f19c #4 2025-01-07 14:37:19 ~25 sec windows 📄log
✖️ 526f19c #4 2025-01-07 14:38:26 ~1 min tests 📄log
✔️ 526f19c #4 2025-01-07 14:41:38 ~4 min macos 📦zip
✔️ 526f19c #4 2025-01-07 14:41:39 ~4 min linux 📦zip
✔️ 526f19c #4 2025-01-07 14:42:00 ~5 min android 📦aar
✔️ 526f19c #4 2025-01-07 14:42:02 ~5 min ios 📦zip
✖️ 526f19c #4 2025-01-07 14:42:50 ~5 min tests-rpc 📄log
✔️ 526f19c #4 2025-01-07 14:45:53 ~9 min macos 📦zip
54f6a89 #5 2025-01-09 13:35:45 ~25 sec windows 📄log
✖️ 54f6a89 #5 2025-01-09 13:36:48 ~1 min tests 📄log
✔️ 54f6a89 #5 2025-01-09 13:40:23 ~5 min macos 📦zip
✔️ 54f6a89 #5 2025-01-09 13:40:38 ~5 min linux 📦zip
✔️ 54f6a89 #5 2025-01-09 13:40:39 ~5 min ios 📦zip
✔️ 54f6a89 #5 2025-01-09 13:40:45 ~5 min android 📦aar
✖️ 54f6a89 #5 2025-01-09 13:41:21 ~6 min tests-rpc 📄log
✔️ 54f6a89 #5 2025-01-09 13:44:36 ~9 min macos 📦zip
54f6a89 #6 2025-01-10 22:02:59 ~26 sec windows 📄log
✖️ 54f6a89 #6 2025-01-10 22:03:58 ~1 min tests 📄log
✖️ 54f6a89 #6 2025-01-10 22:07:08 ~4 min tests-rpc 📄log
✔️ 54f6a89 #6 2025-01-10 22:07:33 ~5 min linux 📦zip
✔️ 54f6a89 #6 2025-01-10 22:07:39 ~5 min macos 📦zip
✔️ 54f6a89 #6 2025-01-10 22:07:55 ~5 min ios 📦zip
✔️ 54f6a89 #6 2025-01-10 22:08:13 ~5 min android 📦aar
✔️ 54f6a89 #6 2025-01-10 22:12:46 ~10 min macos 📦zip
8582775 #7 2025-01-13 12:02:36 ~25 sec windows 📄log
✖️ 8582775 #7 2025-01-13 12:03:38 ~1 min tests 📄log
✔️ 8582775 #7 2025-01-13 12:07:04 ~5 min linux 📦zip
✔️ 8582775 #7 2025-01-13 12:07:20 ~5 min android 📦aar
✔️ 8582775 #7 2025-01-13 12:07:21 ~5 min ios 📦zip
✔️ 8582775 #7 2025-01-13 12:07:27 ~5 min macos 📦zip
✖️ 8582775 #7 2025-01-13 12:07:55 ~5 min tests-rpc 📄log
✔️ 8582775 #7 2025-01-13 12:12:08 ~10 min macos 📦zip
Commit #️⃣ Finished (UTC) Duration Platform Result
0945def #8 2025-01-13 12:16:49 ~24 sec windows 📄log
✖️ 0945def #8 2025-01-13 12:17:37 ~1 min tests 📄log
✔️ 0945def #8 2025-01-13 12:21:07 ~4 min linux 📦zip
✔️ 0945def #8 2025-01-13 12:21:40 ~5 min macos 📦zip
✖️ 0945def #8 2025-01-13 12:21:41 ~5 min tests-rpc 📄log
✔️ 0945def #8 2025-01-13 12:21:49 ~5 min ios 📦zip
✔️ 0945def #8 2025-01-13 12:22:56 ~6 min android 📦aar
✔️ 0945def #8 2025-01-13 12:26:00 ~9 min macos 📦zip
6903ae1 #9 2025-01-13 14:42:50 ~26 sec windows 📄log
✔️ 6903ae1 #9 2025-01-13 14:46:11 ~3 min macos 📦zip
✔️ 6903ae1 #9 2025-01-13 14:47:24 ~5 min linux 📦zip
✖️ 6903ae1 #9 2025-01-13 14:47:33 ~5 min tests-rpc 📄log
✔️ 6903ae1 #9 2025-01-13 14:47:52 ~5 min android 📦aar
✖️ 6903ae1 #9 2025-01-13 14:48:59 ~6 min tests 📄log
✔️ 6903ae1 #9 2025-01-13 14:53:18 ~10 min macos 📦zip
✔️ 6903ae1 #9 2025-01-13 15:56:15 ~1 hr 14 min ios 📦zip

@friofry friofry force-pushed the ab/issue-5997-rpc-providers-integration branch 2 times, most recently from e0e484f to d2795bc Compare December 6, 2024 13:23
@friofry
Copy link
Contributor Author

friofry commented Jan 6, 2025

todo:

  • > As discussed and agreed, we should ensure now that default status-provided networks are not stored in the database anymore, because it makes no sense and only adds confusion. We should only persist user-provided secrets here, but the status-provided secrets must remain built-in in the binary.

#6151 (comment)

@friofry
Copy link
Contributor Author

friofry commented Jan 6, 2025

@friofry friofry force-pushed the ab/issue-5997-rpc-providers-configuration branch 2 times, most recently from 1ecae6e to afb9730 Compare January 6, 2025 18:16
@friofry friofry force-pushed the ab/issue-5997-rpc-providers-integration branch from d2795bc to 526f19c Compare January 7, 2025 14:36
@friofry friofry requested a review from igor-sirotin January 7, 2025 14:49
@friofry friofry force-pushed the ab/issue-5997-rpc-providers-integration branch from 526f19c to 54f6a89 Compare January 9, 2025 13:34
@friofry friofry force-pushed the ab/issue-5997-rpc-providers-configuration branch 2 times, most recently from 1c51d12 to c1b756f Compare January 10, 2025 15:50
Base automatically changed from ab/issue-5997-rpc-providers-configuration to develop January 10, 2025 22:02
Copy link

codecov bot commented Jan 10, 2025

Codecov Report

Attention: Patch coverage is 59.81162% with 256 lines in your changes missing coverage. Please review.

Project coverage is 19.13%. Comparing base (f7d73be) to head (54f6a89).
Report is 7 commits behind head on develop.

Files with missing lines Patch % Lines
rpc/network/db/rpc_provider_db.go 59.85% 44 Missing and 13 partials ⚠️
params/networkhelper/provider_utils.go 43.52% 45 Missing and 3 partials ⚠️
rpc/network/db/network_db.go 60.65% 32 Missing and 16 partials ⚠️
rpc/network/network.go 42.64% 32 Missing and 7 partials ⚠️
params/networkhelper/validate.go 0.00% 27 Missing ⚠️
rpc/network/db/utils.go 63.63% 10 Missing and 6 partials ⚠️
rpc/client.go 73.80% 8 Missing and 3 partials ⚠️
api/defaults.go 0.00% 6 Missing ⚠️
params/network_config.go 86.66% 2 Missing ⚠️
services/connector/commands/test_helpers.go 0.00% 1 Missing ⚠️
... and 1 more

❗ There is a different number of reports uploaded between BASE (f7d73be) and HEAD (54f6a89). Click for more details.

HEAD has 1 upload less than BASE
Flag BASE (f7d73be) HEAD (54f6a89)
unit 1 0
Additional details and impacted files
@@             Coverage Diff              @@
##           develop    #6178       +/-   ##
============================================
- Coverage    60.05%   19.13%   -40.92%     
============================================
  Files          832      817       -15     
  Lines       109840   108221     -1619     
============================================
- Hits         65961    20713    -45248     
- Misses       36148    84685    +48537     
+ Partials      7731     2823     -4908     
Flag Coverage Δ
functional 19.13% <59.81%> (?)
unit ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
api/default_networks.go 100.00% <100.00%> (ø)
node/get_status_node.go 37.64% <100.00%> (-13.24%) ⬇️
params/config.go 23.23% <ø> (-45.80%) ⬇️
protocol/requests/create_account.go 16.66% <ø> (-16.67%) ⬇️
services/connector/commands/test_helpers.go 0.00% <0.00%> (-100.00%) ⬇️
services/connector/test_helpers.go 0.00% <0.00%> (-100.00%) ⬇️
params/network_config.go 86.66% <86.66%> (ø)
api/defaults.go 62.91% <0.00%> (-12.92%) ⬇️
rpc/client.go 57.54% <73.80%> (-6.81%) ⬇️
rpc/network/db/utils.go 63.63% <63.63%> (ø)
... and 5 more

... and 642 files with indirect coverage changes

* default_networks.go
  * explicit provider initialization with more granular config (rps limiter, order)
  * token overrides made more flexible, support not only infura and grove
* get_status_node.go
  * override status-proxy auth instead of passing override config to rpc/client.go
* config.go
  * ProviderConfig removed
* client.go
  * Now any provider can be enabled/disabled (if user wants to use only his custom RPC urls)
  * Use bearer auth instead of URL auth
  * Provider order is defined by default_networks.go
* add method to make a deepcopy of a network + tests
* improved logging
* improved memory allocation
@friofry friofry force-pushed the ab/issue-5997-rpc-providers-integration branch from 54f6a89 to 8582775 Compare January 13, 2025 12:01
Copy link
Collaborator

@igor-sirotin igor-sirotin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't check the wallet logic at all


rpcProviders := []params.RpcProvider{
// Proxy providers
*params.NewProxyProvider(chainID, "proxy-nodefleet", proxyUrl(stageName, "nodefleet", chainName, networkName), false),
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like all these strings ("proxy-nodefleet", "nodefleet", etc) should be defined as const variables.

require.True(t, strings.Contains(n.FallbackURL, ganacheURL))
// Check direct providers for tokens
for _, provider := range n.RpcProviders {
if provider.Type == params.EmbeddedDirectProviderType {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🙏

Suggested change
if provider.Type == params.EmbeddedDirectProviderType {
if provider.Type != params.EmbeddedDirectProviderType {
continue

@@ -117,19 +118,25 @@ func OverrideEmbeddedProxyProviders(networks []params.Network, enabled bool, use
return updatedNetworks
}

func deepCopyNetworks(networks []params.Network) []params.Network {
func DeepCopyNetwork(network params.Network) params.Network {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps this could be a method of params.network?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with Igor's suggestion for this and the function below.

Comment on lines +300 to +301
if provider.AuthType != params.NoAuth {
switch provider.AuthType {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why check if NoAuth, if the switch/case already does it? 🤔

return fmt.Errorf("failed to begin transaction: %w", err)
}
defer func() {
if p := recover(); p != nil {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, this is an smart thing to rollback in case of a panic 👍
Perhaps we should do this for all db transactions? Or at least write it down as a guideline?

@@ -102,3 +102,14 @@ func CompareNetworksList(t require.TestingT, expectedNetworks, actualNetworks []
CompareNetworks(t, expectedNetwork, network)
}
}

func ConvertNetworksToPointers(networks []params.Network) []*params.Network {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
func ConvertNetworksToPointers(networks []params.Network) []*params.Network {
func DereferenceNetworks(networks []params.Network) []*params.Network {

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Though Dereference is the opposite thing 🤔 😄

Copy link
Contributor

@saledjenic saledjenic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@friofry nice and clean. Thanks.

*params.NewProxyProvider(chainID, "proxy-grove", proxyUrl(stageName, "grove", chainName, networkName), false),
// Direct providers
*params.NewDirectProvider(chainID, "direct-infura", "https://mainnet.infura.io/v3/", true),
*params.NewDirectProvider(chainID, "direct-grove", "https://eth-archival.rpc.grove.city/v1/", false),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dlipicar mentioned today that grove updated the format of their endpoints, maybe you can align with that?

@@ -117,19 +118,25 @@ func OverrideEmbeddedProxyProviders(networks []params.Network, enabled bool, use
return updatedNetworks
}

func deepCopyNetworks(networks []params.Network) []params.Network {
func DeepCopyNetwork(network params.Network) params.Network {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with Igor's suggestion for this and the function below.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: No status
Development

Successfully merging this pull request may close these issues.

4 participants