chore(tests/load): C-chain load testing #3914

qdm12 · 2025-04-28T15:07:59Z

Why this should be merged

How this works

./bin/ginkgo -v ./tests/load/c -- --avalanchego-path=$PWD/build/avalanchego

Configuration is in Go code for now, in the tests/load/c/ginkgo_test.go file

How this was tested

Need to be documented in RELEASES.md?

…ebsocket

tests/load/ginkgo_test.go

maru-ava · 2025-04-29T16:54:45Z

tests/load/ginkgo_test.go

+	flagVars = e2e.RegisterFlagsWithDefaultOwner("avalanchego-load")
+}
+
+var _ = ginkgo.SynchronizedBeforeSuite(func() []byte {


Note that it isn't necessary to use ginkgo if there is only going to be a single test entrypoint and there is no need for test parallelism. For details as to how this can work, follow the call chain in one of the antithesis tests starting from here:

https://github.com/ava-labs/avalanchego/blob/master/tests/antithesis/avalanchego/main.go#L53

Would love to but my attempts have failed so far, given the tight coupling of e2e tests code with ginkgo. For example NewTestEnvironment requires a bunch of ginkgo things, and it's rather hard to separate. Any tip on this I am missing out on?

It requires TestContext, not ginkgo. As per the antithesis testing link, it's possible to substitute tests.SimpleTestContext for tests.fixture.e2e.GinkgoTestContext to be able to use NewTestEnvironment without ginkgo. The main requirement of SimpleTestFixture is to invoke defer tc.Cleanup() to ensure the calls to tc.DeferCleanup(...) get executed before exit.

Removed usage of ginkgo in favor of UTs: 3b5e69a

.github/workflows/ci.yml

maru-ava · 2025-05-23T09:16:35Z

tests/fixture/tmpnet/utils.go

@@ -72,6 +73,23 @@ func GetNodeURIs(nodes []*Node) []NodeURI {
 	return uris
 }

+// GetNodeWebsocketURIs returns a list of websocket URIs for the given nodes and


As per the example of TestEnvironment.GetNodeURIs, a valid deferCleanup method is required that ensures forwarding tunnels are scoped to the lifetime of a test.

maru-ava · 2025-05-23T09:18:42Z

tests/load/c/contracts/generate_abi_bindings.sh

+    --out="${CONTRACTS_DIR}/${CONTRACT_NAME}.bindings.go"
+  echo "Generated ${CONTRACT_NAME}.bindings.go"
+done
+rm -r "${TEMPDIR}"


Is it intended that this temporary directory persist in the event of any error in generation? (vs using a cleanup function configured to run on exit e.g. trap cleanup EXIT)

maru-ava · 2025-05-23T09:21:18Z

tests/load/c/execute.go

+			err = result.err
+			cancel()
+		default:
+			panic("unreachable")


(No action required) Why isn't it appropriate to just return an error here?

maru-ava · 2025-05-23T09:22:28Z

tests/load/c/execute.go

+	return agents, nil
+}
+
+func createAgent(ctx context.Context, endpoint string, key *secp256k1.PrivateKey,


(No action required) Arbitrary line break? The convention appears to be all or none of the arguments are separated by line break.

maru-ava · 2025-05-23T10:00:50Z

tests/load/c/prometheus.template.yml

@@ -0,0 +1,17 @@
+global:
+  scrape_interval: 10s


Maybe configure this scrape interval to tmpnet.prometheusScrapeInterval (exporting in the process) rather than hard-coding? That would ensure a value compatible with the network shutdown delay ensuring a final metrics scrape.

maru-ava · 2025-05-23T10:08:36Z

tests/fixture/tmpnet/network.go

@@ -1073,9 +1078,58 @@ func MetricsLinkForNetwork(networkUUID string, startTime string, endTime string)
 		endTime = "now"
 	}
 	return fmt.Sprintf(
-		"https://grafana-poc.avax-dev.network/d/kBQpRdWnk/avalanche-main-dashboard?&var-filter=network_uuid%%7C%%3D%%7C%s&var-filter=is_ephemeral_node%%7C%%3D%%7Cfalse&from=%s&to=%s",
+		"https://%s/d/kBQpRdWnk/avalanche-main-dashboard?&var-filter=network_uuid%%7C%%3D%%7C%s&var-filter=is_ephemeral_node%%7C%%3D%%7Cfalse&from=%s&to=%s",


You're right, this should be composed with BuildMonitoringURLForNetwork. Maybe a separable change too.

maru-ava · 2025-05-23T10:09:16Z

tests/load/prometheus.go

+			"targets": []string{s.addr},
+			"labels": map[string]string{
+				"network_uuid":  networkUUID,
+				"network_owner": networkOwner,


Why is the owner relevant when there is only ever a single network for a load test?

I added this when I was testing my hypothesis, it's not relevant anymore.

maru-ava · 2025-05-23T10:16:41Z

Taskfile.yml

+    cmds:
+      - task: generate-load-contract-bindings
+      - task: build
+      - cmd: ./bin/ginkgo -v ./tests/load/c -- --avalanchego-path=$PWD/build/avalanchego --start-metrics-collector --start-logs-collector


run-monitored-tmpnnet-cmd automatically configures collector start and checks if the monitoring credentials are available

none of the other tasks in this file explicitly enable collector start because doing so when credentials are not set in the environment will result in failure

this task should not be enabling collector start - that is the responsibility of the caller (i.e. to start collectors manually or invoke the task with arguments (e.g. task [my task] -- --start-metrics-collector)

That suggests test-load-local is unnecessary?

Merged test-load-local into test-load and removed arguments to enable collector start: 3d211a7

maru-ava · 2025-05-23T10:52:17Z

tests/load/c/execute.go

+
+func createAgent(ctx context.Context, endpoint string, key *secp256k1.PrivateKey,
+	tracker *load.Tracker[common.Hash],
+) (load.Agent[common.Hash], error) {


(No action required) Not really loving the readability of a generic interface like this... But also not sure how long this pattern will need to persist - hopefully we're moving towards a design where the transaction ID doesn't have to be known to the orchestrator.

RodrigoVillar · 2025-05-23T13:24:21Z

As discussed offline, going to take over development of this PR. My first tasks will be the following:

Switching from using ginkgo to having the load tests be unit tests (as referenced here, there isn't a need to use ginkgo).
Addressing minor nits from reviews above

qdm12 added 23 commits April 28, 2025 14:23

initial commit

b713278

Split listeners in poll and subscription depending if connection is w…

863adcc

…ebsocket

Always generate keys except for the prefunded key

c6030db

Lower poll period from 1s to 50ms

a560a94

Abort fund distribution if no key needs it

7942c76

Ginkgo setup

f6e300c

Define client interfaces for poll and subscriber eth clients

a934818

Observe block times working for both polling and websocket

8224d4e

Use ginkgo context

57ad33b

Fail ginkgo properly using ginkgo.GinkgoT().Error

29502bd

Rely on ginkgo to handle OS signals

a213610

Simplify unneeded run function

2a859a2

Remove unneeded dependencies.go file

2be38c1

Unexport symbols

8701f17

Use all prefunded keys instead of only one

e5f2f79

Thread safe tracker

ee6132e

CI setup

35450dc

Split issuer and listener

5ea102a

Rename package names issuer and generator to issue and generate

3be8f03

Mark remaining in flight transactions as failed when exiting listener

0251e6f

Fix listener poll implementation

8658ca0

Use env network default nodes directly

66eb147

Limit number of nodes to 1

50095f6

maru-ava reviewed Apr 29, 2025

View reviewed changes

tests/load/ginkgo_test.go Outdated Show resolved Hide resolved

maru-ava reviewed Apr 29, 2025

View reviewed changes

michaelkaplan13 and others added 3 commits April 30, 2025 10:23

Fix block production

92616fc

Fix listen subscription exit

c2ebb6f

Minor comment fixes

e323368

qdm12 force-pushed the qdm12/tests/load branch from 1f7fd9d to e2adf7b Compare April 30, 2025 08:23

Log out average duration per block at the end

e13e3ae

qdm12 added 24 commits May 22, 2025 11:25

Shorten GenerateMonitoringConfig comment

da8b904

Use github.com/ava-labs/avalanchego/utils/perms

c7d2beb

Use require.NoError in load_test.go

45d4f86

Remove unneeded network.Nodes[:nodesCount]

2aba1cd

Remove unneeded logger from metrics server

d3d8661

Use test context logger

d45e40b

Use two DeferCleanup

e21dfbd

Remove unused fields in c/issuers/opcode

eeb7581

Remove unneeded maxTipCap

4c00861

Remove unneeded NonceAt method in client interface

fd97d80

Issuer picks a tx type according to a weight

4f22990

Merge both load types together

e742788

chore: newTxOpts always set max tip cap to 1

4ce9359

Max fee cap set per transaction type in issuer

5736025

Adjust max fee cap for zero self transfers

7e6af87

Adjust weights to lower contract deployment

177cbbe

Fix pinning of libevm/cmd/abigen

4d61ab1

More conservative min and max TPS

75a659a

Add test-load-local task for local monitoring

72c67d1

Move body of SynchronizedBeforeSuite to BeforeAll

d9ea754

Revert unrelated auto-formatting diffs

3952c3a

Update local monitoring instructions to not pollute the source tree

5c6221b

Merge branch 'master' into qdm12/tests/load

c1bda80

Fix pinning of abigen (again)

308e096

maru-ava reviewed May 23, 2025

View reviewed changes

RodrigoVillar added 4 commits May 23, 2025 13:50

remove ginkgo from load tests

3b5e69a

single test-load task

3d211a7

remove network owner label

ffad23f

lint

0e1e388

chore(tests/load): C-chain load testing #3914

Are you sure you want to change the base?

chore(tests/load): C-chain load testing #3914

Conversation

qdm12 commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why this should be merged

How this works

How this was tested

Need to be documented in RELEASES.md?

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maru-ava May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RodrigoVillar commented May 23, 2025

Uh oh!

Uh oh!

qdm12 commented Apr 28, 2025 •

edited

Loading

maru-ava May 23, 2025 •

edited

Loading