CCIPv2: RPCLatency test, Full chaos suite #16320

skudasov · 2025-02-11T15:05:07Z

No description provided.

github-actions · 2025-02-11T15:16:33Z

AER Report: CI Core

aer_workflow , commit , Clean Go Tidy & Generate , Detect Changes , Scheduled Run Frequency , GolangCI Lint (integration-tests/load) , GolangCI Lint (integration-tests) , GolangCI Lint (deployment) , Core Tests (go_core_tests) , test-scripts , Core Tests (go_core_tests_integration) , Core Tests (go_core_ccip_deployment_tests) , Core Tests (go_core_fuzz) , Core Tests (go_core_race_tests) , lint , SonarQube Scan

1. Port allocation failure in tests: go_core_ccip_deployment_tests

Source of Error:

Run tests	2025-02-27T06:59:42.4230703Z --- FAIL: TestFundMCMSignersChangeset_VerifyPreconditions (0.31s)
Run tests	2025-02-27T06:59:42.4231224Z panic: freeport: cannot allocate port block [recovered]
Run tests	2025-02-27T06:59:42.4231711Z 	panic: freeport: cannot allocate port block
Run tests	2025-02-27T06:59:42.4231910Z 
Run tests	2025-02-27T06:59:42.4232022Z goroutine 219 [running]:
Run tests	2025-02-27T06:59:42.4233845Z testing.tRunner.func1.2({0x9a5d820, 0xb9c9750})
Run tests	2025-02-27T06:59:42.4234485Z 	/opt/hostedtoolcache/go/1.24.0/x64/src/testing/testing.go:1734 +0x21c
Run tests	2025-02-27T06:59:42.4234953Z testing.tRunner.func1()
Run tests	2025-02-27T06:59:42.4235698Z 	/opt/hostedtoolcache/go/1.24.0/x64/src/testing/testing.go:1737 +0x35e
Run tests	2025-02-27T06:59:42.4236204Z panic({0x9a5d820?, 0xb9c9750?})
Run tests	2025-02-27T06:59:42.4236719Z 	/opt/hostedtoolcache/go/1.24.0/x64/src/runtime/panic.go:787 +0x132
Run tests	2025-02-27T06:59:42.4237211Z github.com/hashicorp/consul/sdk/freeport.alloc()
Run tests	2025-02-27T06:59:42.4237904Z 	/home/runner/go/pkg/mod/github.com/hashicorp/consul/[email protected]/freeport/freeport.go:274 +0xad
Run tests	2025-02-27T06:59:42.4238870Z github.com/hashicorp/consul/sdk/freeport.initialize()
Run tests	2025-02-27T06:59:42.4241822Z 	/home/runner/go/pkg/mod/github.com/hashicorp/consul/[email protected]/freeport/freeport.go:124 +0x2d7
Run tests	2025-02-27T06:59:42.4242376Z sync.(*Once).doSlow(0x6?, 0x0?)
Run tests	2025-02-27T06:59:42.4242888Z 	/opt/hostedtoolcache/go/1.24.0/x64/src/sync/once.go:78 +0xab
Run tests	2025-02-27T06:59:42.4243254Z sync.(*Once).Do(...)
Run tests	2025-02-27T06:59:42.4243689Z 	/opt/hostedtoolcache/go/1.24.0/x64/src/sync/once.go:69
Run tests	2025-02-27T06:59:42.4244145Z github.com/hashicorp/consul/sdk/freeport.Take(0x1)
Run tests	2025-02-27T06:59:42.4244829Z 	/home/runner/go/pkg/mod/github.com/hashicorp/consul/[email protected]/freeport/freeport.go:303 +0xe5
Run tests	2025-02-27T06:59:42.4245618Z github.com/hashicorp/consul/sdk/freeport.GetN({0xba37220, 0xc000602fc0}, 0x1)
Run tests	2025-02-27T06:59:42.4246405Z 	/home/runner/go/pkg/mod/github.com/hashicorp/consul/[email protected]/freeport/freeport.go:427 +0x48
Run tests	2025-02-27T06:59:42.4247168Z github.com/hashicorp/consul/sdk/freeport.GetOne({0xba37220, 0xc000602fc0})
Run tests	2025-02-27T06:59:42.4247971Z 	/home/runner/go/pkg/mod/github.com/hashicorp/consul/[email protected]/freeport/freeport.go:449 +0x36
Run tests	2025-02-27T06:59:42.4248979Z github.com/smartcontractkit/chainlink/deployment/environment/memory.solChain(0xc000602fc0, 0xacf8842c5346530f, 0xc0015d50d8)
Run tests	2025-02-27T06:59:42.4250034Z 	/home/runner/work/chainlink/chainlink/deployment/environment/memory/chain.go:254 +0x10d
Run tests	2025-02-27T06:59:42.4250978Z github.com/smartcontractkit/chainlink/deployment/environment/memory.GenerateChainsSol(0xc000602fc0, 0x1)
Run tests	2025-02-27T06:59:42.4251910Z 	/home/runner/work/chainlink/chainlink/deployment/environment/memory/chain.go:176 +0x2d3
Run tests	2025-02-27T06:59:42.4252889Z github.com/smartcontractkit/chainlink/deployment/environment/memory.NewMemoryChainsSol(0xc000602fc0?, 0xc000d7de30?)
Run tests	2025-02-27T06:59:42.4253848Z 	/home/runner/work/chainlink/chainlink/deployment/environment/memory/environment.go:84 +0x25
Run tests	2025-02-27T06:59:42.4254966Z github.com/smartcontractkit/chainlink/deployment/environment/memory.NewMemoryEnvironment(_, {_, _}, _, {0x0, 0x1, 0x0, 0x0, 0x0, {0x0, ...}})
Run tests	2025-02-27T06:59:42.4255980Z 	/home/runner/work/chainlink/chainlink/deployment/environment/memory/environment.go:222 +0xa5
Run tests	2025-02-27T06:59:42.4257113Z github.com/smartcontractkit/chainlink/deployment/common/changeset/solana_test.TestFundMCMSignersChangeset_VerifyPreconditions(0xc000602fc0)
Run tests	2025-02-27T06:59:42.4258191Z 	/home/runner/work/chainlink/chainlink/deployment/common/changeset/solana/fund_mcm_pdas_test.go:57 +0xa8
Run tests	2025-02-27T06:59:42.4258876Z testing.tRunner(0xc000602fc0, 0xb303398)
Run tests	2025-02-27T06:59:42.4259316Z 	/opt/hostedtoolcache/go/1.24.0/x64/src/testing/testing.go:1792 +0xf4
Run tests	2025-02-27T06:59:42.4259734Z created by testing.(*T).Run in goroutine 1
Run tests	2025-02-27T06:59:42.4260165Z 	/opt/hostedtoolcache/go/1.24.0/x64/src/testing/testing.go:1851 +0x413
Run tests	2025-02-27T06:59:42.4260768Z FAIL	github.com/smartcontractkit/chainlink/deployment/common/changeset/solana	0.871s

Why: The test TestFundMCMSignersChangeset_VerifyPreconditions failed due to the inability to allocate a port block using the freeport library. This is likely caused by the exhaustion of available ports within the ephemeral port range.

Suggested fix: Ensure that the ephemeral port range is not exhausted by either increasing the range or reducing the number of concurrent tests that require port allocation. Alternatively, mock the port allocation in tests.

2. Port allocation failure in tests: go_core_ccip_deployment_tests

Source of Error:

Run tests	2025-02-27T06:59:42.4274992Z --- FAIL: Test_Deploy (0.02s)
Run tests	2025-02-27T06:59:42.4275565Z panic: freeport: cannot allocate port block [recovered]
Run tests	2025-02-27T06:59:42.4275960Z 	panic: freeport: cannot allocate port block
Run tests	2025-02-27T06:59:42.4276217Z 
Run tests	2025-02-27T06:59:42.4276332Z goroutine 370 [running]:
Run tests	2025-02-27T06:59:42.4276662Z testing.tRunner.func1.2({0x5dc2d60, 0x72daf10})
Run tests	2025-02-27T06:59:42.4277156Z 	/opt/hostedtoolcache/go/1.24.0/x64/src/testing/testing.go:1734 +0x21c
Run tests	2025-02-27T06:59:42.4277571Z testing.tRunner.func1()
Run tests	2025-02-27T06:59:42.4278006Z 	/opt/hostedtoolcache/go/1.24.0/x64/src/testing/testing.go:1737 +0x35e
Run tests	2025-02-27T06:59:42.4278402Z panic({0x5dc2d60?, 0x72daf10?})
Run tests	2025-02-27T06:59:42.4278790Z 	/opt/hostedtoolcache/go/1.24.0/x64/src/runtime/panic.go:787 +0x132
Run tests	2025-02-27T06:59:42.4279276Z github.com/hashicorp/consul/sdk/freeport.alloc()
Run tests	2025-02-27T06:59:42.4279851Z 	/home/runner/go/pkg/mod/github.com/hashicorp/consul/[email protected]/freeport/freeport.go:274 +0xad
Run tests	2025-02-27T06:59:42.4280419Z github.com/hashicorp/consul/sdk/freeport.initialize()
Run tests	2025-02-27T06:59:42.4281042Z 	/home/runner/go/pkg/mod/github.com/hashicorp/consul/[email protected]/freeport/freeport.go:124 +0x2d7
Run tests	2025-02-27T06:59:42.4281589Z sync.(*Once).doSlow(0xc002f00a60?, 0xc002b3d4d8?)
Run tests	2025-02-27T06:59:42.4282001Z 	/opt/hostedtoolcache/go/1.24.0/x64/src/sync/once.go:78 +0xab
Run tests	2025-02-27T06:59:42.4282328Z sync.(*Once).Do(...)
Run tests	2025-02-27T06:59:42.4282636Z 	/opt/hostedtoolcache/go/1.24.0/x64/src/sync/once.go:69
Run tests	2025-02-27T06:59:42.4283101Z github.com/hashicorp/consul/sdk/freeport.Take(0x1)
Run tests	2025-02-27T06:59:42.4283738Z 	/home/runner/go/pkg/mod/github.com/hashicorp/consul/[email protected]/freeport/freeport.go:303 +0xe5
Run tests	2025-02-27T06:59:42.4284394Z github.com/hashicorp/consul/sdk/freeport.GetN({0x7324528, 0xc000682c40}, 0x1)
Run tests	2025-02-27T06:59:42.4285048Z 	/home/runner/go/pkg/mod/github.com/hashicorp/consul/[email protected]/freeport/freeport.go:427 +0x48
Run tests	2025-02-27T06:59:42.4286135Z github.com/smartcontractkit/chainlink/deployment/environment/memory.NewNodes(0xc000682c40, 0xff, 0xc0031e04e0, 0xc0031e0540, 0x1, 0x0, {0x0, {0x0, 0x0, 0x0, ...}, ...}, ...)
Run tests	2025-02-27T06:59:42.4287330Z 	/home/runner/work/chainlink/chainlink/deployment/environment/memory/environment.go:177 +0xa5
Run tests	2025-02-27T06:59:42.4288311Z github.com/smartcontractkit/chainlink/deployment/environment/memory.NewMemoryEnvironment(_, {_, _}, _, {0x2, 0x0, 0x0, 0x1, 0x0, {0x0, ...}})
Run tests	2025-02-27T06:59:42.4289210Z 	/home/runner/work/chainlink/chainlink/deployment/environment/memory/environment.go:223 +0x10c
Run tests	2025-02-27T06:59:42.4290079Z github.com/smartcontractkit/chainlink/deployment/keystone/changeset/workflowregistry.Test_Deploy(0xc000682c40)
Run tests	2025-02-27T06:59:42.4290992Z 	/home/runner/work/chainlink/chainlink/deployment/keystone/changeset/workflowregistry/deploy_test.go:22 +0x88
Run tests	2025-02-27T06:59:42.4291726Z testing.tRunner(0xc000682c40, 0x6d53660)
Run tests	2025-02-27T06:59:42.4292236Z 	/opt/hostedtoolcache/go/1.24.0/x64/src/testing/testing.go:1792 +0xf4
Run tests	2025-02-27T06:59:42.4292654Z created by testing.(*T).Run in goroutine 1
Run tests	2025-02-27T06:59:42.4293090Z 	/opt/hostedtoolcache/go/1.24.0/x64/src/testing/testing.go:1851 +0x413
Run tests	2025-02-27T06:59:42.6842742Z FAIL	github.com/smartcontractkit/chainlink/deployment/keystone/changeset/workflowregistry	1.301s

Why: The test Test_Deploy failed due to the inability to allocate a port block using the freeport library. This is likely caused by the exhaustion of available ports within the ephemeral port range.

Suggested fix: Ensure that the ephemeral port range is not exhausted by either increasing the range or reducing the number of concurrent tests that require port allocation. Alternatively, mock the port allocation in tests.

3. Linting issues: lint

Source of Error:

Check Golangci-lint Matrix Results	2025-02-27T06:51:26.5284912Z ##[error]Process completed with exit code 1.

Why: The linting step failed because at least one 'GolangCI Lint' matrix job failed. Specific linting errors were detected in the codebase.

Suggested fix: Review the detailed linting errors and fix the issues in the codebase. Ensure that all code adheres to the linting rules specified in the GolangCI Lint configuration.

4. Linting issues in specific files: GolangCI Lint (integration-tests/load)

Source of Error:

Golang Lint (integration-tests/load)	2025-02-27T06:51:14.9233363Z ##[error]integration-tests/load/ccip/destination_gun.go:132:2: commentFormatting: put a space between `//` and comment text (gocritic)
Golang Lint (integration-tests/load)	2025-02-27T06:51:14.9234538Z 	//acc.GasLimit = *m.testConfig.GasLimit
Golang Lint (integration-tests/load)	2025-02-27T06:51:14.9234983Z 	^
Golang Lint (integration-tests/load)	2025-02-27T06:51:14.9236735Z ##[error]integration-tests/load/ccip/ccip_chaos_test.go:5:1: File is not properly formatted (goimports)
Golang Lint (integration-tests/load)	2025-02-27T06:51:14.9237507Z 	"fmt"
Golang Lint (integration-tests/load)	2025-02-27T06:51:14.9257982Z ^
Golang Lint (integration-tests/load)	2025-02-27T06:51:14.9260220Z ##[error]integration-tests/load
</cicore>
  
    

















<operatoruici>

## AER Report: [Operator UI CI](https://github.com/smartcontractkit/chainlink/actions/runs/13560543377) ran successfully :white_check_mark:

[aer_workflow](https://github.com/smartcontractkit/chainlink/actions/runs/13560561732/job/37902746405) , [commit](https://github.com/smartcontractkit/chainlink/commit/1754348117fc13d8731b62fc00a357e0b97b6e75)

</operatoruici>

# Conflicts: # integration-tests/load/go.mod # integration-tests/load/go.sum

# Conflicts: # integration-tests/go.mod # integration-tests/go.sum # integration-tests/load/go.mod # integration-tests/load/go.sum # integration-tests/testconfig/ccip/ccip.toml

# Conflicts: # integration-tests/go.mod

0xAustinWang · 2025-02-24T14:42:00Z

integration-tests/load/ccip/ccip_test.go

@@ -2,6 +2,7 @@ package ccip

 import (
 	"context"
+	"github.com/smartcontractkit/chainlink/integration-tests/testconfig/ccip"


should this trigger goimports rule?

Good catch, it should, however, let me fix it

cl-sonarqube-production · 2025-02-27T07:01:39Z

Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube

Catch issues before they fail your Quality Gate with our IDE extension SonarQube IDE

skudasov force-pushed the aw/LoadTest-aws-chaos branch 3 times, most recently from 681f1c8 to 7a0db62 Compare February 11, 2025 16:03

Base automatically changed from aw/LoadTest-aws to develop February 12, 2025 18:53

0xAustinWang and others added 13 commits February 13, 2025 09:28

changes for pr

ab06c76

add concurrency

483c602

add logging, make the setup functional

9c38005

PR comments

7fe4592

all changes, mostly imports and parallelization of changesets

0d774c3

parallelization

5e668c2

lint

a2ccb46

lint

8f5dcc2

revert changeset parallelization, go mods

3b02278

chaos tests

2ece164

golint, goimports

4461526

refactor, bump CTF

3d2c791

rebase and add cfg path

c8ccec1

skudasov force-pushed the aw/LoadTest-aws-chaos branch from fb515ef to c8ccec1 Compare February 13, 2025 08:32

skudasov added 11 commits February 13, 2025 10:13

update go mod

29698e4

more experiments for RMNv2

894cdb8

Merge branch 'develop' into aw/LoadTest-aws-chaos

13c5ede

# Conflicts: # integration-tests/load/go.mod # integration-tests/load/go.sum

merge, update go.mod

e1a6ddb

Merge branch 'develop' into aw/LoadTest-aws-chaos

ffd4894

# Conflicts: # integration-tests/go.mod # integration-tests/go.sum # integration-tests/load/go.mod # integration-tests/load/go.sum # integration-tests/testconfig/ccip/ccip.toml

merge again

fd66e1d

merge the tests together

181fa79

Merge branch 'develop' into aw/LoadTest-aws-chaos

79d5cc1

# Conflicts: # integration-tests/go.mod

merge develop

6565760

rpc latency by default

08106b0

40 minutes

64b1328

merge

5f6df38

skudasov marked this pull request as ready for review February 24, 2025 14:02

skudasov requested review from a team as code owners February 24, 2025 14:02

skudasov requested review from chudilka1, winder, asoliman92, makramkd, dimkouv, mateusz-sekara, 0xAustinWang and 0xnogo February 24, 2025 14:02

skudasov changed the title ~~chaos tests~~ CCIPv2: RPCLatency test, Full chaos suite Feb 24, 2025

0xAustinWang approved these changes Feb 24, 2025

View reviewed changes

0xAustinWang previously approved these changes Feb 24, 2025

View reviewed changes

Merge remote-tracking branch 'origin' into aw/LoadTest-aws-chaos

fa5da7f

0xAustinWang dismissed their stale review via fa5da7f February 25, 2025 07:15

0xAustinWang and others added 4 commits February 25, 2025 15:16

add more chains that suport chaos

3be5e63

fix goimports

02d7b78

Merge branch 'develop' into aw/LoadTest-aws-chaos

82268d5

pushing changes

c107307

0xAustinWang requested review from a team as code owners February 26, 2025 14:27

0xAustinWang added 3 commits February 27, 2025 13:58

Merge branch 'develop' into aw/LoadTest-aws-chaos

7092af1

go mod tidy

575fcb1

build issues

1754348

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CCIPv2: RPCLatency test, Full chaos suite #16320

CCIPv2: RPCLatency test, Full chaos suite #16320

skudasov commented Feb 11, 2025

github-actions bot commented Feb 11, 2025 •

edited

Loading

0xAustinWang Feb 24, 2025

skudasov Feb 25, 2025

cl-sonarqube-production bot commented Feb 27, 2025

CCIPv2: RPCLatency test, Full chaos suite #16320

Are you sure you want to change the base?

CCIPv2: RPCLatency test, Full chaos suite #16320

Conversation

skudasov commented Feb 11, 2025

github-actions bot commented Feb 11, 2025 • edited Loading

AER Report: CI Core

1. Port allocation failure in tests: go_core_ccip_deployment_tests

2. Port allocation failure in tests: go_core_ccip_deployment_tests

3. Linting issues: lint

4. Linting issues in specific files: GolangCI Lint (integration-tests/load)

0xAustinWang Feb 24, 2025

Choose a reason for hiding this comment

skudasov Feb 25, 2025

Choose a reason for hiding this comment

cl-sonarqube-production bot commented Feb 27, 2025

Quality Gate failed

github-actions bot commented Feb 11, 2025 •

edited

Loading