Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test: flaky unit tests #6398

Merged
merged 4 commits into from
Feb 6, 2024
Merged

test: flaky unit tests #6398

merged 4 commits into from
Feb 6, 2024

Conversation

nazarhussain
Copy link
Contributor

Motivation

Make the tests suit behave stable.

Description

Fix following flaky tests.

@lodestar/api:  FAIL  test/unit/beacon/genericServerTest/config.test.ts > beacon / config > Run generic server test
@lodestar/api: Error: listen EADDRINUSE: address already in use ::1:55300

@lodestar/validator:  FAIL  test/unit/utils/clock.test.ts > util / Clock > getCurrentSlot > 'should return next slot after 11.5s'
@lodestar/validator: AssertionError: expected 8059733 to be 8059734 // Object.is equality

Steps to test or reproduce

Run all tests

@nazarhussain nazarhussain requested a review from a team as a code owner February 6, 2024 14:03
@nazarhussain nazarhussain self-assigned this Feb 6, 2024
@nazarhussain nazarhussain changed the title fix: fix flaky unit tests fix: flaky unit tests Feb 6, 2024
@nazarhussain nazarhussain changed the title fix: flaky unit tests test: flaky unit tests Feb 6, 2024
@codecov Codecov
Copy link

codecov bot commented Feb 6, 2024 •
edited
Loading

Codecov Report

Merging #6398 (a73adb8) into unstable (c4bf385) will not change coverage.
Report is 3 commits behind head on unstable.
The diff coverage is n/a.

❗ Current head a73adb8 differs from pull request most recent head 6d349b9. Consider uploading reports for the commit 6d349b9 to get more accurate results

Additional details and impacted files 
@@            Coverage Diff            @@
##           unstable    #6398   +/-   ##
=========================================
  Coverage     60.14%   60.14%           
=========================================
  Files           407      407           
  Lines         46512    46512           
  Branches       1551     1551           
=========================================
  Hits          27975    27975           
  Misses        18505    18505           
  Partials         32       32

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 6, 2024 •
edited
Loading

⚠️ Performance Alert ⚠️

Possible performance regression was detected for some benchmarks.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold.

Benchmark suite Current: 6d511bf Previous: cb754f6 Ratio
forkChoice updateHead vc 600000 bc 64 eq 300000 53.658 ms/op 16.394 ms/op 3.27
Full benchmark results
Benchmark suite Current: 6d511bf Previous: cb754f6 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 788.14 us/op 548.88 us/op 1.44
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 169.84 us/op 114.11 us/op 1.49
BLS verify - blst-native 1.7089 ms/op 1.3223 ms/op 1.29
BLS verifyMultipleSignatures 3 - blst-native 3.4398 ms/op 2.7878 ms/op 1.23
BLS verifyMultipleSignatures 8 - blst-native 8.0627 ms/op 6.1552 ms/op 1.31
BLS verifyMultipleSignatures 32 - blst-native 28.358 ms/op 22.497 ms/op 1.26
BLS verifyMultipleSignatures 64 - blst-native 55.174 ms/op 44.399 ms/op 1.24
BLS verifyMultipleSignatures 128 - blst-native 109.06 ms/op 89.758 ms/op 1.22
BLS deserializing 10000 signatures 1.0421 s/op 952.69 ms/op 1.09
BLS deserializing 100000 signatures 10.053 s/op 9.6456 s/op 1.04
BLS verifyMultipleSignatures - same message - 3 - blst-native 1.4615 ms/op 1.3793 ms/op 1.06
BLS verifyMultipleSignatures - same message - 8 - blst-native 1.6147 ms/op 1.5217 ms/op 1.06
BLS verifyMultipleSignatures - same message - 32 - blst-native 2.4165 ms/op 2.3593 ms/op 1.02
BLS verifyMultipleSignatures - same message - 64 - blst-native 3.6851 ms/op 4.5593 ms/op 0.81
BLS verifyMultipleSignatures - same message - 128 - blst-native 5.8904 ms/op 5.9495 ms/op 0.99
BLS aggregatePubkeys 32 - blst-native 28.011 us/op 26.249 us/op 1.07
BLS aggregatePubkeys 128 - blst-native 107.47 us/op 101.79 us/op 1.06
notSeenSlots=1 numMissedVotes=1 numBadVotes=10 69.195 ms/op 53.187 ms/op 1.30
notSeenSlots=1 numMissedVotes=0 numBadVotes=4 53.205 ms/op 52.958 ms/op 1.00
notSeenSlots=2 numMissedVotes=1 numBadVotes=10 45.221 ms/op 34.074 ms/op 1.33
getSlashingsAndExits - default max 170.87 us/op 198.19 us/op 0.86
getSlashingsAndExits - 2k 439.96 us/op 591.91 us/op 0.74
proposeBlockBody type=full, size=empty 6.1079 ms/op 5.5570 ms/op 1.10
isKnown best case - 1 super set check 350.00 ns/op 560.00 ns/op 0.63
isKnown normal case - 2 super set checks 420.00 ns/op 569.00 ns/op 0.74
isKnown worse case - 16 super set checks 337.00 ns/op 536.00 ns/op 0.63
CheckpointStateCache - add get delete 5.7630 us/op 6.5020 us/op 0.89
validate api signedAggregateAndProof - struct 2.8236 ms/op 2.8785 ms/op 0.98
validate gossip signedAggregateAndProof - struct 2.8213 ms/op 2.8414 ms/op 0.99
validate gossip attestation - vc 640000 1.4091 ms/op 1.3996 ms/op 1.01
batch validate gossip attestation - vc 640000 - chunk 32 174.03 us/op 180.55 us/op 0.96
batch validate gossip attestation - vc 640000 - chunk 64 151.01 us/op 149.47 us/op 1.01
batch validate gossip attestation - vc 640000 - chunk 128 156.23 us/op 146.81 us/op 1.06
batch validate gossip attestation - vc 640000 - chunk 256 147.43 us/op 131.03 us/op 1.13
pickEth1Vote - no votes 1.3441 ms/op 1.2645 ms/op 1.06
pickEth1Vote - max votes 17.629 ms/op 13.339 ms/op 1.32
pickEth1Vote - Eth1Data hashTreeRoot value x2048 26.306 ms/op 17.733 ms/op 1.48
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 40.513 ms/op 35.791 ms/op 1.13
pickEth1Vote - Eth1Data fastSerialize value x2048 737.51 us/op 624.76 us/op 1.18
pickEth1Vote - Eth1Data fastSerialize tree x2048 8.6512 ms/op 4.4230 ms/op 1.96
bytes32 toHexString 583.00 ns/op 669.00 ns/op 0.87
bytes32 Buffer.toString(hex) 328.00 ns/op 306.00 ns/op 1.07
bytes32 Buffer.toString(hex) from Uint8Array 578.00 ns/op 486.00 ns/op 1.19
bytes32 Buffer.toString(hex) + 0x 329.00 ns/op 307.00 ns/op 1.07
Object access 1 prop 0.25800 ns/op 0.20000 ns/op 1.29
Map access 1 prop 0.17000 ns/op 0.15200 ns/op 1.12
Object get x1000 8.2250 ns/op 7.6350 ns/op 1.08
Map get x1000 0.89300 ns/op 0.84500 ns/op 1.06
Object set x1000 72.948 ns/op 54.734 ns/op 1.33
Map set x1000 57.971 ns/op 43.403 ns/op 1.34
Return object 10000 times 0.27770 ns/op 0.24300 ns/op 1.14
Throw Error 10000 times 4.3992 us/op 3.8881 us/op 1.13
fastMsgIdFn sha256 / 200 bytes 3.7380 us/op 3.3520 us/op 1.12
fastMsgIdFn h32 xxhash / 200 bytes 443.00 ns/op 330.00 ns/op 1.34
fastMsgIdFn h64 xxhash / 200 bytes 457.00 ns/op 372.00 ns/op 1.23
fastMsgIdFn sha256 / 1000 bytes 13.207 us/op 11.572 us/op 1.14
fastMsgIdFn h32 xxhash / 1000 bytes 617.00 ns/op 419.00 ns/op 1.47
fastMsgIdFn h64 xxhash / 1000 bytes 638.00 ns/op 419.00 ns/op 1.52
fastMsgIdFn sha256 / 10000 bytes 116.37 us/op 102.64 us/op 1.13
fastMsgIdFn h32 xxhash / 10000 bytes 2.2860 us/op 1.9380 us/op 1.18
fastMsgIdFn h64 xxhash / 10000 bytes 1.5820 us/op 1.3200 us/op 1.20
send data - 1000 256B messages 25.921 ms/op 19.809 ms/op 1.31
send data - 1000 512B messages 33.526 ms/op 26.864 ms/op 1.25
send data - 1000 1024B messages 47.751 ms/op 38.932 ms/op 1.23
send data - 1000 1200B messages 42.389 ms/op 38.857 ms/op 1.09
send data - 1000 2048B messages 64.590 ms/op 49.694 ms/op 1.30
send data - 1000 4096B messages 56.695 ms/op 27.933 ms/op 2.03
send data - 1000 16384B messages 137.36 ms/op 115.69 ms/op 1.19
send data - 1000 65536B messages 637.78 ms/op 522.50 ms/op 1.22
enrSubnets - fastDeserialize 64 bits 1.6650 us/op 1.7090 us/op 0.97
enrSubnets - ssz BitVector 64 bits 672.00 ns/op 554.00 ns/op 1.21
enrSubnets - fastDeserialize 4 bits 273.00 ns/op 243.00 ns/op 1.12
enrSubnets - ssz BitVector 4 bits 641.00 ns/op 567.00 ns/op 1.13
prioritizePeers score -10:0 att 32-0.1 sync 2-0 139.08 us/op 121.50 us/op 1.14
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 190.71 us/op 160.07 us/op 1.19
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 254.66 us/op 220.65 us/op 1.15
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 511.44 us/op 366.45 us/op 1.40
prioritizePeers score 0:0 att 64-1 sync 4-1 526.12 us/op 399.38 us/op 1.32
array of 16000 items push then shift 2.0277 us/op 1.7211 us/op 1.18
LinkedList of 16000 items push then shift 13.441 ns/op 10.681 ns/op 1.26
array of 16000 items push then pop 161.73 ns/op 129.35 ns/op 1.25
LinkedList of 16000 items push then pop 11.614 ns/op 9.4230 ns/op 1.23
array of 24000 items push then shift 3.0787 us/op 2.5483 us/op 1.21
LinkedList of 24000 items push then shift 16.053 ns/op 9.3050 ns/op 1.73
array of 24000 items push then pop 244.17 ns/op 154.67 ns/op 1.58
LinkedList of 24000 items push then pop 12.078 ns/op 8.7480 ns/op 1.38
intersect bitArray bitLen 8 8.5910 ns/op 5.8180 ns/op 1.48
intersect array and set length 8 124.11 ns/op 66.336 ns/op 1.87
intersect bitArray bitLen 128 49.318 ns/op 36.693 ns/op 1.34
intersect array and set length 128 1.6217 us/op 967.83 ns/op 1.68
bitArray.getTrueBitIndexes() bitLen 128 2.7810 us/op 1.4640 us/op 1.90
bitArray.getTrueBitIndexes() bitLen 248 4.4510 us/op 2.4100 us/op 1.85
bitArray.getTrueBitIndexes() bitLen 512 8.5420 us/op 4.8360 us/op 1.77
Buffer.concat 32 items 1.3370 us/op 1.0870 us/op 1.23
Uint8Array.set 32 items 3.2700 us/op 1.9120 us/op 1.71
Set add up to 64 items then delete first 6.1044 us/op 4.7750 us/op 1.28
OrderedSet add up to 64 items then delete first 7.9664 us/op 5.9941 us/op 1.33
Set add up to 64 items then delete last 6.2596 us/op 5.0287 us/op 1.24
OrderedSet add up to 64 items then delete last 8.3308 us/op 6.0948 us/op 1.37
Set add up to 64 items then delete middle 6.6884 us/op 4.7583 us/op 1.41
OrderedSet add up to 64 items then delete middle 10.481 us/op 7.1985 us/op 1.46
Set add up to 128 items then delete first 12.445 us/op 9.7144 us/op 1.28
OrderedSet add up to 128 items then delete first 16.447 us/op 13.001 us/op 1.27
Set add up to 128 items then delete last 11.418 us/op 9.5932 us/op 1.19
OrderedSet add up to 128 items then delete last 16.943 us/op 12.033 us/op 1.41
Set add up to 128 items then delete middle 11.973 us/op 9.4470 us/op 1.27
OrderedSet add up to 128 items then delete middle 24.809 us/op 18.582 us/op 1.34
Set add up to 256 items then delete first 26.231 us/op 19.733 us/op 1.33
OrderedSet add up to 256 items then delete first 34.783 us/op 26.170 us/op 1.33
Set add up to 256 items then delete last 22.883 us/op 18.822 us/op 1.22
OrderedSet add up to 256 items then delete last 33.767 us/op 24.143 us/op 1.40
Set add up to 256 items then delete middle 25.563 us/op 19.523 us/op 1.31
OrderedSet add up to 256 items then delete middle 60.101 us/op 46.228 us/op 1.30
transfer serialized Status (84 B) 2.0840 us/op 1.6900 us/op 1.23
copy serialized Status (84 B) 1.6360 us/op 1.2600 us/op 1.30
transfer serialized SignedVoluntaryExit (112 B) 2.6630 us/op 1.8030 us/op 1.48
copy serialized SignedVoluntaryExit (112 B) 1.6480 us/op 1.3840 us/op 1.19
transfer serialized ProposerSlashing (416 B) 2.5750 us/op 2.0760 us/op 1.24
copy serialized ProposerSlashing (416 B) 2.3800 us/op 1.9320 us/op 1.23
transfer serialized Attestation (485 B) 3.5570 us/op 2.1590 us/op 1.65
copy serialized Attestation (485 B) 3.4480 us/op 1.9330 us/op 1.78
transfer serialized AttesterSlashing (33232 B) 3.6190 us/op 2.0500 us/op 1.77
copy serialized AttesterSlashing (33232 B) 9.6550 us/op 7.1600 us/op 1.35
transfer serialized Small SignedBeaconBlock (128000 B) 3.5540 us/op 2.6700 us/op 1.33
copy serialized Small SignedBeaconBlock (128000 B) 34.263 us/op 18.460 us/op 1.86
transfer serialized Avg SignedBeaconBlock (200000 B) 3.9090 us/op 3.1820 us/op 1.23
copy serialized Avg SignedBeaconBlock (200000 B) 49.123 us/op 30.251 us/op 1.62
transfer serialized BlobsSidecar (524380 B) 5.9750 us/op 3.7690 us/op 1.59
copy serialized BlobsSidecar (524380 B) 397.36 us/op 90.652 us/op 4.38
transfer serialized Big SignedBeaconBlock (1000000 B) 4.2420 us/op 3.9770 us/op 1.07
copy serialized Big SignedBeaconBlock (1000000 B) 263.90 us/op 165.56 us/op 1.59
pass gossip attestations to forkchoice per slot 4.9115 ms/op 4.7441 ms/op 1.04
forkChoice updateHead vc 100000 bc 64 eq 0 822.19 us/op 691.23 us/op 1.19
forkChoice updateHead vc 600000 bc 64 eq 0 6.1641 ms/op 8.8591 ms/op 0.70
forkChoice updateHead vc 1000000 bc 64 eq 0 8.4869 ms/op 7.5317 ms/op 1.13
forkChoice updateHead vc 600000 bc 320 eq 0 4.8796 ms/op 4.3029 ms/op 1.13
forkChoice updateHead vc 600000 bc 1200 eq 0 5.2705 ms/op 4.3211 ms/op 1.22
forkChoice updateHead vc 600000 bc 7200 eq 0 6.1132 ms/op 5.6189 ms/op 1.09
forkChoice updateHead vc 600000 bc 64 eq 1000 11.872 ms/op 11.190 ms/op 1.06
forkChoice updateHead vc 600000 bc 64 eq 10000 14.108 ms/op 11.894 ms/op 1.19
forkChoice updateHead vc 600000 bc 64 eq 300000 53.658 ms/op 16.394 ms/op 3.27
computeDeltas 500000 validators 300 proto nodes 7.5840 ms/op 6.6313 ms/op 1.14
computeDeltas 500000 validators 1200 proto nodes 7.1059 ms/op 6.7779 ms/op 1.05
computeDeltas 500000 validators 7200 proto nodes 7.0956 ms/op 6.6595 ms/op 1.07
computeDeltas 750000 validators 300 proto nodes 10.271 ms/op 10.049 ms/op 1.02
computeDeltas 750000 validators 1200 proto nodes 10.346 ms/op 9.9207 ms/op 1.04
computeDeltas 750000 validators 7200 proto nodes 10.463 ms/op 10.104 ms/op 1.04
computeDeltas 1400000 validators 300 proto nodes 20.518 ms/op 20.093 ms/op 1.02
computeDeltas 1400000 validators 1200 proto nodes 20.719 ms/op 19.740 ms/op 1.05
computeDeltas 1400000 validators 7200 proto nodes 23.084 ms/op 20.855 ms/op 1.11
computeDeltas 2100000 validators 300 proto nodes 33.030 ms/op 30.900 ms/op 1.07
computeDeltas 2100000 validators 1200 proto nodes 28.783 ms/op 30.338 ms/op 0.95
computeDeltas 2100000 validators 7200 proto nodes 29.969 ms/op 29.257 ms/op 1.02
altair processAttestation - 250000 vs - 7PWei normalcase 2.5761 ms/op 2.9732 ms/op 0.87
altair processAttestation - 250000 vs - 7PWei worstcase 4.0761 ms/op 4.2030 ms/op 0.97
altair processAttestation - setStatus - 1/6 committees join 207.58 us/op 197.11 us/op 1.05
altair processAttestation - setStatus - 1/3 committees join 374.92 us/op 380.99 us/op 0.98
altair processAttestation - setStatus - 1/2 committees join 521.03 us/op 510.41 us/op 1.02
altair processAttestation - setStatus - 2/3 committees join 682.68 us/op 623.70 us/op 1.09
altair processAttestation - setStatus - 4/5 committees join 920.12 us/op 880.31 us/op 1.05
altair processAttestation - setStatus - 100% committees join 1.0748 ms/op 1.0229 ms/op 1.05
altair processBlock - 250000 vs - 7PWei normalcase 11.796 ms/op 12.079 ms/op 0.98
altair processBlock - 250000 vs - 7PWei normalcase hashState 39.890 ms/op 37.690 ms/op 1.06
altair processBlock - 250000 vs - 7PWei worstcase 52.394 ms/op 41.652 ms/op 1.26
altair processBlock - 250000 vs - 7PWei worstcase hashState 107.57 ms/op 104.64 ms/op 1.03
phase0 processBlock - 250000 vs - 7PWei normalcase 3.7330 ms/op 3.0934 ms/op 1.21
phase0 processBlock - 250000 vs - 7PWei worstcase 37.213 ms/op 34.530 ms/op 1.08
altair processEth1Data - 250000 vs - 7PWei normalcase 904.44 us/op 715.75 us/op 1.26
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15 19.203 us/op 17.176 us/op 1.12
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219 78.814 us/op 50.923 us/op 1.55
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42 24.249 us/op 22.845 us/op 1.06
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18 24.501 us/op 20.326 us/op 1.21
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020 239.53 us/op 221.68 us/op 1.08
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777 1.8345 ms/op 1.4775 ms/op 1.24
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 2.6405 ms/op 1.8075 ms/op 1.46
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 2.4274 ms/op 1.9522 ms/op 1.24
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384 6.2301 ms/op 4.5208 ms/op 1.38
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384 4.3385 ms/op 2.7895 ms/op 1.56
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384 9.3184 ms/op 7.0573 ms/op 1.32
Tree 40 250000 create 833.25 ms/op 374.44 ms/op 2.23
Tree 40 250000 get(125000) 232.02 ns/op 222.82 ns/op 1.04
Tree 40 250000 set(125000) 2.3119 us/op 1.0788 us/op 2.14
Tree 40 250000 toArray() 27.120 ms/op 21.927 ms/op 1.24
Tree 40 250000 iterate all - toArray() + loop 26.708 ms/op 20.944 ms/op 1.28
Tree 40 250000 iterate all - get(i) 78.111 ms/op 71.056 ms/op 1.10
MutableVector 250000 create 19.047 ms/op 13.424 ms/op 1.42
MutableVector 250000 get(125000) 6.9690 ns/op 6.5310 ns/op 1.07
MutableVector 250000 set(125000) 371.89 ns/op 293.10 ns/op 1.27
MutableVector 250000 toArray() 4.1939 ms/op 3.4342 ms/op 1.22
MutableVector 250000 iterate all - toArray() + loop 4.5558 ms/op 3.9579 ms/op 1.15
MutableVector 250000 iterate all - get(i) 1.6062 ms/op 1.5573 ms/op 1.03
Array 250000 create 4.1023 ms/op 3.5807 ms/op 1.15
Array 250000 clone - spread 1.6276 ms/op 1.4360 ms/op 1.13
Array 250000 get(125000) 1.6940 ns/op 1.1810 ns/op 1.43
Array 250000 set(125000) 6.0280 ns/op 4.3900 ns/op 1.37
Array 250000 iterate all - loop 199.72 us/op 170.33 us/op 1.17
effectiveBalanceIncrements clone Uint8Array 300000 151.57 us/op 41.806 us/op 3.63
effectiveBalanceIncrements clone MutableVector 300000 714.00 ns/op 454.00 ns/op 1.57
effectiveBalanceIncrements rw all Uint8Array 300000 228.63 us/op 201.46 us/op 1.13
effectiveBalanceIncrements rw all MutableVector 300000 112.52 ms/op 98.074 ms/op 1.15
phase0 afterProcessEpoch - 250000 vs - 7PWei 133.66 ms/op 114.39 ms/op 1.17
phase0 beforeProcessEpoch - 250000 vs - 7PWei 50.052 ms/op 52.825 ms/op 0.95
altair processEpoch - mainnet_e81889 603.94 ms/op 532.72 ms/op 1.13
mainnet_e81889 - altair beforeProcessEpoch 123.25 ms/op 86.994 ms/op 1.42
mainnet_e81889 - altair processJustificationAndFinalization 31.727 us/op 16.593 us/op 1.91
mainnet_e81889 - altair processInactivityUpdates 9.4697 ms/op 6.9656 ms/op 1.36
mainnet_e81889 - altair processRewardsAndPenalties 88.215 ms/op 45.878 ms/op 1.92
mainnet_e81889 - altair processRegistryUpdates 3.3550 us/op 3.1020 us/op 1.08
mainnet_e81889 - altair processSlashings 763.00 ns/op 443.00 ns/op 1.72
mainnet_e81889 - altair processEth1DataReset 905.00 ns/op 693.00 ns/op 1.31
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.7263 ms/op 1.4429 ms/op 1.20
mainnet_e81889 - altair processSlashingsReset 4.7390 us/op 5.7030 us/op 0.83
mainnet_e81889 - altair processRandaoMixesReset 6.8990 us/op 5.6310 us/op 1.23
mainnet_e81889 - altair processHistoricalRootsUpdate 1.0040 us/op 743.00 ns/op 1.35
mainnet_e81889 - altair processParticipationFlagUpdates 2.9130 us/op 3.2820 us/op 0.89
mainnet_e81889 - altair processSyncCommitteeUpdates 1.0030 us/op 770.00 ns/op 1.30
mainnet_e81889 - altair afterProcessEpoch 127.34 ms/op 120.87 ms/op 1.05
capella processEpoch - mainnet_e217614 2.4527 s/op 2.4569 s/op 1.00
mainnet_e217614 - capella beforeProcessEpoch 556.39 ms/op 550.79 ms/op 1.01
mainnet_e217614 - capella processJustificationAndFinalization 29.142 us/op 20.586 us/op 1.42
mainnet_e217614 - capella processInactivityUpdates 24.392 ms/op 20.308 ms/op 1.20
mainnet_e217614 - capella processRewardsAndPenalties 524.43 ms/op 412.34 ms/op 1.27
mainnet_e217614 - capella processRegistryUpdates 30.934 us/op 37.181 us/op 0.83
mainnet_e217614 - capella processSlashings 716.00 ns/op 523.00 ns/op 1.37
mainnet_e217614 - capella processEth1DataReset 476.00 ns/op 544.00 ns/op 0.88
mainnet_e217614 - capella processEffectiveBalanceUpdates 13.656 ms/op 10.458 ms/op 1.31
mainnet_e217614 - capella processSlashingsReset 4.6480 us/op 4.5790 us/op 1.02
mainnet_e217614 - capella processRandaoMixesReset 9.6650 us/op 4.9160 us/op 1.97
mainnet_e217614 - capella processHistoricalRootsUpdate 893.00 ns/op 695.00 ns/op 1.28
mainnet_e217614 - capella processParticipationFlagUpdates 2.8670 us/op 1.6930 us/op 1.69
mainnet_e217614 - capella afterProcessEpoch 322.55 ms/op 335.29 ms/op 0.96
phase0 processEpoch - mainnet_e58758 527.12 ms/op 472.09 ms/op 1.12
mainnet_e58758 - phase0 beforeProcessEpoch 145.99 ms/op 140.90 ms/op 1.04
mainnet_e58758 - phase0 processJustificationAndFinalization 19.443 us/op 16.422 us/op 1.18
mainnet_e58758 - phase0 processRewardsAndPenalties 56.603 ms/op 54.088 ms/op 1.05
mainnet_e58758 - phase0 processRegistryUpdates 17.856 us/op 10.610 us/op 1.68
mainnet_e58758 - phase0 processSlashings 781.00 ns/op 521.00 ns/op 1.50
mainnet_e58758 - phase0 processEth1DataReset 517.00 ns/op 410.00 ns/op 1.26
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.3655 ms/op 1.1715 ms/op 1.17
mainnet_e58758 - phase0 processSlashingsReset 3.2060 us/op 2.7330 us/op 1.17
mainnet_e58758 - phase0 processRandaoMixesReset 6.0420 us/op 4.0110 us/op 1.51
mainnet_e58758 - phase0 processHistoricalRootsUpdate 634.00 ns/op 362.00 ns/op 1.75
mainnet_e58758 - phase0 processParticipationRecordUpdates 8.7420 us/op 4.7320 us/op 1.85
mainnet_e58758 - phase0 afterProcessEpoch 105.80 ms/op 96.590 ms/op 1.10
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.6620 ms/op 1.4476 ms/op 1.15
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 2.8418 ms/op 1.5441 ms/op 1.84
altair processInactivityUpdates - 250000 normalcase 33.664 ms/op 33.371 ms/op 1.01
altair processInactivityUpdates - 250000 worstcase 28.849 ms/op 33.123 ms/op 0.87
phase0 processRegistryUpdates - 250000 normalcase 13.196 us/op 9.8600 us/op 1.34
phase0 processRegistryUpdates - 250000 badcase_full_deposits 465.47 us/op 434.29 us/op 1.07
phase0 processRegistryUpdates - 250000 worstcase 0.5 168.31 ms/op 155.69 ms/op 1.08
altair processRewardsAndPenalties - 250000 normalcase 55.242 ms/op 62.111 ms/op 0.89
altair processRewardsAndPenalties - 250000 worstcase 51.925 ms/op 61.482 ms/op 0.84
phase0 getAttestationDeltas - 250000 normalcase 8.8387 ms/op 11.423 ms/op 0.77
phase0 getAttestationDeltas - 250000 worstcase 8.6454 ms/op 11.792 ms/op 0.73
phase0 processSlashings - 250000 worstcase 125.23 us/op 133.33 us/op 0.94
altair processSyncCommitteeUpdates - 250000 167.43 ms/op 167.29 ms/op 1.00
BeaconState.hashTreeRoot - No change 503.00 ns/op 674.00 ns/op 0.75
BeaconState.hashTreeRoot - 1 full validator 162.97 us/op 185.64 us/op 0.88
BeaconState.hashTreeRoot - 32 full validator 1.7113 ms/op 1.6074 ms/op 1.06
BeaconState.hashTreeRoot - 512 full validator 16.939 ms/op 19.137 ms/op 0.89
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 196.94 us/op 200.65 us/op 0.98
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 2.4560 ms/op 2.7472 ms/op 0.89
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 35.590 ms/op 33.597 ms/op 1.06
BeaconState.hashTreeRoot - 1 balances 138.50 us/op 145.51 us/op 0.95
BeaconState.hashTreeRoot - 32 balances 1.3389 ms/op 1.1830 ms/op 1.13
BeaconState.hashTreeRoot - 512 balances 13.159 ms/op 14.032 ms/op 0.94
BeaconState.hashTreeRoot - 250000 balances 237.03 ms/op 231.36 ms/op 1.02
aggregationBits - 2048 els - zipIndexesInBitList 19.327 us/op 21.947 us/op 0.88
byteArrayEquals 32 76.991 ns/op 77.535 ns/op 0.99
Buffer.compare 32 56.844 ns/op 58.306 ns/op 0.97
byteArrayEquals 1024 2.1016 us/op 2.1079 us/op 1.00
Buffer.compare 1024 72.846 ns/op 73.945 ns/op 0.99
byteArrayEquals 16384 33.270 us/op 33.803 us/op 0.98
Buffer.compare 16384 275.82 ns/op 270.67 ns/op 1.02
byteArrayEquals 123687377 266.85 ms/op 263.83 ms/op 1.01
Buffer.compare 123687377 7.9378 ms/op 10.284 ms/op 0.77
byteArrayEquals 32 - diff last byte 74.644 ns/op 78.304 ns/op 0.95
Buffer.compare 32 - diff last byte 57.095 ns/op 57.925 ns/op 0.99
byteArrayEquals 1024 - diff last byte 2.1210 us/op 2.2443 us/op 0.95
Buffer.compare 1024 - diff last byte 71.631 ns/op 74.049 ns/op 0.97
byteArrayEquals 16384 - diff last byte 35.226 us/op 35.555 us/op 0.99
Buffer.compare 16384 - diff last byte 282.04 ns/op 261.95 ns/op 1.08
byteArrayEquals 123687377 - diff last byte 262.69 ms/op 261.82 ms/op 1.00
Buffer.compare 123687377 - diff last byte 7.9700 ms/op 8.4688 ms/op 0.94
byteArrayEquals 32 - random bytes 6.1320 ns/op 6.5570 ns/op 0.94
Buffer.compare 32 - random bytes 62.156 ns/op 63.932 ns/op 0.97
byteArrayEquals 1024 - random bytes 5.5990 ns/op 6.3180 ns/op 0.89
Buffer.compare 1024 - random bytes 61.447 ns/op 63.106 ns/op 0.97
byteArrayEquals 16384 - random bytes 5.6380 ns/op 6.3000 ns/op 0.89
Buffer.compare 16384 - random bytes 61.764 ns/op 63.194 ns/op 0.98
byteArrayEquals 123687377 - random bytes 8.5600 ns/op 9.3700 ns/op 0.91
Buffer.compare 123687377 - random bytes 70.430 ns/op 75.220 ns/op 0.94
regular array get 100000 times 46.646 us/op 47.132 us/op 0.99
wrappedArray get 100000 times 46.030 us/op 47.403 us/op 0.97
arrayWithProxy get 100000 times 14.540 ms/op 14.802 ms/op 0.98
ssz.Root.equals 56.934 ns/op 56.130 ns/op 1.01
byteArrayEquals 55.380 ns/op 55.481 ns/op 1.00
Buffer.compare 12.099 ns/op 12.210 ns/op 0.99
shuffle list - 16384 els 7.2139 ms/op 7.2286 ms/op 1.00
shuffle list - 250000 els 106.92 ms/op 108.52 ms/op 0.99
processSlot - 1 slots 18.643 us/op 18.366 us/op 1.02
processSlot - 32 slots 3.7784 ms/op 4.5940 ms/op 0.82
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 64.353 ms/op 63.490 ms/op 1.01
getCommitteeAssignments - req 1 vs - 250000 vc 2.6906 ms/op 2.6030 ms/op 1.03
getCommitteeAssignments - req 100 vs - 250000 vc 3.9675 ms/op 3.7253 ms/op 1.07
getCommitteeAssignments - req 1000 vs - 250000 vc 4.4698 ms/op 4.1100 ms/op 1.09
findModifiedValidators - 10000 modified validators 598.77 ms/op 547.68 ms/op 1.09
findModifiedValidators - 1000 modified validators 509.68 ms/op 462.63 ms/op 1.10
findModifiedValidators - 100 modified validators 508.52 ms/op 427.58 ms/op 1.19
findModifiedValidators - 10 modified validators 464.93 ms/op 433.52 ms/op 1.07
findModifiedValidators - 1 modified validators 423.23 ms/op 413.46 ms/op 1.02
findModifiedValidators - no difference 444.13 ms/op 400.03 ms/op 1.11
compare ViewDUs 5.2441 s/op 4.5912 s/op 1.14
compare each validator Uint8Array 2.3038 s/op 1.6007 s/op 1.44
compare ViewDU to Uint8Array 1.6976 s/op 1.3866 s/op 1.22
migrate state 1000000 validators, 24 modified, 0 new 1.0499 s/op 863.24 ms/op 1.22
migrate state 1000000 validators, 1700 modified, 1000 new 1.3730 s/op 1.0714 s/op 1.28
migrate state 1000000 validators, 3400 modified, 2000 new 1.7027 s/op 1.3535 s/op 1.26
migrate state 1500000 validators, 24 modified, 0 new 994.52 ms/op 827.74 ms/op 1.20
migrate state 1500000 validators, 1700 modified, 1000 new 1.3673 s/op 1.0906 s/op 1.25
migrate state 1500000 validators, 3400 modified, 2000 new 2.1249 s/op 1.3264 s/op 1.60
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 6.5500 ns/op 4.4500 ns/op 1.47
state getBlockRootAtSlot - 250000 vs - 7PWei 1.4677 us/op 557.37 ns/op 2.63
computeProposers - vc 250000 14.423 ms/op 9.3982 ms/op 1.53
computeEpochShuffling - vc 250000 116.91 ms/op 106.92 ms/op 1.09
getNextSyncCommittee - vc 250000 210.92 ms/op 159.91 ms/op 1.32
computeSigningRoot for AttestationData 40.933 us/op 26.436 us/op 1.55
hash AttestationData serialized data then Buffer.toString(base64) 2.4817 us/op 2.4054 us/op 1.03
toHexString serialized data 1.4203 us/op 1.1814 us/op 1.20
Buffer.toString(base64) 274.00 ns/op 245.01 ns/op 1.12

by benchmarkbot/action

nazarhussain
nazarhussain commented Feb 6, 2024
View reviewed changes
packages/api/test/unit/beacon/genericServerTest/beacon.test.ts Outdated
@@ -6,7 +6,7 @@ import {getRoutes} from "../../../../src/beacon/server/beacon.js";
import {runGenericServerTest} from "../../../utils/genericServerTest.js";
import {testData} from "../testData/beacon.js";

describe("beacon / beacon", () => {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All unit tests run concurrently. These tests open up a HTTP Server on random ports and some time those ports collide. Running such tests sequentially is the only way to make sure ports never collide.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we might be able to specify port 0 for the server, this will result in asking the operating system to assign a free (ephemeral) port

nazarhussain
nazarhussain commented Feb 6, 2024
View reviewed changes
packages/validator/src/util/clock.ts
@@ -138,7 +138,7 @@ export class Clock implements IClock {
*/
export function getCurrentSlotAround(config: ChainForkConfig, genesisTime: TimeSeconds): Slot {
const diffInSeconds = Date.now() / 1000 - genesisTime;
const slotsSinceGenesis = Math.round(diffInSeconds / config.SECONDS_PER_SLOT);
const slotsSinceGenesis = Math.floor(diffInSeconds / config.SECONDS_PER_SLOT);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tests fixtures tend to align with Math.floor to use 12.5s still as 12.

nflaig
nflaig reviewed Feb 6, 2024
View reviewed changes
packages/api/test/unit/beacon/genericServerTest/beacon.test.ts Outdated
@@ -6,7 +6,7 @@ import {getRoutes} from "../../../../src/beacon/server/beacon.js";
import {runGenericServerTest} from "../../../utils/genericServerTest.js";
import {testData} from "../testData/beacon.js";

describe("beacon / beacon", () => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we might be able to specify port 0 for the server, this will result in asking the operating system to assign a free (ephemeral) port

packages/validator/test/unit/utils/clock.test.ts Outdated Show resolved Hide resolved
@wemeetagain wemeetagain merged commit 2a8342d into unstable Feb 6, 2024
11 of 15 checks passed
@wemeetagain wemeetagain deleted the nh/flaky-tests branch February 6, 2024 19:39
@wemeetagain wemeetagain mentioned this pull request Feb 7, 2024
Merged
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.16.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wemeetagain wemeetagain wemeetagain approved these changes

@nflaig nflaig Awaiting requested review from nflaig

Assignees

@nazarhussain nazarhussain

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@nazarhussain @wemeetagain @nflaig