Upstream 2.60.2 (#419)

* fix Consensus specification tests CI (erigontech#10391) (erigontech#10396) Cherry-pick: erigontech@bc5fa6f Need this to get PR CI green for v2.60.1 patches, e.g. - erigontech#10390 Co-authored-by: Andrew Ashikhmin <[email protected]> * rpc/handler: do not append null to stream when json may be valid (erigontech#10390) Cherry-pick: erigontech@4d1c954 Relates to: erigontech#10376 * Fixed Bor Log appearing on Ethereum Mainnet (erigontech#10405) (erigontech#10420) Cherry-pick: erigontech@be889f6 Co-authored-by: Giulio rebuffo <[email protected]> * fix gas price not right problem (erigontech#10456) Cherry pick PR erigontech#10451 into the release branch Co-authored-by: mars <[email protected]> * eth_estimateGas: default feeCap to base fee (erigontech#10499) Copy PR erigontech#10495 into the release branch * Add flag for bor waypoint types (erigontech#10501) Cherry pick PR erigontech#10281 into the release branch Co-authored-by: Mark Holt <[email protected]> Co-authored-by: alex.sharov <[email protected]> * try to fix 'method handler crashed' for debug_traceCall of erigontech#9090 (erigontech#10502) Cherry pick PR erigontech#10401 into the release branch Co-authored-by: mars <[email protected]> * diagnostics: cherry pick speedtest disable (erigontech#10509) Cherry pick PR erigontech#10449 into the release branch * Enable DNS p2p discovery on holesky (erigontech#10507) Cherry pick PR erigontech#10460 into the release branch Co-authored-by: Willian Mitsuda <[email protected]> * fix eth_call 'method handler crashed' error when tx has set maxFeePerBlobGas (erigontech#10506) Cherry pick PR erigontech#10452 into the release branch Co-authored-by: mars <[email protected]> * e2: remove overlapped files only after merge (erigontech#10487) Otherwise: if start after `kill -9` in the middle of merge - may remove small files of 1 type of file, but leave small files of another type of files (which merge was not finished) - and leave node in un-mergable state: erigontech#10485 --------- Co-authored-by: awskii <[email protected]> * add flag checking for pruning waypoints (erigontech#10508) Cherry pick PR erigontech#10468 into the release branch Co-authored-by: Mark Holt <[email protected]> * p2p/sentry: sentry doesn't start with ErrNoHead (erigontech#10454) (erigontech#10523) cherry-pick erigontech#10494 to release/2.60 * add lock to purgeMilestoneIDsList (erigontech#10524) Cherry pick PR erigontech#10493 into the release branch Co-authored-by: Mark Holt <[email protected]> * polygon/heimdall: fix checkpoint json marshalling (erigontech#10530) Fixes a recent regression causing unwinds due to checkpoints having zero root hash: ``` [WARN] [05-18|23:58:54.662] [bor] Root hash mismatch while whitelisting checkpoint expected=ac1c57270479250af3ce8eee90075cd8b2ba1bac55353105e063d9a4c87c743e got=0000000000000000000000000000000000000000000000000000000000000000 [WARN] [05-18|23:58:54.662] [bor] Rewinding chain due to checkpoint root hash mismatch number=57125727 ``` Note this has already been fixed on Erigon 3 branch but as part of a non-related PR - https://github.com/ledgerwatch/erigon/pull/10124/files#diff-47d4532f399f2d6a45e6f19944a45c80bac573b4d1b5cb51485d0254229d1b16 * Fix capacity for immediate appends (erigontech#10539) Cherry pick PR erigontech#10528 into the release branch Co-authored-by: Shoham Chakraborty <[email protected]> * core/vm: set tracer-observable value of a delegatecall to match parent value (erigontech#10370) requested by erigontech#9549 port of ethereum/go-ethereum#26632 * params: version 2.60.1 (erigontech#10555) * blobGasPrice should be marshalled as hex (erigontech#10571) Cherry pick PR erigontech#10551 into the release branch * Caplin: Fixed reforwarding of Bls Execution changes (erigontech#10577) Cherry pick PR erigontech#10546 into the release branch Co-authored-by: Giulio rebuffo <[email protected]> * Caplin: Proper "Normalization" of length of ForkVersions to 8 hex characters (erigontech#10578) Cherry pick PR erigontech#10512 into the release branch Co-authored-by: Giulio rebuffo <[email protected]> * Caplin: Update BlobSidecars Beacon API endpoint to the latest specs (erigontech#10580) Cherry pick PR erigontech#10576 into the release branch Co-authored-by: Giulio rebuffo <[email protected]> * bor blocks retire: infinity loop fix (erigontech#10596) Problem: `+1` was added to maxBlockNum instead of minBlockNum for: erigontech#10554 * txpool: EIP-3860 should only apply to create transactions (erigontech#10609) This fixes Issue erigontech#10607 * qa-tests: update 2.60.x test workflows from main (erigontech#10627) * Fix potential p2p shutdown hangup (erigontech#10626) This is a fix for: erigontech#10192 This fixes is a deadlock in v4_udp.go where * Thread A waits on mutex.Lock() in resetTimeout() called after reading listUpdate channel. * Thread B waits on listUpdate <- plist.PushBack(p) called after locking mutex.Lock() This fix decouples the list operations which need locking from the channel operations which don't by storing the changes in local variables. These updates are used for resetting a timeout - which is not order dependent. * downloader: Number of DNS requests seem excessive (erigontech#5145) (erigontech#10739) cherry-pick erigontech#10693 to release * rpc: Fix incorrect txfeecap (erigontech#10643) Cherry pick PR erigontech#10636 to Erigon 2 * downloader: don't block erigon startup if devs deploy new hashes (of same files) (erigontech#10761) * skip hidden files when list files with given extension (erigontech#10654) for erigontech#10644 * qa-tests: backport to release/2.60 improvements made to e3 github action workflows (erigontech#10778) This PR backports improvements that we added to the E3 tests: recording runner name and db version used for testing on MongoDB database. * e2: more snaps (all networks) (erigontech#10794) * e2: configurable hashers amount (erigontech#10785) * Revert "e2: configurable hashers amount" (erigontech#10834) * diagnostics: move E3 changes to E2 (erigontech#10806) Merged all the work done from main branch to keep diagnostics up to date. * Downloader: fix staticpeers flag (erigontech#10798) Cherry pick erigontech#10792 * Fix NewPayload Validation during header download (erigontech#10837) Cherry pick PR erigontech#10093 into the release branch Co-authored-by: Minhyuk Kim <[email protected]> * e2: mainnet blob 9.3M (erigontech#10842) * Fix gas fee calculation for debug calls (erigontech#10880) Cherry pick PR erigontech#10825 into the release branch Co-authored-by: Minhyuk Kim <[email protected]> * Revert "eth_estimateGas: default feeCap to base fee (erigontech#10499)" (erigontech#10904) This reverts PR erigontech#10499. See erigontech#10495 (comment) and PR erigontech#10901 * params: version 2.60.2 (erigontech#10905) * upstream v2.60.2 * fix ci lint --------- Co-authored-by: milen <[email protected]> Co-authored-by: Andrew Ashikhmin <[email protected]> Co-authored-by: Giulio rebuffo <[email protected]> Co-authored-by: mars <[email protected]> Co-authored-by: Mark Holt <[email protected]> Co-authored-by: alex.sharov <[email protected]> Co-authored-by: Dmytro <[email protected]> Co-authored-by: Willian Mitsuda <[email protected]> Co-authored-by: awskii <[email protected]> Co-authored-by: battlmonstr <[email protected]> Co-authored-by: Shoham Chakraborty <[email protected]> Co-authored-by: Michelangelo Riccobene <[email protected]> Co-authored-by: Minhyuk Kim <[email protected]>
node-real · Jun 27, 2024 · 08721af · 08721af
1 parent a533d02
commit 08721af
Show file tree

Hide file tree

Showing 74 changed files with 2,171 additions and 498 deletions.
diff --git a/.github/workflows/qa-clean-exit-block-downloading.yml b/.github/workflows/qa-clean-exit-block-downloading.yml
@@ -13,13 +13,14 @@ on:
   workflow_dispatch:     # Run manually
 
 jobs:
-  long-running-test:
+  clean-exit-bd-test:
     runs-on: [self-hosted, Erigon2]
     env:
       ERIGON_REFERENCE_DATA_DIR: /opt/erigon-versions/reference-version/datadir
       ERIGON_TESTBED_DATA_DIR: /opt/erigon-testbed/datadir
       ERIGON_QA_PATH: /home/qarunner/erigon-qa
       WORKING_TIME_SECONDS: 600
+      CHAIN: mainnet
 
     steps:
     - name: Check out repository
@@ -48,7 +49,7 @@ jobs:
         set +e # Disable exit on error
         
         # Run Erigon, send ctrl-c and check logs
-        python3 $ERIGON_QA_PATH/test_system/qa-tests/clean-exit/run_and_check_clean_exit.py ${{ github.workspace }}/build/bin $ERIGON_TESTBED_DATA_DIR $WORKING_TIME_SECONDS
+        python3 $ERIGON_QA_PATH/test_system/qa-tests/clean-exit/run_and_check_clean_exit.py ${{ github.workspace }}/build/bin $ERIGON_TESTBED_DATA_DIR $WORKING_TIME_SECONDS Erigon2
   
         # Capture monitoring script exit status
         test_exit_status=$?
@@ -85,7 +86,13 @@ jobs:
       if: steps.test_step.outputs.test_executed == 'true'
       env:
         TEST_RESULT: ${{ steps.test_step.outputs.TEST_RESULT }}
-      run: python3 $ERIGON_QA_PATH/test_system/qa-tests/uploads/upload_test_results.py --repo erigon --commit $(git rev-parse HEAD) --test_name clean-exit-block-downloading --outcome $TEST_RESULT --result_file ${{ github.workspace }}/result.json
+      run: |
+        db_version=$(python3 $ERIGON_QA_PATH/test_system/qa-tests/uploads/prod_info.py $ERIGON_REFERENCE_DATA_DIR/../production.ini production erigon_repo_commit)
+        if [ -z "$db_version" ]; then
+          db_version="no-version"
+        fi
+        
+        python3 $ERIGON_QA_PATH/test_system/qa-tests/uploads/upload_test_results.py --repo erigon --commit $(git rev-parse HEAD) --branch ${{ github.ref_name }} --test_name clean-exit-block-downloading --chain $CHAIN --runner ${{ runner.name }} --db_version $db_version --outcome $TEST_RESULT --result_file ${{ github.workspace }}/result.json
 
     - name: Upload test results
       if: steps.test_step.outputs.test_executed == 'true'

diff --git a/.github/workflows/qa-clean-exit-snapshot-downloading.yml b/.github/workflows/qa-clean-exit-snapshot-downloading.yml
@@ -14,12 +14,13 @@ on:
   workflow_dispatch:     # Run manually
 
 jobs:
-  long-running-test:
+  clean-exit-sd-test:
     runs-on: self-hosted
     env:
       ERIGON_DATA_DIR: ${{ github.workspace }}/erigon_data
       ERIGON_QA_PATH: /home/qarunner/erigon-qa
       WORKING_TIME_SECONDS: 600
+      CHAIN: mainnet
 
     steps:
     - name: Check out repository
@@ -45,7 +46,7 @@ jobs:
         set +e # Disable exit on error
         
         # Run Erigon, send ctrl-c and check logs
-        python3 $ERIGON_QA_PATH/test_system/qa-tests/clean-exit/run_and_check_clean_exit.py ${{ github.workspace }}/build/bin $ERIGON_DATA_DIR $WORKING_TIME_SECONDS
+        python3 $ERIGON_QA_PATH/test_system/qa-tests/clean-exit/run_and_check_clean_exit.py ${{ github.workspace }}/build/bin $ERIGON_DATA_DIR $WORKING_TIME_SECONDS Erigon2
   
         # Capture monitoring script exit status
         test_exit_status=$?
@@ -82,7 +83,7 @@ jobs:
       if: steps.test_step.outputs.test_executed == 'true'
       env:
         TEST_RESULT: ${{ steps.test_step.outputs.TEST_RESULT }}
-      run: python3 $ERIGON_QA_PATH/test_system/qa-tests/uploads/upload_test_results.py --repo erigon --commit $(git rev-parse HEAD) --test_name clean-exit-snapshot-downloading --outcome $TEST_RESULT --result_file ${{ github.workspace }}/result.json
+      run: python3 $ERIGON_QA_PATH/test_system/qa-tests/uploads/upload_test_results.py --repo erigon --commit $(git rev-parse HEAD) --branch ${{ github.ref_name }} --test_name clean-exit-snapshot-downloading --chain $CHAIN --runner ${{ runner.name }} --outcome $TEST_RESULT --result_file ${{ github.workspace }}/result.json
 
     - name: Upload test results
       if: steps.test_step.outputs.test_executed == 'true'

diff --git a/.github/workflows/qa-snap-download.yml b/.github/workflows/qa-snap-download.yml
@@ -12,13 +12,14 @@ on:
   workflow_dispatch:     # Run manually
 
 jobs:
-  long-running-test:
+  snap-download-test:
     runs-on: [self-hosted, Erigon2]
+    timeout-minutes: 800
     env:
       ERIGON_DATA_DIR: ${{ github.workspace }}/erigon_data
       ERIGON_QA_PATH: /home/qarunner/erigon-qa
-      TRACKING_TIME_SECONDS: 14400 # 4 hours
-      TOTAL_TIME_SECONDS: 28800 # 8 hours
+      TOTAL_TIME_SECONDS: 43200 # 12 hours
+      CHAIN: mainnet
 
     steps:
     - name: Check out repository
@@ -44,7 +45,7 @@ jobs:
         set +e # Disable exit on error
              
         # Run Erigon, monitor snapshot downloading and check logs
-        python3 $ERIGON_QA_PATH/test_system/qa-tests/snap-download/run_and_check_snap_download.py ${{ github.workspace }}/build/bin $ERIGON_DATA_DIR $TOTAL_TIME_SECONDS
+        python3 $ERIGON_QA_PATH/test_system/qa-tests/snap-download/run_and_check_snap_download.py ${{ github.workspace }}/build/bin $ERIGON_DATA_DIR $TOTAL_TIME_SECONDS Erigon2 $CHAIN
   
         # Capture monitoring script exit status
         test_exit_status=$?
@@ -81,14 +82,14 @@ jobs:
       if: steps.test_step.outputs.test_executed == 'true'
       env:
         TEST_RESULT: ${{ steps.test_step.outputs.TEST_RESULT }}
-      run: python3 $ERIGON_QA_PATH/test_system/qa-tests/uploads/upload_test_results.py --repo erigon --commit $(git rev-parse HEAD) --test_name snap-download --outcome $TEST_RESULT --result_file ${{ github.workspace }}/result.json
+      run: python3 $ERIGON_QA_PATH/test_system/qa-tests/uploads/upload_test_results.py --repo erigon --commit $(git rev-parse HEAD) --branch ${{ github.ref_name }} --test_name snap-download --chain $CHAIN --runner ${{ runner.name }} --outcome $TEST_RESULT --result_file ${{ github.workspace }}/result-$CHAIN.json
 
     - name: Upload test results
       if: steps.test_step.outputs.test_executed == 'true'
       uses: actions/upload-artifact@v4
       with:
         name: test-results
-        path: ${{ github.workspace }}/result.json
+        path: ${{ github.workspace }}/result-${{ env.CHAIN }}.json
 
     - name: Action for Success
       if: steps.test_step.outputs.TEST_RESULT == 'success'

diff --git a/.github/workflows/qa-tip-tracking.yml b/.github/workflows/qa-tip-tracking.yml
@@ -12,14 +12,16 @@ on:
   workflow_dispatch:     # Run manually
 
 jobs:
-  long-running-test:
+  tip-tracking-test:
     runs-on: [self-hosted, Erigon2]
+    timeout-minutes: 600
     env:
       ERIGON_REFERENCE_DATA_DIR: /opt/erigon-versions/reference-version/datadir
       ERIGON_TESTBED_DATA_DIR: /opt/erigon-testbed/datadir
       ERIGON_QA_PATH: /home/qarunner/erigon-qa
       TRACKING_TIME_SECONDS: 14400 # 4 hours
       TOTAL_TIME_SECONDS: 28800 # 8 hours
+      CHAIN: mainnet
 
     steps:
     - name: Check out repository
@@ -50,7 +52,7 @@ jobs:
         # 1. Launch the testbed Erigon instance
         # 2. Allow time for the Erigon to achieve synchronization
         # 3. Begin timing the duration that Erigon maintains synchronization
-        python3 $ERIGON_QA_PATH/test_system/qa-tests/tip-tracking/run_and_check_tip_tracking.py ${{ github.workspace }}/build/bin $ERIGON_TESTBED_DATA_DIR $TRACKING_TIME_SECONDS $TOTAL_TIME_SECONDS
+        python3 $ERIGON_QA_PATH/test_system/qa-tests/tip-tracking/run_and_check_tip_tracking.py ${{ github.workspace }}/build/bin $ERIGON_TESTBED_DATA_DIR $TRACKING_TIME_SECONDS $TOTAL_TIME_SECONDS Erigon2 $CHAIN
   
         # Capture monitoring script exit status
         test_exit_status=$?
@@ -87,14 +89,20 @@ jobs:
       if: steps.test_step.outputs.test_executed == 'true'
       env:
         TEST_RESULT: ${{ steps.test_step.outputs.TEST_RESULT }}
-      run: python3 $ERIGON_QA_PATH/test_system/qa-tests/uploads/upload_test_results.py --repo erigon --commit $(git rev-parse HEAD) --test_name tip-tracking --outcome $TEST_RESULT --result_file ${{ github.workspace }}/result.json
+      run: |
+        db_version=$(python3 $ERIGON_QA_PATH/test_system/qa-tests/uploads/prod_info.py $ERIGON_REFERENCE_DATA_DIR/../production.ini production erigon_repo_commit)
+        if [ -z "$db_version" ]; then
+          db_version="no-version"
+        fi
+      
+        python3 $ERIGON_QA_PATH/test_system/qa-tests/uploads/upload_test_results.py --repo erigon --commit $(git rev-parse HEAD) --branch ${{ github.ref_name }} --test_name tip-tracking --chain $CHAIN --runner ${{ runner.name }} --db_version $db_version --outcome $TEST_RESULT --result_file ${{ github.workspace }}/result-$CHAIN.json
 
     - name: Upload test results
       if: steps.test_step.outputs.test_executed == 'true'
       uses: actions/upload-artifact@v4
       with:
         name: test-results
-        path: ${{ github.workspace }}/result.json
+        path: ${{ github.workspace }}/result-${{ env.CHAIN }}.json
 
     - name: Action for Success
       if: steps.test_step.outputs.TEST_RESULT == 'success'

diff --git a/cmd/diag/_images/dbs/example_databases.png b/cmd/diag/_images/dbs/example_databases.png
diff --git a/cmd/diag/_images/dbs/example_databases_name.png b/cmd/diag/_images/dbs/example_databases_name.png
diff --git a/cmd/diag/_images/dbs/example_databases_name_populated.png b/cmd/diag/_images/dbs/example_databases_name_populated.png
diff --git a/cmd/diag/_images/downloader/example_downloader.png b/cmd/diag/_images/downloader/example_downloader.png
diff --git a/cmd/diag/_images/downloader/example_downloader_files.png b/cmd/diag/_images/downloader/example_downloader_files.png
diff --git a/cmd/diag/_images/downloader/example_downloader_files_downloaded.png b/cmd/diag/_images/downloader/example_downloader_files_downloaded.png
diff --git a/cmd/diag/_images/downloader/example_downloader_files_filename.png b/cmd/diag/_images/downloader/example_downloader_files_filename.png
diff --git a/cmd/diag/_images/local_connection.png b/cmd/diag/_images/local_connection.png
diff --git a/cmd/diag/_images/stages/example_stages.png b/cmd/diag/_images/stages/example_stages.png