CSM performance oracle module (#440)

* add label to FRAME_PREV_REPORT_REF_SLOT

* add withdrawal exit epoch

* add labels to CONTRACT_ON_PAUSE

* linter fix

* fix typo

* fix negative block on forks

* Fix typo in docs

* Fix typo in test function name

* Fix typo in test function name

* feat: attestation and committees

* fix: type

* fix: return -> yield

* fix: pylint in tests

* fix: mypy issues

* fix: test

* chore: change typing for clients

* refactor: strict way for streams `_get_without_fallbacks`

* chore: remove non-mandatory pylint ignore

* make dockerfile buildable

* feat: add CSM performance oracle module

* contracts create

* temp changes

* rename typings

* fix get_latest_blockstamp

* fix test consensus

* tests fixes

* run black

* fix linters

* add blockhash

* fix tests

* fix mypy

* fix linter

* feat: CSM contracts as a separate module

* fix comments

* pylint fixed

* fix lru cache

* test for cache

* fix linter

* fix linters

* remove unused import

* fix: off-by-one error on logs fetching

* chore: op_id -> no_id

* feat: lazy-loading of CSM module

* fix: get ABIs for contracts

* chore: update docs for env variables

* docs: few more corrections to README

* chore: fix timeout_decorator type stub

* fix: remove lru_cache from CSFeeOracle

* feat: collect data for report building

* fix: processing

* fix: review

* fix: ignore

* feat: CSM contracts as a separate module

* fix: off-by-one error on logs fetching

* chore: op_id -> no_id

* feat: lazy-loading of CSM module

* fix: get ABIs for contracts

* chore: update docs for env variables

* docs: few more corrections to README

* chore: fix timeout_decorator type stub

* fix: remove lru_cache from CSFeeOracle

* fix: lint

* unify get call function

* back to HexBytes

* feat: add dummy IPFS provider (#436)

* feat: `get_module_validators_by_node_operators` (#438)

* feat: build CSM oracle report (#437)

* fix: calculated l_ref_slot and r_ref_slot (#441)

* fix: calculated l_ref_slot and r_ref_slot

* fix: calculated l_ref_slot and r_ref_slot

* CSM oracle hot fixes (#442)

* feat: handle thread exception in processing (#443)

* feat: handle thread exception in processing

* fix: remove unused things

* fix: use `ValueError`, shutdown the executor

* fix: linter

* chore: add a comment

* fix: rise `SystemExit` (#446)

* fix: put a stone into a tree (#444)

* feat: `epochs_to_process` (#445)

* feat: `epochs_to_process`

* feat: two sets

* feat: advanced validation

* fix: use `unprocessed_epochs`

* fix: private state properties

* revert: `_data` -> ``data`

* feat: gw3.io provider and MultiIPFSProvider (#447)

* Fix report (#448)

* fix: checkpoint range assert (#449)

* fix: use reference blockstamp as r_slot for stuck events search (#450)

* feat: use attestations endpoint

* feat: fetch stuck events for the given frame only (#452)

* feat: configure workers count for CSM module

* chore: remove DummyIPFSProvider from main

* fix: tests import

* fix: csm `module` property

* fix: logging CID via repr

* chore: remove stone when not needed

* fix: submit_report_data accepts a tuple

* chore: fix mypy

* feat: implement dynamic versions

* fix: black: two lines

* replace wget to culr

* fix: call get_contract_version with block hash

* Close some TODOs (#456)

* fix message

* chore: move iterator to folder

* chore: add timestamp to logs

* feat: add tests to lido contracts

* fix: test slot

* fix pylint

* black fix

* update workflow

* chore: add v2 iterator

* fix tests

* fix: review

* chore: move iterator to folder

* chore: add timestamp to logs

* feat: add tests to lido contracts

* fix: test slot

* fix pylint

* black fix

* update workflow

* fix: review

* fix: pylint

* feat: absolute performance leeway

* fix integration tests

* iterator update

* fix tests

* fix: iterator test

* feat: Multi Third Phase

* squash sr abi

* chore: use TOTAL_BASIS_POINTS const

* fix middleware test

* fix test

* feat: ao-multi-third-phase-partial

* fix: review

* fix tests

* fix: cast to tuple issue

* fix log

* fix: first validator issue

* fix pylint

* fix all tests

* feat: Multi Third Phase

* feat: ao-multi-third-phase-partial

* fix: review

* fix tests

* fix: cast to tuple issue

* fix log

* tweak ejector

* feat: checkpoints refactoring + tests (#463)

* feat: review fixes

* fix: linters

* fix: lru cache

* fix: call from v2 old contract

* fix: linters

* fix: `CSFeeOracle` with no initial epoch

* Update src/modules/csm/csm.py

Co-authored-by: Don Perignom <[email protected]>

* fix: first report l_ref_slot (#466)

* fix: error messages for `validate_for_report` (#467)

* fix: error messages for `validate_for_report`

* fix: use error message explicitly

* fix: Add warning log

* feat: Update python to 3.12

* fix: State network average calculation

* fix: compute operators shares via assigned attestations

* refactor: fix linters

* refactor: calculate distribution

* tests: fix missing assert

* refactor: no cache for stuck_operators

* fix tests and improve cache

* chore: log is_deployed result

Co-authored-by: Raman Siamionau <[email protected]>

* chore: add CSM_MODULE_ADDRESS to .env.example

* chore: add more catch statements for IPFS

* chore: add missing variables to README

* fix: check contracts via _load_contracts

* refactor: CSFeeDistributor calls casts and logs

* chore: debug -> info logs on calls in CSM contracts

* refactor: add Contract suffix to CSM contracts

* Fix: wf

* refactor: InconsistentEvents

* refactor: move TOTAL_BASIS_POINTS out of perf_leeway

* chore: fix sequence assert

* chore: update CSModule abi

* fix linters

* fix: use module address to get keys from keysapi

* fix: remove `no_cache` method for committees

* fix: merge `get` and `get stream` base methods

* feat: check module address

* refactor: extract get_events_in_range into utils

* fix: rework `get_first_non_missed_slot`

* chore: remove events inconsistency check from cs_module

* refactor: use get_events_in_range across the codebase

* fix: add `Final` for env vars

* tests: fix test_calculate_distribution

* fix after merge

* fix: black

* fix: tests

* refactor: extract get_operators_with_stucks_in_range

* refactor: block -> block_identifier

* chore: alias for Shares

* fix: get_stuck_operators_ids return

* tests: rework events utils tests

* fix test names

* fix: calculate_distribution shares as a defaultdict

* docs: CSM-only variables as CSM only in README

* Update src/utils/slot.py

* chore: fix logging for get_events_in_range

* chore: StakingModuleAddress as ChecksumAddress

* chore: update authors

* chore: remove redundant cache

* refactor: move State.load to __init__

* fix: consensus farFutureEpoch const

* chore: add check for l_ref_slot

* fix: `minimum checkpoint step is not reached` is not error (#472)

* fix: review

* fix: test

* feat: modify cache

* comment problematic integration test

* feat: block roots from state, events (#474)

* feat: block roots from state, events

* fix: check events len

* fix: LazyCSM -> CSM

* fix: assert

* feat: check providers for CSM frame

* feat: add DepositContract

* chore: fix linters

---------

Co-authored-by: madlabman <[email protected]>

* fix: linters

* fix: `get_attestation_committees` returns list now (#476)

* fix: `get_attestation_committees` returns list now

* fix: linter

* fix: `_get_block_roots`

* fix: test

* fix: checkpoint slot and tests

* fix: use condition from spec

* chore: set default concurrency for CSM to "2"

* fix: iterator conditions (#477)

* fix: iterator conditions

* fix: black

* fix: use max_available_epoch_to_check

* fix: _get_block_roots

* fix: black

* fix: _select_block_roots

* fix: pylint

* feat: `batched`

* feat: get CSFeeOracle address via CSModule (#478)

* feat: check IPFS provider for CSM (#481)

* feat: check IPFS provider for CSM

* feat: IPFS checker all providers

* feat: IPFS checker mark skipped providers

* fix: review (#480)

* fix: review

* fix: linter

* fix: tests

* fix: no faker for checker

---------

Co-authored-by: F4ever <[email protected]>
Co-authored-by: shuoer86 <[email protected]>
Co-authored-by: vgorkavenko <[email protected]>
Co-authored-by: Vladimir Gorkavenko <[email protected]>
Co-authored-by: F4ever <[email protected]>

.env.example

@@ -2,4 +2,7 @@ CONSENSUS_CLIENT_URI=http://...
 EXECUTION_CLIENT_URI=http://...
 KEYS_API_URI=https://...
 LIDO_LOCATOR_ADDRESS=0x1...
-MEMBER_PRIV_KEY=aaa...
+CSM_ORACLE_ADDRESS=0x...
+CSM_MODULE_ADDRESS=0x...
+MEMBER_PRIV_KEY=aaa...
+

.github/workflows/tests.yml

@@ -13,10 +13,10 @@ jobs:
     steps:
       - uses: actions/checkout@v3
 
-      - name: Set up Python 3.11
+      - name: Set up Python 3.12
         uses: actions/setup-python@v4
         with:
-          python-version: "3.11"
+          python-version: "3.12"
 
       - name: Setup poetry
         run: >
@@ -31,18 +31,22 @@ jobs:
           poetry install --no-interaction --with=dev
 
       - name: Test with pytest
-        run: poetry run pytest -m "not integration and not e2e" --cov=src tests
+        run: poetry run pytest --cov=src tests
+        env:
+          EXECUTION_CLIENT_URI: ${{ secrets.EXECUTION_CLIENT_URI }}
+          CONSENSUS_CLIENT_URI: ${{ secrets.CONSENSUS_CLIENT_URI }}
+          KEYS_API_URI: ${{ secrets.KEYS_API_URI }}
 
   linters:
     runs-on: ubuntu-latest
 
     steps:
       - uses: actions/checkout@v3
 
-      - name: Set up Python 3.11
+      - name: Set up Python 3.12
         uses: actions/setup-python@v4
         with:
-          python-version: "3.11"
+          python-version: "3.12"
 
       - name: Setup poetry
         run: >

.gitignore

@@ -126,6 +126,8 @@ venv.bak/
 .dmypy.json
 dmypy.json
 
+.ruff_cache/
+
 # Pyre type checker
 .pyre/
 
@@ -134,4 +136,7 @@ dmypy.json
 
 # vim
 *.swp
-.direnv
+
+# Cache
+*.pkl
+*.buf

Dockerfile

@@ -1,10 +1,9 @@
-FROM python:3.11-slim as base
+FROM python:3.12.4-slim as base
 
 RUN apt-get update && apt-get install -y --no-install-recommends -qq \
-    gcc=4:10.2.1-1 \
-    libffi-dev=3.3-6 \
-    g++=4:10.2.1-1 \
-    curl=7.74.0-1.3+deb11u7 \
+    libffi-dev=3.4.4-1 \
+    g++=4:12.2.0-3 \
+    curl=7.88.1-10+deb12u5 \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*
 

assets/CSAccounting.json

@@ -0,0 +1 @@
+[{"type":"function","name":"chargeFee","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"},{"name":"amount","type":"uint256","internalType":"uint256"}],"outputs":[],"stateMutability":"nonpayable"},{"type":"function","name":"chargeRecipient","inputs":[],"outputs":[{"name":"","type":"address","internalType":"address"}],"stateMutability":"view"},{"type":"function","name":"claimRewardsStETH","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"},{"name":"stETHAmount","type":"uint256","internalType":"uint256"},{"name":"rewardAddress","type":"address","internalType":"address"},{"name":"cumulativeFeeShares","type":"uint256","internalType":"uint256"},{"name":"rewardsProof","type":"bytes32[]","internalType":"bytes32[]"}],"outputs":[],"stateMutability":"nonpayable"},{"type":"function","name":"claimRewardsUnstETH","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"},{"name":"stEthAmount","type":"uint256","internalType":"uint256"},{"name":"rewardAddress","type":"address","internalType":"address"},{"name":"cumulativeFeeShares","type":"uint256","internalType":"uint256"},{"name":"rewardsProof","type":"bytes32[]","internalType":"bytes32[]"}],"outputs":[],"stateMutability":"nonpayable"},{"type":"function","name":"claimRewardsWstETH","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"},{"name":"wstETHAmount","type":"uint256","internalType":"uint256"},{"name":"rewardAddress","type":"address","internalType":"address"},{"name":"cumulativeFeeShares","type":"uint256","internalType":"uint256"},{"name":"rewardsProof","type":"bytes32[]","internalType":"bytes32[]"}],"outputs":[],"stateMutability":"nonpayable"},{"type":"function","name":"compensateLockedBondETH","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"}],"outputs":[],"stateMutability":"payable"},{"type":"function","name":"depositETH","inputs":[{"name":"from","type":"address","internalType":"address"},{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"}],"outputs":[],"stateMutability":"payable"},{"type":"function","name":"depositStETH","inputs":[{"name":"from","type":"address","internalType":"address"},{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"},{"name":"stETHAmount","type":"uint256","internalType":"uint256"},{"name":"permit","type":"tuple","internalType":"struct ICSAccounting.PermitInput","components":[{"name":"value","type":"uint256","internalType":"uint256"},{"name":"deadline","type":"uint256","internalType":"uint256"},{"name":"v","type":"uint8","internalType":"uint8"},{"name":"r","type":"bytes32","internalType":"bytes32"},{"name":"s","type":"bytes32","internalType":"bytes32"}]}],"outputs":[],"stateMutability":"nonpayable"},{"type":"function","name":"depositWstETH","inputs":[{"name":"from","type":"address","internalType":"address"},{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"},{"name":"wstETHAmount","type":"uint256","internalType":"uint256"},{"name":"permit","type":"tuple","internalType":"struct ICSAccounting.PermitInput","components":[{"name":"value","type":"uint256","internalType":"uint256"},{"name":"deadline","type":"uint256","internalType":"uint256"},{"name":"v","type":"uint8","internalType":"uint8"},{"name":"r","type":"bytes32","internalType":"bytes32"},{"name":"s","type":"bytes32","internalType":"bytes32"}]}],"outputs":[],"stateMutability":"nonpayable"},{"type":"function","name":"feeDistributor","inputs":[],"outputs":[{"name":"","type":"address","internalType":"contract ICSFeeDistributor"}],"stateMutability":"view"},{"type":"function","name":"getActualLockedBond","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"getBond","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"getBondAmountByKeysCount","inputs":[{"name":"keys","type":"uint256","internalType":"uint256"},{"name":"curve","type":"tuple","internalType":"struct ICSBondCurve.BondCurve","components":[{"name":"points","type":"uint256[]","internalType":"uint256[]"},{"name":"trend","type":"uint256","internalType":"uint256"}]}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"getBondAmountByKeysCount","inputs":[{"name":"keys","type":"uint256","internalType":"uint256"},{"name":"curveId","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"getBondAmountByKeysCountWstETH","inputs":[{"name":"keysCount","type":"uint256","internalType":"uint256"},{"name":"curveId","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"getBondAmountByKeysCountWstETH","inputs":[{"name":"keysCount","type":"uint256","internalType":"uint256"},{"name":"curve","type":"tuple","internalType":"struct ICSBondCurve.BondCurve","components":[{"name":"points","type":"uint256[]","internalType":"uint256[]"},{"name":"trend","type":"uint256","internalType":"uint256"}]}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"getBondCurve","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"tuple","internalType":"struct ICSBondCurve.BondCurve","components":[{"name":"points","type":"uint256[]","internalType":"uint256[]"},{"name":"trend","type":"uint256","internalType":"uint256"}]}],"stateMutability":"view"},{"type":"function","name":"getBondCurveId","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"getBondLockRetentionPeriod","inputs":[],"outputs":[{"name":"retention","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"getBondShares","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"getCurveInfo","inputs":[{"name":"curveId","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"tuple","internalType":"struct ICSBondCurve.BondCurve","components":[{"name":"points","type":"uint256[]","internalType":"uint256[]"},{"name":"trend","type":"uint256","internalType":"uint256"}]}],"stateMutability":"view"},{"type":"function","name":"getKeysCountByBondAmount","inputs":[{"name":"amount","type":"uint256","internalType":"uint256"},{"name":"curve","type":"tuple","internalType":"struct ICSBondCurve.BondCurve","components":[{"name":"points","type":"uint256[]","internalType":"uint256[]"},{"name":"trend","type":"uint256","internalType":"uint256"}]}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"getKeysCountByBondAmount","inputs":[{"name":"amount","type":"uint256","internalType":"uint256"},{"name":"curveId","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"getLockedBondInfo","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"tuple","internalType":"struct ICSBondLock.BondLock","components":[{"name":"amount","type":"uint128","internalType":"uint128"},{"name":"retentionUntil","type":"uint128","internalType":"uint128"}]}],"stateMutability":"view"},{"type":"function","name":"getRequiredBondForNextKeys","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"},{"name":"additionalKeys","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"getRequiredBondForNextKeysWstETH","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"},{"name":"additionalKeys","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"getUnbondedKeysCount","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"getUnbondedKeysCountToEject","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"},{"type":"function","name":"lockBondETH","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"},{"name":"amount","type":"uint256","internalType":"uint256"}],"outputs":[],"stateMutability":"nonpayable"},{"type":"function","name":"penalize","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"},{"name":"amount","type":"uint256","internalType":"uint256"}],"outputs":[],"stateMutability":"nonpayable"},{"type":"function","name":"releaseLockedBondETH","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"},{"name":"amount","type":"uint256","internalType":"uint256"}],"outputs":[],"stateMutability":"nonpayable"},{"type":"function","name":"resetBondCurve","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"}],"outputs":[],"stateMutability":"nonpayable"},{"type":"function","name":"setBondCurve","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"},{"name":"curveId","type":"uint256","internalType":"uint256"}],"outputs":[],"stateMutability":"nonpayable"},{"type":"function","name":"settleLockedBondETH","inputs":[{"name":"nodeOperatorId","type":"uint256","internalType":"uint256"}],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"nonpayable"},{"type":"function","name":"totalBondShares","inputs":[],"outputs":[{"name":"","type":"uint256","internalType":"uint256"}],"stateMutability":"view"}]