feat(mirror): Pass epoch config overrides on forknet init

[VanBarbascu]

Context

Forknet is initialized by provisioning a host with a setup directory containing the source data folder. The following commands are executed to prepare the network:

• neard init – generates the configs and keys.
• neard fork-network set-validators – amends the database, inserts the specified set of validators, and creates the genesis config for the new network.
• neard fork-network finalize – completes the forking process.

In this PR, we focus on the first two steps. If we need to override the epoch configs, they will be generated by neard init and then modified by the neard-runner process with the provided overrides. We first apply the general overrides under the "all" key, followed by protocol version-specific ones.

When overriding an epoch config that does not exist, the contents are copied from the previously available epoch config and then overwritten. i.e. let's assume that mainnet only has epoch configs for protocol 72 and 74. If you want to add overrides for p.v. 75, they will be applied on top of the original epoch config for p.v. 74.

When creating the genesis in set-validators, we read the epoch config corresponding to the genesis protocol version.

How to use it

When calling mirror new-test add the --epoch-config-overrides.
This will accept a JSON formatted string that has the protocol version as keys and a list of jq styled overrides sepparated by |.

By protocol version

For example, if you need to update the num_block_producer_seats, num_chunk_validator_seats and num_chunk_producer_seats for protocol version 73, you would use the following argument: { "74": ".num_block_producer_seats = 3 | .validator_selection_config.num_chunk_validator_seats = 22 | .validator_selection_config.num_chunk_producer_seats = 3"}

With all key

A protocol version override only affects a specific epoch config. If you want to apply a change to all epoch configs, add that change under the all key instead.

Priority

Use the all key to apply overrides across all configs simultaneously. Protocol version-specific overrides will then be applied afterward.

Testing

For testing, I used a 6 node forknet with different scenarios:

For backwards compatibility: old binary(2.3.0), new neard-runner

mirror new-test --epoch-length 50 --genesis-protocol-version 71 --num-validators 6 --num-seats 6 --stateless-setup --new-chain-id modknet

Result: network progessed ok with all 6 ndoes as validators.

New binary, new neard-runner, with overrides

mirror new-test --epoch-length 20 --genesis-protocol-version 73 --num-validators 6 --num-seats 6 --stateless-setup --new-chain-id modknet --epoch-config-overrides '{"all": ".validator_selection_config.num_chunk_validator_seats = 22 | .validator_selection_config.num_chunk_producer_seats = 3 | .num_block_producer_seats = 3 | .validator_selection_config.shuffle_shard_assignment_for_chunk_producers = true"}'
Result:

This was for testing purposes. The same result could have been achieved by using the --num-seats 3 parameter and overriding only the shuffle_shard_assignment_for_chunk_producers parameter.

New binary, new neard-runner, with overrides on all and on version 73

mirror new-test --epoch-length 20 --genesis-protocol-version 71 --num-validators 6 --num-seats 6 --stateless-setup --new-chain-id modknet --epoch-config-overrides '{"all": ".validator_selection_config.num_chunk_validator_seats = 22 | .validator_selection_config.num_chunk_producer_seats = 3 | .num_block_producer_seats = 3", "73": ".validator_selection_config.num_chunk_producer_seats = 6"}'

Result:

The network starts at protocol version 71. In the image, it is in the first epoch after the genesis epoch. In the next epoch, protocol version 73 will take effect, along with the num_chunk_producer_seats = 6 override.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.68%. Comparing base (6ccbd62) to head (4d2d660).

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #12421      +/-   ##
==========================================
+ Coverage   71.16%   71.68%   +0.51%     
==========================================
  Files         843      843              
  Lines      170788   170788              
  Branches   170788   170788              
==========================================
+ Hits       121547   122432     +885     
+ Misses      43913    42993     -920     
- Partials     5328     5363      +35     

Flag	Coverage Δ
backward-compatibility	0.16% <ø> (?)
db-migration	0.16% <ø> (ø)
genesis-check	1.27% <ø> (ø)
integration-tests	39.35% <ø> (+1.00%)	⬆️
linux	70.97% <ø> (+<0.01%)	⬆️
linux-nightly	71.26% <ø> (+7.32%)	⬆️
macos	50.65% <ø> (-0.01%)	⬇️
pytests	1.57% <ø> (ø)
sanity-checks	1.38% <ø> (ø)
unittests	64.26% <ø> (+<0.01%)	⬆️
upgradability	0.21% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.