create and load tank snapshots #555

josibake · 2024-09-09T15:20:49Z

Given a url, download the blocks and start a node with this chain. This works by creating a temporary initContainer which downloads the file and untars it into the .bitcoin data dir. Once complete, the initContainer cleans itself up and the pod starts as normal.

Other nodes get this chain by doing IBD, or each node can be specified to start with its own snapshot. This could be useful for things like simulating a hard fork, chain split, etc.

For testing

I mined a signet chain to 200 blocks and created a snapshot using warnet snapshot miner -f "chainstate,blocks,wallets". The network along with the snapshot is saved as a warnet project here: https://github.com/josibake/warnet-projects/tree/main/premined-signet

To test:

Checkout the project: git clone [email protected]:josibake/warnet-projects.git && cd warnet-projects/premined-signet
Make sure you have this PR checked out and install warnet: uv venv && source .venv/bin/activate.fish && uv pip install -e ../../warnet
Run warnet deploy networks/cve_images_on_signet/
Load the wallet for the miner: warnet bitcoin rpc miner loadwallet mining_wallet
Run warnet bitcoin rpc miner -getinfo

You should see something like the following:

Chain: signet
Blocks: 200
Headers: 200
Verification progress: 100.0000%
Difficulty: 0.001126515290698186

Network: in 5, out 0, total 5
Version: 270000
Time offset (s): 0
Proxies: n/a
Min tx relay fee rate (BTC/kvB): 0.00001000

Wallet: mining_wallet
Keypool size: 4000
Transaction fee rate (-paytxfee) (BTC/kvB): 0.00000000

Balance: 5000.00000000

Warnings: (none)

.. which means your miner started with a premined chain and 5000 available coins in the wallet!

Bonus

Check some of the other nodes and verify they are on the same 200 block chain (or doing IBD)
Run the signet miner to make sure you are able to keep mining on top of the premined chain, i.e., warnet run scenarios/signet_miner.py --tank=0 generate --min-nbits --address=tb1qnsv03jqd7tkn2crd7jqv5wxwf2mmxl2l23w7rs --ongoing
After mining a few blocks on the new chain, run warnet down && warnet deploy networks/cve_images_on_signet/ and verify you are back at your starting point of a 200 block chain

TODO:

add a test
add docs for how to create a premined chain and where to upload it

pinheadmz · 2024-09-09T15:26:12Z

resources/charts/bitcoincore/values.yaml

@@ -135,3 +135,6 @@ baseConfig: |
 config: ""

 connect: []
+blocksDownload:
+  enabled: false
+  url: "https://storage.googleapis.com/warnet-signet-chains/0014d33b6e11ca95c4edccd8e986434358d79e919730/premined-signet.tar.gz"


this kind of hard-code is ok in a test but i dont think it belongs in a default

agreed, added "do not merge" to the title of the PR and added this as a todo in the description, will try to fix tomorrow.

if key is set, download and untar a blocks directory into the bitcoin datadir. its up to the user to make sure they are choosing an appropriate snapshot, e.g., if the snapshot was created from a signet chain then it can only be loaded into a signet node with the correct signet challenge. we can expand on this in the future to better export all of the relevant config details in the snapshot, but for now this just adds the ability to load files into pods before the pod starts.

tars a datadir based on a passed in filter (if no filter, zip everything) and copies the zip to the users host.

m3dwards · 2024-09-11T12:45:17Z

This is very cool and I'm sure will be extremely useful!

I've had an initial run through and it has worked as described in the description. I also think the init container architecture sounds good, so concept ack on that.

Few initial thoughts before I dive into code review:

This really needs a new doc. I wouldn't have known to run a command like warnet snapshot miner -f "blocks,chainstate,wallets" I know you have that on TODO.
Snapshots should perhaps be saved in snapshots directory rather than root of current directory?

josibake · 2024-09-11T12:50:49Z

Snapshots should perhaps be saved in snapshots directory rather than root of current directory?

agree. there is an --output flag for the snapshot command, but perhaps better would be to autocreate an empty snapshots directory via warnet init and have the default be there? the problem I see here is that warnet commands can be ran from anywhere , so if you run a warnet command from outside the project directory, and it tries to default to snapshots, the command will fail. perhaps a work around for now would be always require the --output argument so there are no surprises for the user?

m3dwards · 2024-09-11T13:15:26Z

Or create snapshot directory should one not exist?

bdp-DrahtBot · 2024-09-11T13:26:38Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

No conflicts as of last run.

m3dwards · 2024-09-11T13:33:04Z

Code looks ok to me, fancy adding a little doc and we can get this merged?

josibake · 2024-09-11T13:53:55Z

Or create snapshot directory should one not exist?

could do, but generally not a fan of creating dirs randomly on the users system, outside of a scoped warnet project directory. I think for now I'll leave it to "output dir is required" and then as we fine tune this whole project structure / snapshot management we can improve on it.

Code looks ok to me, fancy adding a little doc and we can get this merged?

will do!

allow a user to snapshot a single tank (with a filter) or snapshot all tanks with a filter. this creates a zip that can then be loaded via the initContainer in the first commit

josibake · 2024-09-11T15:40:56Z

@m3dwards added a doc, and also took your suggestion of defaulting to warnet-snapshots and creating the dir if the user doesn't provide one. I chose warnet-snapshots because it's quite annoying to type warnet snapshot and have your terminal try to autocomplete it to the directory snapshots. Probably a more clever fix for this , but this works for now.

m3dwards · 2024-09-11T16:04:10Z

LGTM. Like the doc!

josibake · 2024-09-11T16:05:40Z

thx! working on a test locally but its a bit tricky given interactions with local fs etc, should have it finished tomorrow and will open a PR

willcl-ark · 2024-09-11T19:05:52Z

This has broken apidocs https://github.com/bitcoin-dev-project/warnet/actions/runs/10816493180/job/30007779318

pinheadmz · 2024-09-11T19:10:50Z

@willcl-ark fixed #585

josibake changed the title ~~Start with a premined signet chain~~ POC: Start with a premined signet chain Sep 9, 2024

pinheadmz reviewed Sep 9, 2024

View reviewed changes

josibake mentioned this pull request Sep 9, 2024

Start with a pre-synced chain #466

Closed

josibake changed the title ~~POC: Start with a premined signet chain~~ POC (do not merge): Start with a premined signet chain Sep 9, 2024

josibake force-pushed the premined-signet-chain branch from 6d33d3c to 0fb66d1 Compare September 10, 2024 11:53

josibake changed the title ~~POC (do not merge): Start with a premined signet chain~~ POC (do not merge): create and load tank snapshots Sep 10, 2024

josibake added 2 commits September 10, 2024 14:02

add k8s utility for exporting datadir

dba4a13

tars a datadir based on a passed in filter (if no filter, zip everything) and copies the zip to the users host.

josibake force-pushed the premined-signet-chain branch from 0fb66d1 to cc1b6dc Compare September 10, 2024 12:25

josibake changed the title ~~POC (do not merge): create and load tank snapshots~~ create and load tank snapshots Sep 10, 2024

josibake added this to the Tabconf milestone Sep 10, 2024

bdp-DrahtBot mentioned this pull request Sep 11, 2024

unified HTML landing page (NOT a dashboard!) #531

Merged

6 tasks

m3dwards approved these changes Sep 11, 2024

View reviewed changes

josibake added 2 commits September 11, 2024 17:31

add snapshot cli command

5ba11a1

allow a user to snapshot a single tank (with a filter) or snapshot all tanks with a filter. this creates a zip that can then be loaded via the initContainer in the first commit

docs: add snapshot doc

366465c

josibake force-pushed the premined-signet-chain branch from 228bec2 to 366465c Compare September 11, 2024 15:31

m3dwards merged commit 44a364a into bitcoin-dev-project:main Sep 11, 2024
11 checks passed

willcl-ark mentioned this pull request Sep 11, 2024

Export a snapshot of a running warnet simulation #4

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

create and load tank snapshots #555

create and load tank snapshots #555

josibake commented Sep 9, 2024 •

edited

Loading

pinheadmz Sep 9, 2024

josibake Sep 9, 2024

m3dwards commented Sep 11, 2024

josibake commented Sep 11, 2024

m3dwards commented Sep 11, 2024

bdp-DrahtBot commented Sep 11, 2024 •

edited

Loading

m3dwards commented Sep 11, 2024

josibake commented Sep 11, 2024

josibake commented Sep 11, 2024

m3dwards commented Sep 11, 2024

josibake commented Sep 11, 2024

willcl-ark commented Sep 11, 2024

pinheadmz commented Sep 11, 2024

create and load tank snapshots #555

create and load tank snapshots #555

Conversation

josibake commented Sep 9, 2024 • edited Loading

For testing

Bonus

pinheadmz Sep 9, 2024

Choose a reason for hiding this comment

josibake Sep 9, 2024

Choose a reason for hiding this comment

m3dwards commented Sep 11, 2024

josibake commented Sep 11, 2024

m3dwards commented Sep 11, 2024

bdp-DrahtBot commented Sep 11, 2024 • edited Loading

Conflicts

m3dwards commented Sep 11, 2024

josibake commented Sep 11, 2024

josibake commented Sep 11, 2024

m3dwards commented Sep 11, 2024

josibake commented Sep 11, 2024

willcl-ark commented Sep 11, 2024

pinheadmz commented Sep 11, 2024

josibake commented Sep 9, 2024 •

edited

Loading

bdp-DrahtBot commented Sep 11, 2024 •

edited

Loading