"PV64" Warp Sync

Warp sync extends previous versions of the protocol with full state snapshots. These snapshots can be used to quickly get a full copy of the state at a given block. Every 30,000 blocks, nodes will take a consensus-critical snapshot of that block's state. Any node can fetch these snapshots over the network, enabling a fast sync.

The snapshot format is three-part: block chunks, state chunks, and the manifest. Every chunk is run through snappy compression, and then hashed. Before compression, chunks are created to be approximately CHUNK_SIZE bytes.

CHUNK_SIZE = 4MiB, but this may be subject to change.

Manifest

This contains metadata about the snapshot itself, and is used to coordinate snapshots between nodes. Every snapshot will have a unique manifest, so two identical manifests will refer to the same snapshot.

The manifest is an rlp-encoded list of the following format:

[
    state_hashes: [hash_1: B_32, hash_2: B_32, ...], // a list of all the state chunks in this snapshot
    block_hashes: [hash_1: B_32, hash_2: B_32, ...], // a list of all the block chunks in this snapshot
    state_root: B_32, // the root which the rebuilt state trie should have. used to ensure validity
    block_number: P, // the number of the best block in the snapshot; the one which the state coordinates to.
    block_hash: B_32, // the best block in the snapshot's hash.
]

Block chunks

Block chunks contain raw block data: blocks themselves, and their transaction receipts. The blocks are stored in the "abridged block" format (referred to by AB), and the the receipts are stored in a list: [receipt_1: P, receipt_2: P, ...] (referred to by RC).

Each block chunk is an rlp-encoded list of the following format:

[
    number: P, // number of the first block in the chunk
    hash: B_32, // hash of the first block in the chunk
    td: B_32, // total difficulty of the first block in the chunk
    [abridged_1: AB, receipts_1: RC], // The abridged RLP for the first block, and its receipts.
    [abridged_2: AB, receipts_2: RC], // The abridged RLP for the second block, and its receipts.
    [abridged_3: AB, receipts_3: RC], // ... and so on.
    ...
]

The blocks within are assumed to be sequential, so that their abridged RLP can be assembled into entire blocks simply using the metadata at the beginning.

Abridged block RLP

This differs from the standard block RLP format by omitting some fields which can be reproduced from the metadata at the beginning of the block chunk.

An abridged block takes the following form:

[
    // some header fields
    author: B_20,
    state_root: B_32,
    transactions_root: B_32,
    receipts_root: B_32,
    log_bloom: B_256,
    difficulty: B_32,
    gas_limit: B_32,
    gas_used: B_32,
    timestamp: P,
    extra_data: P,

    // uncles and transactions inline
    [tx_1: P, tx_2: P, ...],
    [uncle_1: P, uncle_2: P, ...],

    // Seal fields
    mix_hash: B_32,
    nonce: B_8,
]

Block Chunk Generation

Definitions: Let NUM(X) denote the number of block X. Let HASH(X) denote the hash of block X. Let TD(X) denote the total difficulty of block X.

Algorithm: Let B be the most recent block. Let B_target = B. Let B_finish = B - 30000 + 1. Let S_current = 0.

Walk backwards from block B until reaching B_finish .

For each block B_x:

• generate the list [abridged: AB, receipts: RC] D_x
• Note its size: S_x = SIZE(D_x).
• Set S_current to S_x.
• If S_current > CHUNK_SIZE:
  • Let S_current = S_current - CHUNK_SIZE
  • Build the chunk [ NUM(B_x+1), HASH(B_x+1), TD(B_{x + 1}), D_{x + 1}, D_{x + 2}, ..., D_Btarget ]
  • Set B_target = B_x.

At the end, if S_current > 0, write out the remaining chunk from block B_finish to B_target: [ NUM(B_finish), HASH(B_finish), TD(B_finish), D_finish, ..., D_Btarget ]