Database Backup

Overview

Chainweb Nodes uses two different type of database to persist chain data:

1. The blockchain history starting from the genesis blocks up to the most recent block for all chains is stored in a single RocksDB database. This is the authoritative source of truth for all information of the respective network (e.g. Kadena Mainnet). All information is cryptographically authenticated and changing a single bit will cause a validation failure and render the database useless.

2. The most recent state of the Pact tables of all smart contracts is stored in SQLite databases. There is one database for each chain. For efficiency and convenience these databases contain provenance information that support fast historic lookups and rewinds e.g. for resolving forks or for replaying older transactions.

   The content of these database can always be rebuilt from the chain database (RocksDB). At startup chainweb-node identifies the most recent consensus state and synchronizes the state of all Pact databases (SQLite) to the respective blocks. Depending on how far off the Pact state of a chain is this can take some time, since the node has to replay all transactions that are missing from the current Pact state.

   If a Pact database for some chain is missing, chainweb-node will rebuild that database from scratch starting with the genesis state. This can take between several hours or even days, depending on the available hardware.

For disaster recovery it is sufficient to just keep a copy of the RocksDB, because the Pact SQLite database can be rebuild from it. However, since recomputing the SQLite databases from scratch can take a long time, it is recommended to also keep occasional copies of these.

Local Backups

The service API of chainweb-node offers endpoints for creating backups while the node is running.

• Schedule backup
• Check backup status

The backup endpoints are disabled by default and must be enabled in the configuration file

chainweb:
  backup:
    api:
      configuration: {}
      enabled: false
    directory: null

or via the command line:

 --enable-backup-api      whether backup-api is enabled or disabled
 --disable-backup-api     unset flag backup-api
 --backup-directory ARG   Directory in which backups will be placed when using
                          the backup API endpoint for backup

The backup directory should be on the same file system or docker volume as the original chainweb-node database, in which case the RocksDB backup will be instantaneous and with almost no space overhead (by using hard links).

By default only RocksDB is backed up. Additional SQLite backups can be requested via an API query parameter. The sqlite backups will take a long time and take as much space as the original DBs. There is also an endpoint to query the status of an update.

There is no builtin retention policy, so you should monitor how much space your backups take and make sure to delete old backups.

It is recommended to create RocksDB backups regularly, at least every few hours. Because space overhead of recent updates is minimal, it is fine to keep at least a few backups around. SQLite backups are more expensive and can be done less regularly, e.g. once a day or every other day. Chainweb-node will replay missing transactions on startup. The time that this takes depends on how old the latest backup is.

Remote Backups

In addition to the local backups it is also recommended to keep additional backups in a secure place that is different from the hardware that chainweb-node is running on. These backups should at least include RocksDB, but for faster recovery it is recommended to also keep somewhat recent backups of the SQLite databases.

SQLite database must be kept secure from unauthorized write access. Therefore backups of SQLite databases should either have appropriately restricted access or one should use file checksums to guarantee that backups haven't be compromised.

One must never use a Pact SQLite database from an untrusted source!

Additional Remarks

Generally, one wants to make sure not to expose the service API (default port is 1848) on a public network without are reverse proxy that protects access.