Fullnode interactive rescan.

[nodech]

Chain "interactive scan"

Motivation

The chain already has scan capabilities, which are used by the wallet
(and separate indexers) to catch up to the tip or to rescan the state of the
blocks for missing data.

But unfortunately, it is not enough for the wallet. The rescanned block can
update the filter, and that means we need to check the same block with the
updated filter. Any indexer using filters will have the same issue. The wallet
is just an indexer with a filter.

Goals

• Rescan to have ability to "repeat" blocks with updated filter.
• Support multiple parallel rescans (allowing more flexible indexers).

rescan interactive Chain API

Chain API only exposes single method that is doing all the processing:
scanInteractive(start, filter, iter)

• start is Block Hash or Number where to start the scan.
• filter initial BloomFilter, if it's null than block will return all TXs.
• iter(chainEntry, txs): scanAction - this is the callback for per block
  iteration as well as the main way to control the flow of the rescan.

iter callback returns scanAction object which has type and controls the
flow of the scan. Action object takes form:
{ type: ACTION_TYPE, ...additional_fields }

Actions:

• NEXT - move on to the next block. This is the default and only behaviour
  non-interactive rescan. ({ type: scanActions.NEXT })
• ABORT - abort the scan process. ({ type: scanAction.ABORT })
• REPEAT - Repeat the last block with the same filter. Note, that if filter
  was passed as a reference and filter was updated, next block scan will use
  updated filter. ({ type: scanAction.REPEAT })
• REPEAT_SET - Repeat the last block but set new filter. This replaces last
  filter with new one and repeats scan of the block.
  ({ type: scanAction.REPEAT_SET, filter: BloomFilter })
• REPEAT_ADD - Add new entries to the filter before repeating the scan of
  the same block. Useful if you don't have reference to the original filter
  that was passed (e.g. in HTTP). Chunks will get added to the filter and then
  will rescan the same block.
  ({ type: scanAction.REPEAT_ADD, chunks: Buffer[] })

"ABORT", "incorrect action", and "internal error" will all abort the rescan
with the appropriate error message. For example, if a reorganization happens on
the chain we are currently on and some other chain becomes the main, it will
abort the scan and throw an error.

Differences between existing scan and interactive scan

In case you are using the behavior as the normal scan - using only NEXT,
there's one big difference. The existing rescan stops the chain from progressing
until the rescan is finished, which in turn avoids reorganizations and
invalidation of the scan. That is not the case with the interactive scan.
It does not stop the chain from progressing, meaning it can end up on an
alternative chain at some point. In that case, the rescan will abort. In turn, it
allows multiple rescans to be active at once.

HTTP API

The HTTP API for the interactive rescan allows only 1 rescan per client (this
can be changed if necessary). However, the server currently does not have any
limits on how many parallel rescans can be executed, which is different from
the normal rescan that blocks the whole chain. The Rescan API uses websockets for
communication, so the websocket connection must be opened first.

NodeClient now has new method rescanInteractive(start, filter). filter
needs to be either encoded BloomFilter or null. If filter is not provided
client.filter which was set previously using set filter will be used.
If you want ALL TXs and you have previously used set filter with that client,
you will need to reset filter first.
Server expects clients to have the hooks necessary for the rescan process
described below:

• rescanInteractive - starts the rescan process.
• client hook: block rescan interactive with args: (rawEntry, rawTXs) and returns
  encoded scanAction. filter in REPEAT_SET action needs to be encoded
  Bloom filter.
• client hook: block rescan interactive abort called with args (message).
  Note, if you are socket.filter (set using set filter), REPEAT_ADD will update
  the socket.filter itself.

Other changes

• Rename tx.test to testAndMaybeUpdate.
• Add tx.test that does not modify the filter.

[coveralls]

coverage: 68.695% (+0.04%) from 68.654%
when pulling 140265e on nodech:interactive-scan
into 349d203 on handshake-org:master.

[nodech]

There is a bug in bsock: bcoin-org/bsock#11 which cases the fullnode interactive rescan to get stuck infinitely. This bug also applies to normal rescan, but it easier to notice with interactive scan.

The interactive rescan allows bigger gaps between .call calls. It can be because chain side got the lock to add new block, another rescan request is being processed or anything else that has acquired the chain lock. If the client closes before socket .call, the next .call will never resolve. Given that interactive rescan acquires the chain lock, this hangs the fullnode. If it did not, it would lead to memory leak.

Now when calling .call on the destroyed client the socket will error right away.