Wallet missing TXs

[nodech]

A wallet, while syncing or rescanning, may miss transactions if it involves an address that is only discovered within the block. Upon discovering an output in the transaction, the wallet will derive additional receive/change addresses, which assists in determining whether the transaction belongs to it. This can occur within a single transaction when one output is derived at point X and another output in the same transaction is at X + lookahead. The lookahead is recalculated after the transaction has been added, which results in the wallet not recognizing outputs that it owns. Another scenario involves a different transaction with an X + lookahead derivation. In this case, there is an issue where the FullNode/SPVNode will not include those transactions in the list initially because the provided filter did not track them. In such instances, we need to request the block again with an updated filter.

This issue can also pertain to inputs, but it is not as critical. If we were unaware of the credit that the transaction was spending, it indicates we were not tracking it to begin with, so we are not losing sight of the money; we simply miss the credit that was already spent.

This situation would not normally occur, but it is possible if a user operates two different devices with the same master/account key. One wallet might fail to derive lookahead addresses without receiving funds on them, while another wallet remains unaware that the lookahead addresses have been generated.

Both scenarios have distinct requirements and solutions. For the first scenario, it is sufficient to remove the transaction from the database and reinsert it. This ensures that no outputs were previously overlooked. The second scenario requires cooperation from the blockchain when rescanning or adding a block (if the block connect event is also filtered). The node sends the block to the wallet; the wallet checks the list of transactions, and if the filter was updated, it updates the filter and requests the node to resend the same block.

Both scenarios require the block to be disconnected and reconnected until the filter remains unchanged. This could lead to extensive loops if a transaction were crafted specifically to exploit this behavior. However, the limit is set by deriving addresses to the necessary depth and the transaction limits within the block. In theory, 1,000 transactions with 100 outputs exploiting this behavior could lead to 100,000 disconnects and reconnects. This issue can be avoided by setting a limit for the wallet on how many times to rescan the same block.

Alternatively if the wallet knows that this issue has happened in the past, on import of the wallet user can derive addresses before running the rescan, but this needs a lot of caution not to cause the further gaps in the derivation.

There are still some cases that can't be recovered if the gap is big enough and only solution is to increase the lookahead.

On track PR/Issues

• Fix missing transaction with filter updates #831
• Fullnode interactive rescan. #856
• WalletDB rescan deadlock fix #868
• Wallet: Add open/close and connect/disconnect events. #859

Related

• wallet: fix missing tx when rescan with filter update #519