12/WAKU2-FILTER: Handle Client failure

[jimstir]

Reference issue: vacp2p/rfc#475
Author: rshiv
Currently a filter node attempts to push messages indefinitely even if the peer client node is offline. A node should be able to drop the peer once it’s determined that a client node is unreachable.

The issue was introduced here #469 .

Acceptance criteria

• Consensus on how to solve the issue amongst the solutions presented.
• Update RFC in 12/WAKU2-FILTER

Possible Solutions

The solution of the problem needs to be with the existing libp2p connection between server and client, rather than new connection for heartbeat.

As the waku message handler handles the message push during init , we check for dial failure at that point, which is already implemented to maintain a map as, <peer, offtimestart>, where offtimestart is the first failure epoch. We remove it from peer list if failure persist ,given it exceeds a certain time limit and clear the peer from the map as well.
Time to remove can be configured by the filter node to provide the flexibility of adaptive server node.

Notes

There are 2 more approaches to solve the problem however they have their tradeoffs.

1. The existing libp2p connection has a callback for connection failure and can be used as a fallback point to stop the message push service and drop the peer. The solution will clean the memory organically as the connections drop, however, as adaptive nodes are expected to be resource restrictive, if we forbid the push request after first failure , it will cause repetitive filter calls.

2. Adding a heartbeat mechanism using libp2p ping protocol between the peers, which tries the connection every x minutes, failure for 5 times cause the server to drop the peer. This approach will rely on continuous polling which will effect the network bandwith.