[Epic: js-waku] Reliability Protocol for Resource-Restricted Clients

[chair28980]

Our user story:

1. As a developer, I want a simple API to register hooks to process messages on one or several content topics: feat(filter)!: new simpler filter API  js-waku#2092
2. As a developer, I do not care about shards or pubsub topics, I am only using autosharding
3. As developer, I want fair reliability when receiving messages (e..g redundant filter nodes + filter ping + store sync)
4. As a developer, I do not want the same incoming message to be processed several times (e.g. filter redundancy should be abstracted)
5. As a developer, I want fair reliability when sending messages.

And our implementation notes:

Store

• implement store-v3: feat!: store v3 js-waku#2036

Combination

• Message verification and retry: feat: message verification and retry js-waku#2075
• Message retry for LightPush: feat(lightpush): add retries to failing peers js-waku#2069
• Continuous discovery: chore(peer-exchange): support updates of previously discovered peer's addresses  js-waku#2051

Liveness of a node:

• Node health state - feat: introduce node health as a metric  js-waku#2070
• Filter subscription recovery after offline state feat: improve offline recoverability of Filter  js-waku#2024
• (questionable) node recovery strategies feat: node recovery strategies js-waku#2076

Testing

• unit testing: feat: unit testing @waku/sdk js-waku#2078 and feat: unit testing @waku/core js-waku#2077
• Test pipeline - test: create scripts for running light-push/filter and measuring ratio of messages sent and received js-waku#2007

Filter:

• ensure number of peers to subscribe to is configurable, documented and respected by implementation;
• dynamic peer management: feat: peer management for protocols (with disconnection management) js-waku#2002
  • if filter ping fails on node 1 but not node 2, disconnect from node 1 and replace: feat(filter): peer/subscription renewal with recurring Filter pings js-waku#2052
  • introduce a monitor if no messages are coming from peer A but are coming from peer B - drop A: feat: validate messages for individual filter nodes & perform renewals js-waku#2057
• automatically connect and subscribe to 2-3 filter nodes: feat: SDK for redundant usage of filter/lightpush js-waku#1463;
• seen cache to ensure dupe messages are not bubbled up;
• when subscribicing, starting running filter ping: feat: run filter ping during filter subscription js-waku#1995;
• keep alive feature feat: add keep alive to Filter js-waku#1970
• filter pings should work per peer, instead of per subscription

LightPush:

• ensure number of peers to use is configurable and documented
• dynamic peer management: feat: peer management for protocols (with disconnection management) js-waku#2002
  • if any node from the pool rejects messages - replace it;
• connect to 2/3 lightpush node when sending messages;

---

First revision

Tasks (obsolete)

• write interop tests to cover non-relay client user stories;
• protocol reliability + exposing additional configs to enable / disable behavior + unit tests:
  • Filter/LightPush uses more than one peer: feat: SDK for redundant usage of filter/lightpush js-waku#1463
• orient protocols to be more RFC oriented + unit tests: chore: protocol implementations in @waku/core should be as unopinionated as possible js-waku#1886
  • Filter: chore(lightpush)!: move protocol implementation to @waku/sdk (1/n) js-waku#1964
  • LightPush: chore(lightpush)!: move protocol implementation opinions to @waku/sdk js-waku#1887
  • Store: chore(store)!: move protocol implementation opinions to @waku/sdk js-waku#1913

We can de-scope some of these streams in the interest of time.

Second revision

Filter:

• ensure number of peers to subscribe to is configurable, documented and respected by implementation;
• Filter subscription recovery after offline state feat: improve offline recoverability of Filter  js-waku#2024

LightPush:

• ensure number of peers to use is
  • configurable
  • documented

Store

• implement store-v3: feat!: store v3 js-waku#2036
• peer renewal for store peers;

Liveness of a node:

• feat: introduce node health as a metric  js-waku#2070
• [ ]

Testing

• unit testing;
• test: create scripts for running light-push/filter and measuring ratio of messages sent and received js-waku#2007

[chair28980]

@waku-org/js-waku-developers please proceed with creating the tasks required in the js-waku repository, please tag them with the E:js-waku Improve Reliability label, and include them in Task list above in the description.

[fryorcraken]

@waku-org/js-waku-developers please note that we now expect an RFC out of this work. Also, need to understand what is done in status-go and align. @chaitanyaprem would be a good person to discuss with as he has some familiarity with usage of req-res protocols in status-go

[weboko]

in status-go

I was planning to schedule knowledge sharing after Status offsite is done

we now expect an RFC out of this work

after knowledge sharing and revisiting completed work we will discuss RFC internally

[fryorcraken]

is there an understanding of what of the change listed above would give you most ROI?

[danisharora099]

is there an understanding of what of the change listed above would give you most ROI?

A lot of high-impact items have been merged, or are in progress. Other than that, imo:

• all protocols: disconnection management (P1)

• filter: validating filter messages from each node against each other to do disconnection (P3)

• filter: running recurring store queries to validate incoming filter messages (dependant on store v3) (P2)

• filter: pinging per peer, instead of per subscription (P3)

• lightpush: seems quite stable once disconnection management is implemented

• store: store v3

• client/node stability:

  • ensure peers are redialed (if not dialed already)
  • recover filter subscriptions
  • running store query upon reconnection to receive missed filter messages

[weboko]

Linking some relevant status-go issues that makes sense to take a look at in the scope of js-waku:

• bug: find out why filter subscriptions are taking time to stabilize when non fleet nodes are used status-im/status-go#5344
• feat: unsubscribe in case invalid push requests are received from server go-waku#1124
• fix: add peer addresses on expiry if peer is discovered again go-waku#1128

[danisharora099]

status-im/status-go#5344

Based on status-im/status-go#5350 and status-im/status-go#5344 (comment). it seems that status-go did not filter peers based on shards when discovered using Peer Exchange, thus attempting to open subscriptions with peers with unsupported pubsub topics.

waku-org/go-waku#1124

Investigating this: waku-org/go-waku#1124 (comment)

waku-org/go-waku#1128

Very interesting. Though we haven't run into a problem like this before (perhaps because of the nature of js-waku node being short lived, thus peer updating its addresses was never observed) -- but this seems to be a very important check to introduce. Opened an issue here: waku-org/js-waku#2051

[chaitanyaprem]

This is the master issue i am using to track all fixes/changes wrt lightclient reliability being done in go-waku/status-go. #219
@danisharora099 , @weboko you can refer to this to see if something that can be applied to js-waku and also can be added to the rfc.

[weboko]

Another point to consider from @jm-clius comment here - waku-org/specs#18 (review):

• verify delivered messages by tracking them by Filter subscription to some node that is different from the one used during LightPush.

Let's discuss at js-waku PM meeting, @danisharora099

[danisharora099]

Strictly based on the newly merged Reliability RFC, derived features and their current state in the js-waku codebase are:

Node Health Management

• Implement node health monitoring for connection states: feat: introduce node health as a metric  js-waku#2070
  • Define conditions for Unhealthy, Minimally Healthy, and Sufficiently Healthy states.

Peers and Connection Management

• Maintain a pool of reliable service nodes for
  • LightPush: feat(lightpush): peer management for protocols js-waku#2003
  • Filter: feat(filter): use protocol peer management js-waku#2047
• Replace failing service nodes upon detection: feat: peer management for protocols (with disconnection management) js-waku#2002
  • Filter: feat(filter): peer/subscription renewal with recurring Filter pings js-waku#2052, feat: validate messages for individual filter nodes & perform renewals js-waku#2057
  • LightPush: feat(lightpush): peer management for protocols js-waku#2003
• Implement preferences for selecting service nodes based on multiaddresses, protocols, and shard participation: feat!: protocols filter peers per shard js-waku#1756, feat!: set peer-exchange with default bootstrap js-waku#1469
• Continuously discover and update service node information: chore(peer-exchange): support updates of previously discovered peer's addresses  js-waku#2051
  • Ensure up-to-date Peer Store with ENR and Libp2p multiaddresses.

LightPush Protocol Enhancements

• Implement redundant message sending to multiple service nodes: feat: SDK for redundant usage of filter/lightpush js-waku#1463
• Implement retry mechanism for failed LightPush requests with increasing intervals: feat(lightpush): add retries to failing peers js-waku#2069
• Implement verification of message delivery using and retry feat: message verification and retry js-waku#2075
  • Store (v3) protocol: feat!: store v3 js-waku#2036
  • Filter protocol

Filter Protocol Enhancements

• Implement regular pings to maintain active subscriptions: feat: add keep alive to Filter js-waku#1970
• Implement redundant Filter subscriptions for message loss mitigation: feat: SDK for redundant usage of filter/lightpush js-waku#1463
• Handle duplicate messages using deterministic message hashing: feat: SDK for redundant usage of filter/lightpush js-waku#1463
• Implement offline recoverability mechanisms for network disconnections: feat: offline state recovery for Filter subscription js-waku#2049
• Initiate Filter pings and replace failing nodes upon network reconnection: feat: run filter ping during filter subscription js-waku#1995

[weboko]

Nice, good job at summarizing!
Let's discuss it at our PM meeting tomorrow as I believe some of the features we might want to make optional such as Store / Filter message verification and retry.

Also to note: we are waiting for update to Light Push to implement some of the strategies in the RFC such as renewing peers only in some cases and doing retries in others.

Upd#1: additional task waku-org/js-waku#2074
Upd#2: one more waku-org/js-waku#2076

[danisharora099]

This epic's deliverables are now all merged in, and can be closed! (minus the unit tests issues)
cc @weboko @chair28980

[weboko]

Unit testing is decoupled into debt and will be addressed independently.
Other things are described in #186 (comment) and will be a fast follow up in dogfooding work stream.