Merge branch 'master' into update-change-log

docs/contributors/continuous-integration.md

@@ -8,7 +8,7 @@ The CI setup exists on the Status.im Jenkins instance:
 
 https://ci.infra.status.im/job/nim-waku/
 
-It currently consists four jobs:
+It currently consists of four jobs:
 
 * [manual](https://ci.infra.status.im/job/nim-waku/job/manual/) - For manually executing builds using parameters.
 * [deploy-wakuv1-test](https://ci.infra.status.im/job/nim-waku/job/deploy-wakuv1-test/) - Builds every new commit in `master` and deploys to `wakuv1.test` fleet.

docs/contributors/release-process.md

@@ -4,7 +4,7 @@ How to do releases.
 
 For more context, see https://trunkbaseddevelopment.com/branch-for-release/
 
-## How to to do releases
+## How to do releases
 
 ### Before release
 
@@ -84,6 +84,7 @@ Ensure all items in this list are ticked:
    > Clients are reachable via the corresponding channels on the Vac Discord server.
    > It should be enough to inform clients on the `#nwaku` and `#announce` channels on Discord.
    > Informal conversations with specific repo maintainers are often part of this process.
+   - Check if nwaku configuration parameters changed. If so [update fleet configuration](https://www.notion.so/Fleet-Ownership-7532aad8896d46599abac3c274189741?pvs=4#d2d2f0fe4b3c429fbd860a1d64f89a64) in [infra-nim-waku](https://github.com/status-im/infra-nim-waku)
    - Deploy release to the `wakuv2.prod` fleet from [Jenkins](https://ci.infra.status.im/job/nim-waku/job/deploy-wakuv2-prod/).
    - Ensure that nodes successfully start up and monitor health using [Grafana](https://grafana.infra.status.im/d/qrp_ZCTGz/nim-waku-v2?orgId=1) and [Kibana](https://kibana.infra.status.im/goto/a7728e70-eb26-11ec-81d1-210eb3022c76).
    - If necessary, revert by deploying the previous release. Download logs and open a bug report issue.

docs/operators/README.md

@@ -13,7 +13,7 @@ Nwaku (formerly `nim-waku`) aims to be a lightweight and robust Waku v2 client.
 It serves as the reference implementation for researchers,
 who extend the client in parallel to spec development.
 As such, it is first in line to support innovative and new Waku v2 protocols,
-but configurable enough to serve the adaptive needs of a various operators.
+but configurable enough to serve the adaptive needs of various operators.
 We are also developing a set of operator-focused tools to monitor and maintain a running nwaku node.
 
 This guide provides step-by-step tutorials covering how to build and configure your own nwaku node,
@@ -32,4 +32,4 @@ For bug reports, please [tag your issue with the `bug` label](https://github.com
 
 If you believe the reported issue requires critical attention, please [use the `critical` label](https://github.com/waku-org/nwaku/issues/new?labels=critical,bug) to assist with triaging.
 
-To get help, or participate in the conversation, join the [Vac Discord](https://discord.gg/KNj3ctuZvZ) server.
+To get help, or participate in the conversation, join the [Vac Discord](https://discord.gg/KNj3ctuZvZ) server.

docs/operators/how-to/configure-store.md

@@ -35,13 +35,13 @@ There is a set of configuration options to customize the waku store protocol's m
   + The time retention policy,`time:<duration-in-seconds>` (e.g., `time:14400`)
   + The capacity retention policy,`capacity:<messages-count>` (e.g, `capacity:25000`)
   + The size retention policy,`size:<size-in-gb-mb>` (e.g, `size:25Gb`)
-  + To disable the retention policy, explicitly, set this option to to `""`, an empty string.
+  + To disable the retention policy, explicitly, set this option to `""`, an empty string.
 * `--store-message-db-url`: The message store database url option controls the message storage engine. This option follows the [_SQLAlchemy_ database URL format](https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls).
 
   + SQLite engine: The only database engine supported by the nwaku node. The database URL has this shape: `sqlite://<database-file-path>`. If the `<database-file-path>` is not an absolute path (preceded by a `/` character), the file will be created in the current working directory. The SQLite engine also supports to select a non-persistent in-memory database by setting the `<database-file-path>` to `:memory:`.
   + In the case you don't want to use a persistent message store; set the `--store-message-db-url` to an empty string, `""`. This will instruct the node to use the fallback in-memory message store.
 
-By default the node message store will be configured with a time retention policy set to `14400` seconds (4 hours). Additionaly, by default, the node message store will use the SQLite database engine to store historical messages in order to persist these between restarts. 
+By default the node message store will be configured with a time retention policy set to `14400` seconds (4 hours). Additionally, by default, the node message store will use the SQLite database engine to store historical messages in order to persist these between restarts. 
 
 > :warning: Note the 3 slashes, `///`,  after the SQLite database URL schema. The third slash indicates that it is an absolute path: `/mnt/nwaku/data/db1/store.sqlite3`
 
@@ -55,4 +55,4 @@ wakunode2 \
 ### How much resources should I allocate?
 
 Currently store service nodes use, by default, a message store backed by an in-disk SQLite database. Most Waku messages average a size of 1KB - 2KB, implying a minimum memory requirement of at least ~250MB
-for a typical store capacity of 100k messages. Note, however, that the allowable maximum size for Waku messages is up to 1MB.
+for a typical store capacity of 100k messages. Note, however, that the allowable maximum size for Waku messages is up to 1MB.

docs/operators/how-to/configure.md

@@ -81,7 +81,7 @@ provided via environment variables.
 
 ### Configuration file
 
-The third configuration mechanism in order of precedence is the configuration via a TOML file. The previous mechanims take precedence over this mechanism as explained above.
+The third configuration mechanism in order of precedence is the configuration via a TOML file. The previous mechanisms take precedence over this mechanism as explained above.
 
 The configuration file follows the [TOML](https://toml.io/en/) format:
 
@@ -90,7 +90,7 @@ log-level = "DEBUG"
 tcp-port = 65000
 ```
 
-The path to the TOML file can be specified usin one of the previous configuration mechanisms:
+The path to the TOML file can be specified using one of the previous configuration mechanisms:
 
 * By passing the `--config-file` command line option:
   ```shell

docs/tutorial/rln-chat2-live-testnet.md

@@ -9,7 +9,7 @@ sending more than one message per epoch.
 At the time of this tutorial, the epoch duration is set to `10` seconds.
 You can inspect the current epoch value by checking the following [constant variable](https://github.com/status-im/nim-waku/blob/21cac6d491a6d995a7a8ba84c85fecc7817b3d8b/waku/v2/protocol/waku_rln_relay/constants.nim#L245) in the nim-waku codebase.
 Your messages will be routed via test fleets and will arrive at other live chat2 clients that are running in rate-limited mode over the same content topic i.e., `/toy-chat/3/mingde/proto`.
-Your samp activity will be detected by them and a proper message will be shown on their console.  
+Your spam activity will be detected by them and a proper message will be shown on their console.  
 
 # Set up
 ## Build chat2

tests/testlib/sequtils.nim

@@ -1,5 +1,2 @@
 proc toString*(bytes: seq[byte]): string =
   cast[string](bytes)
-
-proc toBytes*(str: string): seq[byte] =
-  cast[seq[byte]](str)

tests/wakunode_rest/test_rest_admin.nim

@@ -1,7 +1,7 @@
 {.used.}
 
 import
-  std/sequtils,
+  std/[sequtils,strformat],
   stew/shims/net,
   testutils/unittests,
   presto, presto/client as presto_client,
@@ -10,6 +10,7 @@ import
 import
   ../../waku/waku_core,
   ../../waku/waku_node,
+  ../../waku/waku_filter_v2/client,
   ../../waku/node/peer_manager,
   ../../waku/waku_api/rest/server,
   ../../waku/waku_api/rest/client,
@@ -26,13 +27,15 @@ suite "Waku v2 Rest API - Admin":
   var node1 {.threadvar.}: WakuNode
   var node2 {.threadvar.}: WakuNode
   var node3 {.threadvar.}: WakuNode
+  var peerInfo1 {.threadvar.}: RemotePeerInfo
   var peerInfo2 {.threadvar.}: RemotePeerInfo
   var peerInfo3 {.threadvar.}: RemotePeerInfo
   var restServer {.threadvar.}: RestServerRef
   var client{.threadvar.}: RestClientRef
 
   asyncSetup:
     node1 = newTestWakuNode(generateSecp256k1Key(), parseIpAddress("127.0.0.1"), Port(60600))
+    peerInfo1 = node1.switch.peerInfo
     node2 = newTestWakuNode(generateSecp256k1Key(), parseIpAddress("127.0.0.1"), Port(60602))
     peerInfo2 = node2.switch.peerInfo
     node3 = newTestWakuNode(generateSecp256k1Key(), parseIpAddress("127.0.0.1"), Port(60604))
@@ -94,3 +97,58 @@ suite "Waku v2 Rest API - Admin":
       getRes.status == 200
       $getRes.contentType == $MIMETYPE_JSON
       getRes.data.len() == 0
+
+  asyncTest "Get filter data":
+    await allFutures(node1.mountFilter(), node2.mountFilterClient(), node3.mountFilterClient())
+
+    let
+      contentFiltersNode2 = @[DefaultContentTopic, ContentTopic("2"), ContentTopic("3")]
+      contentFiltersNode3 = @[ContentTopic("3"), ContentTopic("4")]
+      pubsubTopicNode2 = DefaultPubsubTopic
+      pubsubTopicNode3 = PubsubTopic("/waku/2/custom-waku/proto")
+
+      expectedFilterData2 = fmt"(peerId: ""{$peerInfo2}"", filterCriteria:" &
+        fmt" @[(pubsubTopic: ""{pubsubTopicNode2}"", contentTopic: ""{contentFiltersNode2[0]}""), " &
+        fmt"(pubsubTopic: ""{pubsubTopicNode2}"", contentTopic: ""{contentFiltersNode2[1]}""), " &
+        fmt"(pubsubTopic: ""{pubsubTopicNode2}"", contentTopic: ""{contentFiltersNode2[2]}"")]"
+
+      expectedFilterData3 = fmt"(peerId: ""{$peerInfo3}"", filterCriteria:" &
+        fmt" @[(pubsubTopic: ""{pubsubTopicNode3}"", contentTopic: ""{contentFiltersNode3[0]}""), " &
+        fmt"(pubsubTopic: ""{pubsubTopicNode3}"", contentTopic: ""{contentFiltersNode3[1]}"")]"
+
+    let
+      subscribeResponse2 = await node2.wakuFilterClient.subscribe(
+        peerInfo1, pubsubTopicNode2, contentFiltersNode2
+      )
+      subscribeResponse3 = await node3.wakuFilterClient.subscribe(
+        peerInfo1, pubsubTopicNode3, contentFiltersNode3
+      )
+
+    assert subscribeResponse2.isOk(), $subscribeResponse2.error
+    assert subscribeResponse3.isOk(), $subscribeResponse3.error
+
+    let getRes = await client.getFilterSubscriptions()
+
+    check:
+      getRes.status == 200
+      $getRes.contentType == $MIMETYPE_JSON
+      getRes.data.len() == 2
+      ($getRes.data).contains(expectedFilterData2)
+      ($getRes.data).contains(expectedFilterData3)
+
+  asyncTest "Get filter data - no filter subscribers":
+    await node1.mountFilter()
+
+    let getRes = await client.getFilterSubscriptions()
+
+    check:
+      getRes.status == 200
+      $getRes.contentType == $MIMETYPE_JSON
+      getRes.data.len() == 0
+
+  asyncTest "Get filter data - filter not mounted":
+    let getRes = await client.getFilterSubscriptionsFilterNotMounted()
+
+    check:
+      getRes.status == 400
+      getRes.data == "Error: Filter Protocol is not mounted to the node"

waku/README.md

@@ -166,7 +166,7 @@ It is possible to configure an IPv4 DNS domain name that resolves to the node's
 wakunode2 --dns4-domain-name=mynode.example.com
 ```
 
-This allows for the node's publically announced `multiaddrs` to use the `/dns4` scheme.
+This allows for the node's publicly announced `multiaddrs` to use the `/dns4` scheme.
 In addition, nodes with domain name and [secure websocket configured](#enabling-websocket),
 will generate a discoverable ENR containing the `/wss` multiaddr with `/dns4` domain name.
 This is necessary to verify domain certificates when connecting to this node over secure websocket.

waku/waku_api/rest/admin/client.nim

@@ -33,3 +33,11 @@ proc getPeers*():
 proc postPeers*(body: seq[string]):
       RestResponse[string]
       {.rest, endpoint: "/admin/v1/peers", meth: HttpMethod.MethodPost.}
+
+proc getFilterSubscriptions*():
+    RestResponse[seq[FilterSubscription]]
+    {.rest, endpoint: "/admin/v1/filter/subscriptions", meth: HttpMethod.MethodGet.}
+
+proc getFilterSubscriptionsFilterNotMounted*():
+    RestResponse[string]
+    {.rest, endpoint: "/admin/v1/filter/subscriptions", meth: HttpMethod.MethodGet.}

waku/waku_api/rest/admin/handlers.nim

@@ -4,8 +4,7 @@ else:
   {.push raises: [].}
 
 import
-  std/strformat,
-  std/sequtils,
+  std/[strformat,sequtils,sets,tables],
   stew/byteutils,
   chronicles,
   json_serialization,
@@ -32,6 +31,7 @@ logScope:
   topics = "waku node rest admin api"
 
 const ROUTE_ADMIN_V1_PEERS* = "/admin/v1/peers"
+const ROUTE_ADMIN_V1_FILTER_SUBS* = "/admin/v1/filter/subscriptions"
 
 type PeerProtocolTuple = tuple[multiaddr: string, protocol: string, connected: bool]
 
@@ -111,6 +111,30 @@ proc installAdminV1PostPeersHandler(router: var RestRouter, node: WakuNode) =
 
     return RestApiResponse.ok()
 
+proc installAdminV1GetFilterSubsHandler(router: var RestRouter, node: WakuNode) =
+  router.api(MethodGet, ROUTE_ADMIN_V1_FILTER_SUBS) do () -> RestApiResponse:
+
+    if node.wakuFilter.isNil():
+      return RestApiResponse.badRequest("Error: Filter Protocol is not mounted to the node")
+
+    var
+      subscriptions: seq[FilterSubscription] = @[]
+      filterCriteria: seq[FilterTopic]
+
+    for (peerId, criteria) in node.wakuFilter.subscriptions.pairs():
+      filterCriteria = criteria.toSeq().mapIt(FilterTopic(pubsubTopic: it[0],
+        contentTopic: it[1]))
+
+      subscriptions.add(FilterSubscription(peerId: $peerId, filterCriteria: filterCriteria))
+
+    let resp = RestApiResponse.jsonResponse(subscriptions, status=Http200)
+    if resp.isErr():
+      error "An error ocurred while building the json respose: ", error=resp.error
+      return RestApiResponse.internalServerError(fmt("An error ocurred while building the json respose: {resp.error}"))
+
+    return resp.get()
+
 proc installAdminApiHandlers*(router: var RestRouter, node: WakuNode) =
   installAdminV1GetPeersHandler(router, node)
   installAdminV1PostPeersHandler(router, node)
+  installAdminV1GetFilterSubsHandler(router, node)

waku/waku_api/rest/admin/openapi.yaml

@@ -49,6 +49,26 @@ paths:
           description: Cannot connect to one or more peers.
         '5XX':
           description: Unexpected error.
+  /admin/v1/filter/subscriptions:
+    get:
+      summary: Get filter protocol subscribers
+      description: Retrieve information about the serving filter subscriptions
+      operationId: getFilterInfo
+      tags:
+        - admin
+      responses:
+        '200':
+          description: Information about subscribed filter peers and topics
+          content:
+            application/json:
+              schema:
+                type: array
+                items:
+                  $ref: '#/components/schemas/FilterSubscription'
+        '400':
+          description: Filter Protocol is not mounted to the node
+        '5XX':
+          description: Unexpected error.
 
 components:
   schemas:
@@ -72,3 +92,24 @@ components:
                 type: string
               connected:
                 type: boolean
+
+    FilterSubscription:
+      type: object
+      required:
+        - peerId
+        - filterCriteria
+      properties:
+        peerId:
+          type: string
+        filterCriteria:
+          type: array
+          items:
+            type: object
+            required:
+              - pubsubTopic
+              - contentTopic
+            properties:
+              pubsubTopic:
+                type: string
+              contentTopic:
+                type: string