diff --git a/docs/contributors/continuous-integration.md b/docs/contributors/continuous-integration.md index 9e4332ba5c..69554f07d4 100644 --- a/docs/contributors/continuous-integration.md +++ b/docs/contributors/continuous-integration.md @@ -8,7 +8,7 @@ The CI setup exists on the Status.im Jenkins instance: https://ci.infra.status.im/job/nim-waku/ -It currently consists four jobs: +It currently consists of four jobs: * [manual](https://ci.infra.status.im/job/nim-waku/job/manual/) - For manually executing builds using parameters. * [deploy-wakuv1-test](https://ci.infra.status.im/job/nim-waku/job/deploy-wakuv1-test/) - Builds every new commit in `master` and deploys to `wakuv1.test` fleet. diff --git a/docs/contributors/release-process.md b/docs/contributors/release-process.md index ebcf47e604..7b14d8ae57 100644 --- a/docs/contributors/release-process.md +++ b/docs/contributors/release-process.md @@ -4,7 +4,7 @@ How to do releases. For more context, see https://trunkbaseddevelopment.com/branch-for-release/ -## How to to do releases +## How to do releases ### Before release @@ -84,6 +84,7 @@ Ensure all items in this list are ticked: > Clients are reachable via the corresponding channels on the Vac Discord server. > It should be enough to inform clients on the `#nwaku` and `#announce` channels on Discord. > Informal conversations with specific repo maintainers are often part of this process. + - Check if nwaku configuration parameters changed. If so [update fleet configuration](https://www.notion.so/Fleet-Ownership-7532aad8896d46599abac3c274189741?pvs=4#d2d2f0fe4b3c429fbd860a1d64f89a64) in [infra-nim-waku](https://github.com/status-im/infra-nim-waku) - Deploy release to the `wakuv2.prod` fleet from [Jenkins](https://ci.infra.status.im/job/nim-waku/job/deploy-wakuv2-prod/). - Ensure that nodes successfully start up and monitor health using [Grafana](https://grafana.infra.status.im/d/qrp_ZCTGz/nim-waku-v2?orgId=1) and [Kibana](https://kibana.infra.status.im/goto/a7728e70-eb26-11ec-81d1-210eb3022c76). - If necessary, revert by deploying the previous release. Download logs and open a bug report issue. diff --git a/docs/operators/README.md b/docs/operators/README.md index 98bfc87a9d..58509775d0 100644 --- a/docs/operators/README.md +++ b/docs/operators/README.md @@ -13,7 +13,7 @@ Nwaku (formerly `nim-waku`) aims to be a lightweight and robust Waku v2 client. It serves as the reference implementation for researchers, who extend the client in parallel to spec development. As such, it is first in line to support innovative and new Waku v2 protocols, -but configurable enough to serve the adaptive needs of a various operators. +but configurable enough to serve the adaptive needs of various operators. We are also developing a set of operator-focused tools to monitor and maintain a running nwaku node. This guide provides step-by-step tutorials covering how to build and configure your own nwaku node, @@ -32,4 +32,4 @@ For bug reports, please [tag your issue with the `bug` label](https://github.com If you believe the reported issue requires critical attention, please [use the `critical` label](https://github.com/waku-org/nwaku/issues/new?labels=critical,bug) to assist with triaging. -To get help, or participate in the conversation, join the [Vac Discord](https://discord.gg/KNj3ctuZvZ) server. \ No newline at end of file +To get help, or participate in the conversation, join the [Vac Discord](https://discord.gg/KNj3ctuZvZ) server. diff --git a/docs/operators/how-to/configure-store.md b/docs/operators/how-to/configure-store.md index d340ce9057..b8befe8fc1 100644 --- a/docs/operators/how-to/configure-store.md +++ b/docs/operators/how-to/configure-store.md @@ -35,13 +35,13 @@ There is a set of configuration options to customize the waku store protocol's m + The time retention policy,`time:` (e.g., `time:14400`) + The capacity retention policy,`capacity:` (e.g, `capacity:25000`) + The size retention policy,`size:` (e.g, `size:25Gb`) - + To disable the retention policy, explicitly, set this option to to `""`, an empty string. + + To disable the retention policy, explicitly, set this option to `""`, an empty string. * `--store-message-db-url`: The message store database url option controls the message storage engine. This option follows the [_SQLAlchemy_ database URL format](https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls). + SQLite engine: The only database engine supported by the nwaku node. The database URL has this shape: `sqlite://`. If the `` is not an absolute path (preceded by a `/` character), the file will be created in the current working directory. The SQLite engine also supports to select a non-persistent in-memory database by setting the `` to `:memory:`. + In the case you don't want to use a persistent message store; set the `--store-message-db-url` to an empty string, `""`. This will instruct the node to use the fallback in-memory message store. -By default the node message store will be configured with a time retention policy set to `14400` seconds (4 hours). Additionaly, by default, the node message store will use the SQLite database engine to store historical messages in order to persist these between restarts. +By default the node message store will be configured with a time retention policy set to `14400` seconds (4 hours). Additionally, by default, the node message store will use the SQLite database engine to store historical messages in order to persist these between restarts. > :warning: Note the 3 slashes, `///`, after the SQLite database URL schema. The third slash indicates that it is an absolute path: `/mnt/nwaku/data/db1/store.sqlite3` @@ -55,4 +55,4 @@ wakunode2 \ ### How much resources should I allocate? Currently store service nodes use, by default, a message store backed by an in-disk SQLite database. Most Waku messages average a size of 1KB - 2KB, implying a minimum memory requirement of at least ~250MB -for a typical store capacity of 100k messages. Note, however, that the allowable maximum size for Waku messages is up to 1MB. \ No newline at end of file +for a typical store capacity of 100k messages. Note, however, that the allowable maximum size for Waku messages is up to 1MB. diff --git a/docs/operators/how-to/configure.md b/docs/operators/how-to/configure.md index b7f70d4c51..f052b22be0 100644 --- a/docs/operators/how-to/configure.md +++ b/docs/operators/how-to/configure.md @@ -81,7 +81,7 @@ provided via environment variables. ### Configuration file -The third configuration mechanism in order of precedence is the configuration via a TOML file. The previous mechanims take precedence over this mechanism as explained above. +The third configuration mechanism in order of precedence is the configuration via a TOML file. The previous mechanisms take precedence over this mechanism as explained above. The configuration file follows the [TOML](https://toml.io/en/) format: @@ -90,7 +90,7 @@ log-level = "DEBUG" tcp-port = 65000 ``` -The path to the TOML file can be specified usin one of the previous configuration mechanisms: +The path to the TOML file can be specified using one of the previous configuration mechanisms: * By passing the `--config-file` command line option: ```shell diff --git a/docs/tutorial/rln-chat2-live-testnet.md b/docs/tutorial/rln-chat2-live-testnet.md index 8dfc853086..13f5563ac4 100644 --- a/docs/tutorial/rln-chat2-live-testnet.md +++ b/docs/tutorial/rln-chat2-live-testnet.md @@ -9,7 +9,7 @@ sending more than one message per epoch. At the time of this tutorial, the epoch duration is set to `10` seconds. You can inspect the current epoch value by checking the following [constant variable](https://github.com/status-im/nim-waku/blob/21cac6d491a6d995a7a8ba84c85fecc7817b3d8b/waku/v2/protocol/waku_rln_relay/constants.nim#L245) in the nim-waku codebase. Your messages will be routed via test fleets and will arrive at other live chat2 clients that are running in rate-limited mode over the same content topic i.e., `/toy-chat/3/mingde/proto`. -Your samp activity will be detected by them and a proper message will be shown on their console. +Your spam activity will be detected by them and a proper message will be shown on their console. # Set up ## Build chat2 diff --git a/tests/testlib/sequtils.nim b/tests/testlib/sequtils.nim index b47ba75f50..5fd3d414f7 100644 --- a/tests/testlib/sequtils.nim +++ b/tests/testlib/sequtils.nim @@ -1,5 +1,2 @@ proc toString*(bytes: seq[byte]): string = cast[string](bytes) - -proc toBytes*(str: string): seq[byte] = - cast[seq[byte]](str) diff --git a/tests/wakunode_rest/test_rest_admin.nim b/tests/wakunode_rest/test_rest_admin.nim index 2bf66d624e..a418e6ab6d 100644 --- a/tests/wakunode_rest/test_rest_admin.nim +++ b/tests/wakunode_rest/test_rest_admin.nim @@ -1,7 +1,7 @@ {.used.} import - std/sequtils, + std/[sequtils,strformat], stew/shims/net, testutils/unittests, presto, presto/client as presto_client, @@ -10,6 +10,7 @@ import import ../../waku/waku_core, ../../waku/waku_node, + ../../waku/waku_filter_v2/client, ../../waku/node/peer_manager, ../../waku/waku_api/rest/server, ../../waku/waku_api/rest/client, @@ -26,6 +27,7 @@ suite "Waku v2 Rest API - Admin": var node1 {.threadvar.}: WakuNode var node2 {.threadvar.}: WakuNode var node3 {.threadvar.}: WakuNode + var peerInfo1 {.threadvar.}: RemotePeerInfo var peerInfo2 {.threadvar.}: RemotePeerInfo var peerInfo3 {.threadvar.}: RemotePeerInfo var restServer {.threadvar.}: RestServerRef @@ -33,6 +35,7 @@ suite "Waku v2 Rest API - Admin": asyncSetup: node1 = newTestWakuNode(generateSecp256k1Key(), parseIpAddress("127.0.0.1"), Port(60600)) + peerInfo1 = node1.switch.peerInfo node2 = newTestWakuNode(generateSecp256k1Key(), parseIpAddress("127.0.0.1"), Port(60602)) peerInfo2 = node2.switch.peerInfo node3 = newTestWakuNode(generateSecp256k1Key(), parseIpAddress("127.0.0.1"), Port(60604)) @@ -94,3 +97,58 @@ suite "Waku v2 Rest API - Admin": getRes.status == 200 $getRes.contentType == $MIMETYPE_JSON getRes.data.len() == 0 + + asyncTest "Get filter data": + await allFutures(node1.mountFilter(), node2.mountFilterClient(), node3.mountFilterClient()) + + let + contentFiltersNode2 = @[DefaultContentTopic, ContentTopic("2"), ContentTopic("3")] + contentFiltersNode3 = @[ContentTopic("3"), ContentTopic("4")] + pubsubTopicNode2 = DefaultPubsubTopic + pubsubTopicNode3 = PubsubTopic("/waku/2/custom-waku/proto") + + expectedFilterData2 = fmt"(peerId: ""{$peerInfo2}"", filterCriteria:" & + fmt" @[(pubsubTopic: ""{pubsubTopicNode2}"", contentTopic: ""{contentFiltersNode2[0]}""), " & + fmt"(pubsubTopic: ""{pubsubTopicNode2}"", contentTopic: ""{contentFiltersNode2[1]}""), " & + fmt"(pubsubTopic: ""{pubsubTopicNode2}"", contentTopic: ""{contentFiltersNode2[2]}"")]" + + expectedFilterData3 = fmt"(peerId: ""{$peerInfo3}"", filterCriteria:" & + fmt" @[(pubsubTopic: ""{pubsubTopicNode3}"", contentTopic: ""{contentFiltersNode3[0]}""), " & + fmt"(pubsubTopic: ""{pubsubTopicNode3}"", contentTopic: ""{contentFiltersNode3[1]}"")]" + + let + subscribeResponse2 = await node2.wakuFilterClient.subscribe( + peerInfo1, pubsubTopicNode2, contentFiltersNode2 + ) + subscribeResponse3 = await node3.wakuFilterClient.subscribe( + peerInfo1, pubsubTopicNode3, contentFiltersNode3 + ) + + assert subscribeResponse2.isOk(), $subscribeResponse2.error + assert subscribeResponse3.isOk(), $subscribeResponse3.error + + let getRes = await client.getFilterSubscriptions() + + check: + getRes.status == 200 + $getRes.contentType == $MIMETYPE_JSON + getRes.data.len() == 2 + ($getRes.data).contains(expectedFilterData2) + ($getRes.data).contains(expectedFilterData3) + + asyncTest "Get filter data - no filter subscribers": + await node1.mountFilter() + + let getRes = await client.getFilterSubscriptions() + + check: + getRes.status == 200 + $getRes.contentType == $MIMETYPE_JSON + getRes.data.len() == 0 + + asyncTest "Get filter data - filter not mounted": + let getRes = await client.getFilterSubscriptionsFilterNotMounted() + + check: + getRes.status == 400 + getRes.data == "Error: Filter Protocol is not mounted to the node" \ No newline at end of file diff --git a/waku/README.md b/waku/README.md index b9d7b0cda4..27030a8966 100644 --- a/waku/README.md +++ b/waku/README.md @@ -166,7 +166,7 @@ It is possible to configure an IPv4 DNS domain name that resolves to the node's wakunode2 --dns4-domain-name=mynode.example.com ``` -This allows for the node's publically announced `multiaddrs` to use the `/dns4` scheme. +This allows for the node's publicly announced `multiaddrs` to use the `/dns4` scheme. In addition, nodes with domain name and [secure websocket configured](#enabling-websocket), will generate a discoverable ENR containing the `/wss` multiaddr with `/dns4` domain name. This is necessary to verify domain certificates when connecting to this node over secure websocket. diff --git a/waku/waku_api/rest/admin/client.nim b/waku/waku_api/rest/admin/client.nim index 7c0ee8a663..9768536560 100644 --- a/waku/waku_api/rest/admin/client.nim +++ b/waku/waku_api/rest/admin/client.nim @@ -33,3 +33,11 @@ proc getPeers*(): proc postPeers*(body: seq[string]): RestResponse[string] {.rest, endpoint: "/admin/v1/peers", meth: HttpMethod.MethodPost.} + +proc getFilterSubscriptions*(): + RestResponse[seq[FilterSubscription]] + {.rest, endpoint: "/admin/v1/filter/subscriptions", meth: HttpMethod.MethodGet.} + +proc getFilterSubscriptionsFilterNotMounted*(): + RestResponse[string] + {.rest, endpoint: "/admin/v1/filter/subscriptions", meth: HttpMethod.MethodGet.} diff --git a/waku/waku_api/rest/admin/handlers.nim b/waku/waku_api/rest/admin/handlers.nim index bf4b5ebed4..6621908b54 100644 --- a/waku/waku_api/rest/admin/handlers.nim +++ b/waku/waku_api/rest/admin/handlers.nim @@ -4,8 +4,7 @@ else: {.push raises: [].} import - std/strformat, - std/sequtils, + std/[strformat,sequtils,sets,tables], stew/byteutils, chronicles, json_serialization, @@ -32,6 +31,7 @@ logScope: topics = "waku node rest admin api" const ROUTE_ADMIN_V1_PEERS* = "/admin/v1/peers" +const ROUTE_ADMIN_V1_FILTER_SUBS* = "/admin/v1/filter/subscriptions" type PeerProtocolTuple = tuple[multiaddr: string, protocol: string, connected: bool] @@ -111,6 +111,30 @@ proc installAdminV1PostPeersHandler(router: var RestRouter, node: WakuNode) = return RestApiResponse.ok() +proc installAdminV1GetFilterSubsHandler(router: var RestRouter, node: WakuNode) = + router.api(MethodGet, ROUTE_ADMIN_V1_FILTER_SUBS) do () -> RestApiResponse: + + if node.wakuFilter.isNil(): + return RestApiResponse.badRequest("Error: Filter Protocol is not mounted to the node") + + var + subscriptions: seq[FilterSubscription] = @[] + filterCriteria: seq[FilterTopic] + + for (peerId, criteria) in node.wakuFilter.subscriptions.pairs(): + filterCriteria = criteria.toSeq().mapIt(FilterTopic(pubsubTopic: it[0], + contentTopic: it[1])) + + subscriptions.add(FilterSubscription(peerId: $peerId, filterCriteria: filterCriteria)) + + let resp = RestApiResponse.jsonResponse(subscriptions, status=Http200) + if resp.isErr(): + error "An error ocurred while building the json respose: ", error=resp.error + return RestApiResponse.internalServerError(fmt("An error ocurred while building the json respose: {resp.error}")) + + return resp.get() + proc installAdminApiHandlers*(router: var RestRouter, node: WakuNode) = installAdminV1GetPeersHandler(router, node) installAdminV1PostPeersHandler(router, node) + installAdminV1GetFilterSubsHandler(router, node) diff --git a/waku/waku_api/rest/admin/openapi.yaml b/waku/waku_api/rest/admin/openapi.yaml index 2ce64e46c6..f6118ad702 100644 --- a/waku/waku_api/rest/admin/openapi.yaml +++ b/waku/waku_api/rest/admin/openapi.yaml @@ -49,6 +49,26 @@ paths: description: Cannot connect to one or more peers. '5XX': description: Unexpected error. + /admin/v1/filter/subscriptions: + get: + summary: Get filter protocol subscribers + description: Retrieve information about the serving filter subscriptions + operationId: getFilterInfo + tags: + - admin + responses: + '200': + description: Information about subscribed filter peers and topics + content: + application/json: + schema: + type: array + items: + $ref: '#/components/schemas/FilterSubscription' + '400': + description: Filter Protocol is not mounted to the node + '5XX': + description: Unexpected error. components: schemas: @@ -72,3 +92,24 @@ components: type: string connected: type: boolean + + FilterSubscription: + type: object + required: + - peerId + - filterCriteria + properties: + peerId: + type: string + filterCriteria: + type: array + items: + type: object + required: + - pubsubTopic + - contentTopic + properties: + pubsubTopic: + type: string + contentTopic: + type: string diff --git a/waku/waku_api/rest/admin/types.nim b/waku/waku_api/rest/admin/types.nim index ab6b8f22c0..1e051e633d 100644 --- a/waku/waku_api/rest/admin/types.nim +++ b/waku/waku_api/rest/admin/types.nim @@ -25,6 +25,16 @@ type type WakuPeers* = seq[WakuPeer] +type + FilterTopic* = object + pubsubTopic*: string + contentTopic*: string + +type + FilterSubscription* = object + peerId*: string + filterCriteria*: seq[FilterTopic] + #### Serialization and deserialization proc writeValue*(writer: var JsonWriter[RestJson], value: ProtocolState) @@ -41,6 +51,20 @@ proc writeValue*(writer: var JsonWriter[RestJson], value: WakuPeer) writer.writeField("protocols", value.protocols) writer.endRecord() +proc writeValue*(writer: var JsonWriter[RestJson], value: FilterTopic) + {.raises: [IOError].} = + writer.beginRecord() + writer.writeField("pubsubTopic", value.pubsubTopic) + writer.writeField("contentTopic", value.contentTopic) + writer.endRecord() + +proc writeValue*(writer: var JsonWriter[RestJson], value: FilterSubscription) + {.raises: [IOError].} = + writer.beginRecord() + writer.writeField("peerId", value.peerId) + writer.writeField("filterCriteria", value.filterCriteria) + writer.endRecord() + proc readValue*(reader: var JsonReader[RestJson], value: var ProtocolState) {.gcsafe, raises: [SerializationError, IOError].} = var @@ -101,6 +125,66 @@ proc readValue*(reader: var JsonReader[RestJson], value: var WakuPeer) protocols: protocols.get() ) +proc readValue*(reader: var JsonReader[RestJson], value: var FilterTopic) + {.gcsafe, raises: [SerializationError, IOError].} = + var + pubsubTopic: Option[string] + contentTopic: Option[string] + + for fieldName in readObjectFields(reader): + case fieldName + of "pubsubTopic": + if pubsubTopic.isSome(): + reader.raiseUnexpectedField("Multiple `pubsubTopic` fields found", "FilterTopic") + pubsubTopic = some(reader.readValue(string)) + of "contentTopic": + if contentTopic.isSome(): + reader.raiseUnexpectedField("Multiple `contentTopic` fields found", "FilterTopic") + contentTopic = some(reader.readValue(string)) + else: + unrecognizedFieldWarning() + + if pubsubTopic.isNone(): + reader.raiseUnexpectedValue("Field `pubsubTopic` is missing") + + if contentTopic.isNone(): + reader.raiseUnexpectedValue("Field `contentTopic` are missing") + + value = FilterTopic( + pubsubTopic: pubsubTopic.get(), + contentTopic: contentTopic.get() + ) + +proc readValue*(reader: var JsonReader[RestJson], value: var FilterSubscription) + {.gcsafe, raises: [SerializationError, IOError].} = + var + peerId: Option[string] + filterCriteria: Option[seq[FilterTopic]] + + for fieldName in readObjectFields(reader): + case fieldName + of "peerId": + if peerId.isSome(): + reader.raiseUnexpectedField("Multiple `peerId` fields found", "FilterSubscription") + peerId = some(reader.readValue(string)) + of "filterCriteria": + if filterCriteria.isSome(): + reader.raiseUnexpectedField("Multiple `filterCriteria` fields found", "FilterSubscription") + filterCriteria = some(reader.readValue(seq[FilterTopic])) + else: + unrecognizedFieldWarning() + + if peerId.isNone(): + reader.raiseUnexpectedValue("Field `peerId` is missing") + + if filterCriteria.isNone(): + reader.raiseUnexpectedValue("Field `filterCriteria` are missing") + + value = FilterSubscription( + peerId: peerId.get(), + filterCriteria: filterCriteria.get() + ) + ## Utility for populating WakuPeers and ProtocolState func `==`*(a, b: ProtocolState): bool {.inline.} = return a.protocol == b.protocol