Merge branch 'master' into feat--waku-rendezvous

.github/ISSUE_TEMPLATE/bump_dependencies.md

@@ -12,6 +12,7 @@ assignees: ''
 Update `nwaku` "vendor" dependencies.
 
 ### Items to bump
+- [ ] negentropy
 - [ ] dnsclient.nim ( update to the latest tag version )
 - [ ] nim-bearssl
 - [ ] nimbus-build-system

.github/ISSUE_TEMPLATE/prepare_release.md

@@ -20,6 +20,7 @@ All items below are to be completed by the owner of the given release.
 - [ ] Create release branch
 - [ ] Assign release candidate tag to the release branch HEAD. e.g. v0.30.0-rc.0
 - [ ] Generate and edit releases notes in CHANGELOG.md
+- [ ] Review possible update of [config-options](https://github.com/waku-org/docs.waku.org/blob/develop/docs/guides/nwaku/config-options.md)
 - [ ] _End user impact_: Summarize impact of changes on Status end users (can be a comment in this issue).
 - [ ] **Validate release candidate**
 
@@ -43,7 +44,7 @@ All items below are to be completed by the owner of the given release.
       - [ ] Send and receive messages in a community
       - [ ] Close one instance, send messages with second instance, reopen first instance and confirm messages sent while offline are retrieved from store
     - [ ] Perform checks based _end user impact_
-    - [ ] Inform other (Waku and Status) CCs to point their instance to `status.staging` for a few days. Ping Status colleagues from their Discord server or Status community (not blocking point.)
+    - [ ] Inform other (Waku and Status) CCs to point their instance to `status.staging` for a few days. Ping Status colleagues from their Discord server or [Status community](https://status.app/c/G3kAAMSQtb05kog3aGbr3kiaxN4tF5xy4BAGEkkLwILk2z3GcoYlm5hSJXGn7J3laft-tnTwDWmYJ18dP_3bgX96dqr_8E3qKAvxDf3NrrCMUBp4R9EYkQez9XSM4486mXoC3mIln2zc-TNdvjdfL9eHVZ-mGgs=#zQ3shZeEJqTC1xhGUjxuS4rtHSrhJ8vUYp64v6qWkLpvdy9L9) (not blocking point.)
     - [ ] Ask Status-QA to perform sanity checks (as described above) + checks based on _end user impact_; do specify the version being tested
     - [ ] Ask Status-QA or infra to run the automated Status e2e tests against `status.staging`
     - [ ] Get other CCs sign-off: they comment on this PR "used app for a week, no problem", or problem reported, resolved and new RC

.github/workflows/ci.yml

@@ -117,7 +117,7 @@ jobs:
 
           export MAKEFLAGS="-j1"
           export NIMFLAGS="--colors:off -d:chronicles_colors:none"
-          
+
           make V=1 LOG_LEVEL=DEBUG QUICK_AND_DIRTY_COMPILER=1 POSTGRES=$postgres_enabled test testwakunode2
 
   build-docker-image:
@@ -141,25 +141,36 @@ jobs:
       nim_wakunode_image: ${{ needs.build-docker-image.outputs.image }}
       test_type: node-optional
       debug: waku*
-  
+
   lint:
     name: "Lint"
     runs-on: ubuntu-latest
+    needs: build
     steps:
-      - name: Checkout
-        uses: actions/checkout@v4
+      - name: Checkout code
+        uses: actions/checkout@v3
+
+      - name: Get submodules hash
+        id: submodules
+        run: |
+          echo "hash=$(git submodule status | awk '{print $1}' | sort | shasum -a 256 | sed 's/[ -]*//g')" >> $GITHUB_OUTPUT
+
+      - name: Cache submodules
+        uses: actions/cache@v3
         with:
-          fetch-depth: 2  # In PR, has extra merge commit: ^1 = PR, ^2 = base
+          path: |
+            vendor/
+            .git/modules
+          key: ${{ runner.os }}-vendor-modules-${{ steps.submodules.outputs.hash }}
+
+      - name: Build nph
+        run: |
+          make build-nph
 
       - name: Check nph formatting
-        # Pin nph to a specific version to avoid sudden style differences.
-        # Updating nph version should be accompanied with running the new
-        # version on the fluffy directory.
         run: |
-          VERSION="v0.5.1"
-          ARCHIVE="nph-linux_x64.tar.gz"
-          curl -L "https://github.com/arnetheduck/nph/releases/download/${VERSION}/${ARCHIVE}" -o ${ARCHIVE}
-          tar -xzf ${ARCHIVE}
           shopt -s extglob  # Enable extended globbing
-          ./nph examples waku tests tools apps *.@(nim|nims|nimble)
+          NPH=$(make print-nph-path)
+          echo "using nph at ${NPH}"
+          "${NPH}" examples waku tests tools apps *.@(nim|nims|nimble)
           git diff --exit-code

.github/workflows/pre-release.yml

@@ -123,7 +123,7 @@ jobs:
           ref: master
 
       - name: download artifacts
-        uses: actions/download-artifact@v2
+        uses: actions/download-artifact@v4
 
       - name: prep variables
         id: vars

.github/workflows/release-assets.yml

@@ -58,7 +58,7 @@ jobs:
           tar -cvzf ${{steps.vars.outputs.nwaku}} ./build/
 
       - name: Upload asset
-        uses: actions/upload-artifact@v2.2.3
+        uses: actions/upload-artifact@v4.4.0
         with:
           name: ${{steps.vars.outputs.nwaku}}
           path: ${{steps.vars.outputs.nwaku}}

.gitmodules

@@ -169,3 +169,13 @@
 	url = https://github.com/nim-lang/db_connector.git
 	ignore = untracked
 	branch = master
+[submodule "vendor/negentropy"]
+	ignore = untracked
+	path = vendor/negentropy
+	url = https://github.com/waku-org/negentropy.git
+	branch = master
+[submodule "vendor/nph"]
+	ignore = untracked
+	branch = master
+	path = vendor/nph
+	url = https://github.com/arnetheduck/nph.git

CHANGELOG.md

@@ -1,3 +1,87 @@
+## v0.32.0 (2024-08-30)
+
+#### Notes:
+
+* A new `discv5-only` CLI flag was introduced, which if set to true will perform optimizations for nodes that only run the DiscV5 service
+* The `protected-topic` CLI config item has been deprecated in favor of the new `protected-shard` configuration. Protected topics are still supported and will be completely removed in two releases time for `v0.34.0`
+
+### Release highlights
+
+* Merged Nwaku Sync protocol for synchronizing store nodes
+* Added Store Resume mechanism to retrieve messages sent when the node was offline
+
+### Features
+
+- Nwaku Sync ([#2403](https://github.com/waku-org/nwaku/issues/2403)) ([2cc86c51](https://github.com/waku-org/nwaku/commit/2cc86c51))
+- misc. updates for discovery network analysis ([#2930](https://github.com/waku-org/nwaku/issues/2930)) ([4340eb75](https://github.com/waku-org/nwaku/commit/4340eb75))
+- store resume ([#2919](https://github.com/waku-org/nwaku/issues/2919)) ([aed2a113](https://github.com/waku-org/nwaku/commit/aed2a113))
+
+### Bug Fixes
+
+- return on insert error ([#2956](https://github.com/waku-org/nwaku/issues/2956)) ([5f0fbd78](https://github.com/waku-org/nwaku/commit/5f0fbd78))
+- network monitor improvements ([#2939](https://github.com/waku-org/nwaku/issues/2939)) ([80583237](https://github.com/waku-org/nwaku/commit/80583237))
+- add back waku discv5 metrics ([#2927](https://github.com/waku-org/nwaku/issues/2927)) ([e4e01fab](https://github.com/waku-org/nwaku/commit/e4e01fab))
+- update and shift unittest ([#2934](https://github.com/waku-org/nwaku/issues/2934)) ([08973add](https://github.com/waku-org/nwaku/commit/08973add))
+- handle rln-relay-message-limit  ([#2867](https://github.com/waku-org/nwaku/issues/2867)) ([8d107b0d](https://github.com/waku-org/nwaku/commit/8d107b0d))
+
+### Changes
+
+- libwaku retrieve my enr and adapt golang example ([#2987](https://github.com/waku-org/nwaku/issues/2987)) ([1ff9f1dd](https://github.com/waku-org/nwaku/commit/1ff9f1dd))
+- run `ANALYZE messages` regularly for better db performance ([#2986](https://github.com/waku-org/nwaku/issues/2986)) ([32f2d85d](https://github.com/waku-org/nwaku/commit/32f2d85d))
+- liteprotocoltester for simulation and for fleets ([#2813](https://github.com/waku-org/nwaku/issues/2813)) ([f4fa73e9](https://github.com/waku-org/nwaku/commit/f4fa73e9))
+- lock in nph version and add pre-commit hook ([#2938](https://github.com/waku-org/nwaku/issues/2938)) ([d63e3430](https://github.com/waku-org/nwaku/commit/d63e3430))
+- logging received message info via onValidated observer ([#2973](https://github.com/waku-org/nwaku/issues/2973)) ([e8bce67d](https://github.com/waku-org/nwaku/commit/e8bce67d))
+- deprecating protected topics in favor of protected shards ([#2983](https://github.com/waku-org/nwaku/issues/2983)) ([e51ffe07](https://github.com/waku-org/nwaku/commit/e51ffe07))
+- rename NsPubsubTopic ([#2974](https://github.com/waku-org/nwaku/issues/2974)) ([67439057](https://github.com/waku-org/nwaku/commit/67439057))
+- install dig ([#2975](https://github.com/waku-org/nwaku/issues/2975)) ([d24b56b9](https://github.com/waku-org/nwaku/commit/d24b56b9))
+- print WakuMessageHash as hex strings ([#2969](https://github.com/waku-org/nwaku/issues/2969)) ([2fd4eb62](https://github.com/waku-org/nwaku/commit/2fd4eb62))
+- updating dependencies for release 0.32.0 ([#2971](https://github.com/waku-org/nwaku/issues/2971)) ([dfd42a7c](https://github.com/waku-org/nwaku/commit/dfd42a7c))
+- bump negentropy to latest master ([#2968](https://github.com/waku-org/nwaku/issues/2968)) ([b36cb075](https://github.com/waku-org/nwaku/commit/b36cb075))
+- keystore: verbose error message when credential is not found ([#2943](https://github.com/waku-org/nwaku/issues/2943)) ([0f11ee14](https://github.com/waku-org/nwaku/commit/0f11ee14))
+- upgrade peer exchange mounting  ([#2953](https://github.com/waku-org/nwaku/issues/2953)) ([42f1bed0](https://github.com/waku-org/nwaku/commit/42f1bed0))
+- replace statusim.net instances with status.im ([#2941](https://github.com/waku-org/nwaku/issues/2941)) ([f534549a](https://github.com/waku-org/nwaku/commit/f534549a))
+- updating doc reference to https rpc ([#2937](https://github.com/waku-org/nwaku/issues/2937)) ([bb7bba35](https://github.com/waku-org/nwaku/commit/bb7bba35))
+- Simplification of store legacy code ([#2931](https://github.com/waku-org/nwaku/issues/2931)) ([d4e8a0da](https://github.com/waku-org/nwaku/commit/d4e8a0da))
+- add peer filtering by cluster for waku peer exchange ([#2932](https://github.com/waku-org/nwaku/issues/2932)) ([b4618f98](https://github.com/waku-org/nwaku/commit/b4618f98))
+- return all connected peers from REST API ([#2923](https://github.com/waku-org/nwaku/issues/2923)) ([a29eca77](https://github.com/waku-org/nwaku/commit/a29eca77))
+- adding lint job to the CI ([#2925](https://github.com/waku-org/nwaku/issues/2925)) ([086cc8ed](https://github.com/waku-org/nwaku/commit/086cc8ed))
+- improve sonda dashboard ([#2918](https://github.com/waku-org/nwaku/issues/2918)) ([6d385cef](https://github.com/waku-org/nwaku/commit/6d385cef))
+- Add new custom built and test target to make in order to enable easy build or test single nim modules ([#2913](https://github.com/waku-org/nwaku/issues/2913)) ([ad25f437](https://github.com/waku-org/nwaku/commit/ad25f437))
+
+This release supports the following [libp2p protocols](https://docs.libp2p.io/concepts/protocols/):
+| Protocol | Spec status | Protocol id |
+| ---: | :---: | :--- |
+| [`11/WAKU2-RELAY`](https://github.com/vacp2p/rfc-index/blob/main/waku/standards/core/11/relay.md) | `stable` | `/vac/waku/relay/2.0.0` |
+| [`12/WAKU2-FILTER`](https://github.com/vacp2p/rfc-index/blob/main/waku/standards/core/12/filter.md) | `draft` | `/vac/waku/filter/2.0.0-beta1` <br />`/vac/waku/filter-subscribe/2.0.0-beta1` <br />`/vac/waku/filter-push/2.0.0-beta1` |
+| [`13/WAKU2-STORE`](https://github.com/vacp2p/rfc-index/blob/main/waku/standards/core/13/store.md) | `draft` | `/vac/waku/store/2.0.0-beta4` |
+| [`19/WAKU2-LIGHTPUSH`](https://github.com/vacp2p/rfc-index/blob/main/waku/standards/core/19/lightpush.md) | `draft` | `/vac/waku/lightpush/2.0.0-beta1` |
+| [`66/WAKU2-METADATA`](https://github.com/waku-org/specs/blob/master/standards/core/metadata.md) | `raw` | `/vac/waku/metadata/1.0.0` |
+| [`WAKU-SYNC`](https://github.com/waku-org/specs/blob/feat--waku-sync/standards/core/sync.md) | `draft` | `/vac/waku/sync/1.0.0` |
+
+## v0.31.1 (2024-08-02)
+
+### Changes
+
+- Optimize hash queries with lookup table ([#2933](https://github.com/waku-org/nwaku/issues/2933)) ([6463885bf](https://github.com/waku-org/nwaku/commit/6463885bf))
+
+### Bug fixes
+
+* Use of detach finalize when needed [2966](https://github.com/waku-org/nwaku/pull/2966)
+* Prevent legacy store from creating new partitions as that approach blocked the database.
+[2931](https://github.com/waku-org/nwaku/pull/2931)
+
+* lightpush better feedback in case the lightpush service node does not have peers [2951](https://github.com/waku-org/nwaku/pull/2951)
+
+This release supports the following [libp2p protocols](https://docs.libp2p.io/concepts/protocols/):
+| Protocol | Spec status | Protocol id |
+| ---: | :---: | :--- |
+| [`11/WAKU2-RELAY`](https://github.com/vacp2p/rfc-index/blob/main/waku/standards/core/11/relay.md) | `stable` | `/vac/waku/relay/2.0.0` |
+| [`12/WAKU2-FILTER`](https://github.com/vacp2p/rfc-index/blob/main/waku/standards/core/12/filter.md) | `draft` | `/vac/waku/filter/2.0.0-beta1` <br />`/vac/waku/filter-subscribe/2.0.0-beta1` <br />`/vac/waku/filter-push/2.0.0-beta1` |
+| [`WAKU2-STORE`](https://github.com/waku-org/specs/blob/master/standards/core/store.md) | `draft` | `/vac/waku/store-query/3.0.0` |
+| [`13/WAKU2-STORE`](https://github.com/vacp2p/rfc-index/blob/main/waku/standards/core/13/store.md) | `draft` | `/vac/waku/store/2.0.0-beta4` |
+| [`19/WAKU2-LIGHTPUSH`](https://github.com/vacp2p/rfc-index/blob/main/waku/standards/core/19/lightpush.md) | `draft` | `/vac/waku/lightpush/2.0.0-beta1` |
+| [`66/WAKU2-METADATA`](https://github.com/waku-org/specs/blob/master/standards/core/metadata.md) | `raw` | `/vac/waku/metadata/1.0.0` |
+
 ## v0.31.0 (2024-07-16)
 ### Notes
 

Dockerfile

@@ -7,7 +7,7 @@ ARG NIM_COMMIT
 ARG LOG_LEVEL=TRACE
 
 # Get build tools and required header files
-RUN apk add --no-cache bash git build-base pcre-dev linux-headers curl jq
+RUN apk add --no-cache bash git build-base openssl-dev pcre-dev linux-headers curl jq
 
 WORKDIR /app
 COPY . .
@@ -41,7 +41,7 @@ LABEL version="unknown"
 EXPOSE 30303 60000 8545
 
 # Referenced in the binary
-RUN apk add --no-cache libgcc pcre-dev libpq-dev
+RUN apk add --no-cache libgcc pcre-dev libpq-dev bind-tools
 
 # Fix for 'Error loading shared library libpcre.so.3: No such file or directory'
 RUN ln -s /usr/lib/libpcre.so /usr/lib/libpcre.so.3

Makefile

@@ -8,7 +8,8 @@ BUILD_SYSTEM_DIR := vendor/nimbus-build-system
 EXCLUDED_NIM_PACKAGES := vendor/nim-dnsdisc/vendor
 LINK_PCRE := 0
 LOG_LEVEL := TRACE
-
+NPH := vendor/nph/src/nph
+FORMAT_MSG := "\\x1B[95mFormatting:\\x1B[39m"
 # we don't want an error here, so we can handle things later, in the ".DEFAULT" target
 -include $(BUILD_SYSTEM_DIR)/makefiles/variables.mk
 
@@ -98,7 +99,7 @@ ifeq (, $(shell which cargo))
 endif
 
 anvil: rustup
-ifeq (, $(shell which anvil))
+ifeq (, $(shell which anvil 2> /dev/null))
 # Install Anvil if it's not installed
 	./scripts/install_anvil.sh
 endif
@@ -165,6 +166,34 @@ clean-librln:
 # Extend clean target
 clean: | clean-librln
 
+######################
+###   NEGENTROPY   ###
+######################
+.PHONY: negentropy
+
+LIBNEGENTROPY_BUILDDIR := $(CURDIR)/vendor/negentropy/cpp
+LIBNEGENTROPY_FILE := libnegentropy.a
+
+deps: | negentropy
+
+clean: | negentropy-clean
+
+$(LIBNEGENTROPY_FILE):
+	$(MAKE) -C $(LIBNEGENTROPY_BUILDDIR) && \
+		cp $(LIBNEGENTROPY_BUILDDIR)/${LIBNEGENTROPY_FILE} ${LIBNEGENTROPY_FILE}
+
+negentropy: | $(LIBNEGENTROPY_FILE)
+    ## Pass libnegentropy and it's deps to linker.
+    $(eval LIBNEGENTROPY_PATH := $(shell if [ -f "$(LIBNEGENTROPY_FILE)" ]; then echo "$(LIBNEGENTROPY_FILE)"; else echo "./$(LIBNEGENTROPY_FILE)"; fi))
+    $(eval NIM_PARAMS += --passL:$(LIBNEGENTROPY_PATH) --passL:-lcrypto --passL:-lssl --passL:-lstdc++)
+ifeq ($(detected_OS),Darwin)
+    $(eval NIM_PARAMS += --passL:-L/opt/homebrew/lib/)
+endif
+
+negentropy-clean:
+	$(MAKE) -C $(LIBNEGENTROPY_BUILDDIR) clean && \
+		rm ${LIBNEGENTROPY_FILE}
+
 
 #################
 ## Waku Common ##
@@ -241,6 +270,36 @@ networkmonitor: | build deps librln
 	echo -e $(BUILD_MSG) "build/$@" && \
 		$(ENV_SCRIPT) nim networkmonitor $(NIM_PARAMS) waku.nims
 
+############
+## Format ##
+############
+.PHONY: build-nph install-nph clean-nph print-nph-path
+
+build-nph:
+		$(ENV_SCRIPT) nim c vendor/nph/src/nph.nim
+
+GIT_PRE_COMMIT_HOOK := .git/hooks/pre-commit
+
+install-nph: build-nph
+ifeq ("$(wildcard $(GIT_PRE_COMMIT_HOOK))","")
+	cp ./scripts/git_pre_commit_format.sh $(GIT_PRE_COMMIT_HOOK)
+else
+	echo "$(GIT_PRE_COMMIT_HOOK) already present, will NOT override"
+	exit 1
+endif
+
+nph/%: build-nph
+	echo -e $(FORMAT_MSG) "nph/$*" && \
+		$(NPH) $*
+
+clean-nph:
+	rm -f $(NPH)
+
+# To avoid hardcoding nph binary location in several places
+print-nph-path:
+	echo "$(NPH)"
+
+clean: | clean-nph
 
 ###################
 ## Documentation ##
@@ -373,6 +432,7 @@ cwaku_example: | build libwaku
 		./examples/cbindings/base64.c \
 		-lwaku -Lbuild/ \
 		-pthread -ldl -lm \
+		-lnegentropy -Lvendor/negentropy/cpp/ \
 		-lminiupnpc -Lvendor/nim-nat-traversal/vendor/miniupnp/miniupnpc/build/ \
 		-lnatpmp -Lvendor/nim-nat-traversal/vendor/libnatpmp-upstream/ \
 		vendor/nim-libbacktrace/libbacktrace_wrapper.o \
@@ -385,6 +445,7 @@ cppwaku_example: | build libwaku
 		./examples/cpp/base64.cpp \
 		-lwaku -Lbuild/ \
 		-pthread -ldl -lm \
+		-lnegentropy -Lvendor/negentropy/cpp/ \
 		-lminiupnpc -Lvendor/nim-nat-traversal/vendor/miniupnp/miniupnpc/build/ \
 		-lnatpmp -Lvendor/nim-nat-traversal/vendor/libnatpmp-upstream/ \
 		vendor/nim-libbacktrace/libbacktrace_wrapper.o \
@@ -411,3 +472,4 @@ release-notes:
 			sed -E 's@#([0-9]+)@[#\1](https://github.com/waku-org/nwaku/issues/\1)@g'
 # I could not get the tool to replace issue ids with links, so using sed for now,
 # asked here: https://github.com/bvieira/sv4git/discussions/101
+

README.md

@@ -9,7 +9,7 @@ The nwaku repository implements Waku, and provides tools related to it.
 - Examples of Waku usage.
 - Various tests of above.
 
-For more details see the [source code](waku/v2/README.md)
+For more details see the [source code](waku/README.md)
 
 ## How to Build & Run
 
@@ -53,7 +53,8 @@ If you encounter difficulties building the project on WSL, consider placing the
 #### Nim Runtime
 This repository is bundled with a Nim runtime that includes the necessary dependencies for the project.
 
-Before you can utilise the runtime you'll need to build the project, as detailed in a previous section. This will generate a `vendor` directory containing various dependencies, including the `nimbus-build-system` which has the bundled nim runtime.
+Before you can utilise the runtime you'll need to build the project, as detailed in a previous section.
+This will generate a `vendor` directory containing various dependencies, including the `nimbus-build-system` which has the bundled nim runtime.
 
 After successfully building the project, you may bring the bundled runtime into scope by running:
 ```bash
@@ -84,6 +85,24 @@ Binary will be created as `<path to your test file.nim>.bin` under the `build` d
 make test/tests/common/test_enr_builder.nim
 ```
 
+## Formatting
+
+Nim files are expected to be formatted using the [`nph`](https://github.com/arnetheduck/nph) version present in `vendor/nph`.
+
+You can easily format file with the `make nph/<relative path to nim> file` command.
+For example:
+
+```
+make nph/waku/waku_core.nim
+```
+
+A convenient git hook is provided to automatically format file at commit time.
+Run the following command to install it:
+
+```shell
+make install-nph
+```
+
 ### Examples
 
 Examples can be found in the examples folder.

apps/chat2/chat2.nim

@@ -6,7 +6,7 @@ when not (compileOption("threads")):
 
 {.push raises: [].}
 
-import std/[strformat, strutils, times, options, random]
+import std/[strformat, strutils, times, options, random, sequtils]
 import
   confutils,
   chronicles,
@@ -379,7 +379,9 @@ proc processInput(rfd: AsyncFD, rng: ref HmacDrbgContext) {.async.} =
     raise newException(ConfigurationError, "rln-relay-cred-path MUST be passed")
 
   if conf.relay:
-    await node.mountRelay(conf.topics.split(" "))
+    let shards =
+      conf.shards.mapIt(RelayShard(clusterId: conf.clusterId, shardId: uint16(it)))
+    await node.mountRelay(shards)
 
   await node.mountLibp2pPing()