From 6d6ed7aead7b89eddb58394442d2120b729a75f2 Mon Sep 17 00:00:00 2001 From: sudeep Date: Thu, 14 Dec 2023 21:20:42 +0530 Subject: [PATCH 1/9] setup documentation for w3up --- README.md | 71 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 70 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 73a9132..f6af1c0 100644 --- a/README.md +++ b/README.md @@ -8,6 +8,13 @@ - [Upload a file](#upload-a-file) - [Download content (given cid)](#download-content-given-cid) - [Find the cid given some content](#find-the-cid-given-some-content) + - [migration to UCAN and capabilities setup](#migration-to-ucan-and-capabilities-setup) + - [setting up w3cli](#setting-up-w3cli) + - [installation](#installation) + - [login and check spaces](#login-and-check-spaces) + - [generate ucan key](#generate-ucan-key) + - [create delegation to store/add and upload/add](#create-delegation-to-storeadd-and-uploadadd) + - [communicate to the operator](#communicate-to-the-operator) - [Running ipfs-pinner server with docker](#running-ipfs-pinner-server-with-docker) - [Docker Volume setup](#docker-volume-setup) - [Port mapping setup](#port-mapping-setup) @@ -24,7 +31,7 @@ ## Introduction - A wrapper on top of ipfs node, utilising go-ipfs as a library. -- Extended support for custom file upload endpoints provided by pinata & web3.storage. +- Extended support for custom file upload endpoints provided by web3.storage. - Content archive file generation and lightweight deterministic CID generation on client side (using CARs). - It can be used as a go library (see `binary/main.go` for usage) or as a http server. @@ -107,6 +114,68 @@ There's a timeout (check code for value) for the download request, if it doesn't {"cid": "bafkreicszve3ewhhrgobm366mdctki2m2qwzide5e54zh5aifnesg3ofne"}% ``` + +## migration to UCAN and capabilities setup +- web3.storage is sunsetting its custom upload endpoint (on 9th January, 2024), and we need to migrate from using that to w3up. +- w3up uses UCAN which is a capabilities-based authorization system (learn more [here](https://web3.storage/docs/concepts/ucans-and-web3storage/)). +- In this setup, the "central account" (owned by Covalent) sets up a "space" (think namespace for data). The central account (controlled by the email) is delegated the capabilty to operate on this space. +- among other capabilties, the central account can delegate certain capabilities (like uploading to space) to other **agents**. This has to be done at our end, and scripts will be made available for it in this repo. +- once an agent is granted the capability, we share the credentials with the operators, who run ipfs-pinner with it, and can then upload or fetch. + + +### setting up w3cli + +- Create a web3.storage account in the [console](https://console.web3.storage/). +- Create a space which you want to use to upload artifacts. We want to use different spaces for different artifacts to keep a clear separation. + +We'll use w3cli to login and create a new space and register. + +#### installation +```bash +➜ npm install -g @web3-storage/w3cli + +➜ w3 --version +w3, 7.0.3 +``` + +#### login and check spaces +```bash +➜ w3 login sudeep@covalenthq.com + +➜ w3 space ls + did:key:z6MkgSK6VEu3bvrAFtYNyjsnzG7dVXzYi3yT5TasEgeaQrCe mock_artifacts + +➜ w3 space use did:key:z6MkgSK6VEu3bvrAFtYNyjsnzG7dVXzYi3yT5TasEgeaQrCe +did:key:z6MkgSK6VEu3bvrAFtYNyjsnzG7dVXzYi3yT5TasEgeaQrCe +``` + +The did key is the identifier for this space. Now let's generate some DIDs for an operator and delegate upload capabilities to it. + +#### generate ucan key +```bash +➜ npx ucan-key ed --json +{ + "did": "did:key:z6MkpzWw1fDZYMpESgVKFAT87SZAuHiCQZVBC3hmQjB18Nzj", + "key": "MgCbc48J8n+BMdzA4XxwYOaKmdu5Ov34jE71U8vV07IVIjO0BnJa05mNMcB8GSz1lib014QAhvAxorG6zACrstm6PBGA=" +} +``` + +#### create delegation to store/add and upload/add + +```bash +➜ w3 delegation create -c 'store/add' -c 'upload/add' did:key:z6MkpzWw1fDZYMpESgVKFAT87SZAuHiCQZVBC3hmQjB18Nzj | base64 +``` + + +Copy the output. This is the delegation string. + +#### communicate to the operator + +Provide the operator with the `did`, `key` and `delegation` string. These will be passed to operator's setup of the +ipfs-pinner, which can then make the delegations. + + + ## Running ipfs-pinner server with docker We can also run the ipfs-pinner server via docker. From a1232fbf6d2f8ea1f99bcd25b97edefa4f6c9eaa Mon Sep 17 00:00:00 2001 From: sudeep Date: Thu, 4 Jan 2024 15:54:27 +0530 Subject: [PATCH 2/9] use w3cli for making uploads - apart from upload using `w3 up`, there is some setup required. - the agent needs to be setup by the help of `w3 whoami` - The space needs to be setup by importing the delegation proof --- go.mod | 1 + go.sum | 14 +++++ pinclient/client_create_request.go | 24 ++++++-- pinclient/pinclient.go | 50 ++++++++------- pinner.go | 26 ++++++-- server/main.go | 79 +++++++++++++----------- w3up/w3up.go | 98 ++++++++++++++++++++++++++++++ 7 files changed, 220 insertions(+), 72 deletions(-) create mode 100644 w3up/w3up.go diff --git a/go.mod b/go.mod index 238c355..9be8dab 100644 --- a/go.mod +++ b/go.mod @@ -160,6 +160,7 @@ require ( github.com/spaolacci/murmur3 v1.1.0 // indirect github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7 // indirect github.com/ucarion/urlpath v0.0.0-20200424170820-7ccc79b76bbb // indirect + github.com/web3-storage/go-ucanto v0.1.0 // indirect github.com/whyrusleeping/base32 v0.0.0-20170828182744-c30ac30633cc // indirect github.com/whyrusleeping/cbor v0.0.0-20171005072247-63513f603b11 // indirect github.com/whyrusleeping/cbor-gen v0.0.0-20230818171029-f91ae536ca25 // indirect diff --git a/go.sum b/go.sum index 57b133e..dbd2ccf 100644 --- a/go.sum +++ b/go.sum @@ -47,6 +47,7 @@ github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03 github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo= github.com/Jorropo/jsync v1.0.1 h1:6HgRolFZnsdfzRUj+ImB9og1JYOxQoReSywkHOGSaUU= github.com/Jorropo/jsync v1.0.1/go.mod h1:jCOZj3vrBCri3bSU3ErUYvevKlnbssrXeCivybS5ABQ= +github.com/Kubuxu/go-os-helper v0.0.1/go.mod h1:N8B+I7vPCT80IcP58r50u4+gEEcsZETFUpAzWW2ep1Y= github.com/OneOfOne/xxhash v1.2.2 h1:KMrpdQIwFcEqXDklaen+P1axHaj9BSKzvpUUfnHldSE= github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU= github.com/aead/siphash v1.0.1/go.mod h1:Nywa3cDsYNNK3gaciGTWPwHt0wlpNV15vwmswBAUSII= @@ -87,6 +88,7 @@ github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghf github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44= github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= +github.com/cheekybits/genny v1.0.0/go.mod h1:+tQajlRqAUrPI7DOSpB0XAqZYtQakVtB7wXkRAgjxjQ= github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI= github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI= github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU= @@ -140,6 +142,8 @@ github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDD github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk= github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY= github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto= +github.com/eapache/go-xerial-snappy v0.0.0-20180814174437-776d5712da21/go.mod h1:+020luEh2TKB4/GOp8oxxtq0Daoen/Cii55CzbTV6DU= +github.com/eapache/queue v1.1.0/go.mod h1:6eCeP0CKFpHLu8blIFXhExK/dRa7WDZfr6jVFPTqq+I= github.com/elastic/gosigar v0.12.0/go.mod h1:iXRIGg2tLnu7LBdpqzyQfGDEidKCfWcCMS0WKyPWoMs= github.com/elastic/gosigar v0.14.2 h1:Dg80n8cr90OZ7x+bAax/QjoW/XqTI11RmA79ZwIm9/4= github.com/elastic/gosigar v0.14.2/go.mod h1:iXRIGg2tLnu7LBdpqzyQfGDEidKCfWcCMS0WKyPWoMs= @@ -172,6 +176,7 @@ github.com/go-errors/errors v1.0.1/go.mod h1:f4zRHt4oKfwPJE5k8C9vpYG+aDHdBFUsgrm github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU= github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8= github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8= +github.com/go-logfmt/logfmt v0.5.1/go.mod h1:WYhtIu8zTZfxdn5+rREduYbwxfcBr/Vr6KEVveWlfTs= github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= github.com/go-logr/logr v1.2.4 h1:g01GSCwiDw2xSZfjJ2/T9M+S6pFdcNtFYsp+Y43HYDQ= github.com/go-logr/logr v1.2.4/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= @@ -344,6 +349,7 @@ github.com/ipfs/go-graphsync v0.14.4/go.mod h1:yT0AfjFgicOoWdAlUJ96tQ5AkuGI4r1ta github.com/ipfs/go-ipfs-blockstore v1.3.0 h1:m2EXaWgwTzAfsmt5UdJ7Is6l4gJcaM/A12XwJyvYvMM= github.com/ipfs/go-ipfs-blockstore v1.3.0/go.mod h1:KgtZyc9fq+P2xJUiCAzbRdhhqJHvsw8u2Dlqy2MyRTE= github.com/ipfs/go-ipfs-blocksutil v0.0.1 h1:Eh/H4pc1hsvhzsQoMEP3Bke/aW5P5rVM1IWFJMcGIPQ= +github.com/ipfs/go-ipfs-blocksutil v0.0.1/go.mod h1:Yq4M86uIOmxmGPUHv/uI7uKqZNtLb449gwKqXjIsnRk= github.com/ipfs/go-ipfs-chunker v0.0.5 h1:ojCf7HV/m+uS2vhUGWcogIIxiO5ubl5O57Q7NapWLY8= github.com/ipfs/go-ipfs-delay v0.0.0-20181109222059-70721b86a9a8/go.mod h1:8SP1YXK1M1kXuc4KJZINY3TQQ03J2rwBG9QfXmbRPrw= github.com/ipfs/go-ipfs-delay v0.0.1 h1:r/UXYyRcddO6thwOnhiznIAiSvxMECGgtv35Xs1IeRQ= @@ -422,12 +428,15 @@ github.com/jbenet/goprocess v0.1.4/go.mod h1:5yspPrukOVuOLORacaBi858NqyClJPQxYZl github.com/jellevandenhooff/dkim v0.0.0-20150330215556-f50fe3d243e1/go.mod h1:E0B/fFc00Y+Rasa88328GlI/XbtyysCtTHZS8h7IrBU= github.com/jessevdk/go-flags v0.0.0-20141203071132-1679536dcc89/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI= github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI= +github.com/jpillora/backoff v1.0.0/go.mod h1:J/6gKK9jxlEcS3zixgDgUAsiuZ7yrSoa/FX5e0EB2j4= github.com/jrick/logrotate v1.0.0/go.mod h1:LNinyqDIJnpAur+b8yyulnQw/wDuN1+BYKlTRt3OuAQ= github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU= +github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU= github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk= github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo= github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU= +github.com/julienschmidt/httprouter v1.3.0/go.mod h1:JR6WtHb+2LUe8TCKY3cZOxFyyO8IZAc4RVcycCCAKdM= github.com/kami-zh/go-capturer v0.0.0-20171211120116-e492ea43421d/go.mod h1:P2viExyCEfeWGU259JnaQ34Inuec4R38JCyBx2edgD0= github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q= github.com/kisielk/errcheck v1.2.0/go.mod h1:/BMXB+zMLi60iA8Vv6Ksmxu/1UDYcXs4uQLJ+jE2L00= @@ -546,6 +555,7 @@ github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrk github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= +github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= github.com/mr-tron/base58 v1.1.0/go.mod h1:xcD2VGqlgYjBdcBLw+TuYLr8afG+Hj8g2eTVqeSzSU8= github.com/mr-tron/base58 v1.1.2/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc= github.com/mr-tron/base58 v1.1.3/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc= @@ -591,6 +601,7 @@ github.com/multiformats/go-varint v0.0.5/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXS github.com/multiformats/go-varint v0.0.6/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE= github.com/multiformats/go-varint v0.0.7 h1:sWSGR+f/eu5ABZA2ZpYKBILXTTs9JWpdEM/nEGOHFS8= github.com/multiformats/go-varint v0.0.7/go.mod h1:r8PUYw/fD/SjBCiKOoDlGF6QawOELpZAu9eioSos/OU= +github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U= github.com/neelance/astrewrite v0.0.0-20160511093645-99348263ae86/go.mod h1:kHJEU3ofeGjhHklVoIGuVj85JJwZ6kWPaJwCIxgnFmo= github.com/neelance/sourcemap v0.0.0-20151028013722-8c68805598ab/go.mod h1:Qr6/a/Q4r9LP1IltGz7tA7iOK1WonHEYhu1HRBA7ZiM= github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A= @@ -688,6 +699,7 @@ github.com/shurcooL/users v0.0.0-20180125191416-49c67e49c537/go.mod h1:QJTqeLYED github.com/shurcooL/webdavfs v0.0.0-20170829043945-18c3829fa133/go.mod h1:hKmq5kWdCj2z2KEozexVbfEZIWiTjhE0+UjmZgPqehw= github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo= github.com/sirupsen/logrus v1.7.0/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0= +github.com/sirupsen/logrus v1.8.1/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0= github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc= github.com/smartystreets/assertions v1.2.0 h1:42S6lae5dvLc7BrLu/0ugRtcFVjoJNMC/N3yZFZkDFs= github.com/smartystreets/assertions v1.2.0/go.mod h1:tcbTF8ujkAEcZ8TElKY+i30BzYlVhC/LOxJk7iOWnoo= @@ -745,6 +757,8 @@ github.com/warpfork/go-testmark v0.12.1 h1:rMgCpJfwy1sJ50x0M0NgyphxYYPMOODIJHhsX github.com/warpfork/go-wish v0.0.0-20200122115046-b9ea61034e4a/go.mod h1:x6AKhvSSexNrVSrViXSHUEbICjmGXhtgABaHIySUSGw= github.com/warpfork/go-wish v0.0.0-20220906213052-39a1cc7a02d0 h1:GDDkbFiaK8jsSDJfjId/PEGEShv6ugrt4kYsC5UIDaQ= github.com/warpfork/go-wish v0.0.0-20220906213052-39a1cc7a02d0/go.mod h1:x6AKhvSSexNrVSrViXSHUEbICjmGXhtgABaHIySUSGw= +github.com/web3-storage/go-ucanto v0.1.0 h1:Hg6jO7OLLeDLtmseQZWQGBFJMp+5t5OWAFVL5Y3LsOs= +github.com/web3-storage/go-ucanto v0.1.0/go.mod h1:XD6zahQ8HLh8Z2CSK7xYcTR0A7oCVYXTZB7fFtYqGF8= github.com/whyrusleeping/base32 v0.0.0-20170828182744-c30ac30633cc h1:BCPnHtcboadS0DvysUuJXZ4lWVv5Bh5i7+tbIyi+ck4= github.com/whyrusleeping/base32 v0.0.0-20170828182744-c30ac30633cc/go.mod h1:r45hJU7yEoA81k6MWNhpMj/kms0n14dkzkxYHoB96UM= github.com/whyrusleeping/cbor v0.0.0-20171005072247-63513f603b11 h1:5HZfQkwe0mIfyDmc1Em5GqlNRzcdtlv4HTNmdpt7XH0= diff --git a/pinclient/client_create_request.go b/pinclient/client_create_request.go index 1f9e5ca..bb3d041 100644 --- a/pinclient/client_create_request.go +++ b/pinclient/client_create_request.go @@ -4,14 +4,20 @@ import ( "net/http" "github.com/covalenthq/ipfs-pinner/core" + "github.com/web3-storage/go-ucanto/did" ) type ClientCreateRequest struct { ps core.PinningService pinningServiceBaseUrl string filePinBaseUrl string - bearerToken string - httpClient *http.Client + //bearerToken string + + W3_AgentKey string + W3_AgentDid did.DID + W3_DelegationProofPath string + + httpClient *http.Client } func NewClientRequest(ps core.PinningService) ClientCreateRequest { @@ -21,8 +27,18 @@ func NewClientRequest(ps core.PinningService) ClientCreateRequest { return request } -func (r ClientCreateRequest) BearerToken(token string) ClientCreateRequest { - r.bearerToken = token +func (r ClientCreateRequest) W3AgentKey(key string) ClientCreateRequest { + r.W3_AgentKey = key + return r +} + +func (r ClientCreateRequest) W3AgentDid(did did.DID) ClientCreateRequest { + r.W3_AgentDid = did + return r +} + +func (r ClientCreateRequest) DelegationProofPath(proofPath string) ClientCreateRequest { + r.W3_DelegationProofPath = proofPath return r } diff --git a/pinclient/pinclient.go b/pinclient/pinclient.go index e48cf6c..574fa89 100644 --- a/pinclient/pinclient.go +++ b/pinclient/pinclient.go @@ -12,6 +12,7 @@ import ( core "github.com/covalenthq/ipfs-pinner/core" ihttp "github.com/covalenthq/ipfs-pinner/http" "github.com/covalenthq/ipfs-pinner/openapi" + "github.com/covalenthq/ipfs-pinner/w3up" "github.com/ipfs/go-cid" "github.com/multiformats/go-multibase" "github.com/pkg/errors" @@ -27,17 +28,18 @@ type Client struct { client *openapi.APIClient ps core.PinningService cidVersion int + w3up *w3up.W3up } -func NewClient(request ClientCreateRequest, cidVersion int) PinServiceAPI { +func NewClient(request ClientCreateRequest, cidVersion int, w3up *w3up.W3up) PinServiceAPI { // assuming we are getting a supported pinning service request config := openapi.NewConfiguration() if request.httpClient == nil { request.httpClient = ihttp.NewHttpClient(nil) } config.UserAgent = UserAgent - bearer := fmt.Sprintf("Bearer %s", request.bearerToken) - config.AddDefaultHeader("Authorization", bearer) + //bearer := fmt.Sprintf("Bearer %s", request.bearerToken) + //config.AddDefaultHeader("Authorization", bearer) config.Servers = openapi.ServerConfigurations{ openapi.ServerConfiguration{ URL: request.pinningServiceBaseUrl, @@ -50,7 +52,7 @@ func NewClient(request ClientCreateRequest, cidVersion int) PinServiceAPI { } config.HTTPClient = request.httpClient - return &Client{client: openapi.NewAPIClient(config), ps: request.ps, cidVersion: cidVersion} + return &Client{client: openapi.NewAPIClient(config), ps: request.ps, cidVersion: cidVersion, w3up: w3up} } func (c *Client) IsIPFSSupportedFor(ps core.PinningService) bool { @@ -90,24 +92,12 @@ func (c *Client) Add(ctx context.Context, cid cid.Cid, opts ...AddOption) (core. } func (c *Client) UploadFile(ctx context.Context, file *os.File) (cid.Cid, error) { - var err error - var fcid core.CidGetter - switch c.ps { - case core.Pinata: - fcid, err = c.uploadFileViaPinata(ctx, file) - - case core.Web3Storage: - fcid, err = c.uploadFileViaWeb3Storage(ctx, file) - - default: - logger.Fatalf("only pinata supported for file upload") - } - + fcid, err := c.uploadFileViaWeb3Storage(ctx, file) if err != nil { return cid.Undef, fmt.Errorf("%w", err) } - return fcid.GetCid(), nil + return fcid, nil } func (c *Client) ServiceType() core.PinningService { @@ -154,13 +144,21 @@ func (c *Client) uploadFileViaPinata(ctx context.Context, file *os.File) (core.P return core.NewPinataResponseGetter(*result), nil } -func (c *Client) uploadFileViaWeb3Storage(ctx context.Context, file *os.File) (core.Web3StorageResponseGetter, error) { - poster := c.client.FilepinApi.Web3StorageCarUpload(ctx) - result, httpresp, err := poster.Body(file).Execute() - if err != nil { - err := httperr(httpresp, err) - return nil, err - } +func (c *Client) uploadFileViaWeb3Storage(ctx context.Context, file *os.File) (cid.Cid, error) { + return c.w3up.UploadCarFile(file) + + // if err != nil { + // return nil, err + // } + + // return + + // poster := c.client.FilepinApi.Web3StorageCarUpload(ctx) + // result, httpresp, err := poster.Body(file).Execute() + // if err != nil { + // err := httperr(httpresp, err) + // return nil, err + // } - return core.NewWeb3StorageResponseGetter(*result), nil + // return core.NewWeb3StorageResponseGetter(*result), nil } diff --git a/pinner.go b/pinner.go index 637a7b7..ae514d4 100644 --- a/pinner.go +++ b/pinner.go @@ -5,16 +5,16 @@ package pinner import ( + "log" + car "github.com/covalenthq/ipfs-pinner/car" "github.com/covalenthq/ipfs-pinner/core" "github.com/covalenthq/ipfs-pinner/coreapi" "github.com/covalenthq/ipfs-pinner/dag" "github.com/covalenthq/ipfs-pinner/pinclient" - logging "github.com/ipfs/go-log/v2" + "github.com/covalenthq/ipfs-pinner/w3up" ) -var logger = logging.Logger("ipfs-pinner") - type pinnerNode struct { ipfsCore coreapi.CoreExtensionAPI carExporter car.CarExporterAPI @@ -26,12 +26,28 @@ func NewPinnerNode(req PinnerNodeCreateRequest) PinnerNode { node := pinnerNode{} ipfsNode, err := core.CreateIpfsNode(req.cidComputeOnly) if err != nil { - logger.Fatal("error initializing ipfs node: ", err) + log.Fatal("error initializing ipfs node: ", err) } node.ipfsCore = coreapi.NewCoreExtensionApi(ipfsNode) - node.pinApiClient = pinclient.NewClient(req.pinServiceRequest, req.cidVersion) + //SETUP W3UP + log.Print("setting up w3up for uploads....") + w3up := w3up.NewW3up(req.pinServiceRequest.W3_AgentKey, req.pinServiceRequest.W3_AgentDid, req.pinServiceRequest.W3_DelegationProofPath) + agentDid, err := w3up.WhoAmI() + if err != nil { + log.Fatal("error getting agent did: ", err) + } + log.Printf("w3up agent did: %s\n", agentDid.String()) + + spaceDid, err := w3up.SpaceAdd() + if err != nil { + log.Fatal("error adding space: ", err) + } + log.Printf("w3up space did: %s\n", spaceDid.String()) + log.Print("w3up setup complete") + + node.pinApiClient = pinclient.NewClient(req.pinServiceRequest, req.cidVersion, w3up) node.carExporter = car.NewCarExporter(node.ipfsCore) node.unixfsService = dag.NewUnixfsAPI(node.ipfsCore, req.cidVersion, req.cidComputeOnly, req.ipfsFetchUrls) diff --git a/server/main.go b/server/main.go index a895ce6..766a363 100644 --- a/server/main.go +++ b/server/main.go @@ -9,7 +9,6 @@ import ( "io" "log" "net/http" - "net/url" "os" "os/signal" "strconv" @@ -22,6 +21,8 @@ import ( pinner "github.com/covalenthq/ipfs-pinner" "github.com/covalenthq/ipfs-pinner/core" client "github.com/covalenthq/ipfs-pinner/pinclient" + "github.com/web3-storage/go-ucanto/did" + "github.com/web3-storage/go-ucanto/principal/ed25519/signer" ) const ( @@ -38,53 +39,62 @@ func NewState() *State { return &State{status: OK} } +type Config struct { + portNumber int + w3AgentKey string + w3AgentDid did.DID + delegationProofPath string + ipfsGatewayUrls []string +} + var ( - emptyBytes = []byte("") - WEB3_JWT = "WEB3_JWT" + emptyBytes = []byte("") + DOWNLOAD_TIMEOUT = 12 * time.Minute // download can take a lot of time if it's not locally present UPLOAD_TIMEOUT = 60 * time.Second // uploads of around 6MB files happen in less than 10s typically ) func main() { portNumber := flag.Int("port", 3001, "port number for the server") - token := flag.String("jwt", "", "JWT token for web3.storage") + + w3AgentKey := flag.String("w3-agent-key", "", "w3 agent key") + w3DelegationFile := flag.String("w3-delegation-file", "", "w3 delegation file") + ipfsGatewayUrls := flag.String("ipfs-gateway-urls", "https://w3s.link/ipfs/%s,https://dweb.link/ipfs/%s,https://ipfs.io/ipfs/%s", "comma separated list of ipfs gateway urls") flag.Parse() core.Version() - setUpAndRunServer(*portNumber, *token, *ipfsGatewayUrls) -} -func setUpAndRunServer(portNumber int, token, ipfsGatewayUrls string) { - mux := http.NewServeMux() - httpState := NewState() - if token == "" { - var present bool - token, present = os.LookupEnv(WEB3_JWT) - if !present { - log.Fatalf("token (%s) not found in env", WEB3_JWT) - } + if *w3AgentKey == "" { + log.Fatalf("w3 agent key is required") } - var ipfsGatewayUrlArr []string - if ipfsGatewayUrls != "" { - ipfsGatewayUrlArr = strings.Split(ipfsGatewayUrls, ",") - for _, ipfsUrlStr := range ipfsGatewayUrlArr { - if !strings.Contains(ipfsUrlStr, "%s") { - log.Fatalf("ipfs gateway url %s does not contain %%s", ipfsUrlStr) - } + if *w3DelegationFile == "" { + log.Fatalf("w3 delegation file is required") + } - if _, err := url.Parse(fmt.Sprintf(ipfsUrlStr, "sample_cid")); err != nil { - log.Fatalf("ipfs gateway url %s is not a valid url", ipfsUrlStr) - } - } - } else { - log.Fatalf("ipfs gateway urls not found in params") + _, err := os.ReadFile(*w3DelegationFile) + if err != nil { + log.Fatalf("error reading delegation proof file: %w", err) } - clientCreateReq := client.NewClientRequest(core.Web3Storage).BearerToken(token) + agentSigner, err := signer.Parse(*w3AgentKey) + if err != nil { + log.Fatalf("error parsing agent signer: %w", err) + } + + log.Printf("agent did: %s", agentSigner.DID().DID().String()) + + setUpAndRunServer(Config{*portNumber, *w3AgentKey, agentSigner.DID().DID(), *w3DelegationFile, strings.Split(*ipfsGatewayUrls, ",")}) +} + +func setUpAndRunServer(config Config) { + mux := http.NewServeMux() + httpState := NewState() + + clientCreateReq := client.NewClientRequest(core.Web3Storage).W3AgentKey(config.w3AgentKey).W3AgentDid(config.w3AgentDid).DelegationProofPath(config.delegationProofPath) // check if cid compute true works with car uploads - nodeCreateReq := pinner.NewNodeRequest(clientCreateReq, ipfsGatewayUrlArr).CidVersion(1).CidComputeOnly(false) + nodeCreateReq := pinner.NewNodeRequest(clientCreateReq, config.ipfsGatewayUrls).CidVersion(1).CidComputeOnly(false) node := pinner.NewPinnerNode(*nodeCreateReq) mux.Handle("/upload", recoveryWrapper(uploadHttpHandler(node))) @@ -97,7 +107,7 @@ func setUpAndRunServer(portNumber int, token, ipfsGatewayUrls string) { log.Print("Listening...") serverStopCommandSetup() - err := http.ListenAndServe(":"+strconv.Itoa(portNumber), mux) + err := http.ListenAndServe(":"+strconv.Itoa(config.portNumber), mux) if err != nil { log.Println("error listening and serving on TCP network: %w", err) } @@ -280,12 +290,7 @@ func uploadHandler(contents string, node pinner.PinnerNode) (cid.Cid, error) { } defer carf.Close() // should delete the file due to unlink - - err = syscall.Unlink(carf.Name()) - if err != nil { - log.Printf("%v", err) - return cid.Undef, err - } + defer syscall.Unlink(carf.Name()) log.Printf("car file location: %s\n", carf.Name()) diff --git a/w3up/w3up.go b/w3up/w3up.go new file mode 100644 index 0000000..15a3382 --- /dev/null +++ b/w3up/w3up.go @@ -0,0 +1,98 @@ +package w3up + +import ( + "encoding/json" + "fmt" + "log" + "os" + "os/exec" + "strings" + + "github.com/ipfs/go-cid" + "github.com/web3-storage/go-ucanto/did" +) + +type W3up struct { + principal string + agentDid did.DID + delegationProofPath string + w3up_dir string +} + +func NewW3up(principal string, did did.DID, delegationProofPath string) *W3up { + tmpDir, err := os.MkdirTemp("", "w3up_config") + if err != nil { + log.Fatal("error creating temp dir for w3up config: ", err) + } + return &W3up{principal: principal, agentDid: did, delegationProofPath: delegationProofPath, w3up_dir: tmpDir} +} + +func (w3up *W3up) WhoAmI() (did.DID, error) { + command := fmt.Sprintf("W3_STORE_NAME=%s W3_PRINCIPAL=\"%s\" w3 whoami", w3up.w3up_dir, w3up.principal) + cmd := exec.Command("bash", "-c", command) + output, err := cmd.CombinedOutput() + if err != nil { + log.Fatal("error initializing w3up: ", string(output)) + return did.DID{}, err + } + + pdid, err := did.Parse(strings.TrimSpace(string(output))) + if err != nil { + log.Fatal("error parsing did: ", err) + return did.DID{}, err + } + return pdid, nil +} + +func (w3up *W3up) SpaceAdd() (did.DID, error) { + command := fmt.Sprintf("W3_STORE_NAME=%s w3 space add %s", w3up.w3up_dir, w3up.delegationProofPath) + cmd := exec.Command("bash", "-c", command) + output, err := cmd.CombinedOutput() + if err != nil { + log.Fatal("error adding w3up space: ", string(output)) + return did.DID{}, err + } + + pdid, err := did.Parse(strings.TrimSpace(string(output))) + if err != nil { + log.Fatal("error parsing did: ", err) + return did.DID{}, err + } + + return pdid, nil +} + +func (w3up *W3up) UploadCarFile(carFile *os.File) (cid.Cid, error) { + command := fmt.Sprintf("W3_STORE_NAME=%s w3 up %s --json --no-wrap --car", w3up.w3up_dir, carFile.Name()) + cmd := exec.Command("bash", "-c", command) + output, err := cmd.CombinedOutput() + if err != nil { + log.Fatal("error uploading car file: ", string(output)) + return cid.Undef, err + } + + jsons := strings.TrimSpace(string(output)) + + log.Printf("w3 up output: %s\n", jsons) + + var result map[string]interface{} + err = json.Unmarshal([]byte(jsons), &result) + if err != nil { + log.Fatal("error parsing json: ", err) + return cid.Undef, err + } + + root, ok := result["root"].(map[string]interface{}) + if !ok { + log.Fatal("error type asserting root") + return cid.Undef, fmt.Errorf("error type asserting root") + } + + rcid, ok := root["/"].(string) + if !ok { + log.Fatal("error type asserting cid") + return cid.Undef, fmt.Errorf("error type asserting cid") + } + + return cid.Parse(rcid) +} From 49d05246bb80cd9927e1c93683dc8882634ed1b4 Mon Sep 17 00:00:00 2001 From: sudeep Date: Thu, 4 Jan 2024 15:59:32 +0530 Subject: [PATCH 3/9] update documentation for working witj w3cli based ipfs-pinner --- README.md | 33 +++++++++++++++++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index f6af1c0..5373829 100644 --- a/README.md +++ b/README.md @@ -15,6 +15,7 @@ - [generate ucan key](#generate-ucan-key) - [create delegation to store/add and upload/add](#create-delegation-to-storeadd-and-uploadadd) - [communicate to the operator](#communicate-to-the-operator) + - [operator invocation](#operator-invocation) - [Running ipfs-pinner server with docker](#running-ipfs-pinner-server-with-docker) - [Docker Volume setup](#docker-volume-setup) - [Port mapping setup](#port-mapping-setup) @@ -163,7 +164,7 @@ The did key is the identifier for this space. Now let's generate some DIDs for a #### create delegation to store/add and upload/add ```bash -➜ w3 delegation create -c 'store/add' -c 'upload/add' did:key:z6MkpzWw1fDZYMpESgVKFAT87SZAuHiCQZVBC3hmQjB18Nzj | base64 +➜ w3 delegation create -c 'store/add' -c 'upload/add' did:key:z6MkpzWw1fDZYMpESgVKFAT87SZAuHiCQZVBC3hmQjB18Nzj -o proof.out ``` @@ -171,10 +172,38 @@ Copy the output. This is the delegation string. #### communicate to the operator -Provide the operator with the `did`, `key` and `delegation` string. These will be passed to operator's setup of the +Provide the operator with the `did`, `key` string + `proof.out` file. These will be passed to operator's setup of the ipfs-pinner, which can then make the delegations. +#### operator invocation + +the operator can pass the `key` for `-w3-agent-key` and proof file in `-w3-delegation-file` flag. + +```bash +go run server/main.go -w3-agent-key -w3-delegation-file ./proof.out +ipfs-pinner +ipfs-pinner Version: 0.1.16 +Architecture: arm64 +Go Version: go1.20.5 +Operating System: darwin +GOPATH=/Users/sudeep/go/ +GOROOT=/usr/local/go +2024/01/04 15:52:05 agent did: did:key:z6MkoLvhaiE9NRYs3vJcynCM8CeyP8hXduWhE5Ter2U2x93y +generating 2048-bit RSA keypair...done +peer identity: QmY49BMJdGneQjJAbTPrGSqaQcLjpCE1WFkRBP6XZEHd6i +2024/01/04 15:52:09 setting up w3up for uploads.... +2024/01/04 15:52:10 w3up agent did: did:key:z6MkoLvhaiE9NRYs3vJcynCM8CeyP8hXduWhE5Ter2U2x93y +2024/01/04 15:52:10 w3up space did: did:key:z6MkgSK6VEu3bvrAFtYNyjsnzG7dVXzYi3yT5TasEgeaQrCe +2024/01/04 15:52:10 w3up setup complete +2024/01/04 15:52:10 Listening... +2024/01/04 15:52:15 generated dag has root cid: bafybeigvijf76lcsjwcmkr6rmzovoiiqdog3muqs5vnplvf4jxh47shfiu +2024/01/04 15:52:15 car file location: /var/folders/w0/bf3y1c7d6ys15tq97ffk5qhw0000gn/T/3475885728.car +2024/01/04 15:53:06 w3 up output: {"root":{"/":"bafybeigvijf76lcsjwcmkr6rmzovoiiqdog3muqs5vnplvf4jxh47shfiu"}} +2024/01/04 15:53:28 uploaded file has root cid: bafybeigvijf76lcsjwcmkr6rmzovoiiqdog3muqs5vnplvf4jxh47shfiu +``` + + ## Running ipfs-pinner server with docker From 0fa37bd2523b5ab01a0b580f7cd0e90380b0aa1b Mon Sep 17 00:00:00 2001 From: sudeep Date: Thu, 4 Jan 2024 17:16:12 +0530 Subject: [PATCH 4/9] edits Dockerfile to use w3cli for upload to web3.storage --- Dockerfile | 6 ++++-- README.md | 17 ++++++++++------- 2 files changed, 14 insertions(+), 9 deletions(-) diff --git a/Dockerfile b/Dockerfile index d0eeefc..4654244 100644 --- a/Dockerfile +++ b/Dockerfile @@ -9,13 +9,15 @@ RUN go mod download && CGO_ENABLED=0 GOOS=linux go build -a -ldflags="-s -w" -o FROM alpine:3.18.3 RUN mkdir /app WORKDIR /app -RUN apk update && apk add --no-cache bash=5.2.15-r5 +RUN apk update && apk add --no-cache bash=5.2.15-r5 nodejs npm git && npm install -g @web3-storage/w3cli COPY --from=builder --chmod=700 /build/ipfs-server /app +RUN apk del git && rm -rf /var/cache/apk/* /root/.npm /tmp/* + HEALTHCHECK --interval=10s --timeout=5s CMD wget --no-verbose --tries=1 --spider localhost:3001/health ENTRYPOINT [ "/bin/bash", "-l", "-c" ] -CMD [ "./ipfs-server -port 3001 -jwt $WEB3_JWT" ] +CMD [ "./ipfs-server -port 3001 -w3-agent-key $W3_AGENT_KEY -w3-delegation-file $W3_DELEGATION_FILE" ] # ipfs-pinner API server; EXPOSE 3001 diff --git a/README.md b/README.md index 5373829..72d2010 100644 --- a/README.md +++ b/README.md @@ -48,18 +48,19 @@ To avoid this issue, the merkle DAG thus generated is exported into special file ## Running ipfs-pinner server -1. Set the environment variable `WEB3_JWT` +1. Get the agent key, did and delegation proof from Covalent -2. to start a server which listens for request on 3001 port, run: +2. build the server and run: ```bash -make clean server-dbg run +make clean server-dbg ``` NOTE: If you want more control over CLI params, you can run the server binary (after `make clean server-dbg`): ```bash ./build/bin/server -jwt -port 3001 +./build/bin/server -w3-agent-key -w3-delegation-file ``` NOTE: If you get some error when running this, check if the diagnostic is there in [known issues](#known-issues) @@ -215,9 +216,9 @@ for ipfs-pinner to function properly with docker, we need Docker run command should have: -- Volumes for data persistence +- Volumes for data persistence; volumes containing the delegation proof - Port mappings -- JWT token passed in the env +- W3up agent key passed in the env ```bash docker buildx create --name builder --use --platform=linux/amd64,linux/arm64 && docker buildx build --platform=linux/amd64,linux/arm64 . -t gcr.io/covalent-project/ipfs-pinner:latest @@ -228,8 +229,10 @@ Now, we can run the container: ```bash docker container run --detach --name ipfs-pinner-instance \ --volume /tmp/data/.ipfs/:/root/.ipfs/ \ + --volume /Users/sudeep/repos/ipfs/w3up-testing/w3up_proof:/root/w3up_proof/ \ -p 3001:3001 \ - --env WEB3_JWT=$WEB3_JWT \ + --env W3_AGENT_KEY=$W3_AGENT_KEY \ + --env W3_DELEGATION_FILE=/root/w3up_proof/proof.out ``` @@ -335,7 +338,7 @@ Users would sometimes want to maintain a different volume to fulfil large storag ipfs-pinner currently uses some known IPFS gateways to fetch content. These gateways are expected to be run and maintained for a long time, but if you need to update the gateways list due to one of the going down, or a more efficient gateway being introduced etc. you can change the list: ```bash -./build/bin/server -jwt -port 3001 -ipfs-gateway-urls "https://w3s.link/ipfs/%s,https://dweb.link/ipfs/%s,https://ipfs.io/ipfs/%s" +./build/bin/server -ipfs-gateway-urls "https://w3s.link/ipfs/%s,https://dweb.link/ipfs/%s,https://ipfs.io/ipfs/%s" ##OTHER PARAMS ``` The `-ipfs-gateways-urls` is a comma separated list of http urls with a `%s` present in it, which is formatted to replace the IPFS content identifier (CID) in it. From 53b75511c2fed569eb273807b792d4074043407e Mon Sep 17 00:00:00 2001 From: sudeep Date: Thu, 4 Jan 2024 18:08:20 +0530 Subject: [PATCH 5/9] linter fixes and fix golangci-lint workflow --- .github/workflows/golangci-lint.yml | 9 ++++---- binary/main.go | 7 +----- go.mod | 4 ++-- go.sum | 12 ---------- pinclient/pinclient.go | 35 ----------------------------- server/main.go | 11 ++++++--- 6 files changed, 16 insertions(+), 62 deletions(-) diff --git a/.github/workflows/golangci-lint.yml b/.github/workflows/golangci-lint.yml index 230bfde..f3e0c56 100644 --- a/.github/workflows/golangci-lint.yml +++ b/.github/workflows/golangci-lint.yml @@ -20,17 +20,18 @@ jobs: steps: - uses: actions/checkout@v3 - uses: actions/setup-go@v3 - - name: Set up Go 1.19 + - name: Set up Go 1.20 uses: actions/setup-go@v3 with: - go-version: 1.19 + go-version: 1.20.5 id: go - run: go version - name: Lint run: | - curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s v1.53.3 - ./bin/golangci-lint run --timeout=3m + curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s v1.53.2 + go mod tidy + ./bin/golangci-lint run -v --timeout=3m --go=1.20.5 # Optional: working directory, useful for monorepos # working-directory: somedir diff --git a/binary/main.go b/binary/main.go index b4ef36b..fc93ae8 100644 --- a/binary/main.go +++ b/binary/main.go @@ -18,12 +18,7 @@ var WEB3_JWT = "WEB3_JWT" var UPLOAD_FILE = "./main.go" // uploading current file itself func main() { - token, present := os.LookupEnv(WEB3_JWT) - if !present { - log.Fatalf("token (%s) not found in env", WEB3_JWT) - } - - clientCreateReq := client.NewClientRequest(core.Web3Storage).BearerToken(token) + clientCreateReq := client.NewClientRequest(core.Web3Storage) // check if cid compute true works with car uploads nodeCreateReq := pinner.NewNodeRequest(clientCreateReq, []string{"https://w3s.link/ipfs/%s"}).CidVersion(1).CidComputeOnly(false) node := pinner.NewPinnerNode(*nodeCreateReq) diff --git a/go.mod b/go.mod index 9be8dab..c3769b7 100644 --- a/go.mod +++ b/go.mod @@ -13,7 +13,9 @@ require ( github.com/ipld/go-ipld-prime v0.21.0 github.com/multiformats/go-multiaddr v0.10.1 github.com/multiformats/go-multibase v0.2.0 + github.com/multiformats/go-multihash v0.2.3 github.com/pkg/errors v0.9.1 + github.com/web3-storage/go-ucanto v0.1.0 github.com/ybbus/httpretry v1.0.2 golang.org/x/oauth2 v0.8.0 ) @@ -136,7 +138,6 @@ require ( github.com/multiformats/go-multiaddr-dns v0.3.1 // indirect github.com/multiformats/go-multiaddr-fmt v0.1.0 // indirect github.com/multiformats/go-multicodec v0.9.0 // indirect - github.com/multiformats/go-multihash v0.2.3 // indirect github.com/multiformats/go-multistream v0.4.1 // indirect github.com/multiformats/go-varint v0.0.7 // indirect github.com/onsi/ginkgo/v2 v2.11.0 // indirect @@ -160,7 +161,6 @@ require ( github.com/spaolacci/murmur3 v1.1.0 // indirect github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7 // indirect github.com/ucarion/urlpath v0.0.0-20200424170820-7ccc79b76bbb // indirect - github.com/web3-storage/go-ucanto v0.1.0 // indirect github.com/whyrusleeping/base32 v0.0.0-20170828182744-c30ac30633cc // indirect github.com/whyrusleeping/cbor v0.0.0-20171005072247-63513f603b11 // indirect github.com/whyrusleeping/cbor-gen v0.0.0-20230818171029-f91ae536ca25 // indirect diff --git a/go.sum b/go.sum index dbd2ccf..4c0477a 100644 --- a/go.sum +++ b/go.sum @@ -47,7 +47,6 @@ github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03 github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo= github.com/Jorropo/jsync v1.0.1 h1:6HgRolFZnsdfzRUj+ImB9og1JYOxQoReSywkHOGSaUU= github.com/Jorropo/jsync v1.0.1/go.mod h1:jCOZj3vrBCri3bSU3ErUYvevKlnbssrXeCivybS5ABQ= -github.com/Kubuxu/go-os-helper v0.0.1/go.mod h1:N8B+I7vPCT80IcP58r50u4+gEEcsZETFUpAzWW2ep1Y= github.com/OneOfOne/xxhash v1.2.2 h1:KMrpdQIwFcEqXDklaen+P1axHaj9BSKzvpUUfnHldSE= github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU= github.com/aead/siphash v1.0.1/go.mod h1:Nywa3cDsYNNK3gaciGTWPwHt0wlpNV15vwmswBAUSII= @@ -88,7 +87,6 @@ github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghf github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44= github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= -github.com/cheekybits/genny v1.0.0/go.mod h1:+tQajlRqAUrPI7DOSpB0XAqZYtQakVtB7wXkRAgjxjQ= github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI= github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI= github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU= @@ -142,8 +140,6 @@ github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDD github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk= github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY= github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto= -github.com/eapache/go-xerial-snappy v0.0.0-20180814174437-776d5712da21/go.mod h1:+020luEh2TKB4/GOp8oxxtq0Daoen/Cii55CzbTV6DU= -github.com/eapache/queue v1.1.0/go.mod h1:6eCeP0CKFpHLu8blIFXhExK/dRa7WDZfr6jVFPTqq+I= github.com/elastic/gosigar v0.12.0/go.mod h1:iXRIGg2tLnu7LBdpqzyQfGDEidKCfWcCMS0WKyPWoMs= github.com/elastic/gosigar v0.14.2 h1:Dg80n8cr90OZ7x+bAax/QjoW/XqTI11RmA79ZwIm9/4= github.com/elastic/gosigar v0.14.2/go.mod h1:iXRIGg2tLnu7LBdpqzyQfGDEidKCfWcCMS0WKyPWoMs= @@ -176,7 +172,6 @@ github.com/go-errors/errors v1.0.1/go.mod h1:f4zRHt4oKfwPJE5k8C9vpYG+aDHdBFUsgrm github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU= github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8= github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8= -github.com/go-logfmt/logfmt v0.5.1/go.mod h1:WYhtIu8zTZfxdn5+rREduYbwxfcBr/Vr6KEVveWlfTs= github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= github.com/go-logr/logr v1.2.4 h1:g01GSCwiDw2xSZfjJ2/T9M+S6pFdcNtFYsp+Y43HYDQ= github.com/go-logr/logr v1.2.4/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= @@ -349,7 +344,6 @@ github.com/ipfs/go-graphsync v0.14.4/go.mod h1:yT0AfjFgicOoWdAlUJ96tQ5AkuGI4r1ta github.com/ipfs/go-ipfs-blockstore v1.3.0 h1:m2EXaWgwTzAfsmt5UdJ7Is6l4gJcaM/A12XwJyvYvMM= github.com/ipfs/go-ipfs-blockstore v1.3.0/go.mod h1:KgtZyc9fq+P2xJUiCAzbRdhhqJHvsw8u2Dlqy2MyRTE= github.com/ipfs/go-ipfs-blocksutil v0.0.1 h1:Eh/H4pc1hsvhzsQoMEP3Bke/aW5P5rVM1IWFJMcGIPQ= -github.com/ipfs/go-ipfs-blocksutil v0.0.1/go.mod h1:Yq4M86uIOmxmGPUHv/uI7uKqZNtLb449gwKqXjIsnRk= github.com/ipfs/go-ipfs-chunker v0.0.5 h1:ojCf7HV/m+uS2vhUGWcogIIxiO5ubl5O57Q7NapWLY8= github.com/ipfs/go-ipfs-delay v0.0.0-20181109222059-70721b86a9a8/go.mod h1:8SP1YXK1M1kXuc4KJZINY3TQQ03J2rwBG9QfXmbRPrw= github.com/ipfs/go-ipfs-delay v0.0.1 h1:r/UXYyRcddO6thwOnhiznIAiSvxMECGgtv35Xs1IeRQ= @@ -428,15 +422,12 @@ github.com/jbenet/goprocess v0.1.4/go.mod h1:5yspPrukOVuOLORacaBi858NqyClJPQxYZl github.com/jellevandenhooff/dkim v0.0.0-20150330215556-f50fe3d243e1/go.mod h1:E0B/fFc00Y+Rasa88328GlI/XbtyysCtTHZS8h7IrBU= github.com/jessevdk/go-flags v0.0.0-20141203071132-1679536dcc89/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI= github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI= -github.com/jpillora/backoff v1.0.0/go.mod h1:J/6gKK9jxlEcS3zixgDgUAsiuZ7yrSoa/FX5e0EB2j4= github.com/jrick/logrotate v1.0.0/go.mod h1:LNinyqDIJnpAur+b8yyulnQw/wDuN1+BYKlTRt3OuAQ= github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU= -github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU= github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk= github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo= github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU= -github.com/julienschmidt/httprouter v1.3.0/go.mod h1:JR6WtHb+2LUe8TCKY3cZOxFyyO8IZAc4RVcycCCAKdM= github.com/kami-zh/go-capturer v0.0.0-20171211120116-e492ea43421d/go.mod h1:P2viExyCEfeWGU259JnaQ34Inuec4R38JCyBx2edgD0= github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q= github.com/kisielk/errcheck v1.2.0/go.mod h1:/BMXB+zMLi60iA8Vv6Ksmxu/1UDYcXs4uQLJ+jE2L00= @@ -555,7 +546,6 @@ github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrk github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= -github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= github.com/mr-tron/base58 v1.1.0/go.mod h1:xcD2VGqlgYjBdcBLw+TuYLr8afG+Hj8g2eTVqeSzSU8= github.com/mr-tron/base58 v1.1.2/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc= github.com/mr-tron/base58 v1.1.3/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc= @@ -601,7 +591,6 @@ github.com/multiformats/go-varint v0.0.5/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXS github.com/multiformats/go-varint v0.0.6/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE= github.com/multiformats/go-varint v0.0.7 h1:sWSGR+f/eu5ABZA2ZpYKBILXTTs9JWpdEM/nEGOHFS8= github.com/multiformats/go-varint v0.0.7/go.mod h1:r8PUYw/fD/SjBCiKOoDlGF6QawOELpZAu9eioSos/OU= -github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U= github.com/neelance/astrewrite v0.0.0-20160511093645-99348263ae86/go.mod h1:kHJEU3ofeGjhHklVoIGuVj85JJwZ6kWPaJwCIxgnFmo= github.com/neelance/sourcemap v0.0.0-20151028013722-8c68805598ab/go.mod h1:Qr6/a/Q4r9LP1IltGz7tA7iOK1WonHEYhu1HRBA7ZiM= github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A= @@ -699,7 +688,6 @@ github.com/shurcooL/users v0.0.0-20180125191416-49c67e49c537/go.mod h1:QJTqeLYED github.com/shurcooL/webdavfs v0.0.0-20170829043945-18c3829fa133/go.mod h1:hKmq5kWdCj2z2KEozexVbfEZIWiTjhE0+UjmZgPqehw= github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo= github.com/sirupsen/logrus v1.7.0/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0= -github.com/sirupsen/logrus v1.8.1/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0= github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc= github.com/smartystreets/assertions v1.2.0 h1:42S6lae5dvLc7BrLu/0ugRtcFVjoJNMC/N3yZFZkDFs= github.com/smartystreets/assertions v1.2.0/go.mod h1:tcbTF8ujkAEcZ8TElKY+i30BzYlVhC/LOxJk7iOWnoo= diff --git a/pinclient/pinclient.go b/pinclient/pinclient.go index 574fa89..d2351f2 100644 --- a/pinclient/pinclient.go +++ b/pinclient/pinclient.go @@ -16,12 +16,8 @@ import ( "github.com/ipfs/go-cid" "github.com/multiformats/go-multibase" "github.com/pkg/errors" - - logging "github.com/ipfs/go-log/v2" ) -var logger = logging.Logger("ipfs-pinner") - const UserAgent = "ipfs-pinner" type Client struct { @@ -128,37 +124,6 @@ func httperr(resp *http.Response, e error) error { return errors.Wrapf(e, "remote pinning service returned http error %d", resp.StatusCode) } -func (c *Client) uploadFileViaPinata(ctx context.Context, file *os.File) (core.PinataResponseGetter, error) { - //ctx = context.WithValue(ctx, openapi.ContextServerIndex, 1) // index = 1 is the file pin url - - poster := c.client.FilepinApi.PinataFileUpload(ctx) - opt := openapi.NewPinataOptions() - opt.SetCidVersion(string(rune(c.cidVersion))) - - result, httpresp, err := poster.PinataOptions(*opt).File(file).Execute() - if err != nil { - err := httperr(httpresp, err) - return nil, err - } - - return core.NewPinataResponseGetter(*result), nil -} - func (c *Client) uploadFileViaWeb3Storage(ctx context.Context, file *os.File) (cid.Cid, error) { return c.w3up.UploadCarFile(file) - - // if err != nil { - // return nil, err - // } - - // return - - // poster := c.client.FilepinApi.Web3StorageCarUpload(ctx) - // result, httpresp, err := poster.Body(file).Execute() - // if err != nil { - // err := httperr(httpresp, err) - // return nil, err - // } - - // return core.NewWeb3StorageResponseGetter(*result), nil } diff --git a/server/main.go b/server/main.go index 766a363..663830d 100644 --- a/server/main.go +++ b/server/main.go @@ -75,12 +75,12 @@ func main() { _, err := os.ReadFile(*w3DelegationFile) if err != nil { - log.Fatalf("error reading delegation proof file: %w", err) + log.Fatalf("error reading delegation proof file: %v", err) } agentSigner, err := signer.Parse(*w3AgentKey) if err != nil { - log.Fatalf("error parsing agent signer: %w", err) + log.Fatalf("error parsing agent signer: %v", err) } log.Printf("agent did: %s", agentSigner.DID().DID().String()) @@ -290,7 +290,12 @@ func uploadHandler(contents string, node pinner.PinnerNode) (cid.Cid, error) { } defer carf.Close() // should delete the file due to unlink - defer syscall.Unlink(carf.Name()) + defer func() { + err := syscall.Unlink(carf.Name()) + if err != nil { + log.Printf("error in unlinking:%v", err) + } + }() log.Printf("car file location: %s\n", carf.Name()) From 88f6647061de0f7e0b8a3679c92234dd5f8f0e6f Mon Sep 17 00:00:00 2001 From: sudeep Date: Thu, 4 Jan 2024 23:05:18 +0530 Subject: [PATCH 6/9] update ipfs-pinnern version to 0.1.17 --- core/support.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/support.go b/core/support.go index 0f22788..e12a06f 100644 --- a/core/support.go +++ b/core/support.go @@ -17,7 +17,7 @@ const ( // IpfsPinnerVersionMinor is Minor version component of the current release IpfsPinnerVersionMinor = 1 // IpfsPinnerVersionPatch is Patch version component of the current release - IpfsPinnerVersionPatch = 16 + IpfsPinnerVersionPatch = 17 clientIdentifier = "ipfs-pinner" // Client identifier to advertise over the network ) From b9e895857d95099a7fe1c739342df55930e3720b Mon Sep 17 00:00:00 2001 From: sudeep Date: Fri, 5 Jan 2024 14:27:05 +0530 Subject: [PATCH 7/9] revert golangci-lint version --- .github/workflows/golangci-lint.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/golangci-lint.yml b/.github/workflows/golangci-lint.yml index f3e0c56..5026f39 100644 --- a/.github/workflows/golangci-lint.yml +++ b/.github/workflows/golangci-lint.yml @@ -29,7 +29,7 @@ jobs: - name: Lint run: | - curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s v1.53.2 + curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s v1.53.3 go mod tidy ./bin/golangci-lint run -v --timeout=3m --go=1.20.5 From 32437e2cc6de70b6ecb3e61e9b111258cd384eae Mon Sep 17 00:00:00 2001 From: sudeep Date: Fri, 5 Jan 2024 14:29:10 +0530 Subject: [PATCH 8/9] update ipfs-pinner version to 1.0.0 --- core/support.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/core/support.go b/core/support.go index e12a06f..fae71a9 100644 --- a/core/support.go +++ b/core/support.go @@ -13,11 +13,11 @@ const ( Web3Storage PinningService = "web3.storage" Other PinningService = "other" // IpfsPinnerVersionMajor is Major version component of the current release - IpfsPinnerVersionMajor = 0 + IpfsPinnerVersionMajor = 1 // IpfsPinnerVersionMinor is Minor version component of the current release - IpfsPinnerVersionMinor = 1 + IpfsPinnerVersionMinor = 0 // IpfsPinnerVersionPatch is Patch version component of the current release - IpfsPinnerVersionPatch = 17 + IpfsPinnerVersionPatch = 0 clientIdentifier = "ipfs-pinner" // Client identifier to advertise over the network ) From acadce32b8db77467ff2038c4052152001d26e50 Mon Sep 17 00:00:00 2001 From: sudeep Date: Fri, 5 Jan 2024 14:38:53 +0530 Subject: [PATCH 9/9] update documentation for instructions about delegation proof file --- README.md | 32 ++++++++++++++++++++++---------- 1 file changed, 22 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 72d2010..c671395 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,8 @@ - [communicate to the operator](#communicate-to-the-operator) - [operator invocation](#operator-invocation) - [Running ipfs-pinner server with docker](#running-ipfs-pinner-server-with-docker) + - [Running the image](#running-the-image) + - [Building the docker image](#building-the-docker-image) - [Docker Volume setup](#docker-volume-setup) - [Port mapping setup](#port-mapping-setup) - [Development](#development) @@ -216,36 +218,46 @@ for ipfs-pinner to function properly with docker, we need Docker run command should have: -- Volumes for data persistence; volumes containing the delegation proof +- Volumes for data persistence; - Port mappings - W3up agent key passed in the env + +### Running the image + +Copy the delegation proof file into the ipfs directory which will be mapped onto the docker image. + ```bash -docker buildx create --name builder --use --platform=linux/amd64,linux/arm64 && docker buildx build --platform=linux/amd64,linux/arm64 . -t gcr.io/covalent-project/ipfs-pinner:latest +mv proof.out /tmp/data/.ipfs/ ``` -Now, we can run the container: +Then one can run the docker container: ```bash docker container run --detach --name ipfs-pinner-instance \ --volume /tmp/data/.ipfs/:/root/.ipfs/ \ - --volume /Users/sudeep/repos/ipfs/w3up-testing/w3up_proof:/root/w3up_proof/ \ -p 3001:3001 \ --env W3_AGENT_KEY=$W3_AGENT_KEY \ - --env W3_DELEGATION_FILE=/root/w3up_proof/proof.out + --env W3_DELEGATION_FILE=/root/.ipfs/proof.out ``` + +### Building the docker image +```bash +docker buildx create --name builder --use --platform=linux/amd64,linux/arm64 && docker buildx build --platform=linux/amd64,linux/arm64 . -t gcr.io/covalent-project/ipfs-pinner:latest +``` + ### Docker Volume setup -There's 1 docker volume that needs to be shared (and persisted) between the container and the host - this `~/.ipfs` directory needs to have its lifecycle unaffected by container lifecycle (since it contains the merklelized nodes, blockstore etc.), and so that is docker volume managed. +There's 1 docker volume that needs to be shared (and persisted) between the container and the host - the `~/.ipfs` directory, which needs to have its lifecycle unaffected by container lifecycle (since it contains the merklelized nodes, blockstore etc.), and so that is docker volume managed. ### Port mapping setup -:4001 : swarm port for p2p (currently disabled) -:8080 - http gateway (used by encapsulated ipfs-node) -:5001: local api (should be bound to 127.0.0.1 only, and must never be exposed publicly as it allows one to control the ipfs node; also used by encapsulated ipfs-node) -:3001: The ipfs-pinner itself exposes its REST API on this port +`:4001` - swarm port for p2p (currently disabled) +`:8080` - http gateway (used by encapsulated ipfs-node) +`:5001` - local api (should be bound to 127.0.0.1 only, and must never be exposed publicly as it allows one to control the ipfs node; also used by encapsulated ipfs-node) +`:3001` - The ipfs-pinner itself exposes its REST API on this port Out of the above, only the swarm port and the REST api port (3001) are essential.