migrating to w3up

.github/workflows/golangci-lint.yml

@@ -20,17 +20,18 @@ jobs:
     steps:
       - uses: actions/checkout@v3
       - uses: actions/setup-go@v3
-      - name: Set up Go 1.19
+      - name: Set up Go 1.20
         uses: actions/setup-go@v3
         with:
-          go-version: 1.19
+          go-version: 1.20.5
         id: go
       - run: go version
 
       - name: Lint
         run: |
           curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s v1.53.3
-          ./bin/golangci-lint run --timeout=3m
+          go mod tidy
+          ./bin/golangci-lint run -v --timeout=3m --go=1.20.5
 
           # Optional: working directory, useful for monorepos
           # working-directory: somedir

Dockerfile

@@ -9,13 +9,15 @@ RUN go mod download && CGO_ENABLED=0 GOOS=linux go build -a -ldflags="-s -w" -o
 FROM alpine:3.18.3
 RUN mkdir /app
 WORKDIR /app
-RUN apk update && apk add --no-cache bash=5.2.15-r5
+RUN apk update && apk add --no-cache bash=5.2.15-r5 nodejs npm git && npm install -g @web3-storage/w3cli
 COPY --from=builder --chmod=700 /build/ipfs-server /app
 
+RUN apk del git && rm -rf /var/cache/apk/* /root/.npm /tmp/*
+
 HEALTHCHECK --interval=10s --timeout=5s CMD wget --no-verbose --tries=1 --spider localhost:3001/health
 
 ENTRYPOINT [ "/bin/bash", "-l", "-c" ]
-CMD [ "./ipfs-server -port 3001 -jwt $WEB3_JWT" ]
+CMD [ "./ipfs-server -port 3001 -w3-agent-key $W3_AGENT_KEY -w3-delegation-file $W3_DELEGATION_FILE" ]
 
 # ipfs-pinner API server;
 EXPOSE 3001

README.md

@@ -8,7 +8,17 @@
     - [Upload a file](#upload-a-file)
     - [Download content (given cid)](#download-content-given-cid)
     - [Find the cid given some content](#find-the-cid-given-some-content)
+  - [migration to UCAN and capabilities setup](#migration-to-ucan-and-capabilities-setup)
+    - [setting up w3cli](#setting-up-w3cli)
+      - [installation](#installation)
+      - [login and check spaces](#login-and-check-spaces)
+      - [generate ucan key](#generate-ucan-key)
+      - [create delegation to store/add and upload/add](#create-delegation-to-storeadd-and-uploadadd)
+      - [communicate to the operator](#communicate-to-the-operator)
+      - [operator invocation](#operator-invocation)
   - [Running ipfs-pinner server with docker](#running-ipfs-pinner-server-with-docker)
+    - [Running the image](#running-the-image)
+    - [Building the docker image](#building-the-docker-image)
     - [Docker Volume setup](#docker-volume-setup)
     - [Port mapping setup](#port-mapping-setup)
   - [Development](#development)
@@ -24,7 +34,7 @@
 ## Introduction
 
 - A wrapper on top of ipfs node, utilising go-ipfs as a library.
-- Extended support for custom file upload endpoints provided by pinata & web3.storage.
+- Extended support for custom file upload endpoints provided by web3.storage.
 - Content archive file generation and lightweight deterministic CID generation on client side (using CARs).
 - It can be used as a go library (see `binary/main.go` for usage) or as a http server.
 
@@ -40,18 +50,19 @@ To avoid this issue, the merkle DAG thus generated is exported into special file
 
 ## Running ipfs-pinner server
 
-1. Set the environment variable `WEB3_JWT`
+1. Get the agent key, did and delegation proof from Covalent
 
-2. to start a server which listens for request on 3001 port, run:
+2. build the server and run:
 
 ```bash
-make clean server-dbg run
+make clean server-dbg
 ```
 
 NOTE: If you want more control over CLI params, you can run the server binary (after `make clean server-dbg`):
 
 ```bash
 ./build/bin/server -jwt <WEB3_JWT> -port 3001
+./build/bin/server -w3-agent-key <AGENT_KEY> -w3-delegation-file <DELEGATION_PROOF_FILE_PATH>
 ```
 
 NOTE: If you get some error when running this, check if the diagnostic is there in [known issues](#known-issues)
@@ -107,6 +118,96 @@ There's a timeout (check code for value) for the download request, if it doesn't
 {"cid": "bafkreicszve3ewhhrgobm366mdctki2m2qwzide5e54zh5aifnesg3ofne"}%
 ```
 
+
+## migration to UCAN and capabilities setup
+- web3.storage is sunsetting its custom upload endpoint (on 9th January, 2024), and we need to migrate from using that to w3up.
+- w3up uses UCAN which is a capabilities-based authorization system (learn more [here](https://web3.storage/docs/concepts/ucans-and-web3storage/)). 
+- In this setup, the "central account" (owned by Covalent) sets up a "space" (think namespace for data). The central account (controlled by the email) is delegated the capabilty to operate on this space.
+- among other capabilties, the central account can delegate certain capabilities (like uploading to space) to other **agents**. This has to be done at our end, and scripts will be made available for it in this repo.
+- once an agent is granted the capability, we share the credentials with the operators, who run ipfs-pinner with it, and can then upload or fetch.
+
+
+### setting up w3cli
+
+- Create a web3.storage account in the [console](https://console.web3.storage/). 
+- Create a space which you want to use to upload artifacts. We want to use different spaces for different artifacts to keep a clear separation.
+
+We'll use w3cli to login and create a new space and register.
+
+#### installation
+```bash
+➜ npm install -g @web3-storage/w3cli
+
+➜ w3 --version
+w3, 7.0.3
+```
+
+#### login and check spaces
+```bash
+➜ w3 login [email protected]
+
+➜ w3 space ls
+  did:key:z6MkgSK6VEu3bvrAFtYNyjsnzG7dVXzYi3yT5TasEgeaQrCe mock_artifacts
+
+➜ w3 space use did:key:z6MkgSK6VEu3bvrAFtYNyjsnzG7dVXzYi3yT5TasEgeaQrCe
+did:key:z6MkgSK6VEu3bvrAFtYNyjsnzG7dVXzYi3yT5TasEgeaQrCe
+```
+
+The did key is the identifier for this space. Now let's generate some DIDs for an operator and delegate upload capabilities to it.
+
+#### generate ucan key
+```bash
+➜ npx ucan-key ed --json
+{
+  "did": "did:key:z6MkpzWw1fDZYMpESgVKFAT87SZAuHiCQZVBC3hmQjB18Nzj",
+  "key": "MgCbc48J8n+BMdzA4XxwYOaKmdu5Ov34jE71U8vV07IVIjO0BnJa05mNMcB8GSz1lib014QAhvAxorG6zACrstm6PBGA="
+}
+```
+
+#### create delegation to store/add and upload/add
+
+```bash
+➜ w3 delegation create -c 'store/add' -c 'upload/add' did:key:z6MkpzWw1fDZYMpESgVKFAT87SZAuHiCQZVBC3hmQjB18Nzj -o proof.out
+```
+
+
+Copy the output. This is the delegation string.
+
+#### communicate to the operator
+
+Provide the operator with the `did`, `key` string + `proof.out` file. These will be passed to operator's setup of the 
+ipfs-pinner, which can then make the delegations.
+
+
+#### operator invocation
+
+the operator can pass the `key` for `-w3-agent-key` and proof file in `-w3-delegation-file` flag.
+
+```bash
+go run server/main.go -w3-agent-key <agent-key> -w3-delegation-file ./proof.out
+ipfs-pinner
+ipfs-pinner Version: 0.1.16
+Architecture: arm64
+Go Version: go1.20.5
+Operating System: darwin
+GOPATH=/Users/sudeep/go/
+GOROOT=/usr/local/go
+2024/01/04 15:52:05 agent did: did:key:z6MkoLvhaiE9NRYs3vJcynCM8CeyP8hXduWhE5Ter2U2x93y
+generating 2048-bit RSA keypair...done
+peer identity: QmY49BMJdGneQjJAbTPrGSqaQcLjpCE1WFkRBP6XZEHd6i
+2024/01/04 15:52:09 setting up w3up for uploads....
+2024/01/04 15:52:10 w3up agent did: did:key:z6MkoLvhaiE9NRYs3vJcynCM8CeyP8hXduWhE5Ter2U2x93y
+2024/01/04 15:52:10 w3up space did: did:key:z6MkgSK6VEu3bvrAFtYNyjsnzG7dVXzYi3yT5TasEgeaQrCe
+2024/01/04 15:52:10 w3up setup complete
+2024/01/04 15:52:10 Listening...
+2024/01/04 15:52:15 generated dag has root cid: bafybeigvijf76lcsjwcmkr6rmzovoiiqdog3muqs5vnplvf4jxh47shfiu
+2024/01/04 15:52:15 car file location: /var/folders/w0/bf3y1c7d6ys15tq97ffk5qhw0000gn/T/3475885728.car
+2024/01/04 15:53:06 w3 up output: {"root":{"/":"bafybeigvijf76lcsjwcmkr6rmzovoiiqdog3muqs5vnplvf4jxh47shfiu"}}
+2024/01/04 15:53:28 uploaded file has root cid: bafybeigvijf76lcsjwcmkr6rmzovoiiqdog3muqs5vnplvf4jxh47shfiu
+```
+
+
+
 ## Running ipfs-pinner server with docker
 
 We can also run the ipfs-pinner server via docker.
@@ -117,34 +218,46 @@ for ipfs-pinner to function properly with docker, we need
 
 Docker run command should have:
 
-- Volumes for data persistence
+- Volumes for data persistence;
 - Port mappings
-- JWT token passed in the env
+- W3up agent key passed in the env
+
+
+### Running the image
+
+Copy the delegation proof file into the ipfs directory which will be mapped onto the docker image.
 
 ```bash
-docker buildx create --name builder --use --platform=linux/amd64,linux/arm64  && docker buildx build --platform=linux/amd64,linux/arm64 . -t gcr.io/covalent-project/ipfs-pinner:latest
+mv proof.out /tmp/data/.ipfs/
 ```
 
-Now, we can run the container:
+Then one can run the docker container:
 
 ```bash
 docker container run --detach --name ipfs-pinner-instance \
        --volume /tmp/data/.ipfs/:/root/.ipfs/  \
        -p 3001:3001  \
-       --env WEB3_JWT=$WEB3_JWT \
+       --env W3_AGENT_KEY=$W3_AGENT_KEY \
+       --env W3_DELEGATION_FILE=/root/.ipfs/proof.out
     <image-id>
 ```
 
+
+### Building the docker image
+```bash
+docker buildx create --name builder --use --platform=linux/amd64,linux/arm64  && docker buildx build --platform=linux/amd64,linux/arm64 . -t gcr.io/covalent-project/ipfs-pinner:latest
+```
+
 ### Docker Volume setup
 
-There's 1 docker volume that needs to be shared (and persisted) between the container and the host - this `~/.ipfs` directory needs to have its lifecycle unaffected by container lifecycle (since it contains the merklelized nodes, blockstore etc.), and so that is docker volume managed.  
+There's 1 docker volume that needs to be shared (and persisted) between the container and the host - the `~/.ipfs` directory, which needs to have its lifecycle unaffected by container lifecycle (since it contains the merklelized nodes, blockstore etc.), and so that is docker volume managed.  
 
 ### Port mapping setup
 
-:4001 : swarm port for p2p  (currently disabled)
-:8080 - http gateway (used by encapsulated ipfs-node)
-:5001: local api (should be bound to 127.0.0.1 only, and must never be exposed publicly as it allows one to control the ipfs node; also used by encapsulated ipfs-node)  
-:3001: The ipfs-pinner itself exposes its REST API on this port
+`:4001` - swarm port for p2p  (currently disabled)  
+`:8080` - http gateway (used by encapsulated ipfs-node)  
+`:5001` - local api (should be bound to 127.0.0.1 only, and must never be exposed publicly as it allows one to control the ipfs node; also used by encapsulated ipfs-node)  
+`:3001` - The ipfs-pinner itself exposes its REST API on this port  
 
 <B> Out of the above, only the swarm port and the REST api port (3001) are essential.</B>  
 
@@ -237,7 +350,7 @@ Users would sometimes want to maintain a different volume to fulfil large storag
 ipfs-pinner currently uses some known IPFS gateways to fetch content. These gateways are expected to be run and maintained for a long time, but if you need to update the gateways list due to one of the going down, or a more efficient gateway being introduced etc. you can change the list:
 
 ```bash
-./build/bin/server -jwt <WEB3_JWT> -port 3001 -ipfs-gateway-urls "https://w3s.link/ipfs/%s,https://dweb.link/ipfs/%s,https://ipfs.io/ipfs/%s"
+./build/bin/server -ipfs-gateway-urls "https://w3s.link/ipfs/%s,https://dweb.link/ipfs/%s,https://ipfs.io/ipfs/%s" ##OTHER PARAMS
 ```
 
 The `-ipfs-gateways-urls` is a comma separated list of http urls with a `%s` present in it, which is formatted to replace the IPFS content identifier (CID) in it.

binary/main.go

@@ -18,12 +18,7 @@ var WEB3_JWT = "WEB3_JWT"
 var UPLOAD_FILE = "./main.go" // uploading current file itself
 
 func main() {
-	token, present := os.LookupEnv(WEB3_JWT)
-	if !present {
-		log.Fatalf("token (%s) not found in env", WEB3_JWT)
-	}
-
-	clientCreateReq := client.NewClientRequest(core.Web3Storage).BearerToken(token)
+	clientCreateReq := client.NewClientRequest(core.Web3Storage)
 	// check if cid compute true works with car uploads
 	nodeCreateReq := pinner.NewNodeRequest(clientCreateReq, []string{"https://w3s.link/ipfs/%s"}).CidVersion(1).CidComputeOnly(false)
 	node := pinner.NewPinnerNode(*nodeCreateReq)

core/support.go

@@ -13,11 +13,11 @@ const (
 	Web3Storage PinningService = "web3.storage"
 	Other       PinningService = "other"
 	// IpfsPinnerVersionMajor is Major version component of the current release
-	IpfsPinnerVersionMajor = 0
+	IpfsPinnerVersionMajor = 1
 	// IpfsPinnerVersionMinor is Minor version component of the current release
-	IpfsPinnerVersionMinor = 1
+	IpfsPinnerVersionMinor = 0
 	// IpfsPinnerVersionPatch is Patch version component of the current release
-	IpfsPinnerVersionPatch = 16
+	IpfsPinnerVersionPatch = 0
 	clientIdentifier       = "ipfs-pinner" // Client identifier to advertise over the network
 )
 

go.mod

@@ -13,7 +13,9 @@ require (
 	github.com/ipld/go-ipld-prime v0.21.0
 	github.com/multiformats/go-multiaddr v0.10.1
 	github.com/multiformats/go-multibase v0.2.0
+	github.com/multiformats/go-multihash v0.2.3
 	github.com/pkg/errors v0.9.1
+	github.com/web3-storage/go-ucanto v0.1.0
 	github.com/ybbus/httpretry v1.0.2
 	golang.org/x/oauth2 v0.8.0
 )
@@ -136,7 +138,6 @@ require (
 	github.com/multiformats/go-multiaddr-dns v0.3.1 // indirect
 	github.com/multiformats/go-multiaddr-fmt v0.1.0 // indirect
 	github.com/multiformats/go-multicodec v0.9.0 // indirect
-	github.com/multiformats/go-multihash v0.2.3 // indirect
 	github.com/multiformats/go-multistream v0.4.1 // indirect
 	github.com/multiformats/go-varint v0.0.7 // indirect
 	github.com/onsi/ginkgo/v2 v2.11.0 // indirect

go.sum

@@ -745,6 +745,8 @@ github.com/warpfork/go-testmark v0.12.1 h1:rMgCpJfwy1sJ50x0M0NgyphxYYPMOODIJHhsX
 github.com/warpfork/go-wish v0.0.0-20200122115046-b9ea61034e4a/go.mod h1:x6AKhvSSexNrVSrViXSHUEbICjmGXhtgABaHIySUSGw=
 github.com/warpfork/go-wish v0.0.0-20220906213052-39a1cc7a02d0 h1:GDDkbFiaK8jsSDJfjId/PEGEShv6ugrt4kYsC5UIDaQ=
 github.com/warpfork/go-wish v0.0.0-20220906213052-39a1cc7a02d0/go.mod h1:x6AKhvSSexNrVSrViXSHUEbICjmGXhtgABaHIySUSGw=
+github.com/web3-storage/go-ucanto v0.1.0 h1:Hg6jO7OLLeDLtmseQZWQGBFJMp+5t5OWAFVL5Y3LsOs=
+github.com/web3-storage/go-ucanto v0.1.0/go.mod h1:XD6zahQ8HLh8Z2CSK7xYcTR0A7oCVYXTZB7fFtYqGF8=
 github.com/whyrusleeping/base32 v0.0.0-20170828182744-c30ac30633cc h1:BCPnHtcboadS0DvysUuJXZ4lWVv5Bh5i7+tbIyi+ck4=
 github.com/whyrusleeping/base32 v0.0.0-20170828182744-c30ac30633cc/go.mod h1:r45hJU7yEoA81k6MWNhpMj/kms0n14dkzkxYHoB96UM=
 github.com/whyrusleeping/cbor v0.0.0-20171005072247-63513f603b11 h1:5HZfQkwe0mIfyDmc1Em5GqlNRzcdtlv4HTNmdpt7XH0=

pinclient/client_create_request.go

@@ -4,14 +4,20 @@ import (
 	"net/http"
 
 	"github.com/covalenthq/ipfs-pinner/core"
+	"github.com/web3-storage/go-ucanto/did"
 )
 
 type ClientCreateRequest struct {
 	ps                    core.PinningService
 	pinningServiceBaseUrl string
 	filePinBaseUrl        string
-	bearerToken           string
-	httpClient            *http.Client
+	//bearerToken           string
+
+	W3_AgentKey            string
+	W3_AgentDid            did.DID
+	W3_DelegationProofPath string
+
+	httpClient *http.Client
 }
 
 func NewClientRequest(ps core.PinningService) ClientCreateRequest {
@@ -21,8 +27,18 @@ func NewClientRequest(ps core.PinningService) ClientCreateRequest {
 	return request
 }
 
-func (r ClientCreateRequest) BearerToken(token string) ClientCreateRequest {
-	r.bearerToken = token
+func (r ClientCreateRequest) W3AgentKey(key string) ClientCreateRequest {
+	r.W3_AgentKey = key
+	return r
+}
+
+func (r ClientCreateRequest) W3AgentDid(did did.DID) ClientCreateRequest {
+	r.W3_AgentDid = did
+	return r
+}
+
+func (r ClientCreateRequest) DelegationProofPath(proofPath string) ClientCreateRequest {
+	r.W3_DelegationProofPath = proofPath
 	return r
 }