Notary Server SGX Attestation Template #492

maceip · 2024-06-05T03:14:52Z

sgx attestation

this pr adds two files to the notary-server:

a gramine manifest template
- this manifest details needed files, mounts, and other hardware required by the enclave; it sets the entrypoint to be executed when the enclave boots, and other gramine specific security / policy settings
a Makefile
- a convenience file to preprocess the manifest template, and runs our binary in SGX, or if there is no SGX hardware, uses gramine-direct for testing / verification.

this PR sets the stage for SGX specific concepts and scaffolding before we deploy on SGX hardware

The SGX Report

The SGX report contains a few values, one of which is MR_ENCLAVE.

This PR also adds a push-hook that generates a new SGX report for that commit, and modifies the project's README.md with the updated SGX Report:

How To Create an SGX Report without SGX:

steps to test this out (gramine is a pain to build on macos, use a *nix:

install gramine; for ubuntu it's (copypasta this whole block):

sudo curl -fsSLo /usr/share/keyrings/gramine-keyring.gpg https://packages.gramineproject.io/gramine-keyring.gpg
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/gramine-keyring.gpg] https://packages.gramineproject.io/ $(lsb_release -sc) main" \
| sudo tee /etc/apt/sources.list.d/gramine.list
sudo curl -fsSLo /usr/share/keyrings/intel-sgx-deb.asc https://download.01.org/intel-sgx/sgx_repo/ubuntu/intel-sgx-deb.key
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/intel-sgx-deb.asc] https://download.01.org/intel-sgx/sgx_repo/ubuntu $(lsb_release -sc) main" \
| sudo tee /etc/apt/sources.list.d/intel-sgx.list

sudo apt-get update
sudo apt-get install gramine

ensure you have the tlsn deps
make
gramine-sgx-gen-private-key
gramine-sgx-sign -v --manifest notary-server.manifest --output notary-server.sgx
gramine-sgx-sigstruct-view notary-server.sig

if all went well you should see something like the following:

Attributes:
    mr_signer: 9a03bd7cd0172e7d04dc886d2f01405b9fb28a02fd08b9e388a387176c5a1edb
    mr_enclave: e9ee7d2d63f95d5cd426e57d20b1e7120a2657af2f45a50879a8e51e0bf5ac07
    isv_prod_id: 0
    isv_svn: 0
    debug_enclave: False

the mr_enclave value is the thing we care about: it won't change unless you change the notary server & it's deps.

if you want to test notary-server, spawned inside libOS on a machine without SGX, you can invoke it like so:
make start-gramine-server

or if you do have SGX, you would run it like so: SGX=1 make start-gramine-server

.sig, .manifest$, and *.sgx need to get added to .gitignore

sinui0 · 2024-06-05T22:40:46Z

Let's open a tee branch and merge related work into there until we're ready to release

maceip · 2024-06-06T08:21:05Z

Let's open a tee branch and merge related work into there until we're ready to release

sgtm; i tried but i get a:

remote: Permission to tlsnotary/tlsn.git denied to maceip.

…ement feat: push hook action to generate a gramine sig for notary-server, add output to readme

heeckhau

Good work. See comments

.gitignore

heeckhau · 2024-06-06T08:46:30Z

.github/workflows/sgx-report.yml

+        with:
+          ref: ${{ github.ref }}


AFAIK this is the default

Suggested change

with:

ref: ${{ github.ref }}

.github/workflows/sgx-report.yml

heeckhau · 2024-06-06T09:01:48Z

.github/workflows/sgx-report.yml

+      - uses: awalsh128/cache-apt-pkgs-action@latest
+        with:
+          packages: rustc cargo gramine cmake clang gramine
+          version: 1.1


What is this version here?

.github/workflows/sgx-report.yml

heeckhau · 2024-06-06T09:11:40Z

.github/workflows/sgx-report.yml

+        run: |
+               echo "GITHUB_ENV: $GITHUB_ENV"
+               echo "SGX_REPORT: $SGX_REPORT"
+      - name: update README


We can avoid changing the readme and regex magic, If write the report to a file, add and commit it.
The README file can link to this file.
Similar to what projects to with checksum files.
What do you think?

I'll postpone reviewing the README file until we make a decision here.

now it creates and uploads a signed-by github artifact

README.md

notary-server/notary-server.manifest.template

notary-server/Makefile

.github/workflows/sgx-report.yml

yuroitaki

Great stuff! got a few minor comments :)

maceip · 2024-06-29T05:38:28Z

will address comments above;

attestation artifacts attested by github:

yuroitaki · 2024-07-05T03:19:43Z

.github/workflows/release.yml

+on:
+  create:
+    branches:
+      - 'release/*'


for releases, we don't use release/* naming; but just tags, like https://github.com/tlsnotary/tlsn/blob/main/.github/workflows/ci.yml#L7-L8

yuroitaki · 2024-07-05T03:21:45Z

.github/workflows/release.yml

+
+jobs:
+  create-gramine-attestation:
+    uses: maceip/tlsn/.github/workflows/gramine-report.yml@sgx-attest


it shouldn't be maceip's here right?

yuroitaki · 2024-07-05T03:27:40Z

notary/server/README.md

@@ -134,3 +134,8 @@ Axum is chosen as the framework to serve HTTP and WebSocket requests from the pr

 #### WebSocket
 Axum's internal implementation of WebSocket uses [tokio_tungstenite](https://docs.rs/tokio-tungstenite/latest/tokio_tungstenite/), which provides a WebSocket struct that doesn't implement [AsyncRead](https://docs.rs/futures/latest/futures/io/trait.AsyncRead.html) and [AsyncWrite](https://docs.rs/futures/latest/futures/io/trait.AsyncWrite.html). Both these traits are required by the TLSN core libraries for the prover and the notary. To overcome this, a [slight modification](./src/service/axum_websocket.rs) of Axum's implementation of WebSocket is used, where [async_tungstenite](https://docs.rs/async-tungstenite/latest/async_tungstenite/) is used instead so that [ws_stream_tungstenite](https://docs.rs/ws_stream_tungstenite/latest/ws_stream_tungstenite/index.html) can be used to wrap on top of the WebSocket struct to get AsyncRead and AsyncWrite implemented.
+
+## Reproduciple Builds & Attestation
+- We are using [Gramine](https://github.com/gramineproject) to generate SGX reports for notary-server;


Suggested change

- We are using [Gramine](https://github.com/gramineproject) to generate SGX reports for notary-server;

- We are using [Gramine](https://github.com/gramineproject) to generate SGX reports for notary-server.

yuroitaki · 2024-07-05T03:27:55Z

notary/server/README.md

@@ -134,3 +134,8 @@ Axum is chosen as the framework to serve HTTP and WebSocket requests from the pr

 #### WebSocket
 Axum's internal implementation of WebSocket uses [tokio_tungstenite](https://docs.rs/tokio-tungstenite/latest/tokio_tungstenite/), which provides a WebSocket struct that doesn't implement [AsyncRead](https://docs.rs/futures/latest/futures/io/trait.AsyncRead.html) and [AsyncWrite](https://docs.rs/futures/latest/futures/io/trait.AsyncWrite.html). Both these traits are required by the TLSN core libraries for the prover and the notary. To overcome this, a [slight modification](./src/service/axum_websocket.rs) of Axum's implementation of WebSocket is used, where [async_tungstenite](https://docs.rs/async-tungstenite/latest/async_tungstenite/) is used instead so that [ws_stream_tungstenite](https://docs.rs/ws_stream_tungstenite/latest/ws_stream_tungstenite/index.html) can be used to wrap on top of the WebSocket struct to get AsyncRead and AsyncWrite implemented.
+
+## Reproduciple Builds & Attestation


Suggested change

## Reproduciple Builds & Attestation

## Reproducible Builds & Attestation

yuroitaki · 2024-07-05T03:29:06Z

notary/server/README.md

+## Reproduciple Builds & Attestation
+- We are using [Gramine](https://github.com/gramineproject) to generate SGX reports for notary-server;
+- Each release of tlsn will include a build artifact attested by github, which includes the gramine signature.
+- if you build and run the notary-server with gramine, you should get the same mr_enclave hash as in our release artifact.


Suggested change

- if you build and run the notary-server with gramine, you should get the same mr_enclave hash as in our release artifact.

- If you build and run the notary-server with gramine, you should get the same `mr_enclave` hash as in our release artifact.

yuroitaki · 2024-07-05T03:36:51Z

.github/workflows/gramine-report.yml

+          execute_install_scripts: true
+      - name: Set PATH
+        run: echo "export PATH=\$PATH:/usr/local/bin:/usr/bin" >> $GITHUB_ENV
+      - name: generate manifest and sig


do we not need to cd into notary/server first?

yuroitaki · 2024-07-05T03:49:39Z

.github/workflows/gramine-report.yml

+name: generate a gramine signature, which can be verified later
+
+on:
+  workflow_call:


is there a reason why we making this modular to be called from another workflow, instead of everything in one workflow file?

yuroitaki · 2024-07-05T04:07:29Z

notary/server/Makefile

+
+ARCH_LIBDIR ?= /lib/$(shell $(CC) -dumpmachine)
+
+SELF_EXE = target/release/notary-server


the target/release/notary-server file is actually located in the parent folder: notary/target/release, instead of notary/server/target/release — will this affect anything?

the latest repo reorg has shifted the target folder to the top level of the repo

yuroitaki

Great stufff! BTW SO SORRY EVERYONE haha i didn't realise each of my previous comments was an independent review that might end up spamming everyone's inbox :(

yuroitaki · 2024-07-05T04:13:01Z

notary/server/Makefile

+
+# Note that we're compiling in release mode regardless of the DEBUG setting passed
+# to Make, as compiling in debug mode results in an order of magnitude's difference in
+# performance that makes testing by running a benchmark with ab painful. The primary goal


question: does ab here mean ab testing?

yuroitaki · 2024-07-05T04:16:32Z

notary/server/Makefile

+	$(GRAMINE) notary-server
+.PHONY: clean
+clean:
+	$(RM) -rf *.token *.sig *.manifest.sgx *.manifest result-* OUTPUT


question: does this command run in notary/server directory? or somewhere else?

yuroitaki · 2024-07-05T04:29:21Z

notary/server/notary-server.manifest.template

+    { uri = "file:fixture/notary/notary.key" },
+    { uri = "file:fixture/notary/notary.pub" },
+    { uri = "file:fixture/auth/whitelist.csv" },
+    { uri = "file:fixture/tls/notary.crt" },


doesn't tls/notary.key also need to be included since it's the TLS key?

tls/notary.key not needed anymore as hosted notary server doesnt use it

same goes for whitelist.csv and notary.crt; though for config.yaml, notary.key and notary.pub — we need to make sure they are the same files used by the hosted notary server

yuroitaki · 2024-08-15T04:41:33Z

notary/server/notary-server.manifest.template

+    { uri = "file:fixture/notary/notary.key" },
+    { uri = "file:fixture/notary/notary.pub" },
+    { uri = "file:fixture/auth/whitelist.csv" },
+    { uri = "file:fixture/tls/notary.crt" },


tls/notary.key not needed anymore as hosted notary server doesnt use it

yuroitaki · 2024-08-15T04:42:46Z

notary/server/notary-server.manifest.template

+    { uri = "file:fixture/notary/notary.key" },
+    { uri = "file:fixture/notary/notary.pub" },
+    { uri = "file:fixture/auth/whitelist.csv" },
+    { uri = "file:fixture/tls/notary.crt" },


same goes for whitelist.csv and notary.crt; though for config.yaml, notary.key and notary.pub — we need to make sure they are the same files used by the hosted notary server

yuroitaki · 2024-08-15T04:43:47Z

notary/server/notary-server.manifest.template

+edmm_enable = true
+
+allowed_files = [
+    "file:fixture/tls",


we dont need this anymore probably? since hosted notary server don't use it

yuroitaki · 2024-08-15T07:20:36Z

notary/server/Makefile

+
+ARCH_LIBDIR ?= /lib/$(shell $(CC) -dumpmachine)
+
+SELF_EXE = target/release/notary-server


the latest repo reorg has shifted the target folder to the top level of the repo

yuroitaki · 2024-08-15T07:25:16Z

notary/server/notary-server.manifest.template

+mounts = [
+    { path = "/lib", uri = "file:{{ gramine.runtimedir() }}" },
+    { path = "{{ arch_libdir }}", uri = "file:{{ arch_libdir }}" },
+    { path = "/fixture", uri = "file:fixture" },


we dont need this anymore probably? since hosted notary server don't use it

feat: gramine manifest.template and Makefile to facilitate tee measur…

58e9487

…ement feat: push hook action to generate a gramine sig for notary-server, add output to readme

maceip force-pushed the sgx-attest branch from 2bd9e1a to 58e9487 Compare June 6, 2024 08:37

Automated SGX Report Gen

5fd5037

heeckhau requested changes Jun 6, 2024

View reviewed changes

yuroitaki reviewed Jun 7, 2024

View reviewed changes

README.md Outdated Show resolved Hide resolved

yuroitaki reviewed Jun 7, 2024

View reviewed changes

notary-server/notary-server.manifest.template Outdated Show resolved Hide resolved

yuroitaki reviewed Jun 7, 2024

View reviewed changes

notary-server/Makefile Outdated Show resolved Hide resolved

yuroitaki reviewed Jun 7, 2024

View reviewed changes

.github/workflows/sgx-report.yml Outdated Show resolved Hide resolved

yuroitaki reviewed Jun 7, 2024

View reviewed changes

[feat] sgx artifacts signed by github

1b06502

maceip force-pushed the sgx-attest branch from ba4d3b5 to 1b06502 Compare June 29, 2024 04:43

maceip added 4 commits July 1, 2024 18:24

upstream rebase

405ecc9

[chore] remove attestation from README

4f148ea

[chore] update README with sgx attestation info

0f2ea42

[chore] update attestation report to use .txt extension

97cd381

yuroitaki reviewed Jul 5, 2024

View reviewed changes

yuroitaki requested changes Jul 5, 2024

View reviewed changes

yuroitaki reviewed Aug 15, 2024

View reviewed changes

yuroitaki mentioned this pull request Sep 13, 2024

ci: azure ssh runner for named feature: tee_quote #580

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Notary Server SGX Attestation Template #492

Notary Server SGX Attestation Template #492

maceip commented Jun 5, 2024 •

edited

Loading

sinui0 commented Jun 5, 2024

maceip commented Jun 6, 2024 •

edited

Loading

heeckhau left a comment

heeckhau Jun 6, 2024

heeckhau Jun 6, 2024

heeckhau Jun 6, 2024

maceip Jul 2, 2024

yuroitaki left a comment

maceip commented Jun 29, 2024

yuroitaki Jul 5, 2024

yuroitaki Jul 5, 2024

yuroitaki Jul 5, 2024

yuroitaki Jul 5, 2024

yuroitaki Jul 5, 2024 •

edited

Loading

yuroitaki Jul 5, 2024

yuroitaki Jul 5, 2024

yuroitaki Jul 5, 2024

yuroitaki Aug 15, 2024

yuroitaki left a comment •

edited

Loading

yuroitaki Jul 5, 2024 •

edited

Loading

yuroitaki Jul 5, 2024

yuroitaki Jul 5, 2024

yuroitaki Aug 15, 2024

yuroitaki Aug 15, 2024

yuroitaki Aug 15, 2024

yuroitaki Aug 15, 2024

yuroitaki Aug 15, 2024

yuroitaki Aug 15, 2024

yuroitaki Aug 15, 2024

	- We are using [Gramine](https://github.com/gramineproject) to generate SGX reports for notary-server;
	- We are using [Gramine](https://github.com/gramineproject) to generate SGX reports for notary-server.

	## Reproduciple Builds & Attestation
	## Reproducible Builds & Attestation

	- if you build and run the notary-server with gramine, you should get the same mr_enclave hash as in our release artifact.
	- If you build and run the notary-server with gramine, you should get the same `mr_enclave` hash as in our release artifact.


		ARCH_LIBDIR ?= /lib/$(shell $(CC) -dumpmachine)

		SELF_EXE = target/release/notary-server

Notary Server SGX Attestation Template #492

Are you sure you want to change the base?

Notary Server SGX Attestation Template #492

Conversation

maceip commented Jun 5, 2024 • edited Loading

sgx attestation

How To Create an SGX Report without SGX:

*.sig, *.manifest$, and *.sgx need to get added to .gitignore

sinui0 commented Jun 5, 2024

maceip commented Jun 6, 2024 • edited Loading

heeckhau left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuroitaki left a comment

Choose a reason for hiding this comment

maceip commented Jun 29, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuroitaki Jul 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuroitaki left a comment • edited Loading

Choose a reason for hiding this comment

yuroitaki Jul 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maceip commented Jun 5, 2024 •

edited

Loading

.sig, .manifest$, and *.sgx need to get added to .gitignore

maceip commented Jun 6, 2024 •

edited

Loading

yuroitaki Jul 5, 2024 •

edited

Loading

yuroitaki left a comment •

edited

Loading

yuroitaki Jul 5, 2024 •

edited

Loading