feat: Create two APIs for data operations #69

binglekruger · 2022-09-28T09:37:46Z

Create a new branch for the Data Service API

Upload data pool
Query data pool for data creator

* feat(ntc-vault-cli): initial layout * style(ntc-vault-cli): add rustfmt.toml * feat(ntc-vault-cli): add clap-based CLI skeleton * feat(ntc-vault-cli): flesh out structure, implement identity subcommand * refactor(ntc-vault-cli): separate code into two crates: core, cli * refactor(ntc-vault-cli): move rand-using code from core to cli * test(ntc-vault-cli): generate_secure_seed smoke test * feat(ntc-vault-cli): data package and JSON Schema work-in-progress * refactor: move crates into rust-workspace * docs: add comment about using "cargo +nightly fmt" * docs: add ARCHITECTURE.md * refactor: rename package "ntc-vault-core" → "ntc-data-packages" * feat(ntc-data-packages): replace anyhow with thiserror * build(ntc-data-packages): drop jsonschema's default features We don't need jsonschema's CLI, or the file / HTTP resolving features. This greatly reduces our dependency tree. * build: declare rust-version * docs: add package descriptions * feat(ntc-data-packages): better validation error messages * ci(rust-workspace): add check and test workflows for GitHub Actions * docs: fix rustdoc link issues * build(rust-workspace): add rust-toolchain.toml, with channel = "stable" * docs(README): add link to ARCHITECTURE.md * deps(ntc-vault-cli): update confy revision for store_path fix Upstream PR: rust-cli/confy#60 (fix: `store_path` should create missing directories, like `load_path`) * refactor(ntc-data-packages): move tests to API-based integration tests Motivation: https://matklad.github.io/2021/02/27/delete-cargo-integration-tests.html * refactor(ntc-vault-cli): move try_exists to compat module * refactor(ntc-vault-cli): VaultIdentityConfig: make pub * deps(ntc-vault-cli): add dev dependencies for CLI tests * test(ntc-vault-cli): add snapshot-based CLI tests * deps(ntc-data-packages): add serde * feat(ntc-data-packages): add Metadata::from_json_bytes * feat(ntc-vault-cli): add fs_io, with read_metadata * feat(ntc-vault-cli): implement more of the "data" subcommand * chore: add todos to keep track of changes that needs to be made to allign with design * chore(deps): update dependencies and bump jsonschema to 0.16 Co-authored-by: Herman <[email protected]>

* build(rust-sgx-workspace): add workspace for SGX code * build(ntc-tee-server): add SGX project for the TEE server * build(ntc-tee-server): bump Rust edition: "2018" → "2021" * build(ntc-tee-server): better error message for missing Enclave_u library * ci(rust-sgx-workspace): add check and test workflows for GitHub Actions * ci: use nested checkout for the multiple repositories to prevent _temp being deleted * style: fix clippy warnings * ci: remove --all-targets build arg to fix enclave builds and checks Co-authored-by: Herman <[email protected]>

Co-authored-by: Jean-Pierre de Villiers <[email protected]>

* feat(ntc-oracle-node): connect to PureStake Indexer using algonaut * feat(ntc-oracle-node): add config crate * feat(ntc-oracle-node): add initial server with axum * feat(ntc-oracle-node): implement get_auth_data * feat(ntc-oracle-node): use ed25519 to sign AuthData with ring-compat * style(ntc-oracle-node): apply rustfmt.toml * refactor(ntc-oracle-node): move auth_data functions into handler * feat(ntc-oracle-node): add anyhow error handling * feat(ntc-oracle-node): add unit test for sign_auth_data * feat(ntc-oracle-node): use serde_with to base64 encode SignedAuthData response * refactor(ntc-oracle-node): use cfg(test) in place of allow(dead_code)

* feat(web-server): copy web-server from Nautilus Wallet * docs(web-server): fix comments in rust-toolchain and rustfmt toml files * refactor: move http-service-impl and sgx-helpers into rust-sgx-workspace * refactor: move sgx-wallet-impl into rust-sgx-workspace but exclude as member * refactor: move sgx-wallet into rust-sgx-workspace but exclude as member * refactor: move sgx-wallet-test into rust-sgx-workspace but exclude as member * refactor: remove web-server * docs(http-service-impl): fix sgx-wallet-impl broken intra doc link * fix(sgx-wallet): fix path to sgx-wallet-impl in Makefile

#67)

…-trusted-compute into data-operations

binglekruger · 2022-10-24T14:05:21Z

Started with two tutorials for using CosmosDB with Mongo APIs (recommended by Bill):
https://www.mongodb.com/developer/languages/rust/rust-mongodb-crud-tutorial/
https://github.com/Azure-Samples/azure-cosmos-db-mongodb-rust-getting-started

Found another helpful tutorial: https://jbarszczewski.com/rust-actix-cosmosdb-mongodb-tutorial-api

Created 2 API functions:

get_data - Display data pools of a specific user
add_data - Add a new data pool to CosmosDB

Set connection string when starting project:

CONNECTION_STRING="mongodb://ntc-data:zkicQepvrK6B1SySo3vDLQpycrEsSt3WByUE1Zg7SYs47CceytRCzIuP3cu3p09GtY2cREJyJtdc9zSjqlxcvA==@ntc-data.mongo.cosmos.azure.com:10255/?ssl=true&replicaSet=globaldb&retrywrites=false&maxIdleTimeMS=120000&appName=@ntc-data@" cargo run

Test POST - (Upload/Add data)

(Copy source code)

curl --header "Content-Type: application/json" \ --request POST \ --data '{"id":"Test Client 01","pool_name":"NTLS third pool","sealed_data":"This is some sealed data"}' \ http://127.0.0.1:8000/api/data

Test GET - (View/Get data)

(Copy source code)

curl --header "Content-Type: application/json" \ --request GET \ --data '{"id":"Test Client 02"}' \ http://127.0.0.1:8000/api/data

billguo99

Run the following to format your code:
cargo make format
cargo make format-toml

billguo99 · 2022-11-01T12:40:16Z

rust-workspace/projects/data-server/src/data_handlers/mod.rs

+use std::sync::Mutex;
+use chrono::prelude::*;
+use serde::Deserialize;
+use std::env;


unused import

billguo99 · 2022-11-01T12:41:04Z

rust-workspace/projects/data-server/src/main.rs

+    env::set_var("RUST_LOG", "actix_web=debug");
+    // Remember to set connection string when starting the server
+    let mongo_url = env::var("CONNECTION_STRING").unwrap();
+    let mut client_options = ClientOptions::parse(&mongo_url).await.unwrap();


remove mut as variable does not have to be mutable

billguo99 · 2022-11-01T12:42:36Z

rust-workspace/projects/data-server/src/data_handlers/mod.rs

+#[derive(Deserialize)]
+pub struct NewDataPool {
+    pub id: String,
+    pub poolName: String, //User defined pool name


can we use snake case name :)
pool_name
sealed_data

billguo99 · 2022-11-01T12:43:55Z

rust-workspace/projects/data-server/src/data_handlers/mod.rs

+}
+
+// This function accepts application data
+async fn get_data(data: web::Data<Mutex<Client>>, existingUser: web::Json<UserData>) -> impl Responder {


snake case naming: existingUser--> existing_user

billguo99 · 2022-11-01T12:44:05Z

rust-workspace/projects/data-server/src/data_handlers/mod.rs

+        .database(MONGO_DB)
+        .collection(MONGO_COLLECTION);
+
+    let userId = &existingUser.id;


Claire-Bear · 2022-11-15T11:09:22Z

Task linked: CU-30ektdj Create 2 APIs (upload data pool & query data pool for data creator)

* feat: add Vault CLI & data package library (#1) * feat(ntc-vault-cli): initial layout * style(ntc-vault-cli): add rustfmt.toml * feat(ntc-vault-cli): add clap-based CLI skeleton * feat(ntc-vault-cli): flesh out structure, implement identity subcommand * refactor(ntc-vault-cli): separate code into two crates: core, cli * refactor(ntc-vault-cli): move rand-using code from core to cli * test(ntc-vault-cli): generate_secure_seed smoke test * feat(ntc-vault-cli): data package and JSON Schema work-in-progress * refactor: move crates into rust-workspace * docs: add comment about using "cargo +nightly fmt" * docs: add ARCHITECTURE.md * refactor: rename package "ntc-vault-core" → "ntc-data-packages" * feat(ntc-data-packages): replace anyhow with thiserror * build(ntc-data-packages): drop jsonschema's default features We don't need jsonschema's CLI, or the file / HTTP resolving features. This greatly reduces our dependency tree. * build: declare rust-version * docs: add package descriptions * feat(ntc-data-packages): better validation error messages * ci(rust-workspace): add check and test workflows for GitHub Actions * docs: fix rustdoc link issues * build(rust-workspace): add rust-toolchain.toml, with channel = "stable" * docs(README): add link to ARCHITECTURE.md * deps(ntc-vault-cli): update confy revision for store_path fix Upstream PR: rust-cli/confy#60 (fix: `store_path` should create missing directories, like `load_path`) * refactor(ntc-data-packages): move tests to API-based integration tests Motivation: https://matklad.github.io/2021/02/27/delete-cargo-integration-tests.html * refactor(ntc-vault-cli): move try_exists to compat module * refactor(ntc-vault-cli): VaultIdentityConfig: make pub * deps(ntc-vault-cli): add dev dependencies for CLI tests * test(ntc-vault-cli): add snapshot-based CLI tests * deps(ntc-data-packages): add serde * feat(ntc-data-packages): add Metadata::from_json_bytes * feat(ntc-vault-cli): add fs_io, with read_metadata * feat(ntc-vault-cli): implement more of the "data" subcommand * chore: add todos to keep track of changes that needs to be made to allign with design * chore(deps): update dependencies and bump jsonschema to 0.16 Co-authored-by: Herman <[email protected]> * feat: add initial TEE server (#3) * build(rust-sgx-workspace): add workspace for SGX code * build(ntc-tee-server): add SGX project for the TEE server * build(ntc-tee-server): bump Rust edition: "2018" → "2021" * build(ntc-tee-server): better error message for missing Enclave_u library * ci(rust-sgx-workspace): add check and test workflows for GitHub Actions * ci: use nested checkout for the multiple repositories to prevent _temp being deleted * style: fix clippy warnings * ci: remove --all-targets build arg to fix enclave builds and checks Co-authored-by: Herman <[email protected]> * license: relicense to AGPL-3.0 (#17) * ci: use ntls-io fork of rust-sgx-sdk-env (#39) Co-authored-by: Jean-Pierre de Villiers <[email protected]> * feat: create oracle-node API (#18) * feat(ntc-oracle-node): connect to PureStake Indexer using algonaut * feat(ntc-oracle-node): add config crate * feat(ntc-oracle-node): add initial server with axum * feat(ntc-oracle-node): implement get_auth_data * feat(ntc-oracle-node): use ed25519 to sign AuthData with ring-compat * style(ntc-oracle-node): apply rustfmt.toml * refactor(ntc-oracle-node): move auth_data functions into handler * feat(ntc-oracle-node): add anyhow error handling * feat(ntc-oracle-node): add unit test for sign_auth_data * feat(ntc-oracle-node): use serde_with to base64 encode SignedAuthData response * refactor(ntc-oracle-node): use cfg(test) in place of allow(dead_code) * feat: add Vault backend (#61) * feat(web-server): copy web-server from Nautilus Wallet * docs(web-server): fix comments in rust-toolchain and rustfmt toml files * refactor: move http-service-impl and sgx-helpers into rust-sgx-workspace * refactor: move sgx-wallet-impl into rust-sgx-workspace but exclude as member * refactor: move sgx-wallet into rust-sgx-workspace but exclude as member * refactor: move sgx-wallet-test into rust-sgx-workspace but exclude as member * refactor: remove web-server * docs(http-service-impl): fix sgx-wallet-impl broken intra doc link * fix(sgx-wallet): fix path to sgx-wallet-impl in Makefile * refactor(rust-sgx-workspace): rename 'wallet' to 'vault' (#62) * refactor(sgx-vault-impl): change vault id to algorand address (#65) * refactor: remove Onfido operations and XRP related code (#66) * feat(rust-sgx-workspace): add Docker build definition for Vault server (#67) * feat: Create two APIs for data operations * Merge branch 'data-operations' of https://github.com/ntls-io/nautilus-trusted-compute into data-operations * feat(data-operations-api): create new project dir * feat: Add base crate * feat: Add basic Actix server * feat: Configure server + add logs_handlers * feat: Fetching logs * feat: Adding logs * refactor: Update log -> data * feat: Add filter functionality * refactor: remove unused imports + fix snake_case * feat(backend-services): initial commit for api * feat(backend-services): initial commit for api * fix(backend-services): add env to .gitignore. * fix (backend-services): .gitignore and datetime * feature (backend-services): add example json and schema files --------- Co-authored-by: Pi Delport <[email protected]> Co-authored-by: Herman <[email protected]> Co-authored-by: Jean-Pierre de Villiers <[email protected]> Co-authored-by: Jean-Pierre de Villiers <[email protected]> Co-authored-by: Bill Guo <[email protected]> Co-authored-by: Bill Guo <[email protected]> Co-authored-by: binglekruger <[email protected]>

PiDelport and others added 11 commits April 28, 2022 11:36

license: relicense to AGPL-3.0 (#17)

1521f59

ci: use ntls-io fork of rust-sgx-sdk-env (#39)

356aa48

Co-authored-by: Jean-Pierre de Villiers <[email protected]>

refactor(rust-sgx-workspace): rename 'wallet' to 'vault' (#62)

d6c260d

refactor(sgx-vault-impl): change vault id to algorand address (#65)

fb26233

refactor: remove Onfido operations and XRP related code (#66)

8e8d6d8

feat(rust-sgx-workspace): add Docker build definition for Vault server (

266a57d

#67)

feat: Create two APIs for data operations

7c3a0db

binglekruger force-pushed the data-operations branch from 5fed21d to 462ed04 Compare October 13, 2022 11:38

binglekruger added 2 commits October 13, 2022 11:39

Merge branch 'data-operations' of https://github.com/ntls-io/nautilus…

9327bad

…-trusted-compute into data-operations

feat(data-operations-api): create new project dir

5a78c00

binglekruger force-pushed the data-operations branch from ccb5652 to 392c6d6 Compare October 14, 2022 09:43

binglekruger added 7 commits October 14, 2022 09:45

feat: Add base crate

ebf31d5

feat: Add basic Actix server

d713a86

feat: Configure server + add logs_handlers

3654c67

feat: Fetching logs

59ad783

feat: Adding logs

96d03a3

refactor: Update log -> data

454e6b0

feat: Add filter functionality

d4aceb8

binglekruger requested review from billguo99, jdvlio and sonasi October 24, 2022 14:05

billguo99 requested changes Nov 1, 2022

View reviewed changes

refactor: remove unused imports + fix snake_case

b1313e2

binglekruger requested a review from billguo99 November 1, 2022 14:32

sonasi approved these changes Dec 8, 2022

View reviewed changes

sonasi force-pushed the data-operations branch from 7fa0869 to b1313e2 Compare January 25, 2023 12:07

sonasi and others added 2 commits February 14, 2023 17:07

Merge branch 'main' into data-operations

73f4b5e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Create two APIs for data operations #69

feat: Create two APIs for data operations #69

binglekruger commented Sep 28, 2022 •

edited

Loading

binglekruger commented Oct 24, 2022 •

edited

Loading

billguo99 left a comment

billguo99 Nov 1, 2022

billguo99 Nov 1, 2022

billguo99 Nov 1, 2022

billguo99 Nov 1, 2022

billguo99 Nov 1, 2022

Claire-Bear commented Nov 15, 2022

feat: Create two APIs for data operations #69

Are you sure you want to change the base?

feat: Create two APIs for data operations #69

Conversation

binglekruger commented Sep 28, 2022 • edited Loading

binglekruger commented Oct 24, 2022 • edited Loading

Set connection string when starting project:

Test POST - (Upload/Add data)

Test GET - (View/Get data)

billguo99 left a comment

Choose a reason for hiding this comment

billguo99 Nov 1, 2022

Choose a reason for hiding this comment

billguo99 Nov 1, 2022

Choose a reason for hiding this comment

billguo99 Nov 1, 2022

Choose a reason for hiding this comment

billguo99 Nov 1, 2022

Choose a reason for hiding this comment

billguo99 Nov 1, 2022

Choose a reason for hiding this comment

Claire-Bear commented Nov 15, 2022

binglekruger commented Sep 28, 2022 •

edited

Loading

binglekruger commented Oct 24, 2022 •

edited

Loading