Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove datastore, fetch latest block #928

Open
wants to merge 73 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
c2fcf30
basic redis setup in the same container
volovyks Oct 2, 2024
f552f24
formatting
volovyks Oct 2, 2024
704ee5f
basic presignature-storage using redis
volovyks Oct 4, 2024
3aa2ac3
presignature storage wrapped in arc and ewlock
volovyks Oct 9, 2024
2f0e5b1
presignature storage redis functions implemented
volovyks Oct 9, 2024
8539fc8
redis presignature storage used in presignature manager
volovyks Oct 11, 2024
e26a21a
develop
volovyks Oct 11, 2024
feba156
serialization for presignature
volovyks Oct 11, 2024
0102d1f
warnings fixed
volovyks Oct 15, 2024
3d5796b
presignature bad parameters error check
volovyks Oct 15, 2024
91d25dd
min refactoring
volovyks Oct 15, 2024
05e8394
added redis config file path to dockerfile, added tf to mount /data v…
kmaus-near Oct 15, 2024
9d28c3d
it fix
volovyks Oct 15, 2024
7ca6a5d
rust 2024 requirements
volovyks Oct 15, 2024
6e3b027
start redis in it
volovyks Oct 16, 2024
6d6fcbb
pull redis in it
volovyks Oct 16, 2024
eefe68a
lock conflict
volovyks Oct 16, 2024
998c897
lock file update
volovyks Oct 16, 2024
e449f8c
redis unit test
volovyks Oct 16, 2024
19fc21d
develop
volovyks Oct 17, 2024
4f82d8b
comments 1
volovyks Oct 17, 2024
90d6d8e
comments 2
volovyks Oct 17, 2024
3bce87b
triple manager functions renamed to match presiganture manager
volovyks Oct 17, 2024
fde93fb
serialization simlified
volovyks Oct 17, 2024
9fa82ca
comments 3
volovyks Oct 17, 2024
1edc9b3
added start command to mpc-node and added redis URL format
kmaus-near Oct 17, 2024
992f334
Merge branch 'serhii/presignature-storage' of https://github.com/near…
kmaus-near Oct 17, 2024
077466d
use account id as a part of redis storage key
volovyks Oct 17, 2024
9737f5a
Merge branch 'serhii/presignature-storage' of github.com:near/mpc-rec…
volovyks Oct 17, 2024
99e8f75
added full redis config, modified dockerfile to use it
kmaus-near Oct 17, 2024
596f13b
unnecessary port expose removed
volovyks Oct 18, 2024
99126b5
remove local redis config
volovyks Oct 18, 2024
d460253
redis config update
volovyks Oct 18, 2024
4a2c5a1
redis file ownership change
kmaus-near Oct 18, 2024
43ceebc
Merge branch 'serhii/presignature-storage' of https://github.com/near…
kmaus-near Oct 18, 2024
8e310a7
fix persistent data issue
kmaus-near Oct 19, 2024
701a4a5
fix redis unit test
volovyks Oct 21, 2024
9aacb9a
extend redis lifecycle
volovyks Oct 21, 2024
01e678e
clippy
volovyks Oct 21, 2024
93c296a
datastore triple storage removed
volovyks Oct 22, 2024
2b6ad95
refactor triple storage
volovyks Oct 23, 2024
c8eccc8
use redis to store triples
volovyks Oct 23, 2024
a55a298
Merge branch 'develop' into serhii/presignature-storage
volovyks Oct 23, 2024
e95f602
Merge branch 'develop' into serhii/triple-storage
volovyks Oct 23, 2024
0f5d078
fix rust typing issue
volovyks Oct 23, 2024
a4baaba
Merge branch 'serhii/triple-storage' into serhii/presignature-storage
volovyks Oct 23, 2024
b764321
2 triple pre-check added
volovyks Oct 24, 2024
f10bafc
add link to redis in setup-env
volovyks Oct 28, 2024
41a46ce
use redis connection pool
volovyks Oct 29, 2024
900d0d5
fmt
volovyks Oct 29, 2024
b83ba1b
change info to debug in storage layer
volovyks Oct 29, 2024
54e3c49
rename count to lem
volovyks Oct 29, 2024
df919c8
develop
volovyks Oct 29, 2024
566a134
Fix locks being acquired incorrectly (#914)
ChaoticTempest Oct 30, 2024
b654fba
chore: removed Arc and RwLock for triple/presignature storage (#915)
ChaoticTempest Oct 30, 2024
b552acf
fix conflicts
volovyks Oct 30, 2024
a3716b1
fix conflicts 2
volovyks Oct 30, 2024
db18173
in progress
volovyks Oct 30, 2024
2a0504a
develop
Nov 1, 2024
f82cb38
remove test output
Nov 1, 2024
d7314e3
redis app storage removed
Nov 7, 2024
b233519
remove starting block parameter
volovyks Nov 7, 2024
e3241c0
remove redundant datastore setup
volovyks Nov 7, 2024
382105c
add in memory variable for latest processed block
volovyks Nov 7, 2024
b78ccea
develop
volovyks Nov 8, 2024
f904e54
remove datastore users and roles from tf
volovyks Nov 8, 2024
2ae0bf6
datastore docker estup removed
volovyks Nov 8, 2024
669373d
fix tf mistake
volovyks Nov 8, 2024
404ddbe
fetch latest block using RPC
volovyks Nov 8, 2024
358a7d1
clippy
volovyks Nov 8, 2024
5b7ab8f
change log order
volovyks Nov 8, 2024
8050190
revert mpc recovery tf changes
volovyks Nov 8, 2024
00bbd96
revert mpc recovery tf changes
volovyks Nov 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 18 additions & 15 deletions chain-signatures/node/src/cli.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
use crate::config::{Config, LocalConfig, NetworkConfig, OverrideConfig};
use crate::gcp::GcpService;
use crate::protocol::{MpcSignProtocol, SignQueue};
use crate::storage::app_data_storage;
use crate::{http_client, indexer, mesh, storage, web};
use clap::Parser;
use deadpool_redis::Runtime;
Expand Down Expand Up @@ -195,14 +196,6 @@ pub fn run(cmd: Cli) -> anyhow::Result<()> {
.build()?;
let gcp_service =
rt.block_on(async { GcpService::init(&account_id, &storage_options).await })?;
let (indexer_handle, indexer) = indexer::run(
&indexer_options,
&mpc_contract_id,
&account_id,
&sign_queue,
&gcp_service,
&rt,
)?;

let key_storage =
storage::secret_storage::init(Some(&gcp_service), &storage_options, &account_id);
Expand All @@ -214,6 +207,23 @@ pub fn run(cmd: Cli) -> anyhow::Result<()> {
let triple_storage = storage::triple_storage::init(&redis_pool, &account_id);
let presignature_storage =
storage::presignature_storage::init(&redis_pool, &account_id);
let app_data_storage = app_data_storage::init(&redis_pool, &account_id);

let mut rpc_client = near_fetch::Client::new(&near_rpc);
if let Some(referer_param) = client_header_referer {
let client_headers = rpc_client.inner_mut().headers_mut();
client_headers.insert(http::header::REFERER, referer_param.parse().unwrap());
}
tracing::info!(rpc_addr = rpc_client.rpc_addr(), "rpc client initialized");

let (indexer_handle, indexer) = indexer::run(
&indexer_options,
&mpc_contract_id,
&account_id,
&sign_queue,
app_data_storage,
rpc_client.clone(),
)?;

let sign_sk = sign_sk.unwrap_or_else(|| account_sk.clone());
let my_address = my_address
Expand All @@ -229,13 +239,6 @@ pub fn run(cmd: Cli) -> anyhow::Result<()> {
let (sender, receiver) = mpsc::channel(16384);

tracing::info!(%my_address, "address detected");
let mut rpc_client = near_fetch::Client::new(&near_rpc);
if let Some(referer_param) = client_header_referer {
let client_headers = rpc_client.inner_mut().headers_mut();
client_headers.insert(http::header::REFERER, referer_param.parse().unwrap());
}

tracing::info!(rpc_addr = rpc_client.rpc_addr(), "rpc client initialized");
let signer = InMemorySigner::from_secret_key(account_id.clone(), account_sk);
let (protocol, protocol_state) = MpcSignProtocol::init(
my_address,
Expand Down
22 changes: 0 additions & 22 deletions chain-signatures/node/src/gcp/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -19,25 +19,3 @@ pub enum SecretStorageError {
#[error("(de)serialization error: {0}")]
SerdeError(#[from] serde_json::Error),
}

#[derive(thiserror::Error, Debug)]
pub enum DatastoreStorageError {
#[error("GCP error: {0}")]
GcpError(#[from] google_datastore1::Error),
#[error("IO error: {0}")]
IoError(#[from] std::io::Error),
#[error("(de)serialization error: {0}")]
SerdeError(#[from] serde_json::Error),
#[error("datastore value conversion error: {0}")]
ConvertError(ConvertError),
#[error("fetch_entities error: `{0}`")]
FetchEntitiesError(String),
#[error("could not find entity: {0}")]
EntityNotFound(String),
}

impl From<ConvertError> for DatastoreStorageError {
fn from(err: ConvertError) -> Self {
DatastoreStorageError::ConvertError(err)
}
}
Loading
Loading