Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(core): multi-pubkey DB support #2093

Open
wants to merge 207 commits into
base: dev
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
207 commits
Select commit Hold shift + click to select a range
aa0909f
WIP: resolve db paths by account keys
onur-ozkan Mar 28, 2024
f051600
WIP: handle db ids optionally
onur-ozkan Mar 29, 2024
c8a434e
use dummy values to fix build errors for development
onur-ozkan Apr 1, 2024
3e0061f
fix conflicts after merge
borngraced Apr 9, 2024
c6146d0
more wip changes and start coin pubkey for indexeddb storage
borngraced Apr 9, 2024
eff374c
Z coin or default pubkey for initializing blocksdb
borngraced Apr 9, 2024
fef8a4e
zcoin - allow db_id for walletdb
borngraced Apr 10, 2024
6247766
wip — SwapsContext
borngraced Apr 10, 2024
fa0f5ab
changes to maker and taker swap db_id
borngraced Apr 10, 2024
947ab8d
complete zcoin dbs multi pubkey support for WASM target
borngraced Apr 10, 2024
ae95d4e
fix clippy/fmt
borngraced Apr 10, 2024
c0d13e7
impl account_db_id for wasm target
borngraced Apr 10, 2024
71c4e9c
complete some todos in swap mod
borngraced Apr 10, 2024
be3dc81
minor changes to DbIdentifier constructor(new)
borngraced Apr 10, 2024
ffbc262
changes to swaps db
borngraced Apr 11, 2024
07c1168
handle recv_swap_msg with db_id
borngraced Apr 11, 2024
1ec96bc
add todos for StateMachineStorage for TakerSwapStorage
borngraced Apr 11, 2024
3ccfa00
add todos for StateMachineStorage for TakerSwapStorage
borngraced Apr 11, 2024
29763e2
get_swap_repr
borngraced Apr 11, 2024
cf749eb
mark account_db_id as todo for native target
borngraced Apr 11, 2024
368441c
impl find_unique_active_account_ids and use for swap_kick_starts
borngraced Apr 11, 2024
7095605
make swap_v2 multi_key
borngraced Apr 11, 2024
8823be2
WIP - start nft multi key support
borngraced Apr 11, 2024
42a22b1
WIP - my_swaps_storage
borngraced Apr 12, 2024
23f0566
fix field assignment outside of initializer
borngraced Apr 12, 2024
65a07d2
WIP - uxto_block_header_storage
borngraced Apr 12, 2024
71ae0b1
wip sqlite
borngraced Apr 12, 2024
61c49bb
merge find_unique_account_ids and find_unique_active_account_ids
borngraced Apr 13, 2024
d184c1e
Merge branch 'dev' into multi-pubkey-dbs
borngraced Apr 13, 2024
bd114fc
refactor find_unique_account_ids
borngraced Apr 14, 2024
f58c80a
debug with logger
borngraced Apr 14, 2024
a99c8a6
wasm clipy
borngraced Apr 14, 2024
5298f2b
WIP nft
borngraced Apr 15, 2024
d9d8e53
use a single find_unique_nft_account_ids for all targets
borngraced Apr 16, 2024
1538f81
make account_db_id return None by default
borngraced Apr 16, 2024
c0355fa
WIP shared, async, sqlite connection manager impl
borngraced Apr 16, 2024
a27ec3e
WIP sql_ctx
borngraced Apr 16, 2024
3407835
WIP sqlite connection manager
borngraced Apr 17, 2024
c27206c
make account_id optional and return None by default
borngraced Apr 17, 2024
ca13b41
WIP rollback sql_ctx
borngraced Apr 17, 2024
3f053bd
WIP rollback sql_ctx
borngraced Apr 17, 2024
f0e8016
rollback sql_ctx
borngraced Apr 17, 2024
f5e5245
Merge remote-tracking branch 'origin/dev' into multi-pubkey-dbs
borngraced Apr 17, 2024
f8d08cf
fix lint
borngraced Apr 17, 2024
2d96232
find_unique_nft_account_ids
borngraced Apr 17, 2024
d2152f8
improve find_unique_nft_account_ids_active and update get_nft_list to…
borngraced Apr 18, 2024
04d2896
update get_nft_transfers to fetch transfers concurrently for multi db_id
borngraced Apr 18, 2024
879438b
use try_join_all for efficiency
borngraced Apr 18, 2024
234a2aa
finish implement async_sqlite_storage_vw for multi key support
borngraced Apr 19, 2024
e2dacfc
finish refactoring nft ctx for multi key support db
borngraced Apr 19, 2024
a00c2c7
replace async_sqlite_connection with v2
borngraced Apr 19, 2024
04e1d97
rename AsyncConnectionArc to AsyncSqliteConnectionArc
borngraced Apr 19, 2024
f904927
WIP sync_sqlite_connection_v2
borngraced Apr 19, 2024
ea778d2
WIP utxo_block_header storage refacting
borngraced Apr 19, 2024
8fa9cec
WIP utxo_block_header storage refacting test fn
borngraced Apr 19, 2024
86649f5
WIP simplify new_from_ctx test fn
borngraced Apr 19, 2024
0699843
WIP sqlite_connection_v2
borngraced Apr 19, 2024
3480e26
fmt
borngraced Apr 19, 2024
b9375e0
finish sqlite refactoring and rename sqlite_connection_v2 to sql_conn…
borngraced Apr 20, 2024
4c3a61c
cargo fmt
borngraced Apr 20, 2024
12537c0
initialize only default pubkey in lp_init
borngraced Apr 20, 2024
77b43f5
cargo clippy
borngraced Apr 20, 2024
fbe1fe2
stats_nodes multi key db
borngraced Apr 21, 2024
795f718
fix find_unique_account_ids todos
borngraced Apr 21, 2024
42f5db0
fix merge conflicts
borngraced Apr 21, 2024
88754f7
minor changes
borngraced Apr 22, 2024
cc3a02f
cargo fmt
borngraced Apr 22, 2024
ee8d9d0
wip changes
borngraced Apr 23, 2024
acd6383
wip changes db_id
borngraced Apr 23, 2024
4d04128
organize lp_coin mods/imports
borngraced Apr 23, 2024
e508b59
wip lp_ordermatch db update
borngraced Apr 23, 2024
d7c4a83
orders_kick_start
borngraced Apr 24, 2024
3a7fa66
wasm shareddb storage
borngraced Apr 24, 2024
4e7ae81
clippy & fmt
borngraced Apr 24, 2024
69d9886
wip lp_ordermatch
borngraced Apr 25, 2024
8796922
Merge remote-tracking branch 'origin/dev' into multi-pubkey-dbs
borngraced Apr 25, 2024
875c9df
cleanup lp_ordermatch.rs
borngraced Apr 25, 2024
4f7824d
merge dev branch and fix conflicts
borngraced Apr 26, 2024
4bba3fb
new sql initialization TODOs
borngraced Apr 26, 2024
637286d
todo for db_id in orders_kick_start
borngraced Apr 26, 2024
7cd60f8
minor changes/improvements
borngraced Apr 29, 2024
b03a0aa
improve and refactor async ocnnection management logic
borngraced Apr 30, 2024
98cb993
wip AccountContext
borngraced Apr 30, 2024
061d1ff
minor changes
borngraced Apr 30, 2024
398b93b
sae dev state
borngraced May 1, 2024
e2877a8
save dev state
borngraced May 1, 2024
577e18b
remove SwapsContext db_id param
borngraced May 1, 2024
9eaeb2e
fix tests
borngraced May 1, 2024
9882ccc
update ordermatch
borngraced May 1, 2024
8a2f4a7
debug db_id f or TakerOrder
borngraced May 1, 2024
4594ff6
fix ordermatch test bug and improve db_id getter
borngraced May 2, 2024
9ff52de
fix wasm test
borngraced May 2, 2024
49b9260
check recent swaps test
borngraced May 2, 2024
6db0236
debug check_my_reecent_swaps rpc
borngraced May 2, 2024
f970a12
remove debug line
borngraced May 2, 2024
094a659
update swaplock with db_id
borngraced May 3, 2024
92b94ac
cargo fmt
borngraced May 3, 2024
6d51390
sql_connection_pool
borngraced May 6, 2024
a4c27b9
fix wasm clippy and add doc comments
borngraced May 6, 2024
1c08ef5
more doc comments
borngraced May 6, 2024
a6d0b51
add hd wallet todos
borngraced May 6, 2024
35b102a
add todos for AccountContext
borngraced May 6, 2024
474d641
impl AsyncSqliteConnPool
borngraced May 6, 2024
0686d0e
code organization and improvements
borngraced May 6, 2024
ca440df
merge dev and fix conflicts
borngraced May 7, 2024
719862b
minor changes
borngraced May 7, 2024
8921e4a
merge dev and fix conflicts
borngraced May 7, 2024
6d6169f
fix hd_wallet todos
borngraced May 7, 2024
6954d62
add more todos
borngraced May 8, 2024
79e7c89
add nft unit test for multikeydb
borngraced May 9, 2024
20a4486
make mm_ctx_with_custom_async_db non wasm32 target
borngraced May 9, 2024
7a2ebaf
refactor db_lock and use tokio sync for handling lock
borngraced May 10, 2024
fdb4f55
fix hd_wallet
borngraced May 11, 2024
6c93752
fix wasm clippy 0
borngraced May 11, 2024
db397e5
fix wasm clippy
borngraced May 12, 2024
393e825
cargo fmt
borngraced May 12, 2024
6e515ae
cargo fmt
borngraced May 12, 2024
22319bc
Merge remote-tracking branch 'origin/dev' into multi-pubkey-dbs
borngraced May 14, 2024
b036c62
implement account db_id for tendermint coin
borngraced May 14, 2024
9dcb662
implement account db_id for tendermint coin
borngraced May 14, 2024
0ac682d
use coin db_id in init_db and a minor change to account_db_ids loop
borngraced May 14, 2024
c409ecc
cleanup test mod in hd_wallet/storage/mod
borngraced May 14, 2024
23bb803
drop connections as early as possible
borngraced May 14, 2024
8281609
refactor db_lock and substitute Mutex for RwLock
borngraced May 15, 2024
115a873
revert db_lock connection list lock handling to mutex
borngraced May 15, 2024
9062bee
improve my_orders_storage
borngraced May 16, 2024
b370bec
implement DbMigrationWatcher
borngraced May 16, 2024
46f8712
handle shared_db_id and minor fixes
borngraced May 17, 2024
5bde012
polish sql_connection_pool and fix review notes
borngraced May 17, 2024
e036383
cargo fmt
borngraced May 17, 2024
fab32e2
abstract db migration from tendermint activation fn and fix other rev…
borngraced May 20, 2024
789ce8d
minor changes to DbMigrationWatcher
borngraced May 20, 2024
ac5cc96
update coins_needed_for_kick_start for new db activation
borngraced May 20, 2024
d8caebd
minor fix
borngraced May 20, 2024
76efdf3
comment out gui_storage_dispatcher and related
borngraced May 21, 2024
24774a4
merge with dev and fix conflicts
borngraced May 21, 2024
6a0d0b6
comment out gui_storage_accounts_functionality unit tests
borngraced May 21, 2024
fbfe747
fix review notes
borngraced May 22, 2024
097a63e
merge with dev and fix conflicts
borngraced May 27, 2024
8a6d0cd
fix error handling
borngraced May 27, 2024
f685f72
fix hd_wallet todos
borngraced May 27, 2024
df3cf1e
remove unneeded comments
borngraced May 27, 2024
c028061
mark use_wacher as true
borngraced May 28, 2024
309a5ac
fix sqlite deadlock and improve api design
borngraced May 29, 2024
3cfd616
cargo clippy
borngraced May 29, 2024
0b5275c
remove todos after review
borngraced May 30, 2024
a53bd38
fix build error
borngraced May 30, 2024
ff42791
revert hd_wallet storage changes
borngraced Jun 4, 2024
4b99dba
impl account_db_id for eth and make account_db_id async
borngraced Jun 6, 2024
2d427d9
impl account_db_id for utxo coins
borngraced Jun 7, 2024
b516435
impl account_db_id for utxo and eth
borngraced Jun 10, 2024
497a006
fix clippy
borngraced Jun 10, 2024
df10264
fix native unit tests and other fixes
borngraced Jun 10, 2024
d32f8f6
fix clippy
borngraced Jun 10, 2024
966607d
fix bugs and improve code
borngraced Jun 11, 2024
19930d3
rename account_shared_db_id to shared_db_id
borngraced Jun 12, 2024
527efbc
use HDACcount shared_db_id for tx_history
borngraced Jun 12, 2024
bb41f41
cleanups and improve eth impl
borngraced Jun 12, 2024
916ce6a
fix clear nft todo
borngraced Jun 12, 2024
0f7c075
add docs and very minor improvs
borngraced Jun 13, 2024
bf12b01
minor changes
borngraced Jun 18, 2024
41b61c1
Merge remote-tracking branch 'origin/dev' into multi-pubkey-dbs
borngraced Jun 18, 2024
54714eb
use device rmd160 for hd_wallet storage
borngraced Jun 19, 2024
7699f55
implement account_db_id for pirate and fix othe review notes
borngraced Jun 20, 2024
e9ef701
use appropriate pubkey in UtxoCoinBuilder::build
borngraced Jun 20, 2024
1b3931b
save shared_db_id as string and use lock
borngraced Jun 24, 2024
4962351
use path_to_db_root in path_to_dbdir
borngraced Jun 24, 2024
f449cdd
make getter fns non async
borngraced Jun 24, 2024
a4ac369
fix other review notes
borngraced Jun 24, 2024
99e6ca6
improve migration watcher -handling
borngraced Jun 24, 2024
a94c791
fix TendermintKeyPair derivation and minor fixes
borngraced Jun 25, 2024
33ba505
hd wallet and some review notes fixes
borngraced Jun 26, 2024
8a01341
fix clippy
borngraced Jun 26, 2024
b19d2ba
minor improvements and fixes
borngraced Jun 27, 2024
f35f371
fix review notes
borngraced Jun 27, 2024
0856b54
merge with dev and fix clippy
borngraced Jun 28, 2024
9a1f89d
cargo fmt
borngraced Jun 28, 2024
5fdee3e
Fix swap restart issue with multi-pubkey DBs
borngraced Jul 1, 2024
9a8494e
fix review notes
borngraced Jul 1, 2024
7c12cce
revert Cargo.toml
borngraced Jul 1, 2024
0b4d808
fix shared_db_id derivation and review notes
borngraced Jul 4, 2024
90f89b7
Merge remote-tracking branch 'origin/dev' into multi-pubkey-dbs
borngraced Jul 5, 2024
8f19481
proper shared_db handling and improvements
borngraced Jul 5, 2024
e643ab3
merge and fix merge conflict
borngraced Jul 8, 2024
a78704d
fix review notes
borngraced Jul 9, 2024
bdd79af
minor fix
borngraced Jul 9, 2024
7e3d3a6
Merge remote-tracking branch 'origin/dev' into multi-pubkey-dbs
borngraced Jul 12, 2024
5470b25
fix review notes
borngraced Jul 19, 2024
d57e394
merge with dev and fix conflicts
borngraced Jul 19, 2024
b690e93
fix clippy
borngraced Jul 22, 2024
1675a53
fix review notes and remove left over
borngraced Jul 24, 2024
5e02fa4
use display_eth_address
borngraced Jul 28, 2024
5f72e83
minor changes and improvements
borngraced Jul 29, 2024
a0cf3cb
cargo fmt
borngraced Jul 29, 2024
0c817e8
Merge remote-tracking branch 'origin/dev' into multi-pubkey-dbs
borngraced Jul 30, 2024
34f9d27
allow swap continuation with correct db_id on swap restart for swap_v2
borngraced Jul 30, 2024
79f3792
improve swap_v2 swap kickstart and eth pubkey deriv
borngraced Jul 30, 2024
117aa44
Fix: Improve db_id validation for taker order processing
borngraced Aug 1, 2024
2f3086c
cleanups
borngraced Aug 2, 2024
c6b3648
merge with dev and fix conflicts
borngraced Aug 5, 2024
06ce83b
save dev state
borngraced Aug 6, 2024
eb9d74a
Merge remote-tracking branch 'origin/dev' into multi-pubkey-dbs
borngraced Aug 6, 2024
5cbfd89
fix clippy
borngraced Aug 6, 2024
cea464b
fix orders_kick_start
borngraced Aug 7, 2024
07a478c
fix minor review notes
borngraced Aug 7, 2024
342fab8
Merge remote-tracking branch 'origin/dev' into multi-pubkey-dbs
shamardy Sep 14, 2024
07177cf
post merge formating
shamardy Sep 14, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions mm2src/coins/eth/v2_activation.rs
Original file line number Diff line number Diff line change
Expand Up @@ -742,7 +742,9 @@ pub(crate) async fn build_address_and_priv_key_policy(

#[cfg(not(target_arch = "wasm32"))]
{
let pubkey = dhash160(activated_key.public().as_bytes()).to_string();
// Skip the first byte of the uncompressed public key before converting to the eth address.
let pubkey = Public::from_slice(&activated_key.public().as_bytes()[1..]);
let pubkey = public_to_address(&pubkey).to_string();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here

run_db_migration_for_new_pubkey(ctx, pubkey)
.await
.map_to_mm(EthActivationV2Error::InternalError)?;
Expand Down Expand Up @@ -1013,10 +1015,11 @@ pub(super) async fn eth_shared_db_id(coin: &EthCoin, ctx: &MmArc) -> Option<Stri

pub(super) async fn eth_account_db_id(coin: &EthCoin) -> Option<String> {
match coin.derivation_method() {
DerivationMethod::HDWallet(hd_wallet) => hd_wallet
.get_enabled_address()
.await
.map(|addr| dhash160(addr.pubkey().as_bytes()).to_string()),
DerivationMethod::HDWallet(hd_wallet) => hd_wallet.get_enabled_address().await.map(|addr| {
// Skip the first byte of the uncompressed public key before converting to the eth address.
let pubkey = Public::from_slice(&addr.pubkey().as_bytes()[1..]);
public_to_address(&pubkey).to_string()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you should use display_eth_address function and strip the 0x at the start

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you should use display_eth_address function and strip the 0x at the start

I think the below would be much simpler than what you implemented

let address = display_eth_address(&addr.address());
address.trim_start_matches("0x").to_string()

}),
_ => None,
}
}
18 changes: 3 additions & 15 deletions mm2src/mm2_db/src/indexed_db/indexed_db.rs
Original file line number Diff line number Diff line change
Expand Up @@ -111,28 +111,20 @@ impl DbIdentifier {
}

pub fn display_db_id(&self) -> String { self.db_id.clone().unwrap_or_else(|| "KOMODEFI".to_string()) }

pub fn db_id(&self) -> String {
self.db_id
.clone()
.unwrap_or_else(|| hex::encode(H160::default().as_slice()))
}
}

pub struct IndexedDbBuilder {
pub db_name: String,
pub db_version: u32,
pub tables: HashMap<String, OnUpgradeNeededCb>,
pub db_id: String,
}

impl IndexedDbBuilder {
pub fn new(db_id: DbIdentifier) -> IndexedDbBuilder {
pub fn new(db_ident: DbIdentifier) -> IndexedDbBuilder {
IndexedDbBuilder {
db_name: db_id.to_string(),
db_name: db_ident.to_string(),
db_version: 1,
tables: HashMap::new(),
db_id: db_id.db_id(),
}
}

Expand All @@ -148,13 +140,12 @@ impl IndexedDbBuilder {
}

pub async fn build(self) -> InitDbResult<IndexedDb> {
let db_id = self.db_id.clone();
let (init_tx, init_rx) = oneshot::channel();
let (event_tx, event_rx) = mpsc::unbounded();

self.init_and_spawn(init_tx, event_rx);
init_rx.await.expect("The init channel must not be closed")?;
Ok(IndexedDb { event_tx, db_id })
Ok(IndexedDb { event_tx })
}

fn init_and_spawn(
Expand Down Expand Up @@ -190,7 +181,6 @@ impl IndexedDbBuilder {

pub struct IndexedDb {
event_tx: DbEventTx,
db_id: String,
}

async fn send_event_recv_response<Event, Item, Error>(
Expand Down Expand Up @@ -248,8 +238,6 @@ impl IndexedDb {
// ignore if the receiver is closed
result_tx.send(Ok(transaction_event_tx)).ok();
}

pub fn get_db_id(&self) -> String { self.db_id.to_string() }
}

pub struct DbTransaction<'transaction> {
Expand Down
15 changes: 12 additions & 3 deletions mm2src/mm2_main/src/lp_native_dex.rs
Original file line number Diff line number Diff line change
Expand Up @@ -460,21 +460,21 @@ async fn init_db_migration_watcher_loop(ctx: MmArc) {
let mut migrations = HashSet::new();
let mut receiver = ctx
.init_db_migration_watcher()
.expect("db_m igration_watcher initialization failed");
.expect("db_migration_watcher initialization failed");

while let Some(db_id) = receiver.next().await {
if migrations.contains(&db_id) {
debug!("{} migrated, skipping migration..", db_id);
continue;
}

// run db migration for db_id if new activated pubkey is unique.
// run db migration for new db_id.
if let Err(err) = run_db_migration_impl(&ctx, Some(&db_id), None).await {
error!("{err:?}");
continue;
};

// insert new db_id to migration list
// insert new db_id to migrated list
migrations.insert(db_id.to_owned());

// Fetch and extend ctx.coins_needed_for_kick_start from new intialized db.
Expand All @@ -500,6 +500,15 @@ async fn run_db_migration_impl(ctx: &MmArc, db_id: Option<&str>, shared_db_id: O
}

pub async fn lp_init_continue(ctx: MmArc) -> MmInitResult<()> {
#[cfg(not(target_arch = "wasm32"))]
{
let dbdir = ctx.dbdir(None);
fs::create_dir_all(&dbdir).map_to_mm(|e| MmInitError::ErrorCreatingDbDir {
path: dbdir.clone(),
error: e.to_string(),
})?;
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be before initialize_wallet_passphrase as it was done before

// This either initializes the cryptographic context or sets up the context for "no login mode".
initialize_wallet_passphrase(&ctx).await?;

You don't need to create the default db dir, only the global one. We probably also need to ensure_dir_is_writable, just in case.


init_ordermatch_context(&ctx)?;
init_p2p(ctx.clone()).await?;

Expand Down
8 changes: 8 additions & 0 deletions mm2src/mm2_main/src/lp_ordermatch.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5428,6 +5428,10 @@ pub async fn orders_kick_start(ctx: &MmArc, db_id: Option<&str>) -> Result<HashS
{
let mut maker_orders_ctx = ordermatch_ctx.maker_orders_ctx.lock();
for order in saved_maker_orders {
// we need to only kickstart orders if the coin is activated with the same db_id as the order's db_id.
if order.db_id().as_deref() != db_id {
continue;
}
coins.insert(order.base.clone());
coins.insert(order.rel.clone());
maker_orders_ctx.add_order(ctx.weak(), order.clone(), None);
Expand All @@ -5436,6 +5440,10 @@ pub async fn orders_kick_start(ctx: &MmArc, db_id: Option<&str>) -> Result<HashS

let mut taker_orders = ordermatch_ctx.my_taker_orders.lock().await;
for order in saved_taker_orders {
// we need to only kickstart orders if the coin is activated with the same db_id as the order's db_id.
if order.db_id().as_deref() != db_id {
continue;
}
coins.insert(order.request.base.clone());
coins.insert(order.request.rel.clone());
taker_orders.insert(order.request.uuid, order);
Expand Down
Loading