Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: TipRouter CLI separate stages #87

Merged
merged 124 commits into from
Mar 11, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
124 commits
Select commit Hold shift + click to select a range
fa58c2f
Prep for testing
ebatsell Jan 6, 2025
4979178
Debug lines
ebatsell Jan 7, 2025
f81018f
snapshot-slot
ebatsell Jan 7, 2025
3988534
...
ebatsell Jan 7, 2025
42dd84a
print out better errors
ebatsell Jan 8, 2025
0497a4f
Desired slot in blcokstore
ebatsell Jan 8, 2025
8fc24f3
rm div_ceil
ebatsell Jan 8, 2025
f2cd56b
...
ebatsell Jan 8, 2025
a5911df
more debug
ebatsell Jan 8, 2025
6c952c1
check blocks
ebatsell Jan 8, 2025
f4171a6
fix underflow bug
ebatsell Jan 9, 2025
22c7d6a
Prep for real epoch rollover
ebatsell Jan 12, 2025
2342107
Revert "Prep for real epoch rollover"
ebatsell Jan 13, 2025
0004501
Remove debug code
ebatsell Jan 13, 2025
1d0cd74
more cleanup
ebatsell Jan 14, 2025
e633aef
docker
ebatsell Jan 14, 2025
9831f17
.....
ebatsell Jan 14, 2025
06a9fee
Nits
ebatsell Jan 16, 2025
eda1355
Merge branch 'master' into lite-operator
ebatsell Jan 17, 2025
181bb6b
Update instruction
ebatsell Jan 17, 2025
b0b0c07
[draft] Operator Improvements (#65)
ebatsell Jan 22, 2025
10f8d99
Remove incremental from backup not main dir
ebatsell Jan 23, 2025
83252da
Reduce logs
ebatsell Jan 23, 2025
d5a1e87
WIP copy pasta a whole bunch of stuff for mev tip claim
tomjohn1028 Jan 23, 2025
6b78aa9
add claim command to CLI
tomjohn1028 Jan 23, 2025
4574cd7
update todo
tomjohn1028 Jan 23, 2025
c952640
update other todo
tomjohn1028 Jan 23, 2025
46cff84
fix bug and tests
tomjohn1028 Jan 24, 2025
80fa8d9
Smart backup (#71)
ebatsell Jan 24, 2025
8eda3aa
Include telegraf stats
ebatsell Jan 24, 2025
4531cb0
WIP: gut the snapshot slot command
tomjohn1028 Jan 24, 2025
7976774
Reduce iteration of submit to ncn
ebatsell Jan 24, 2025
85d1d5b
debug
ebatsell Jan 24, 2025
fae6f6d
WIP testing some stuff
tomjohn1028 Jan 24, 2025
ffdced7
cut initial memo
tomjohn1028 Jan 27, 2025
fe39a97
copy over ledger utils fn
ebatsell Jan 27, 2025
787b911
Link to main process
ebatsell Jan 27, 2025
26dc401
Revert "copy over ledger utils fn"
ebatsell Jan 27, 2025
87d85af
exit bool
ebatsell Jan 28, 2025
67e2a94
rm unneeded
ebatsell Jan 28, 2025
b30a469
Attempt with ledger tool and drop
ebatsell Jan 28, 2025
c4d38bd
Merge branch 'lite-operator' into add-mev-claim
tomjohn1028 Jan 28, 2025
667dd3c
Merge branch 'lite-operator' into snapshot-stage
tomjohn1028 Jan 28, 2025
823c953
add datapoints to other error checks
tomjohn1028 Jan 28, 2025
dff4559
fmt
tomjohn1028 Jan 28, 2025
9543af9
break down other stages
tomjohn1028 Jan 28, 2025
b7d97b0
add create stake meta command
tomjohn1028 Jan 28, 2025
6f200bd
Merge branch 'add-mev-claim' into snapshot-stage
tomjohn1028 Jan 28, 2025
c038f0f
add create merkle tree collection command
tomjohn1028 Jan 28, 2025
69e4893
add create meta merkle command
tomjohn1028 Jan 28, 2025
798a0e0
WIP transition loop
tomjohn1028 Jan 29, 2025
a78e6d4
debugging
ebatsell Jan 29, 2025
b51ee67
last fixes
ebatsell Jan 29, 2025
14bc1e3
rm logs
ebatsell Jan 29, 2025
2513f32
last cleanup
ebatsell Jan 29, 2025
b66f838
Build
ebatsell Jan 29, 2025
1df78dd
final
ebatsell Jan 29, 2025
70429bf
final final
ebatsell Jan 29, 2025
d93a82a
Merge branch 'eb/load-ledger-improvements' into lite-operator
ebatsell Jan 29, 2025
118092a
Merge conflicts
ebatsell Jan 29, 2025
4c4ad79
Merge branch 'lite-operator' into add-mev-claim
tomjohn1028 Jan 30, 2025
8aa6515
Merge branch 'add-mev-claim' into snapshot-stage
tomjohn1028 Jan 30, 2025
c752801
add control flow for epoch handling
tomjohn1028 Jan 30, 2025
922d000
handle loading from file for stake meta and merkle tree collection
tomjohn1028 Jan 30, 2025
45a1d83
fmt and wait for optimal incremental snapshot
tomjohn1028 Jan 30, 2025
8772714
fix naming
tomjohn1028 Feb 4, 2025
2dcabc4
create save path if not exists
tomjohn1028 Feb 4, 2025
8dc3ed1
no snapshot when creating stake meta
tomjohn1028 Feb 4, 2025
38fbdc7
WIP dependencies map, fixing upgrade errors
tomjohn1028 Feb 6, 2025
e329316
WIP fixing compiler errors
tomjohn1028 Feb 6, 2025
bc4b851
fix some more stuff
tomjohn1028 Feb 6, 2025
dc09a4d
cargo build works
tomjohn1028 Feb 6, 2025
3222f27
tests compile
tomjohn1028 Feb 6, 2025
09ae852
oops..now tests compile
tomjohn1028 Feb 6, 2025
07b506e
fix incorrect TokenOwner error
tomjohn1028 Feb 6, 2025
def719c
fix non-zero initialize vault amount
tomjohn1028 Feb 6, 2025
fc9bb2e
fix failing tests
tomjohn1028 Feb 6, 2025
885eeae
pin restaking to git branch
tomjohn1028 Feb 6, 2025
274bb0c
cargo fmt
tomjohn1028 Feb 7, 2025
ceb9984
update workflow solana version
tomjohn1028 Feb 7, 2025
e1018fc
update kinobi generated client
tomjohn1028 Feb 7, 2025
2e32acb
use patched kinobi packages to support anchor 0.31.x
tomjohn1028 Feb 7, 2025
c98dfcb
remove unnecessary patched crates
tomjohn1028 Feb 7, 2025
7fc8a69
debugging CI
tomjohn1028 Feb 7, 2025
6c43ee0
remove debugging log
tomjohn1028 Feb 11, 2025
fac13ef
Merge branch 'master' into upgrade-v2.1.x
tomjohn1028 Feb 11, 2025
9f6f72e
[hopefully] fix ci issues after merging master
tomjohn1028 Feb 11, 2025
8f88948
update dependencies and do some cleanup; still failing vault program …
tomjohn1028 Feb 11, 2025
7552096
update vault program
tomjohn1028 Feb 11, 2025
8c3d08d
remove comment
tomjohn1028 Feb 11, 2025
ceb5886
Merge branch 'master' into upgrade-v2.1.x
tomjohn1028 Feb 12, 2025
2ebf89d
fix bug from merge
tomjohn1028 Feb 12, 2025
4698e44
Merge branch 'upgrade-v2.1.x' into snapshot-stage
tomjohn1028 Feb 12, 2025
eccf6fa
move stage transition loop to process_epoch
tomjohn1028 Feb 12, 2025
85a52f6
load bank from a full snapshot for a given slot
tomjohn1028 Feb 12, 2025
d58b148
clean up lil bit
tomjohn1028 Feb 12, 2025
e7fdd01
update save file naming and remove unnecessary merkle_root_dir arg in…
tomjohn1028 Feb 13, 2025
b857941
handle clippy errors
tomjohn1028 Feb 13, 2025
f1373aa
add the CastVote stage to the transition loop
tomjohn1028 Feb 13, 2025
ce29469
clean up stuff
tomjohn1028 Feb 13, 2025
55503a5
Merge branch 'master' into separate-stages
tomjohn1028 Feb 13, 2025
a454164
update some stuff on review
tomjohn1028 Feb 15, 2025
1e22155
fix command file path bugs
tomjohn1028 Feb 17, 2025
6119999
minor tweaks
tomjohn1028 Feb 17, 2025
f5acc56
make sure backup_snapshots_dir is different from full_snapshots_path
tomjohn1028 Feb 25, 2025
f300fe1
add meta-merkle-tree-dir as save_path alias
tomjohn1028 Feb 28, 2025
ecdab29
load fees from NCN config
tomjohn1028 Feb 28, 2025
d0e3ebc
change enable-snapshots to save-snapshot
tomjohn1028 Feb 28, 2025
336e47c
stop fighting the validators snapshot directory
tomjohn1028 Feb 28, 2025
e694fcc
assert monitored epochs > 0; evict full snapshots from backup dir
tomjohn1028 Feb 28, 2025
8941a29
evict saved TipRouter files when epoch changes
tomjohn1028 Mar 1, 2025
b3dca7b
remove prints
tomjohn1028 Mar 1, 2025
cae65e5
run format
tomjohn1028 Mar 1, 2025
a1c33dc
fix protocol fees; remove old todos
tomjohn1028 Mar 4, 2025
8b82d70
format
tomjohn1028 Mar 4, 2025
74b252a
fix ncn protocol fees
tomjohn1028 Mar 4, 2025
fa08b73
fix bug in evict_same_epoch_incremental
tomjohn1028 Mar 4, 2025
45b96b6
remove todo comment anf format
tomjohn1028 Mar 4, 2025
264f707
Merge branch 'master' into separate-stages
ebatsell Mar 6, 2025
8b03dfe
Attempt faster tx submission
ebatsell Mar 8, 2025
03840a4
Start claiming immediately
ebatsell Mar 8, 2025
dfd38ee
Don't copy full snaps
ebatsell Mar 9, 2025
4be55c1
shuffle
ebatsell Mar 11, 2025
0a6ed06
to_vec
ebatsell Mar 11, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -199,7 +199,7 @@ jobs:
name: jito_tip_router_program.so
path: integration_tests/tests/fixtures/
- uses: taiki-e/install-action@nextest
- run: cargo nextest run --all-features -E 'not test(ledger_utils::tests::test_get_bank_from_ledger_success) and not test(test_meta_merkle_creation_from_ledger)'
- run: cargo nextest run --all-features -E 'not test(ledger_utils::tests)'
env:
SBF_OUT_DIR: ${{ github.workspace }}/integration_tests/tests/fixtures

Expand Down
1 change: 0 additions & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion format.sh
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ cargo fmt --all

print_executing "cargo nextest run --all-features"
cargo build-sbf --sbf-out-dir integration_tests/tests/fixtures
SBF_OUT_DIR=integration_tests/tests/fixtures cargo nextest run --all-features -E 'not test(ledger_utils::tests::test_get_bank_from_ledger_success) and not test(test_meta_merkle_creation_from_ledger)'
SBF_OUT_DIR=integration_tests/tests/fixtures cargo nextest run --all-features -E 'not test(ledger_utils::tests::test_get_bank_from_ledger_success)'

# Code coverage only runs with flag
if [[ "$*" == *"--code-coverage"* ]]; then
Expand Down
32 changes: 31 additions & 1 deletion meta_merkle_tree/src/generated_merkle_tree.rs
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
use std::{fs::File, io::BufReader, path::PathBuf};
use std::{
fs::File,
io::{BufReader, Write},
path::PathBuf,
};

use jito_tip_distribution_sdk::{
jito_tip_distribution::ID as TIP_DISTRIBUTION_ID, CLAIM_STATUS_SEED,
Expand Down Expand Up @@ -119,6 +123,14 @@ impl GeneratedMerkleTreeCollection {

Ok(tree)
}

/// Write a GeneratedMerkleTreeCollection to a filepath
pub fn write_to_file(&self, path: &PathBuf) -> Result<(), MerkleRootGeneratorError> {
let serialized = serde_json::to_string_pretty(&self)?;
let mut file = File::create(path)?;
file.write_all(serialized.as_bytes())?;
Ok(())
}
}

#[derive(Clone, Eq, Debug, Hash, PartialEq, Deserialize, Serialize)]
Expand Down Expand Up @@ -329,6 +341,24 @@ pub struct StakeMetaCollection {
pub slot: Slot,
}

impl StakeMetaCollection {
/// Load a serialized merkle tree from file path
pub fn new_from_file(path: &PathBuf) -> Result<Self, MerkleRootGeneratorError> {
let file = File::open(path)?;
let reader = BufReader::new(file);
let tree: Self = serde_json::from_reader(reader)?;

Ok(tree)
}

/// Write a merkle tree to a filepath
pub fn write_to_file(&self, path: &PathBuf) {
let serialized = serde_json::to_string_pretty(&self).unwrap();
let mut file = File::create(path).unwrap();
file.write_all(serialized.as_bytes()).unwrap();
}
}

#[derive(Clone, Deserialize, Serialize, Debug, PartialEq, Eq)]
pub struct StakeMeta {
#[serde(with = "pubkey_string_conversion")]
Expand Down
11 changes: 6 additions & 5 deletions meta_merkle_tree/src/meta_merkle_tree.rs
Original file line number Diff line number Diff line change
Expand Up @@ -90,10 +90,11 @@ impl MetaMerkleTree {
}

/// Write a merkle tree to a filepath
pub fn write_to_file(&self, path: &PathBuf) {
let serialized = serde_json::to_string_pretty(&self).unwrap();
let mut file = File::create(path).unwrap();
file.write_all(serialized.as_bytes()).unwrap();
pub fn write_to_file(&self, path: &PathBuf) -> Result<()> {
let serialized = serde_json::to_string_pretty(&self)?;
let mut file = File::create(path)?;
file.write_all(serialized.as_bytes())?;
Ok(())
}

pub fn get_node(&self, tip_distribution_account: &Pubkey) -> TreeNode {
Expand Down Expand Up @@ -241,7 +242,7 @@ mod tests {
let path = PathBuf::from("merkle_tree.json");

// serialize merkle distributor to file
merkle_distributor_info.write_to_file(&path);
merkle_distributor_info.write_to_file(&path).unwrap();
// now test we can successfully read from file
let merkle_distributor_read: MetaMerkleTree = MetaMerkleTree::new_from_file(&path).unwrap();

Expand Down
1 change: 0 additions & 1 deletion tip-router-operator-cli/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@ hex = "0.4"
im = "15.1"
itertools = "0.11"
jito-bytemuck = { workspace = true }
jito-restaking-program = { workspace = true }
jito-tip-distribution-sdk = { workspace = true }
jito-tip-payment-sdk = { workspace = true }
jito-tip-router-client = { workspace = true }
Expand Down
Loading
Loading