Archive node feature #2338
Archive node feature #2338
Conversation
- Add archive feature flag to rusk (depends on node) - Add ArchivalData enum to node-data - Add VM event MPSC channel for archival to Rusk - Add with_archivist to RuskNodeBuilder - Split up build and run for RuskNodeBuilder - Add ArchivistSrv to service list - Add Archivist trait to node - Add archive_sender to Rusk & send VM events in accept_transactions - Add SQLiteArchive struct to Node - Add optional sqlx, serde_json dep & archive feature to node - Add migration file with archive db schema - Add an .env file to get sqlx-cli & macros to work - Add SQLiteArchive functionalities & tests - Use hash type alias for [u8; 32] - Add serialization support & tests to node_data ContractEvent - Update gitignore (.vscode) - fmt & todo comments - Update README with archive run info
Neotamandua
force-pushed
the
archive_node
branch
from
07dee9f
to
ef12ae1
Compare
Neotamandua
force-pushed
the
archive_node
branch
from
ef12ae1
to
0c83bf6
Compare
- Fix shutdown problem & get rid of Mutex - Move ContractTxEvent to node-data - Combine ContractTxEvent with ContractEvent - Skip origin serialization if none - Let sqlx queries compile without live db connection - Update README
Neotamandua
force-pushed
the
archive_node
branch
from
0c83bf6
to
13353d5
Compare
Neotamandua
requested review from
ureeves,
goshawk-3,
herr-seppia and
miloszm
There was a problem hiding this comment.
I'd prefer the code not to be so heavily #IFDEF'ed, #[cfg(feature="archive")] is used 22 times. I'd rather limit their usage to module inclusion and some constants, but not for, e.g., structs and method invocations, as in the long run such code is hard to maintain, read and test. Normal in-language optionality could be used in such places instead, so that we still have a common code base, only differing in some modules being included (or not) and in some constants.
Thanks for the input and I agree. I have taken note of this for the next related task, which aims to separate archivers from provisioners. This next task will change the structure again, so it's a good idea imo to tackle it there. |
Merge rusk http ContractEvent with node-data ContractEvent
| /// - The block does not store the events, only the hash of the events. The | ||
| /// archivist will store the events. | ||
| #[cfg(feature = "archive")] | ||
| pub(crate) trait Archivist { |
There was a problem hiding this comment.
if its a trait that's just being used inside the crate and not exported, don't you think it will be more readable with less abstraction and more functional. This is just design philosophy but traits are understandable if they are exported (we're also trying to reduce the number of traits in the wallets from Store/State moving towards functional design)
This PR introduces the "archive" feature flag to rusk and node. Another PR would build on top of this and introduce serving them through an http api.
When run with:
DUSK_CONSENSUS_KEYS_PASS=password cargo r --release --features archive -p rusk -- -s /tmp/example.state, the archive feature is active and starts archiving contract events from the vm into an SQLite database.This was done by adding a new service, ArchivistSrv, to the node's service_list. This service listens on a channel that receives the VM events from a sender that sends them during the execution of the
accept_transactionsfunction of the Rusk struct (the same place where VM events are sent to RUES).The ArchivistSrv, upon receiving new VM events, calls the SQLiteArchive (which manages a SQLite db) and stores all
ContractTxEventevents from a block in a json serialized column.Considerations:
Merge ContractEvent in rusk http lib with the one in node-dataPRAGMA journal_mode=WAL;)cargo install sqlx-cli && cargo sqlx prepare --check -- --all-targets --all-featuresfor the queries