-
Notifications
You must be signed in to change notification settings - Fork 40
feat: prototype implementation of SqliteStorage #549
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A few high-level thoughts to get us started on this...
Each SqliteStorage creates a background thread (well, technically SqliteStore does this) and SqliteClient reused across all shared SqliteStore instances. We could key the background thread by (absolute) path such that multiple invocations of Storage also reuse the same worker thread.
- We probably have to be careful to use actual threads here unless SQLite bindings give us guarantees about blocking behavior. If SQLite blocks on a thread where
tokio
doesn't expect blocking work to happen, we could end up with scheduling deadlocks. Remember: SQLite cannot cooperate with thetokio
runtime where thread pool management / scheduling is concerned. - I'm not sure that handling an internal thread for reads/writes is the right approach here. It seems like we can just as easily assume "do work on current thread," and then wrap the storage in a generic "operations occur on dedicated thread" thing.
If we have a "blocking" storage implementation, the level of abstraction where I would like to deal with that is outside of the storage implementation itself. That would give us more control over scheduling, and we could circumstantially decide whether and how to share threads. |
a7869f4
to
35b0ca8
Compare
Updated the implementation to now use a connection pool (
|
@@ -98,7 +98,7 @@ jobs: | |||
- name: 'Run Rust native target tests' | |||
run: NOOSPHERE_LOG=deafening cargo test --features test-kubo,headers | |||
|
|||
run-test-suite-linux-rocksdb: | |||
run-test-suite-linux-sqlite: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Temporary
MVP of a sqlite-backed Storage provider.
cargo test --features sqlite
. Test timing increased from 27s to 42s (for me).message_channel
functionality fromnoosphere_ns
. Moved this intonoosphere_core
as general utility, but as storage is a core dependency, no go. Copied over for now. Maybe a no-noosphere-dependency crate like noosphere-utils for something like this.