[RFC] API for IndexedDB

[c410-f3r]

Summary

This RFC tries to establish an API for IndexedDB

Motivation

gloo doesn't have an API for IndexedDB, a fundamental technology for asynchronous persistent storage.

Detailed Explanation

Based on the discussions that took place in #3 and the now recently closed #59, I propose an "rust-ish" 1x1 implementation of the idb project that has a better alignment with the goals of the gloo team.

Two APIs are going to provided, a more complex and feature-complete IndexedDbFull and a simpler, quicker and easier IndexedDb that builds on top of IndexedDbFull. This way, the user has the option to choose whatever is best for his use-case.

IndexedDbFull and relatables

impl IndexedDbFull {
    pub fn delete_db(&self, name: &str) {}
    pub fn open(&mut self, database_name: &str, version: f64) -> IndexedDbFullVersionManager {}
    pub fn store(&self, name: &str) -> IndexedDbFullStorageManager;
}

impl IndexedDbFullStoreManager {
    pub fn all_keys(&self) {}
    pub fn clear(&self) { }
    pub fn delete(&self, key: &str) {}
    pub fn get(&self, key: &str) {}
    pub fn set(&self, key: &str, value: &str) {}
    pub fn transaction(&self, mode: &str) {}
}

impl IndexedDbFullVersionManager {
    pub fn upgrade(&self, cb: F) {}
    pub fn blocked(&self, cb: F) {}
    pub fn blocking(&self, cb: F) {}
}

let idb = IndexedDbFull::default();
let v = idb.open("foo-db", 1.0);
v.upgrade(|old, new, transaction| ... );
let s = idb.store("foo-store");
s.set("foo-key", "foo-value");

IndexedDb

Has a single hard-coded database, version and store.

// This is just a demonstration. `IndexedDb` and `IndexedDbFullStoreManager`
// are using the same trait definition for its methods.
impl IndexedDb {
    pub fn all_keys(&self) {}
    pub fn clear(&self) {}
    pub fn delete(&self, key: &str) {}
    pub fn get(&self, key: &str) {}
    pub fn set(&self, key: &str, value: &str) {}
}

let idb = IndexedDb::default();
s.set("foo-key", "foo-value");

The two APIs are using callbacks and their Future and Stream interfaces will have a very similar, if not equal, definition.

Drawbacks, Rationale, and Alternatives

The API focuses on simplicity but there are more complex projects like PouchDB and dexie.js.

Unresolved Questions

Should IndexedDbFullStoreManager, IndexedDbFullVersionManager and IndexedDb use the Fluent Interface pattern?

[yoshuawuyts]

@c410-f3r It's not clear to me why two separate interfaces are proposed. Could you share more about your thinking here?

I have an additional question: should we guarantee that handles onto the database are unique? E.g. if a particular table is opened somewhere, then any subsequent calls to open the same table fail. I haven't completely thought this through yet, but it feels like at this abstraction layer that might help save some (accidental) concurrency problems.

I guess this would in particular be applicable to the IndexedDb interface in the proposal. It doesn't have a constructor other than Default which means that when two handles to it are opened without realizing they're not unique, the underlying data might be mixed and arbitrarily overwritten, leading to database corruption.

[c410-f3r]

@yoshuawuyts

@c410-f3r It's not clear to me why two separate interfaces are proposed. Could you share more about your thinking here?

Previous discussions showed a desire for a simple key/value API but that would effectively limit the user's freedom, preventing him to use some built-in features like version. These two APIs are an attempt to reach a middle ground.

I have an additional question: should we guarantee that handles onto the database are unique?

Nice question. IndexeDB deals with these concurrency problems by versioning the database and for the IndexedDbAPI, we can generate a new version for each modification/transaction (clear, delete, set) or simply block the user request using the onblocked event.

[yoshuawuyts]

@c410-f3r I still not entirely convinced this would be the right way to split the API. Instead I feel we could likely have the best of both worlds through a single API with different constructors. To illustrate:

Generalized API

// Open a database
let db = await? Idb::open("cat_db", 1);
let db = await? Idb::builder("cat_db", 1)
    .on_upgrade(|&db, diff_data| /* handle upgrade code here */)
    .build();

// Open a "store" on a database (idb equivalent of a table)
let store = await? Idb::open_store("cat_db", 1, "cats");
let store = await? db.store("cats");
let store = await? db.store_builder("cats")
    .key_path("id")
    .auto_increment(true)
    .build();

// set, get, del a value.
await? store.set("cat", "chashu");
let val = await? store.get("cat");
assert_eq(val, "chashu");
await? store.delete("cat");

"Convenient" Subset

let store = await? Idb::open_store("my_db", 1, "cats");
await? store.set("cat", "chashu");
let val = await? store.get("cat");
await? store.delete("cat");

note: because of the nature of idb, all of these methods have to be async. I believe going with futures over callbacks is the right approach here.

The convenient subset would allow people to store data rather easily, just by exposing a few convenience functions. This would also allow us to sidestep any type problems about code that wants to be able to take either type of database -- which would seem like could definitely become a thing if someone ever moves from the simple API to the full API because they need more features.

---

Something else I just thought of: idb doesn't accept versions under 1. We could possibly leverage NonZeroUsize for that. With the tradeoff that floats can no longer be used as the version number, which I have a gut feeling about unlikely to be a problem for practical use.

[c410-f3r]

@yoshuawuyts Great idea! This unifies both APIs in a graceful way.
Just an opinion. To avoid user input, it is even possible to provide methods like:

let _ = await? Idb::open_with_automatic_versioning("cat_db");
let _ = await? Idb::open_store_with_latest_version("cat_db", "cats");
let _ = await? Idb::open_fixed_store_with_latest_version("cat_db");

Something else I just thought of: idb doesn't accept versions under 1. We could possibly leverage NonZeroUsize for that

IdbOpenDbOptions requires f64 and with NonZeroUsize, it would be necessary to convert NonZeroUsize -> usize and then cast usize -> f64. I think an assert!(v >= 1.0) should be enough.

[richard-uk1]

One small comment:

The version should be a whole number > 0. The reason it is f64 in web-sys is that all numbers in javascript are f64. This has a consequence that numbers above a certain value are not possible to represent (as f64). This number is exposed as Number.MAX_SAFE_INTEGER in javascript. For this reason, I would use NonZeroU32 (u32::max_value() < Number.MAX_SAFE_INTEGER). If you expose an f64 parameter, you're just giving the library user a footgun - I'm pretty sure fractional versions get rounded through some crazy scheme (and I have no idea what happens for NaN).

Related reading: w3c/IndexedDB#147

[richard-uk1]

Indexedb is hardddddddd!

I've been trying to make progress on a rust-y wrapper, but here are so many moving parts that it's really hard to come up with a good api. For example: https://www.w3.org/TR/IndexedDB-2/#handling-versionchange shows how to reload the page when the same origin on a different browser page requests an upgrade and you have a database open.

I can think of two ways to make progress, either trying to model indexeddb as a state machine and working out what different states it can be in, or trying to list all the things a user might want to do with it.

If category theory taught us anything, it is that the way to handle a hard problem is to decompose it into smaller simpler pieces and then re-assemble them. But I'm not sure if you can do that with the indexeddb api because all parts of it interact.

[c410-f3r]

This thread will be closed since I personally don't have the motivation or time to continue pursuing this API.

Feel free to create another proposal or re-open this issue if needed. Good luck!

[devashishdxt]

I've created a futures based wrapper around indexed db: https://crates.io/crates/rexie

Feedback on API is welcome.