CRDT extension

[gedw99]

I have been playing with this package for a while and noticed that we can run this in a browser or a server.

So I was thinking how cool it might be to have basic CRDT extension. like what they do with Corrosion. This would allow multi master servers with CRDT doing the sync, and catchup.

It would also allow Browsers to do offline editing and then when they go back online they will sync with any of the Servers they find.

https://github.com/superfly/corrosion does this using rust and sqlite.
The HTTP API is very simple:
https://superfly.github.io/corrosion/api/index.html

For example the change feed on a SQL Subscription looks a bit like:

{ "columns": ["sandwich"] }
{ "row":     [1, ["shiitake"]] }
{ "row":     [2, ["ham"]] }
{ "row":     [3, ["grilled cheese"]] }
{ "row":     [4, ["brie and cranberry"]] }
{ "eoq":     { "time": 8e-8, "change_id": 0 } }
{ "change":  ["update", 2, ["smoked meat"], 1] }
{ "change":  ["update", 1, ["smoked meat"], 2] }
{ "change":  ["update", 2, ["ham"], 3] }
{ "change":  ["update", 1, ["burger"], 4] }
{ "change":  ["update", 2, ["smoked meat"], 5] }
// ...

Tables are marked as CRDT ones.
https://superfly.github.io/corrosion/crdts.html

This has some similarities to how Marmot works, which also tracks Tables, however Marmot using NATS and its RAFT to do the sync. https://github.com/maxpert/marmot

https://maxpert.github.io/marmot/internals

https://github.com/maxpert/marmot/blob/master/db/table_change_log_script.tmpl

Marmot cant do runtime DB schema updates. I believe that Corrosion can...

Corrosion only uses CRDT and QUIC to all other DBS. I think it uses Consul to discover the other DB's.

.

[ncruces]

This runs in the browser, but tbh it's not great.

You'll be running SQLite compiled to Wasm, inside the wazero Wasm interpreter, inside Go compiled to Wasm, inside the browsers Wasm VM.

It's nice that it runs as it proves portability of the project, which in my experience helps find problems early, so I actually do try to build it regularly in that and other weird configurations. But it will be slow and far from ideal.

Also, see: #15

[gedw99]

Yes I understand what you mean about browser usage not being the goal and also the perf implications. I have personally found it worthwhile because I can move all computation to the browser when I need isomorphic setup.

I will put up a full demo in a repo soon if anyone is curious.

—-

I have been working on database change feeds on subscription queries . I have been using corrosion as I mentioned.

I am playing with https://github.com/ebuckley/cr-sqlite-go

this is golang. It’s probably that it layers out a path for an extension into go-sqlite3.

Its using the SQLite extension from https://github.com/vlcn-io/js

[gedw99]

Hey @ncruces

I am doing web workers alot for golang . In browser.

it got me thinking about how useful a SQLite WASM would be , acting as a cs he, not for offline.

I noticed that https://github.com/DallasHoff/sqlocal does pretty much this, and is using SQLite-WASM , just like you , but with a different driver . It’s using a VFS called origin private file system.

https://developer.mozilla.org/en-US/docs/Web/API/File_System_API#origin_private_file_system

It would be pretty nice if we could have this for golang .

https://www.notion.com/blog/how-we-sped-up-notion-in-the-browser-with-wasm-sqlite

The use case would be that the WASM holds the rendering code and a cache of the data .
The server holds the exact same code and has the real data in SQLite.
Both codes call SQLite .
The browser code has a “ on cache miss” logical check and gets it from the server if needed .

so each users data in the browser is different and grows based on usage . The intent is not to do offline , so a mutation is forwarded to the server and the local caches version is deleted , forcing a get from the server.
I’m sure the logic is too glossed over here. But the intent at least is clear.

[ncruces]

I'll restate what I said above: compiling the “wazero Wasm interpreter” to Wasm and running it in “another Wasm VM” (browser) works, but makes very little sense. It'll be many megabytes of Wasm (including embed Wasm) to run Wasm in Wasm.

You should be using the SQLite Wasm build and talking to it from/through JavaScript. Even if some of your code is in Go. But that's another project.

Other than that, to the extent that I can support Wasm, I already do: #212; also see #207 (someone got it to work). Also if you want to build a VFS targeting OPFS, the current bindings should be sufficient (there are now 6 or 7 VFSes in the tree).

[gedw99]

Thanks for the reply.

The reason I want to use golang was because I have a large stack of golang that run in the browser as service worker and web workers.

So adding SQLite to the sw with golang would frankly be less work.

I will see if I can do it by studying the code in this repo and let you know.

I will close this issue as it’s clear that supporting browsers is out of scope . Sorry to waste your time @ncruces

[ncruces]

If you really want to, you can. You can probably implement a VFS that's backed by OPFS, but I honestly don't know. And this driver should work: it's tested on wasip1 and I fixed some bugs to ensure it works on wasm.

What I'm not sure is what's the advantage vs. using Go to call the official SQLite Wasm through the official JavaScript API. But I'm really not an expert on browser use.