Consider using Automerge for syncing object across nodes

[justvanrossum]

https://automerge.org/

Thinking:

• A single glyph should probably a "document", in automerge terms

Question:

• There is no Python implementation, would our server need a JS component?

[justvanrossum]

Or https://github.com/yjs/yjs

[justvanrossum]

Or https://github.com/share/sharedb/ Meh.

[justvanrossum]

CRDT

So, the buzzword here is "Conflict-free replicated data type" (CRDT).

It is a very cool concept and mechanism, and it may be useful for us.

These tools are designed to allow data to diverge: edited by two or more parties, then synced later, say, after one of them comes back online. So it is ideal for apps that allow the user to continue working even if the connection to the server was lost, even for a longer time.

The algorithms behind Automerge and Yjs are designed to never cause merge conflicts: it is always defined what will happen if two parties edit something near each other or even overlapping. I can see how this works for text-like data, but I'm not sure this is what you'd want for outline data: we intuitively know that some edits are incompatible, and there is no way to automatically resolve the situation without accepting one person's work will be (partially) lost.

Perhaps something in-between is possible: automatically merge if there is no conflict at all, and always ask the user to review if there is a conflict.

Automerge is cool in that it considers all objects to be immutable, and each mutation results in a new object. This is supposed to be done in an efficient way, but I'm curious how well it would perform with our kind of data (which is tree-like rather than sequence-like).

I need to read more about how Yjs works. Some say it is faster than Automerge.

Beyond collaboration

The mechanism is useful beyond (remote) collaboration:

• undo/redo could be implemented using these tools (edit: Yjs has a complete undo solution)
• working with branches

CRDT and Fontra

Our immediate practical problem is that the official libraries are written in JavaScript (and Rust), and with our current architecture, we need it to work in Python, too. So that's a bit of a show-stopper for immediate experimentation.

The alternative for Fontra is to make something ourselves, which would a huge rabbit hole. But then again, we could limit our requirements. For example:

• Collaborative editing requires a network connection to a server or to a peer. Maybe we don't strictly need offline-yet-collaborative editing.
• We could use a locking system like RoboCJK on a per-glyph basis: if one person is edting a glyph, no one else will be permitted to edit it at the same time.

This is much easier to achieve, but still requires us to build a lot of infrastructure.

We could also say: we will initially have a more limited set of requirements, and intend to upgrade to a proper CRDT-based appraoch later. One question will be: how much work will be wasted by this two-step timeline?

More links

• https://blog.jupyter.org/how-we-made-jupyter-notebooks-collaborative-with-yjs-b8dff6a9d8af
• https://github.com/automerge/automerge-py Automerge for Python: doesn't seem very active or mature

[justvanrossum]

From @davelab6 in #238 (comment):

https://automerge.org/blog/automerge-2/ has impressive numbers, might be relevant