A peer-to-peer data sync framework
It was a proof of concept. I learned a lot, and I'm slowly designing a more robust and verifyable system (see here), but in the meantime you might as well regard it as abandoned.
Mytosis organizes data as one massive object which contains other objects. The root is called the graph, and its children are called nodes.
Graph {
node1
node2
node3
...
}
A key concept is that nodes cannot contain other nodes. To express relationships, Mytosis uses pointers, sometimes called "edges". Those edges are automatically resolved, allowing you to structure your data as a tree, as tables, as key-value pairs, or an interconnected mesh.
Graph {
book {
title: "Hitchhiker's Guide"
author: Pointer(author)
}
author {
name: "Douglas Adams"
authored: Pointer(books)
}
books {
book: Pointer(book)
}
}
To mutate the graph, you declare all fields which changed along with their new values. Mytosis will generate a patch update and give it to your storage and network plugins.
This package is available on npm.
# Install with npm
npm install mytosis --save
# Install with yarn
yarn add mytosisNow you can import it into your project.
// ES Modules
import database from 'mytosis'
// CommonJS
const database = require('mytosis').defaultThe main API surface is intentionally small. It consists of only a few methods.
If you need more fine-grained control, you might find the
graph-crdtdocumentation useful.
DatabaseandContextinherit fromGraphandNode(respectively).
Creates a new database. It can be used with or without the new keyword.
By default, the database operates as an in-memory cache, and can be extended by passing in plugins. There are tons of options, so we'll cover those later.
const db = database()Most methods accept an options object as the last parameter, which supports the following:
options.storage- Overrides the storage defaults. Can be a storage plugin ornull.options.network- Overrides the network defaults. Can be aConnectionGroupornull.
Reads a Node from the database, resolved through a promise.
Node- An object containing only primitives. Nested objects are represented as pointers.
const stats = await db.read('stats')
console.log('Stats:', stats)If the node can't be found, null is returned.
Same options as db.nodes().
Reads many keys simultaneously, resolving with an array of nodes.
Note: This isn't just
Promise.all. It's more performant thandb.read, especially if you're querying the network.
// Reads both `timeline` and `profile/<userid>`.
const [timeline, profile] = await db.nodes(['timeline', `profile/${userId}`])
console.log('Profile:', profile.snapshot())If a node can't be found, it's value is null.
All standard options are supported. In addition...
options.force- Ignore the in-memory cache and force a read from the plugins. Expects a boolean value.
Updates properties on a node. If the node doesn't exist, it's created.
db.write('preferences', {
notificationsDisabled: true,
theme: 'dark',
})Everything in Mytosis is a patch update. You only need to declare the properties you're changing.
Clones the current database into a new in-memory store. Nothing you change will have an effect on the source database until you choose to commit() them.
Hooks and API extensions are shared with branches, but storage and network plugins are ignored.
Some use cases are:
- Describing a collection of changes which should be applied simultaneously
- Making edits which might be later cancelled
const update = db.branch()
// These writes won't affect `db`.
await update.write('settings', {
bio: 'It all started when...',
theme: 'Monokai',
})
await update.write('contact', {
email: '[email protected]',
})
// Write all the changes at once.
await db.commit(update)Note: Changes to the source database will not affect branches.
You may want torebase()your changes before callingcommit().
Applies a collection of changes all at once, represented as a graph. Useful for committing changes pushed from other replicas or branched databases.
It's used under the hood by node.write and db.write.
import { Graph, Node } from 'mytosis'
const graph = new Graph()
const node = new Node({ uid: 'change' })
node.merge({ changes: 'eh, probably' })
graph.merge({ change: node })
await db.commit(graph)Each mutation will emit an "update" event, passing a graph containing only the changes. There's also a "history" event when properties are overwritten. If you keep track of these deltas, you can roll time backwards and forwards.
Reads a primitive value from the node. If the value is a pointer to another node, Mytosis will automatically resolve it.
const weather = await db.write('weather', {
temperature: 25,
})
const temperature = await weather.read('temperature')Read a list of fields in a single request. Pointers are resolved using db.nodes for efficient read batching.
const [name, friends] = await user.fields(['name', 'friends'])Standard options are accepted.
Writes a value to the node. The value can be any primitive.
const company = await db.write('company', {})
await company.write('phone', '456-123-8970')To create a pointer to another node, you can write the reference.
// Create two nodes...
const dave = await db.write('user', { name: 'dave' })
const company = await db.write('company', { name: 'Enterprise Inc.' })
// Link one to the other.
await dave.write('workplace', company)
// Prints "Enterprise Inc."
dave.read('company')
.then(company => company.read('name'))
.then(name => console.log('Name:', name))Each node inherits from an event emitter. Any mutation triggers "update", passing the changes. This is useful for observing real-time data and reacting to changes.
const usage = await db.read('usage-stats')
usage.on('update', (changes) => {
console.log('Fields changed:', ...changes)
})You can read everything in storage as a stream of nodes. Streams use JavaScript async iteration, and completely ignore the in-memory cache, going straight to the storage plugin.
for await (const node of db) {
console.log('Node ID:', String(node))
console.log('Value:', node.snapshot())
}Naturally those queries can be quite expensive, so by default none of the network plugins are used, the query is only executed on the machine which requested it.
If you don't have a storage plugin, or if your storage plugin doesn't support streaming, you'll get an error.
Mytosis is designed to be highly extensible through plugins. You can use it with any storage backend, sync it over the network using any connection prototcol (such as websockets, http, webrtc, or a mix of all three), intercept and transform reads and writes, filter incoming reads or writes, and extend the core API.
These plugins are all contained in the config, and must be defined when the database is instantiated.
You can have more than one config though, as the database function accepts several and will merge them all together. This allows you to create presets, or groupings of plugins and share them as a single unit.
const db = database(config1, config2, config3)Note: Currently, there are no published plugins.
Used to intercept reads & writes. For example, you could use hooks to:
- Reject writes of the wrong data type
- Listen for write events and update a database visualization
- Keep track of how often a value is read for an LRU policy
- Transform values to a different format (like a
Dateobject)
A hook is a function which takes a read or write action and returns the transformed action.
const writeHook = (action) => actionWhen you write a value, mytosis generates an action object which represents the write. It contains details like what network & storage plugins should be called, what the update graph is, what the state will look like when the write has finished, and any merge deltas after completion.
Hooks can change every bit of it.
For example, here's a hook which adds a prefix to every read.
const readHook = (readAction) => ({
...readAction,
keys: readAction.keys.map((key) => `my-prefix/${key}`),
})
const db = database({
hooks: {
before: {
read: {
nodes: readHook,
},
},
},
})Hooks can be asynchronous. If a promise is returned, the entire pipeline is put on pause until it resolves.
// Delay all writes by one second.
const writeHook = async (writeAction) => {
await Promise.delay(1000)
return writeAction
}As a best practice, never mutate the action. Return a new action instead.
These are all the hooks you can register:
const hook = (action) => action
database({
hooks: {
before: {
write: hook,
read: {
nodes: hook,
field: hook,
},
},
after: {
write: hook,
read: {
node: hook,
field: hook,
},
},
}
})Note: You may be tempted to implement security using hooks. Don't. Do it in the router instead.
Used to save data to a persistent cache and read it back later. If a plugin is provided, the first read will attempt to pull from storage, and its return value will be cached in memory.
database({
storage: new StoragePlugin(),
})Writing a storage plugin will look a bit like this...
class StoragePlugin {
async read (action) {} // Read a value
async write (action) {} // Write a set of values
async remove (action) {} // Delete a value
}Note: Mytosis may require
.query()and.list()methods in the future.
Used to send updates and request data from remote sources.
database({
network: new NetworkPlugin(),
})By default, Mytosis will not use your network plugins. Instead, it interfaces with another plugin type, called the router. The router is responsible for answering requests and sending out data.
Network plugins are objects with a .send() method and a message event stream. Each connection declares its connection type and its unique identifier.
Here's an example of a network plugin...
import { Stream } from 'mytosis'
class NetworkPlugin {
type = 'websocket'
id = uuid()
send (message) {}
messages = new Stream(emit => {
myNetworkInterface.on('message', emit)
})
}Mytosis may specify
connectionanddisconnectionevent streams in the future, as well asofflineandephemeralflags.
The router is responsible for sending read requests, pushing out updates, and handling requests from others.
At its core, a router looks like this:
const router = (db, config) => {
config.network.messages.forEach((message) => {
console.log('Incoming message:', message)
})
return {
async push ({ network, update }) {
network.send({ /* your update */ })
}
async pull ({ network, key }) {
network.send({ /* your request */ })
}
}
}
database({ router })push() is invoked for writes, while pull() is called for reads. You're given a group of network connections and complete creative freedom. Whatever the push/pull resolve value is will be given to the user (after merging with whatever storage returns).
Mytosis doesn't have a stance on read or write permissions. Your router is responsible for how you answer unauthorized requests. If a message comes in asking for data they're not authorized to read, pull() (or push()) should reject.
Ultimately, the security mechanism is completely up to you.
Note: Mytosis may publish a routing framework in the near future designed to address permissions and security.
- Better offline editing
- Query support (plugin integration)
- Ability to delete
- Low-level streaming API