Ably JSON Patch data service demo

This is a JSON Patch demo for Ably Realtime, a realtime Data Streaming Network.

At Ably, many of our customers use our platform to push updates to a data object to any number of clients. For example, this could be a score board during a live football match, and customers rely on Ably to provide the limitless scale, reliability and low latency for each update published.

However, when the scoreboard objects become larger than a kilobyte or two, it becomes inefficient, from a bandwidth perspective, to send the entire object each time, and instead it makes sense to only send the deltas (changes).

We built this demo to show how easy it is to use Ably's pub/sub API along with our history API, to build a service that publishes only the changes (deltas) instead of the entire object for each update.

Building on this idea with Open SDSP

At Ably, we're sponsoring an open source protocol called the Streaming Data Sync Protocol (SDSP) that allows publishers of realtime data to efficiently synchronize an object with decoupled subscribers. The protocol is designed to be interoperable with any underlying platforms and transports. The protocol is particularly effective when there is a single publisher and one or more decoupled subscribers (pub/sub pattern).

Whilst this demo shows how trivial it is to build a simple JSON patch service yourself, we'd welcome your input and participation in a standard that will benefit the entire development community at https://github.com/open-sdsp/spec.

Try this demo

Demo app hosted on Heroku

See http://ably-json-patch-demo.herokuapp.com/

Deploy to your Heroku account for free

How this works

Publisher

The publisher works as follows:

1. All data updates are handled with a simple PatchService class. The class is responsible for storing the latest version of the object, and generating diffs for each update.
2. JSON diffs are generated by the Starcounter JSON Patch library. JSON patch is very portable and there are libraries in pretty much every language out there.
3. The PatchService class generates a serial number for each update, and determines whether it should send the entire object or just a diff. If a diff is generated, the last full object serial is included so that clients know how far back to go in the published update history to construct the object.
4. The diff / complete object returned from the PatchService is simply published on an Ably channel using a realtime connection. A REST library could be used too, however it is then the responsibility of the developer to ensure only one update is in-flight at any one time. If there is no single request in-flight lock when using REST, it is plausible that a later patch may arrive before another by Ably and thus the patches will be applied out of sequence.

Subscriber

The subscribe works as follows:

1. The subscriber attaches to the channel that updates are published on.
2. Once attached, it uses Ably's unique untilAttach history feature to retrieve the last published message. untilAttach guarantees that from the moment you are attached, your history query will return all messages published from before that point i.e. you will never have a race condition where either the history query returns duplicate data or missing data.
3. The last published message is inspected. If it contains the entire JSON object, then all updates received on the channel are applied immediately and moving forwards. If however the update contains a patch, then the completeSerial property is compared with the current serial to work out how many messages need to be retrieved to obtain the last published complete object. A subsequent untilAttach history request is then issued to retrieve exactly all messages needed to obtain the last complete object + all subsequent patches up until the point the channel was attached. The object is then constructed by simply applying the JSON patches, any patches that have been received on the channel subsequently, and any moving forwards.

Considerations

• Ably uniquely provides reliable ordering of messages. Unless ordering is maintained from the publisher to the subscriber, it is impractical to simply apply patches as they arrive, so don't try this with other realtime services that don't provide this functionality.
• We have configured this demo to republish the entire message either every 100 updates or every 1 minute, whenever there is an update. This configuration should be changed based on your requirements and frequency of updates. The rationale behind this is that if you don't publish the whole object very frequently, then new clients that connect will have to potentially retrieve a large number of messages to simply build the current object's state. For example, if you configured this demo to only publish the entire object every 1,000 updates, then a client may connect and have to retrieve 1,000 messages (patches) over our history API to construct the current object.
• Error conditions in this demo are caught and logged, but there is little attempt to retry or recover from failure. If you use this code in production you should consider recovery strategies.

Running this demo

On Heroku (the easiest)

By far the easiest way to view the demo is just to click the Heroku button below and follow the instructions.

Locally

If you want to try this out locally, follow these instructions:

1. Go to https://www.ably.io/ and sign up for a free API key
2. Enable history in your app you set up. Go to app, settings and configure the namespace "json-patch" to persist messages. See https://goo.gl/LFbwXj for more info on how history works and how to set it up. History is needed so that new clients can obtain the latest object and apply recent patches
3. Download index.html locally
4. Insert the API key where the placeholder exists const ApiKey = "[INSERT_API_KEY_HERE]"
5. Open index.html in your browser. Click Start publish to start publishing random updates.
6. Open index.html in as many other browser tabs as you like and watch how they all keep in sync. Note that if you attempt to have more than one publisher it will fail because the serial numbers will conflict for each published message.

Questions

Please visit http://support.ably.io/ for access to our knowledgebase and to ask for any assistance.

You can also view the community reported Github issues.

License

Copyright (c) 2018 Ably Real-time Ltd, Licensed under the Apache License, Version 2.0. Refer to LICENSE for the license terms.