Skip to content

Channel Index Processing

Jump to bottom
Adam Fraser edited this page Sep 15, 2015 · 3 revisions

##Write Processing

To optimize index writes, kvChangeIndex batches up pending incoming mutations and writes updates as a batch. Mutations sent via DocChanged are added to a pending channel for processing. A worker goroutine - kvChangeIndex - is continually working that pending go-channel and processing entries it finds as a batch. For each batch, it does the following:

  1. Groups the updates by channel
  2. For each channel set:
a. Updates the channel index blocks for channel presence
b. Adds sequence entries
c. Updates the channel clock
  1. Updates the stable sequence clock

##Read Processing

When talking about read processing, we're really talking about supporting _changes requests. The rest of the replication operations - get, bulk get, revs diff - don't depend on the channel index.

The request handling and routing for _changes processing hasn't been changed significantly, so there's no new code required for things like websocket vs. HTTP, heartbeats, timeouts, etc. The bulk of the changes are related to the change in sequence format, and change notification (waking up continuous or longpoll requests that are waiting for new changes).

###Change Notification

Previously each Sync Gateway node was aware of every change in the system (via it's mutation feed), and so notification was driven by the cache writer. When working with a sharded feed, we need to monitor the state of the index to identify when channels have changed.

Currently we're using a polling model to identify these changes (although it's built to simplify moving to a DCP feed of the index bucket in future). kvChangeIndex has a polling process/goroutine that does the following:

  1. Check whether we have active readers (active _changes requests). If not, we don't need to poll.
  2. Load the stable sequence, and see if it's changed since the last poll. If not, we're done.
  3. Read the channel clock for each active channel, and check with the reader whether this represents changes.
  4. Send a notification (using the existing changeListener) for the changed channels.
Clone this wiki locally