Skip to content

Source Files

Jump to bottom
Jens Alfke edited this page Mar 25, 2014 · 3 revisions

This is a quick tour of the Sync Gateway source files, for the benefit of those wanting to dive into the code.

Sub-packages

The gateway's packages are nested under github.com/couchbaselabs/sync_gateway, and the corresponding source directories are under src/github.com/couchbaselabs/sync_gateway. I'll go through them in bottom-up order.

base

Shared utilities used by the other packages.

  • bucket.go: Defines a wrapper that adapts the go-couchbase package's Bucket to match Walrus's Bucket interface, allowing the two to be used interchangeably by the rest of the code. Also has some factory functions for instantiating buckets.
  • error.go: Error handling utilities.
  • http_listener.go: An implementation of an HTTP listener that can limit the number of simultaneous connections it allows.
  • logging.go: A nice logging utility. You'll see calls to base.LogTo() all over the code.
  • logging_bucket.go: A wrapper around the Bucket interface that logs info about every call.
  • set.go: Implements Set, a reusable "set of strings" data type.
  • util.go: Miscellaneous utility functions.

channels

Various types and functions for working with channels.

  • change_log.go: The widely-used LogEntry (represents a change to the database, i.e. a new revision) and ChangeLog (an array of LogEntrys).
  • channelmapper.go: High-level API to a database's sync function. Adds thread-safety to SyncRunner.
  • set.go: Extra functions for using a base.Set as a set of channel names. Includes channel name validation and some utilities for managing the magic * channel name.
  • sync_runner.go: Low-level API to a database's sync function. The actual implementation that calls into JavaScript is here, but this level is not thread-safe so it should not be called directly.
  • timed_set.go: A map from channel names to sequence numbers.

auth

Manages users and roles and their persistent storage.

  • auth.go: The Authenticator class, a factory for User and Role objects.
  • password_hash.go: Password hashing for authentication.
  • principal.go: Defines the abstract Principal interface and its two sub-interfaces Role and User.
  • role.go: Concrete implementation of Role interface.
  • user.go: Concrete implementation of User interface.

db

The big kahuna, implements the Sync Gateway's functionality.

  • assimilator.go: Optional task that detects un-versioned documents added to the bucket by a Couchbase Server client and adds _sync metadata to them. (Deprecated in favor of bucket shadowing.)
  • attachment.go: Manages converting between saved attachment blobs and a document's _attachments dictionary. Also parses and generates MIME multipart bodies from documents.
  • change_cache.go: In-memory cache of recent database changes, and low-level generator of change feeds.
  • change_listener.go: A goroutine that listens to the bucket's Tap feed to detect changes to documents.
  • changes.go: Generates changes feeds for the REST API to serve.
  • changes_view.go: Runs a view query to generate feeds for changes that are too old to be in the cache anymore.
  • channel_cache: Subcomponent of changeCache that tracks changes on a single channel.
  • crud.go: High-level methods for Creating, Reading, Updating, Deleting documents.
  • database.go: The Database and DatabaseContext structs that represent a Sync Gateway database. (These are sort of misnamed. DatabaseContext does most of the work; Database is just a wrapper around it that remembers what user is accessing the database so it can do access-control checks.)
  • document.go: The document struct that represents the JSON structure of a single document with its metadata.
  • revision.go: Stores older revisions in separate Couchbase documents, and has a variety of functions for working with document revisions.
  • revision_cache.go: An in-memory cache of recently-used revision bodies.
  • revtree.go: RevTree, a representation of a document's revision history that serializes to JSON. (A subcomponent of a document.)
  • sequence_allocator.go: Manages a persistent atomic counter for allocation of revision sequence numbers.
  • shadower.go: Runs document shadowing for a database. Watches the database and the external bucket and propagates changes from one to the other.
  • special_docs.go: Implementation of "local" documents, which are used by client replicators to store checkpoints (and really not for anything else.)
  • statistics.go: Statistics struct, which tracks usage count of a resource, such as the _changes feed.

rest

HTTP handlers that implement the Gateway's REST API, and the startup code.

  • admin_api.go: Various HTTP handlers for the admin API, such as configuring databases and accessing users and roles.
  • api.go: HTTP handlers for high-level resources like /. (Most of these have become admin-only.)
  • bulk_api.go: Handlers for _all_docs, _bulk_docs, _bulk_get, and some admin utilities for dumping data.
  • changes_api.go: Handles _changes feeds.
  • config.go: Reads the Gateway's JSON config files, parses command-line arguments, and starts the HTTP listeners.
  • debug.go: Some debugging tools like the /_expvar handler.
  • doc_api.go: Handlers for CRUD operations on documents and attachments.
  • encoded_response_writer.go: Implements GZip compression of HTTP responses.
  • facebook.go: Facebook authentication handler.
  • handler.go: A struct representing an HTTP request that wraps the http.Request and http.ResponseWriter objects as well as the Database the request operates on. All the HTTP handlers are implemented as methods on this struct.
  • persona.go: Mozilla Persona authentication handler.
  • routing.go: Sets up the routing from incoming request methods/URLs to handlers.
  • server_context.go: A long-lived object that represents a database being served.
  • session_api.go: HTTP handlers for cookie-based login sessions.

Dependencies

There are a number of dependent Go packages. Rather than use the regular Go package manager (go get), which always updates packages to the latest revision available, we add them to the Git repo as submodules, so we can control exactly which revisions we use.

To do this, we change the $GOPATH. The details are in the go.sh script, but basically the GOPATH consists of the root of the repo (for the Gateway sources) and the vendor subdirectory (for the dependent packages). The Git submodules are checked out under vendor/src:

  • github.com/couchbaselabs/go-couchbase: Couchbase Server API
  • github.com/couchbaselabs/go.assert: Assertion utilities for unit tests
  • github.com/couchbaselabs/sync_gateway_admin_ui: HTML source for the Gateway admin UI
  • github.com/couchbaselabs/walrus: Lightweight storage engine for tests and experimentation
  • github.com/dustin/gomemcached: Memcached API (used by go-couchbase)
  • github.com/gorilla/context: Web-server utility for storing request variables (used by gorilla/mux)
  • github.com/gorilla/mux: Rails-like HTTP request routing package
  • github.com/robertkrimen/otto: A pure Go JavaScript interpreter
  • github.com/samuel/go-metrics: Utilities for computing performance metrics; used for logging
  • github.com/tleyden/fakehttp: Mock HTTP server; used by unit tests

Some of the dependent packages don't have Git repositories because they're hosted by Google Code. These are checked in directly. Most of these have several sub-packages of which we use only one or two:

  • code.google.com/p/go.crypto: We use the bcrypt package for password hashing.
  • code.google.com/p/go.net: We use the websocket package for handling WebSocket requests.
  • code.google.com/p/go.text: Walrus uses the collate and language packages for Unicode sorting.
Clone this wiki locally