-
Notifications
You must be signed in to change notification settings - Fork 1
/
index.html
208 lines (196 loc) · 9.55 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
<html>
<head>
<meta charset="utf8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Shardcache</title>
<link href="css/bootstrap.min.css" rel="stylesheet">
<link href="css/bootstrap-theme.min.css" rel="stylesheet">
<link href="css/metathon-style.css" rel="stylesheet">
</head>
<body>
<div class="gradient"> </div>
<div class="container-fluid">
<div class="row header">
<div class="col-md-12">
<a href="#" class="logo">Shardcache</a>
<a href="https://github.com/Shardcached/shardcached">Github Project</a>
<a href="https://github.com/shardcached/">Github Organization</a>
<a href="api">API Reference</a>
</div>
</div>
<div class="row">
<div class="col-md-8">
<div class="project-box-wide ">
<a id="Shardcache_0"></a>
<p>Shardcache is a distributed cache and storage system, originally inspired by
<a href="https://github.com/golang/groupcache">groupcache</a>, intended as a replacement
for <a href="http://memcached.org">memcached</a>. Note that shardcache does not include
code from those projects, and also the protocol for node and communication
is different.</p>
<h2><a id="Key_Features_9"></a>Key Features</h2>
<p>The project is divided into a <code>libshardcache</code> library, implementing both
a client and a node, and a <code>shardcached</code> daemon, which implements a shardcache
node and an optional HTTP frontend to the cache.</p>
<h3><a id="libshardcache_16"></a>libshardcache</h3>
<ul>
<li>
<p>Cache filling algorithm based on adaptive replacement that minimizes access to the
underlying storage</p>
</li>
<li>
<p>Automatically connects to its own peers and handles intra-node communication</p>
</li>
<li>
<p>Ensures the items are fetched from the peers or from the storage only once
even when multiple concurrent requests are accessing the same uncached item</p>
</li>
<li>
<p>Supports SET operations. If the node which receives the SET operation is
responsible for the specified KEY, the new value will be provided to the
underlying storage (and to next GET requests). If the receiving node is not
the responsible for the key, the request will be forwarded (through the
internal communication channel) to the responsible peer which will eventually
store the new value and make it available to all the shardcache nodes.</p>
</li>
<li>
<p>Supports DEL operations. If the node which receives the DEL operation is
responsible for the specified KEY, the key will be removed from the underlying
storage and from the cache. If evict-on-delete is turned on (it is by default)
an evict command will be sent to all other peers to force eviction from their
cache. If the receiving node is not responsible for the key, it will still be
removed from the local cache (if present) and the request will be forwarded
through the internal communication channel to the responsible peer which will
eventually remove the key from its storage and from the cache.</p>
</li>
<li>
<p>Supports EVICT operations. Evicts differ from deletes in the sense that the key
is only unloaded from the cache but not removed from the storage. Forcing
evictions might be very useful to force new values to be visible as soon as
possible after being set.</p>
</li>
<li>
<p>Supports migrations via the MGB (migration-begin), MGA (migration-abort) and MGE
(migration-end) commands. The nodes automatically redistribute the keys taking
the newly added ones into account.</p>
<p>While the migration is in progress (and keys are being redistributed) the behaviour
is the following:</p>
<ul>
<li>
<p>If a get operation arrives, first the old continuum is checked, if not found
the new continuum is checked.</p>
</li>
<li>
<p>If a set/delete operation arrives the new continuum is used to determine the
owner of the key.</p>
</li>
</ul>
<p>Once the migration is completed the continua are swapped and the new continuum
becomes the main one.</p>
</li>
<li>
<p>Supports volatile keys, which have an expiration time and will be automatically
removed when expired. Note that such keys are always kept in memory, regardless of
the storage type, and are never passed to the storage backend. During a migration
the volatile keys eventually not owned by a node will NOT be forwarded to the new
owner but will be instead expired “prematurely” (since they won’t be anyway
available anymore when switching to the new continuum)</p>
</li>
</ul>
<h3><a id="shardcached_71"></a>shardcached</h3>
<ul>
<li>
<p>Implements an <code>libshardcache</code> node with an HTTP frontend.</p>
</li>
<li>
<p>Allows to define ACLs to control which IP addresses can access which keys.</p>
</li>
<li>
<p>Exposes instrumentation counters and the cache index.</p>
</li>
<li>
<p>Supports custom mime-type mappings to use when serving items over HTTP.</p>
</li>
<li>
<p>Supports in-memory only or persistant storage.</p>
</li>
<li>
<p>Provides an API to specify custom pluggable storage backends. Current support
includes:</p>
<ul>
<li>sqlite</li>
<li>mysql</li>
<li>redis</li>
</ul>
</li>
<li>
<p>Supports migrations that can be initiated by new nodes at startup.</p>
</li>
</ul>
<h1><a id="Getting_Started_91"></a>Getting Started</h1>
<p>Shardcache development is done on GitHub. The latest release can be downloaded
from the <a href="https://github.com/Shardcached/shardcached/releases">GitHub releases page</a>,
or the latest development code can be cloned from the
<a href="https://github.com/Shardcached/shardcached">repository</a>.</p>
<p>The project is written in C, so it requires a working build environment. To compile,
invoke <code>make</code> in the toplevel directory. The makefile will fetch or update the
dependencies and build the daemon.</p>
<h3><a id="Usage_103"></a>Usage</h3>
<pre><code>Usage: ./shardcached [OPTION]...
Version: 1.0 (libshardcache: 1.1)
Possible options:
-a <access_log_file> the path where to store the access_log file (defaults to './shardcached_access.log')
-c <config_file> the config file to load
-d <plugins_path> the path where to look for storage plugins (defaults to './')
-f run in foreground
-F force caching
-H disable the HTTP frontend
-i <interval> change the time interval (in seconds) used to report internal stats via syslog (defaults to '0')
-l <ip_address:port> ip_address:port where to listen for incoming http connections
-L enable lazy expiration
-M sets the arc_mode in libshardcache to 'loose' (defaults to : 'strict')
-E <expire_time> set the expiration time for cached items (defaults to: 0)
-e <conn_expire_time> the expiration time for a connection in the pool to trigger a NOOP (defaults to 5000)
-r <mux_timeout_low> set the low timeout passed to iomux_run() calls (in microsecs, defaults to: 100000)
-R <mux_timeout_high> set the high timeout pssed to iomux_run() calls (in microsecs, defaults to: 500000)
-b HTTP url basepath (optional, defaults to '')
-B HTTP url baseadminpath (optional, defaults to '')
-n <nodes> list of nodes participating in the shardcache in the form : 'label:address:port,label2:address2:port2'
-N no storage subsystem, use only the internal libshardcache volatile storage
-m me the label of this node, to identify it among the ones participating in the shardcache
-P <pipelining_max> the maximum amount of requests to handle ahead on the same connection while still serving a response (defaults to: 64)
-S shared secret used for message signing (defaults to : '')
-s cache size in bytes (defaults to : '536870912')
-T <tcp_timeout> tcp timeout (in milliseconds) used for connections opened by libshardcache (defaults to '5000')
-t <type> storage type (available are : 'mem' and 'fs' (defaults to 'mem')
-o <options> comma-separated list of storage options (defaults to '')
-u <username> assume the identity of <username> (only when run as root)
-v increase the log level (can be passed multiple times)
-V output the version number and exit
-w <num_workers> number of shardcache worker threads (defaults to '10')
-W <num_http_workers> number of http worker threads (defaults to '10')
-x <nodes> new list of nodes to migrate the shardcache to. The format to use is the same as for the '-n' option
Builtin storage types:
* mem memory based storage
Options:
- initial_table_size=<size> the initial number of slots in the internal hashtable
- max_table_size=<size> the maximum number of slots that the internal hashtable can be grown up to
* fs filesystem based storage
Options:
- storage_path=<path> the path where to store the keys/values on the filesystem
- tmp_path=<path> the path to a temporary directory to use while new data is being uploaded
</code></pre>
</div>
</div>
<div class="col-md-4 project-page">
<h3>Table of contents</h3>
<p><a href="#Key_Features_9">Key Features</a></p>
<p><a href="#Getting_Started_91">Getting Started</a></p>
</div>
</div>
<div class="row footer">
<div class="col-md-12"></div>
</div>
</div>
</body>
</html>