-
Notifications
You must be signed in to change notification settings - Fork 76
ManPage
Archie L. Cobbs edited this page Oct 23, 2023
·
24 revisions
S3BACKER(1) BSD General Commands Manual S3BACKER(1)
NAME
s3backer -- FUSE-based single file backing store via Amazon S3
SYNOPSIS
s3backer [options] bucket[/subdir] /mount/point
s3backer --nbd [options] bucket[/subdir] /dev/nbdX
s3backer --test [options] dir /mount/point
s3backer --erase [options] bucket[/subdir]
s3backer --reset-mounted-flag [options] bucket[/subdir]
DESCRIPTION
s3backer is a filesystem that contains a single file backed by the Amazon Sim-
ple Storage Service (Amazon S3). As a filesystem, it is very simple: it pro-
vides a single normal file having a fixed size. Underneath, the file is di-
vided up into blocks, and the content of each block is stored in a unique Ama-
zon S3 object. In other words, what s3backer provides is really more like an
S3-backed virtual hard disk device, rather than a filesystem.
In typical usage, a `normal' filesystem is mounted on top of the file exported
by the s3backer filesystem using a loopback mount (or disk image mount on Mac
OS X).
This arrangement has several benefits compared to more complete S3 filesystem
implementations:
o By not attempting to implement a complete filesystem, which is a complex
undertaking and difficult to get right, s3backer can stay very lightweight
and simple. All of the experience and knowledge about how to properly im-
plement filesystems that already exists can be reused.
o By utilizing existing filesystems, you get full UNIX filesystem semantics.
Subtle bugs or missing functionality relating to hard links, extended at-
tributes, POSIX locking, etc. are avoided.
o The gap between normal filesystem semantics and Amazon S3 ``eventual con-
sistency'' is more easily and simply solved when one can interpret S3 ob-
jects as simple device blocks rather than filesystem objects (see below).
o When storing your data on Amazon S3 servers, which are not under your con-
trol, the ability to encrypt and authenticate data becomes a critical is-
sue. s3backer supports secure encryption and authentication. Alter-
nately, the encryption capability built into the Linux loopback device can
be used.
o Since S3 data is accessed over the network, local caching is also very im-
portant for performance reasons. Since s3backer presents the equivalent
of a virtual hard disk to the kernel, most of the filesystem caching can
be done where it should be: in the kernel, via the kernel's page cache.
However s3backer also includes its own internal block cache for increased
performance, using asynchronous worker threads to take advantage of the
parallelism inherent in the network.
Consistency Guarantees
Previously, Amazon S3 made relatively weak guarantees relating to the timing
and consistency of reads vs. writes (collectively known as ``eventual consis-
tency''). s3backer includes logic and configuration parameters to work around
these limitations, allowing the user to guarantee consistency to whatever
level desired, up to and including 100% detection and avoidance of incorrect
data. These are:
1. s3backer enforces a minimum delay between consecutive PUT or DELETE opera-
tions on the same block. This ensures that Amazon S3 doesn't apply these
operations out of order.
2. s3backer maintains an internal block MD5 checksum cache, which enables au-
tomatic detection and rejection of `stale' blocks returned by GET opera-
tions.
This logic is configured by the following command line options:
--md5CacheSize, --md5CacheTime, and --minWriteDelay.
In December of 2020, Amazon announced that S3 is now fully consistent. There-
fore, the MD5 cache is disabled by default in s3backer versions after 1.6.3.
Other S3-compatible services may still require it however.
To guarantee consistency, --md5CacheTime and --minWriteDelay should be greater
than the maximum S3 convergence time.
Zeroed Block Optimization
As a simple optimization, s3backer does not store blocks containing all ze-
roes; instead, they are simply deleted. Conversely, reads of non-existent
blocks will contain all zeroes. In other words, the backed file is always
maximally sparse.
As a result, blocks do not need to be created before being used and no special
initialization is necessary when creating a new filesystem.
s3backer also uses a simple bitmap to track blocks that are known to be zero
on an ongoing basis. When the --listBlocks flag is given (recommended),
s3backer will list all existing blocks at startup to initialize this bitmap.
This especially affects performance with operations that read or write a large
number of zero blocks, such as filesystem initialization, fstrim(8), bulk
copies, etc.
File and Block Size Auto-Detection
As a convenience, whenever the first block of the backed file is written,
s3backer includes as meta-data (in the ``x-amz-meta-s3backer-filesize''
header) the total size of the file. Along with the size of the block itself,
this value can be checked and/or auto-detected later when the filesystem is
remounted, eliminating the need for the --blockSize or --size flags to be ex-
plicitly provided and avoiding accidental mis-interpretation of an existing
filesystem.
Block Cache
s3backer includes support for an internal block cache to increase performance.
The block cache cache is completely separate from the MD5 cache which only
stores MD5 checksums transiently and whose sole purpose is to mitigate ``even-
tual consistency''. The block cache is a traditional cache containing cached
data blocks. When full, clean blocks are evicted as necessary in LRU order.
Reads of cached blocks will return immediately with no network traffic.
Writes to the cache also return immediately and trigger an asynchronous write
operation to the network via a separate worker thread. Because the kernel
typically writes blocks through FUSE filesystems one at a time, performing
writes asynchronously allows s3backer to take advantage of the parallelism in-
herent in the network, vastly improving write performance.
When the cache is full, least recently accessed blocks are evicted first (but
see also the --blockCacheNumProtected flag).
The block cache can be configured to store the cached data in a local file in-
stead of in memory. This permits larger cache sizes and allows s3backer to
reload cached data after a restart. Reloaded data is verified via MD5 check-
sum with Amazon S3 before reuse.
Having said all that, Linux users may want to consider instead using the ker-
nel "bcache" mechanism for local caching of blocks.
The block cache is configured by the following command line options:
--blockCacheFile, --blockCacheMaxDirty, --blockCacheNoVerify,
--blockCacheNumProtected, --blockCacheSize, --blockCacheSync,
--blockCacheThreads, --blockCacheTimeout, --blockCacheWriteDelay, and
--blockCacheRecoverDirtyBlocks.
Read Ahead
s3backer implements a simple read-ahead algorithm in the block cache. When a
configurable number of blocks are read in order, block cache worker threads
are awoken to begin reading subsequent blocks into the block cache. Read
ahead continues as long as the kernel continues reading blocks sequentially.
The kernel typically requests blocks one at a time, so having multiple worker
threads already reading the next few blocks improves read performance by tak-
ing advantage of the parallelism inherent in the network.
Note that the kernel implements a read ahead algorithm as well; its behavior
should be taken into consideration. By default, s3backer passes the -o
max_readahead=0 option to FUSE.
Read ahead is configured by the --readAhead and --readAheadTrigger command
line options.
Encryption and Authentication
s3backer supports encryption via the --encrypt, --password, and --passwordFile
flags. When encryption is enabled, SHA1 HMAC authentication is also automati-
cally enabled, and s3backer rejects any blocks that are not properly encrypted
and signed.
Encrypting at the s3backer layer is preferable to encrypting at an upper layer
(e.g., at the loopback device layer), because if the data s3backer sees is al-
ready encrypted it can't optimize away zeroed blocks or do meaningful compres-
sion.
Compression
s3backer supports block-level compression, which minimizes transfer time and
storage costs.
Compression is configured via the --compress and --compress-level flags. Com-
pression is automatically enabled when encryption is enabled.
Server Side Encryption
s3backer supports server side encryption via the --sse flag.
Read-Only Access
An Amazon S3 account is not required in order to use s3backer. The filesystem
must already exist and have S3 objects with ACL's configured for public read
access (see --accessType below); users should perform the looback mount with
the read-only flag (see mount(8)) and provide the --readOnly flag to s3backer.
This mode of operation facilitates the creation of public, read-only filesys-
tems.
Simultaneous Mounts
Although it functions over the network, the s3backer filesystem is not a dis-
tributed filesystem and does not support simultaneous read/write mounts.
(This is not something you would normally do with a hard-disk partition ei-
ther.) As a safety measure, s3backer attempts to detect this situation using
an 'already mounted' flag in the data store, and will fail to start if it
does.
This detection may produce a false positive if a former s3backer process was
not shutdown cleanly; if so, the --reset-mounted-flag flag can be used to re-
set the 'already mounted' flag. But see also BUGS below.
Statistics File
s3backer populates the filesystem with a human-readable statistics file. Use
--statsFilename to change the name of this file (default `stats'). The sta-
tistics can be reset to zero by attempting to remove the file.
NBD Plugin
On platforms with ndbkit(1), s3backer is also installed as an nbdkit(1)
plugin.
When using the NBD plugin, command line flags like --blockSize=SIZE should in-
stead be specified as plugin parameters like blockSize=SIZE; for boolean
flags, specify a value of true. The bucket name (with optional subdirectory)
must be specified with the bucket parameter.
To avoid conflicts with other nbdkit(1) flags, any s3backer flag can be speci-
fied with a s3b_ prefix, for example s3b_force=true.
As an alternative to launching nbdkit(1) directly, the --nbd flag causes
s3backer to function as a wrapper process for managing an NBD session; see be-
low.
Logging
In normal operation s3backer will log via syslog(3). When run with the -d or
-f flags, s3backer will log to standard error.
OPTIONS
Each command line flag has two forms, for example --accessFile=FILE and -o
accessFile=FILE. Only the first form is shown below. Either form many be
used; both are equivalent. The second form allows mount options to be speci-
fied directly in /etc/fstab and passed seamlessly to s3backer by FUSE.
--accessFile=FILE
Specify a file containing `accessID:accessKey' pairs, one per-line.
Blank lines and lines beginning with a `#' are ignored. If no
--accessKey or --accessKeyEnv is specified, this file will be searched
for the entry matching the access ID specified via --accessId; if no
--accessId is specified, the first entry in this file will be used.
Default value is $HOME/.s3backer_passwd.
--accessId=ID
Specify Amazon S3 access ID. Specify an empty string to force no ac-
cess ID. If no access ID is specified (and none is found in the ac-
cess file) then s3backer will still function, but only reads of pub-
licly available filesystems will work.
--accessKey=KEY
Specify Amazon S3 access key.
To avoid publicizing this secret via the command line, use
--accessKeyEnv or --accessFile instead of this flag.
--accessKeyEnv=VARNAME
Specify an environment variable containing the Amazon S3 access key.
--accessType=TYPE
Specify the Amazon S3 access privilege ACL type for newly written
blocks. The value must be one of `private', `public-read', `public-
read-write', or `authenticated-read'. Default is `private'.
--accessEC2IAM=ROLE
Download access credentials and security token in JSON document form
from
http://169.254.169.254/latest/meta-data/iam/security-credentials/ROLE
every five minutes.
This option allows S3 credentials to be provided automatically via the
specified IAM role to s3backer when running on an Amazon EC2 instance.
--authVersion=TYPE
Specify how to authenticate requests. There are two supported authen-
tication methods: aws2 is the original AWS authentication scheme.
aws4 is the newer, recommended authentication scheme.
aws4 is the default setting starting in version 1.4, and is required
for certain non-US regions, while aws2 may still be required by some
non-Amazon S3 providers.
--baseURL=URL
Specify the base URL, which must end in a forward slash. Default is
`http://s3.amazonaws.com/'.
Use this flag to specify FIPS and dual-stack endpoints.
Note: the region name is used in authentication, so if you include a
region name you probably also need to specify it via --region.
--blockCacheFile=FILE
Specify a file in which to store cached data blocks. Without this
flag, the block cache lives entirely in process memory and the cached
data disappears when s3backer is stopped. The file will be created if
it doesn't exist.
Cache files that have been created by previous invocations of s3backer
are reusable as long as they were created with the same configured
block size (if not, startup will fail). This is true even if s3backer
was stopped abruptly, e.g., due to a system crash; however, this guar-
antee rests on the assumption that the filesystem containing the cache
file will not reorder writes across calls to fsync(2).
If an existing cache is used but was created with a different size,
s3backer will automatically expand or shrink the file at startup.
When shrinking, blocks that don't fit in the new, smaller cache are
discarded. This process also compacts the cache file to the extent
possible.
By default, only clean cache blocks are recoverable after a restart.
This means a system crash will cause dirty blocks in the cache to be
lost (of course, that is the case with an in-memory cache as well).
With the newer cache file format introduced in release 1.5.0, you can
recover these dirty blocks by specifying the
--blockCacheRecoverDirtyBlocks option. This will cause any dirty
blocks in the cache file to be made writable again on startup. If
your cache file was created with a prior release of s3backer or you do
not specify this option, dirty blocks in the cache file are discarded
on startup. The window of this data loss can be limited by
--blockCacheWriteDelay.
By default, when having reloaded the cache from a cache file, s3backer
will verify the MD5 checksum of each reloaded block with Amazon S3 be-
fore its first use. This verify operation does not require actually
reading the block's data, and therefore is relatively quick. This
guards against the cached data having unknowingly gotten out of sync
since the cache file was last used, a situation that is otherwise im-
possible for s3backer to detect.
--blockCacheMaxDirty=NUM
Specify a limit on the number of dirty blocks in the block cache.
When this limit is reached, subsequent write attempts will block until
an existing dirty block is successfully written (and therefore becomes
no longer dirty). This flag limits the amount of inconsistency there
can be with respect to the underlying S3 data store.
The default value is zero, which means no limit.
--blockCacheNoVerify
Disable the MD5 verification of blocks loaded from a cache file speci-
fied via --blockCacheFile. Using this flag is dangerous; use only
when you are sure the cached file is uncorrupted and the data it con-
tains is up to date.
--blockCacheSize=SIZE
Specify the block cache size (in number of blocks). Each entry in the
cache will consume approximately block size plus 20 bytes. A value of
zero disables the block cache. Default value is 1000.
--blockCacheThreads=NUM
Set the size of the thread pool associated with the block cache (if
enabled). This bounds the number of simultaneous writes that can oc-
cur to the network. Default value is 20.
--blockCacheTimeout=MILLIS
Specify the maximum time a clean entry can remain in the block cache
before it will be forcibly evicted and its associated memory freed. A
value of zero means there is no timeout; in this case, the number of
entries in the block cache will never decrease, eventually reaching
the maximum size configured by --blockCacheSize and staying there.
Configure a non-zero value if the memory usage of the block cache is a
concern. Default value is zero (no timeout).
--blockCacheWriteDelay=MILLIS
Specify the maximum time a dirty block can remain in the block cache
before it must be written out to the network. Blocks may be written
sooner when there is cache pressure. A value of zero configures a
``write-through'' policy; greater values configure a ``write-back''
policy. Larger values increase performance when a small number of
blocks are accessed repeatedly, at the cost of greater inconsistency
with the underlying S3 data store. Default value is 250 milliseconds.
--blockCacheSync
Forces synchronous writes in the block cache layer. Instead of re-
turning immediately and scheduling the actual write to operation hap-
pen later, write requests will not return until the write has com-
pleted. This flag is a stricter requirement than
--blockCacheWriteDelay=0, which merely causes the writes to be initi-
ated as soon as possible (but still after the write request returns).
This flag requires --blockCacheWriteDelay to be zero. Using this flag
is likely to drastically reduce write performance.
--blockCacheRecoverDirtyBlocks
An unclean dismount may leave dirty blocks (blocks written to the lo-
cal cache file, but not yet flushed to S3) in the cache file.
If this option is set, s3backer will recover any such dirty blocks and
eventually write them back to S3. If this option is not specified,
all dirty data in the cache file are discarded on startup.
If the filesystem has been mounted since the cache file was last used,
s3backer will refuse to mount. This is verified by checking a unique
32-bit mount token in the cache file against the 'already mounted'
flag in the data store.
This flag requires --blockCacheFile to be set.
--blockCacheNumProtected=NUM
Preferentially retain the first NUM blocks in the block cache.
Some upper filesystems store highly active data (e.g., write journal)
at the beginning of the filesystem. This option can be used to im-
prove performance by reducing network reads for these regions of the
file. With this option enabled, blocks after the first NUM blocks
will be evicted before any protected blocks.
--blockCacheFileAdvise
Immediately after being read or written by the kernel, data in the
block cache file can end up being cached twice by the kernel: once in
the cache for the block cache file, and again in the cache for the
s3backer file.
When this flag is given, posix_fadvise(2) is used whenever data is
read or written to/from the block cache file to tell the kernel not to
cache that data on behalf of the block cache file.
This flag is ignored if --blockCacheFile is not specified.
--blockHashPrefix
Prepend random prefixes (generated deterministically from the block
number) to block object names. This spreads requests more evenly
across the namespace, and prevents heavy access to a narrow range of
blocks from all being directed to the same backend server.
As with --prefix or subdir, this flag must be used consistently once a
disk image is established.
--blockSize=SIZE
Specify the block size. This must be a power of two and should be a
multiple of the kernel's native page size. The size may have an op-
tional suffix 'K' for kilobytes, 'M' for megabytes, etc. The total
number of blocks must fit into 32 bits.
s3backer supports partial block operations, though this forces a read
before each write; use of the block cache and proper alignment of the
s3backer block size with the intended use (e.g., the block size of the
`upper' filesystem) will help minimize the extra reads. Note that
even when filesystems are configured for large block sizes, the kernel
will often still write page-sized blocks.
s3backer will attempt to auto-detect the block size by reading block
number zero at startup. If this option is not specified, the auto-de-
tected value will be used. If this option is specified but disagrees
with the auto-detected value, s3backer will exit with an error unless
--force is also given. If auto-detection fails because block number
zero does not exist, and this option is not specified, then the de-
fault value of 4K (4096) is used.
--cacert=FILE
Specify SSL certificate file to be used when verifying the remote
server's identity when operating over SSL connections. Equivalent to
the --cacert flag documented in curl(1).
--compress[=ALGORITHM]
Compress blocks using the specified compression algorithm before send-
ing them over the network. This should result in less network traffic
(in both directions) and lower storage costs.
The ALGORITHM is optional; currently, the default value is deflate.
Also supported is zstd.
This flag only enables compression of newly written blocks; decompres-
sion is always enabled and applied when appropriate. Therefore, it is
safe to switch this flag on or off between different invocations of
s3backer on the same filesystem.
This flag is implied when --encrypt is used.
When using an encrypted upper layer filesystem, this flag adds no
value because the data will not be compressible.
--compress-level=LEVEL
Configure the compression level for the selected compression algo-
rithm, which must be specified via --compress.
The interpretation of LEVEL depends on the algorithm; for deflate, it
must be an integer between 1 (maximum speed) and 9 (maximum compres-
sion), inclusive.
If this flag is omitted, the default compression level for the se-
lected algorithm is used.
--configFile=FILE
Insert command line flags and arguments read from the specified file
in place of this flag.
Leading and trailing whitespace is trimmed from each line, and blank
lines and lines starting with # are ignored. Each line contains ex-
actly one command line argument (including internal whitespace, if
any).
Nested --configFile flags are allowed.
This option may also be specified as -F FILE.
--directIO
Disable kernel caching of the backed file. This will force the kernel
to always pass reads and writes directly to s3backer. This may reduce
performance but also eliminates one source of inconsistency.
--debug
Enable logging of debug messages. This includes logging HTTP error
response payloads.
Note that this flag is different from -d, which is a flag to FUSE;
however, the -d FUSE flag implies this flag.
--debug-http
Configure libcurl(3) to print debugging information (e.g., HTTP re-
quest and response headers) to standard error.
With this flag you would normally also specify -f so s3backer runs in
the foreground and standard error is visible.
--defaultContentEncoding=VALUE
Use this to workaround S3 backends that fail to send back the
Content-Encoding header that was sent to them by s3backer. If a block
read response contains no Content-Encoding header, this value will be
substituted.
If you get errors complaining that the content was expected to be en-
crypted, try setting this to deflate,encrypt-AES-128-CBC.
--encrypt[=CIPHER]
Enable encryption and authentication of block data. See your OpenSSL
documentation for a list of supported ciphers; the default if no ci-
pher is specified is AES-128 CBC.
Currently, only block ciphers are supported.
The encryption password may be supplied via one of --password or
--passwordFile. If neither flag is given, s3backer will ask for the
password at startup.
Note: the actual key used is derived by hashing the password, the
bucket name, the prefix name (if any), and the block number. There-
fore, encrypted data cannot be ported to different buckets or pre-
fixes.
This flag implies --compress.
--erase
Completely erase the file system by deleting all non-zero blocks,
clear the 'already mounted' flag, and then exit. User confirmation is
required unless the --force flag is also given. Note, no simultaneous
mount detection is performed in this case.
This operation bypasses the caching layers, so any leftover cache file
must be manually deleted.
--filename=NAME
Specify the name of the backed file that appears in the s3backer
filesystem. Default is `file'.
--fileMode=MODE
Specify the UNIX permission bits for the backed file that appears in
the s3backer filesystem. Default is 0600, unless --readOnly is speci-
fied, in which case the default is 0400.
--force
Proceed even if the value specified by --blockSize or --size disagrees
with the auto-detected value, or s3backer detects that another
s3backer instance is still mounted on top of the same S3 bucket (and
prefix). In any of these cases, proceeding will lead to corrupted
data, so the --force flag should be avoided for normal use.
The simultaneous mount detection can produce a false positive when a
previous s3backer instance was not shut down cleanly. In this case,
don't use --force but rather run s3backer once with the
--reset-mounted-flag flag.
If --erase is given, --force causes s3backer to proceed without user
confirmation.
-h --help
Print a help message and exit.
--http11
Restrict to HTTP version 1.1. There have been reports of errors
and/or reduced performance with some S3-compatible backends when
HTTP/2 is used. This flag can be used to prevent HTTP/2 negotiation.
--initialRetryPause=MILLIS
Specify the initial pause time in milliseconds before the first retry
attempt after failed HTTP operations. Failures include network fail-
ures and timeouts, HTTP errors, and reads of stale data (i.e., MD5
mismatch); s3backer will make multiple retry attempts using an expo-
nential backoff algorithm, starting with this initial retry pause
time. Default value is 200ms. See also --maxRetryPause.
--insecure
Do not verify the remote server's identity when operating over SSL
connections. Equivalent to the --insecure flag documented in curl(1).
--keyLength
Override the length of the generated block encryption key.
Versions of s3backer prior to 1.3.6 contained a bug where the length
of the generated encryption key was fixed but system-dependent, caus-
ing it to be possibly incompatible on different systems for some ci-
phers. In version 1.3.6, this bug was corrected; however, in some
cases this changed the generated key length, making the encryption no
longer compatible with previously written data. This flag can be used
to force the older, fixed key length. The value you want to use is
whatever is defined for EVP_MAX_KEY_LENGTH on your system, typically
64.
It is an error to specify a value smaller than the cipher's natural
key length; however, a value of zero is allowed and is equivalent to
not specifying anything.
--listBlocks
After starting, initiate a background query to determine which blocks
already exist. This enables optimizations whereby, for each block
that does not yet exist, reads return zeroes and zeroed writes are
omitted, thereby eliminating any network access.
This flag is useful when creating a new backed file, or any time it is
expected that a large number of zeroed blocks will be read or written,
such as when initializing a new filesystem.
In general, use of this flag is recommended, but it does create addi-
tional network traffic during startup in proportion to the number of
blocks that already exist.
--listBlocksThreads=NUM
To minimize startup delay, the initial block enumeration of
--listBlocks is performed in parallel using multiple threads. This
flag configures the number of threads used.
Default value is 16.
--maxUploadSpeed=BITSPERSEC
--maxDownloadSpeed=BITSPERSEC
These flags set a limit on the bandwidth utilized for individual block
uploads and downloads (i.e., the setting applies on a per-thread ba-
sis). The limits only apply to HTTP payload data and do not include
any additional overhead from HTTP or TCP headers, etc.
The value is measured in bits per second, and abbreviations like
`256k', `1m', etc. may be used. By default, there is no fixed limit.
Use of these flags may also require setting the --timeout flag to a
higher value.
--maxRetryPause=MILLIS
Specify the total amount of time in milliseconds s3backer should pause
when retrying failed HTTP operations before giving up. Failures in-
clude network failures and timeouts, HTTP errors, and reads of stale
data (i.e., MD5 mismatch); s3backer will make multiple retry attempts
using an exponential backoff algorithm, up to this maximum total retry
pause time. This value does not include the time it takes to perform
the HTTP operations themselves (use --timeout for that). Default
value is 30000 (30 seconds). See also --initialRetryPause.
--minWriteDelay=MILLIS
Specify a minimum time in milliseconds between the successful comple-
tion of a write and the initiation of another write to the same block.
This delay ensures that S3 doesn't receive the writes out of order.
This value must be set to zero when --md5CacheSize is set to zero (MD5
cache disabled). Default value is 500ms.
--md5CacheSize=SIZE
Specify the size of the MD5 checksum cache (in number of blocks). If
the cache is full when a new block is written, the write will block
until there is room. Therefore, it is important to configure
--md5CacheTime and --md5CacheSize according to the frequency of writes
to the filesystem overall and to the same block repeatedly. Alter-
nately, a value equal to the number of blocks in the filesystem elimi-
nates this problem but consumes the most memory when full (each entry
in the cache is approximately 40 bytes). A value of zero disables the
MD5 cache. Default value is zero (disabled).
--md5CacheTime=MILLIS
Specify in milliseconds the time after a block has been successfully
written for which the MD5 checksum of the block's contents should be
cached, for the purpose of detecting stale data during subsequent
reads. A value of zero means `infinite' and provides a guarantee
against reading stale data; however, you should only do this when
--md5CacheSize is configured to be equal to the number of blocks; oth-
erwise deadlock will (eventually) occur. This value must be at least
as big as --minWriteDelay. This value must be set to zero when
--md5CacheSize is set to zero (MD5 cache disabled). Default value is
10 seconds.
The MD5 checksum cache is not persisted across restarts. Therefore,
to ensure the same eventual consistency protection while s3backer is
not running, you must delay at least --md5CacheTime milliseconds be-
tween stopping and restarting s3backer.
--nbd Create a Network Block Device (NBD) instead of a FUSE filesystem.
This is probably a more logical and direct way to do things on systems
that support NBD.
s3backer will start up appropriate instances of nbdkit(1) and
nbd-client(1) in order to attach the S3 volume to the specified de-
vice.
When invoked in this mode, only --doubleDash style flags are sup-
ported, with the exception of -f and -d; any other flags (including
any FUSE-specific flags) are disallowed. The --nbd flag may not ap-
pear in a --configFile file.
To detach the S3 volume, just kill the s3backer process and it will
automatically disconnect the NBD client and shut down the NBD server.
--nbd-flag=string
--nbd-param=string
Add string as an nbdkit(1) command line parameter, either before or
after the plugin name (respectively).
--noAutoDetect
Disable block and file size auto-detection at startup. If this flag
is given, then the block size defaults to 4096 and the --size flag is
required.
--noCurlCache
Disable caching of cURL handles. This flag will reduce performance;
it exists to help in certain debugging situations.
--password=PASSWORD
Supply the password for encryption and authentication as a command-
line parameter.
--passwordFile=FILE
Read the password for encryption and authentication from (the first
line of) the specified file.
--prefix=STRING
Specify a prefix to prepend to the resource names within bucket that
identify each block. By using different prefixes, multiple indepen-
dent s3backer disks can live in the same S3 bucket.
Instead of using this flag, it is recommended to specify a prefix im-
plicitly by appending a slash and a subdirectory name to the bucket
name (the S3 bucket namespace is flat, so the notion of a subdirectory
is not real). The configured prefix then becomes the given subdirec-
tory name, followed by slash. The AWS web console will then display
the virtual disk's blocks grouped into a subfolder.
For example, specifying bucket mybucket/foo/bar is equivalent to spec-
ifying bucket mybucket along with the flag --prefix=foo/bar/ (note the
additional trailing slash).
Extraneous slash characters in a subdirectory name are disallowed.
This flag (or subdir) must be used consistently once a disk image is
established.
The default prefix is the empty string.
--quiet
Suppress progress output during initial startup.
--readAhead=NUM
Configure the number of blocks of read ahead. This determines how
many blocks will be read into the block cache ahead of the last block
read by the kernel when read ahead is active. This option has no ef-
fect if the block cache is disabled. Default value is 4 in FUSE mode,
zero in NBD mode.
--readAheadTrigger=NUM
Configure the number of blocks that must be read consecutively before
the read ahead algorithm is triggered. Once triggered, read ahead
will continue as long as the kernel continues reading blocks sequen-
tially. This option has no effect if the block cache is disabled.
Default value is 2 in FUSE mode, zero in NBD mode.
--readOnly
Assume the filesystem is going to be mounted read-only, and return
EROFS in response to any attempt to write. This flag also changes the
default mode of the backed file from 0600 to 0400 and disables the MD5
checksum cache.
--region=REGION
Specify an AWS region. This flag changes the default base URL to in-
clude the region name and automatically sets the --vhost flag, unless
the --no-vhost flag is used.
The default region is us-east-1.
--reset-mounted-flag
Reset the 'already mounted' flag on the underlying S3 data store.
s3backer detects simultaneous mounts by checking a special flag. If a
previous invocation of s3backer was not shut down cleanly, the flag
may not have been cleared. Running s3backer --erase will clear it
manually. But see also BUGS below.
--size=SIZE
Specify the size (in bytes) of the backed file to be exported by the
filesystem. The size may have an optional suffix 'K' for kilobytes,
'M' for megabytes, 'G' for gigabytes, 'T' for terabytes, 'E' for ex-
abytes, 'Z' for zettabytes (if supported), or 'Y' for yottabytes (if
supported). The total number of blocks must fit into 32 bits. Suf-
fixes represent powers of two.
s3backer will attempt to auto-detect the size of the backed file by
reading block number zero. If this option is not specified, the auto-
detected value will be used. If this option is specified but dis-
agrees with the auto-detected value, s3backer will exit with an error
unless --force is also given.
--sse=TYPE
Enable server side encryption. This adds the
x-amz-server-side-encryption header to all PUT requests.
The value must be either AES256 or aws:kms. In the latter case, a
customer-supplied key is used for encryption, and the --sse-key-id
flag is required.
--sse-key-id=ID
Specify the ID of the customer-supplied key for server-side encryp-
tion. This flag is required if --sse is given, ignored otherwise.
--ssl Equivalent to --baseURL https://s3.amazonaws.com/
--statsFilename=NAME
Specify the name of the human-readable statistics file that appears in
the s3backer filesystem. A value of empty string disables the appear-
ance of this file. Default is `stats'.
--storageClass=TYPE
Specify storage class.
Valid values are: STANDARD, STANDARD_IA, ONEZONE_IA,
REDUCED_REDUNDANCY, INTELLIGENT_TIERING, GLACIER, DEEP_ARCHIVE, and
OUTPOSTS.
The default is STANDARD.
--test Operate in local test mode. Filesystem blocks are stored as regular
files in the directory dir. No network traffic occurs.
Note if dir is a relative pathname (and -f is not given) it will be
resolved relative to the root directory.
--test-delays
In test mode, introduce random I/O delays.
--test-discard
In test mode, discard all data written, return zero for all data
blocks read, and perform no I/O operations. This mode is useful for
isolating the FUSE and s3backer performance overhead.
--test-errors
In test mode, introduce random I/O errors.
--timeout=SECONDS
Specify a time limit in seconds for one HTTP operation attempt. This
limits the entire operation including connection time (if not already
connected) and data transfer time. The default is 30 seconds; this
value may need to be adjusted upwards to avoid premature timeouts on
slower links and/or when using a large number of block cache worker
threads.
See also --maxRetryPause.
--version
Output version and exit.
--vhost
Force virtual hosted style requests. For example, this will cause
s3backer to use the URL http://mybucket.s3.amazonaws.com/path/uri in-
stead of http://s3.amazonaws.com/mybucket/path/uri.
This flag is required when S3 buckets have been created with location
constraints (for example `EU buckets'). Put another way, this flag is
required for buckets defined outside of the US region. This flag is
automatically set when the --region flag is used, unless the
--no-vhost flag is used.
--no-vhost
Disable virtual hosted style requests (the default).
In addition, s3backer accepts all of the generic FUSE options as well. Here
is a partial list:
-o uid=UID
Override the user ID of the backed file, which defaults to the current
user ID.
-o gid=GID
Override the group ID of the backed file, which defaults to the cur-
rent group ID.
-o sync_read
Do synchronous reads.
-o max_readahead=NUM
Set maximum read-ahead (in bytes).
-f Run in the foreground (do not fork). Causes logging to be sent to
standard error.
-d Enable FUSE debug mode. Implies -f.
-s Run in single-threaded mode.
In addition, s3backer passes the following flags which are optimized for
s3backer to FUSE (unless overridden by the user on the command line):
-o kernel_cache
-o fsname=<baseURL><bucket>/<prefix>
-o subtype=s3backer
-o use_ino
-o entry_timeout=31536000
-o negative_timeout=31536000
-o max_readahead=0
-o attr_timeout=0
-o default_permissions
-o allow_other
-o nodev
-o nosuid
FILES
$HOME/.s3backer_passwd
Contains Amazon S3 `accessID:accessKey' pairs.
/run/s3backer-nbd
Contains UNIX socket files used by nbd-client and nbdkit(1) when run-
ning in --nbd mode. For security reasons this directory is only read-
able by root.
SEE ALSO
curl(1), losetup(8), mount(8), nbdkit(1), umount(8), fusermount(8).
s3backer: FUSE-based single file backing store via Amazon S3,
https://github.com/archiecobbs/s3backer.
Amazon Simple Storage Service (Amazon S3), http://aws.amazon.com/s3.
FUSE: Filesystem in Userspace, http://fuse.sourceforge.net/.
MacFUSE: A User-Space File System Implementation Mechanism for Mac OS X,
http://code.google.com/p/macfuse/.
FUSE for OS X, https://osxfuse.github.io/.
Google Search for `linux page cache',
http://www.google.com/search?q=linux+page+cache.
Linux Block Layer Cache (bcache),
https://www.kernel.org/doc/Documentation/bcache.txt.
BUGS
Due to a design flaw in FUSE, an unmount of the s3backer filesystem will com-
plete successfully before s3backer has finished writing back all dirty blocks.
Therefore, when using the block cache, attempts to remount the same bucket and
prefix may fail with an 'already mounted' error while the former s3backer
process finishes flushing its cache. Before assuming a false positive and us-
ing --reset-mounted-flag, ensure that any previous s3backer process attached
to the same bucket and prefix has exited. See issue #40 for details.
For cache space efficiency, s3backer uses 32 bit values to index individual
blocks. Therefore, the block size must be increased beyond the default 4K
when very large filesystems (greater than 16 terabytes) are created.
On Mac OS X, the kernel imposes its own timeout (600 seconds) on FUSE opera-
tions, and automatically unmounts the filesystem when this limit is reached.
This can happen when a combination of --maxRetryPause and/or --timeout set-
tings allow HTTP retries to take longer than this value. A warning is emitted
on startup in this case.
Filesystem size is limited by the maximum allowable size of a single file.
The default block size of 4k is non-optimal from a compression and cost per-
spective. Typically, users will want a larger value to maximize compression
and minimize transaction costs, e.g., 1m.
When running under NBD, the stats file in not available.
AUTHOR
Archie L. Cobbs <[email protected]>
BSD September 7, 2009 BSD