-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
45 changed files
with
2,812 additions
and
2,850 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,5 @@ | ||
# | ||
# s3fs - FUSE-based file system backed by Amazon S3 | ||
# ossfs - FUSE-based file system backed by InspurCloud OSS | ||
# | ||
# Copyright(C) 2007 Randy Rizun <[email protected]> | ||
# | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,5 @@ | ||
# | ||
# s3fs - FUSE-based file system backed by Amazon S3 | ||
# ossfs - FUSE-based file system backed by InspurCloud OSS | ||
# | ||
# Copyright(C) 2007 Randy Rizun <[email protected]> | ||
# | ||
|
@@ -72,9 +72,10 @@ missing | |
*.log | ||
*.trs | ||
default_commit_hash | ||
src/s3fs | ||
src/ossfs | ||
src/test_* | ||
test/s3proxy-* | ||
dist | ||
|
||
# | ||
# Local variables: | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,5 @@ | ||
# | ||
# s3fs - FUSE-based file system backed by Amazon S3 | ||
# ossfs - FUSE-based file system backed by InspurCloud OSS | ||
# | ||
# Copyright(C) 2007 Randy Rizun <[email protected]> | ||
# | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,5 @@ | ||
###################################################################### | ||
# s3fs - FUSE-based file system backed by Amazon S3 | ||
# ossfs - FUSE-based file system backed by InspurCloud OSS | ||
# | ||
# Copyright 2007-2008 Randy Rizun <[email protected]> | ||
# | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,77 +1,51 @@ | ||
s3fs | ||
==== | ||
|
||
s3fs allows Linux and macOS to mount an S3 bucket via FUSE. | ||
s3fs preserves the native object format for files, allowing use of other | ||
tools like [AWS CLI](https://github.com/aws/aws-cli). | ||
[![Build Status](https://travis-ci.org/s3fs-fuse/s3fs-fuse.svg?branch=master)](https://travis-ci.org/s3fs-fuse/s3fs-fuse) | ||
|
||
Features | ||
-------- | ||
|
||
* large subset of POSIX including reading/writing files, directories, symlinks, mode, uid/gid, and extended attributes | ||
* compatible with Amazon S3, Google Cloud Storage, and other S3-based object stores | ||
* large files via multi-part upload | ||
* renames via server-side copy | ||
* optional server-side encryption | ||
* data integrity via MD5 hashes | ||
* in-memory metadata caching | ||
* local disk data caching | ||
* user-specified regions, including Amazon GovCloud | ||
* authenticate via v2 or v4 signatures | ||
# OSSFS | ||
|
||
Installation from pre-built packages | ||
------------------------------------ | ||
### 简介 | ||
|
||
Some systems provide pre-built packages: | ||
ossfs能让您在Linux/Mac OS X 系统中把inspur oss的bucket挂载到本地文件系统中,您能够便捷的通过本地文件系统操作OSS上的对象,实现数据的共享。 | ||
|
||
* On Debian 9 and Ubuntu 16.04 or newer: | ||
### 功能 | ||
|
||
``` | ||
sudo apt-get install s3fs | ||
``` | ||
ossfs基于s3fs构建,具有s3fs的全部功能。主要功能包括: | ||
|
||
* On SUSE 12 or newer and openSUSE 42.1 or newer: | ||
* 支持posix文件系统的大部分功能,包括文件读写,目录,链接操作,权限,uid/gid,以及扩展属性(extended attributes)。 | ||
* 通过oss的multipart功能上传大文件。 | ||
* 通过server-side copy实现重命名。 | ||
* MD5 校验保证数据完整性。 | ||
* 元数据缓存。 | ||
* 数据缓存。 | ||
* 自定义region。 | ||
|
||
``` | ||
sudo zypper in s3fs | ||
``` | ||
### 安装 | ||
|
||
* On Fedora 27 and newer: | ||
``` | ||
sudo yum install s3fs-fuse | ||
``` | ||
#### 预编译的安装包 | ||
|
||
* On RHEL/CentOS 7 and newer through EPEL repositories: | ||
``` | ||
sudo yum install epel-release | ||
sudo yum install s3fs-fuse | ||
``` | ||
我们为常见的linux发行版制作了安装包: | ||
|
||
* On Amazon Linux through EPEL repositories: | ||
- Ubuntu-16.04 | ||
- CentOS-7.0 | ||
|
||
``` | ||
sudo amazon-linux-extras install epel | ||
sudo yum install s3fs-fuse | ||
``` | ||
对于Ubuntu,安装命令为: | ||
|
||
* On macOS, install via [Homebrew](https://brew.sh/): | ||
|
||
``` | ||
brew cask install osxfuse | ||
brew install s3fs | ||
``` | ||
``` | ||
sudo apt-get update | ||
sudo apt-get install fuse libcurl4-gnutls libxml2 libssl | ||
sudo dpkg -i ossfs_package | ||
``` | ||
|
||
Compilation and installation from sources | ||
----------------------------------------- | ||
对于CentOS6.5及以上,安装命令为: | ||
|
||
These are generic instructions to compile from the master branch, and should work on almost any GNU/Linux, macOS, BSD, or similar. | ||
``` | ||
sudo yum localinstall ossfs_package | ||
``` | ||
|
||
If you want specific instructions for some distributions, check the [wiki](https://github.com/s3fs-fuse/s3fs-fuse/wiki/Installation-Notes). | ||
#### 源码安装 | ||
|
||
Keep in mind using the pre-built packages when available. | ||
|
||
1. Ensure your system satisfies build and runtime dependencies for: | ||
1. 保证具有以下依赖: | ||
|
||
* fuse >= 2.8.4 | ||
* automake | ||
|
@@ -81,120 +55,103 @@ Keep in mind using the pre-built packages when available. | |
* libxml2 | ||
* openssl | ||
|
||
2. Then compile from master via the following commands: | ||
可以采用一下命令进行安装依赖 | ||
|
||
Ubuntu: | ||
|
||
``` | ||
sudo apt-get install automake autotools-dev g++ git libcurl4-gnutls-dev \ | ||
libfuse-dev libssl-dev libxml2-dev make pkg-config | ||
``` | ||
|
||
CentOS: | ||
|
||
``` | ||
sudo yum install automake gcc-c++ git libcurl-devel libxml2-devel \ | ||
fuse-devel make openssl-devel | ||
``` | ||
|
||
|
||
2. 编译安装: | ||
|
||
``` | ||
git clone https://github.com/s3fs-fuse/s3fs-fuse.git | ||
cd s3fs-fuse | ||
cd ossfs-fuse | ||
./autogen.sh | ||
./configure | ||
make | ||
sudo make install | ||
``` | ||
|
||
Examples | ||
-------- | ||
|
||
s3fs supports the standard | ||
[AWS credentials file](https://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html) | ||
stored in `${HOME}/.aws/credentials`. Alternatively, s3fs supports a custom passwd file. | ||
|
||
The default location for the s3fs password file can be created: | ||
### 运行 | ||
|
||
* using a .passwd-s3fs file in the users home directory (i.e. ${HOME}/.passwd-s3fs) | ||
* using the system-wide /etc/passwd-s3fs file | ||
|
||
Enter your credentials in a file `${HOME}/.passwd-s3fs` and set | ||
owner-only permissions: | ||
设置access key/id信息,将其存放在`${HOME}/.passwd-ossfs`或者`/etc/passwd-ossfs`文件中, | ||
注意这个文件的权限必须正确设置,建议设为600. | ||
|
||
将access key/id信息保存到`${HOME}/.passwd-ossfs`中,并设置权限 | ||
``` | ||
echo ACCESS_KEY_ID:SECRET_ACCESS_KEY > ${HOME}/.passwd-s3fs | ||
chmod 600 ${HOME}/.passwd-s3fs | ||
echo ACCESS_KEY_ID:SECRET_ACCESS_KEY > ${HOME}/.passwd-ossfs | ||
chmod 600 ${HOME}/.passwd-ossfs | ||
``` | ||
|
||
Run s3fs with an existing bucket `mybucket` and directory `/path/to/mountpoint`: | ||
将已创建好的桶`mybucket`挂载到指定目录`/path/to/mountpoint`: | ||
|
||
``` | ||
s3fs mybucket /path/to/mountpoint -o passwd_file=${HOME}/.passwd-s3fs | ||
ossfs mybucket /path/to/mountpoint | ||
``` | ||
#### 示例 | ||
|
||
If you encounter any errors, enable debug output: | ||
挂载: | ||
|
||
``` | ||
s3fs mybucket /path/to/mountpoint -o passwd_file=${HOME}/.passwd-s3fs -o dbglevel=info -f -o curldbg | ||
``` | ||
|
||
You can also mount on boot by entering the following line to `/etc/fstab`: | ||
|
||
``` | ||
s3fs#mybucket /path/to/mountpoint fuse _netdev,allow_other 0 0 | ||
``` | ||
|
||
or | ||
|
||
``` | ||
mybucket /path/to/mountpoint fuse.s3fs _netdev,allow_other 0 0 | ||
``` | ||
|
||
If you use s3fs with a non-Amazon S3 implementation, specify the URL and path-style requests: | ||
将`mybucket`这个bucket挂载到`/tmp/ossfs`目录下,AccessKeyId是`osstestAccessKeyId`, | ||
AccessKeySecret是`osstestAccessKeySecret`,oss endpoint是`http://oss.cn-north-3.inspurcloudoss.com/` | ||
|
||
``` | ||
s3fs mybucket /path/to/mountpoint -o passwd_file=${HOME}/.passwd-s3fs -o url=https://url.to.s3/ -o use_path_request_style | ||
echo osstestAccessKeyId:osstestAccessKeySecret > ${HOME}/.passwd-ossfs | ||
chmod 600 ${HOME}/.passwd-ossfs | ||
mkdir /tmp/ossfs | ||
ossfs mybucket /tmp/ossfs -ourl=http://oss.cn-north-3.inspurcloudoss.com/ | ||
``` | ||
|
||
or(fstab) | ||
``` | ||
s3fs#mybucket /path/to/mountpoint fuse _netdev,allow_other,use_path_request_style,url=https://url.to.s3/ 0 0 | ||
``` | ||
卸载bucket: | ||
|
||
To use IBM IAM Authentication, use the `-o ibm_iam_auth` option, and specify the Service Instance ID and API Key in your credentials file: | ||
```bash | ||
umount /tmp/ossfs # root user | ||
fusermount -u /tmp/ossfs # non-root user | ||
``` | ||
echo SERVICEINSTANCEID:APIKEY > /path/to/passwd | ||
``` | ||
The Service Instance ID is only required when using the `-o create_bucket` option. | ||
|
||
Note: You may also want to create the global credential file first | ||
#### 常用设置 | ||
|
||
``` | ||
echo ACCESS_KEY_ID:SECRET_ACCESS_KEY > /etc/passwd-s3fs | ||
chmod 600 /etc/passwd-s3fs | ||
``` | ||
* 如果使用ossfs的机器是浪潮云ECS,可以使用内网域名来**避免流量收费**和**提高速度**: | ||
|
||
Note2: You may also need to make sure `netfs` service is start on boot | ||
``` | ||
ossfs mybucket /tmp/ossfs -ourl=http://oss-innet.cn-north-3.inspurcloudoss.com/ | ||
``` | ||
|
||
* 遇到错误可以打开debug模式 | ||
|
||
Limitations | ||
----------- | ||
``` | ||
ossfs mybucket /path/to/mountpoint -o passwd_file=${HOME}/.passwd-ossfs -o dbglevel=info -f -o curldbg | ||
``` | ||
|
||
Generally S3 cannot offer the same performance or semantics as a local file system. More specifically: | ||
* 使用`ossfs --help`来查看更多可用的参数 | ||
|
||
* random writes or appends to files require rewriting the entire file | ||
* metadata operations such as listing directories have poor performance due to network latency | ||
* [eventual consistency](https://en.wikipedia.org/wiki/Eventual_consistency) can temporarily yield stale data([Amazon S3 Data Consistency Model](https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel)) | ||
* no atomic renames of files or directories | ||
* no coordination between multiple clients mounting the same bucket | ||
* no hard links | ||
* inotify detects only local modifications, not external ones by other clients or tools | ||
### 局限性 | ||
|
||
References | ||
---------- | ||
与本地文件系统相比,ossfs提供的文件系统具有一些局限性。包括: | ||
|
||
* [goofys](https://github.com/kahing/goofys) - similar to s3fs but has better performance and less POSIX compatibility | ||
* [s3backer](https://github.com/archiecobbs/s3backer) - mount an S3 bucket as a single file | ||
* [S3Proxy](https://github.com/gaul/s3proxy) - combine with s3fs to mount Backblaze B2, EMC Atmos, Microsoft Azure, and OpenStack Swift buckets | ||
* [s3ql](https://github.com/s3ql/s3ql/) - similar to s3fs but uses its own object format | ||
* [YAS3FS](https://github.com/danilop/yas3fs) - similar to s3fs but uses SNS to allow multiple clients to mount a bucket | ||
* 随机或者追加写文件会导致整个文件的重写。 | ||
* 元数据操作,例如list directory,性能较差,因为需要远程访问oss服务器。 | ||
* 文件/文件夹的rename操作不是原子的。 | ||
* 多个客户端挂载同一个oss bucket时,依赖用户自行协调各个客户端的行为。例如避免多个客户端写同一个文件等等。 | ||
* 不支持hard link。 | ||
* 不建议用在高并发读/写的场景,这样会让系统的load升高 | ||
|
||
Frequently Asked Questions | ||
-------------------------- | ||
* [FAQ wiki page](https://github.com/s3fs-fuse/s3fs-fuse/wiki/FAQ) | ||
* [s3fs on Stack Overflow](https://stackoverflow.com/questions/tagged/s3fs) | ||
* [s3fs on Server Fault](https://serverfault.com/questions/tagged/s3fs) | ||
### 相关链接 | ||
|
||
License | ||
------- | ||
* [s3fs](https://github.com/s3fs-fuse/s3fs-fuse) - 通过fuse接口,挂载s3 bucket到本地文件系统。 | ||
|
||
Copyright (C) 2010 Randy Rizun <[email protected]> | ||
### License | ||
|
||
Licensed under the GNU GPL version 2 | ||
Copyright (C) 2010 Randy Rizun [email protected] | ||
|
||
Licensed under the GNU GPL version 2 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.