Support multi s3 bucket

[rongfengliang]

Currently I do some tests. juicefs only support on bucket. any plan for support multi bucket

[davies]

Multiple buckets is support in latest main branch, please build it from source.

   --multi-buckets           use top level of directories as buckets (default: false)

[rongfengliang]

@davies from docs I see that's for gateway. Can we support adding multi buckets for juicefs format

[xiaogaozi]

@rongfengliang Yes you can, the format command supports --shards option, you could specify multi-buckets for one JuiceFS file system. For more information, please see #349 here.

[rongfengliang]

@rongfengliang Yes you can, the format command supports --shards option, you could specify multi-buckets for one JuiceFS file system. For more information, please see #349 here.

ok thanks!

[moonsphere]

@rongfengliang @xiaogaozi , sorry to interrupt, may I take the liberty to ask why we need multiple buckets instead of one bucket? do we meet any questions when storing all data in one bucket?

(I'm also a developer that wants to use s3 to store some data and confuse in "store all data in one buckets" vs "store them in multiple buckets"...but I'm freshmen to aws s3 and guess juicefs should have a lot experience about it~ so I found this issue :)

Thanks :)

[davies]

This is no problem to put all the object in single bucket, if the underlying object storage can store them efficiently. In practice, some of the system may have some limit on number of objects, or the performance degrades with billions of objects, multiple buckets will be useful in these cases.

[rongfengliang]

@davies Will support Multi s3 endpoint not only Multi bucket. We use minio as our s3 storage, but minio is not better for scale up

[davies]

It's doable. You can create a DNS for each endpoint, for example, http://endpoint1.example.com:9000, then use http://endpoint%d.example.com:9000 as the bucket when format a JuiceFS volume.

[rongfengliang]

@davies use domain is a better choice. but if current we only use IP addresses maybe need support Multi endpoint

[solracsf]

May I ask what's the difference between --shards and --multi-buckets?

[davies]

There is no --template, use --bucket instead

[xunfeng1980]

Hi,minio gateway for juice can support multi juicefs filesystem？ --multi-buckets not is multi juicefs filesystem.

[xiaogaozi]

@xunfeng1980 No, each S3 gateway can only correspond to one JuiceFS file system, so you need to start multiple S3 gateway services.

[Sinnevy]

@xiaogaozi 可以仅用一个s3网关控制多个bucket，来进行多租户管理吗，每个bucket对应一个租户，期望可以通过s3网关创建bucket并设置容量

[davies]

使用完整功能的 gateway, 再结合 --multi-buckets 参数，应该可以实现多租户。