Very slow performance on git clone to a mounted folder

[thiagoalvescosta]

Very slow performance on git clone with a thousands of small files.

1. Mount a folder using AWS MemoryDB Redis as metadata database on EC2 instance on same subnet
2. Enter into the mounted folder and execute time git clone https://github.com/thiagoalvescosta/jose123.git
   Result:
   real 2m12.502s
   user 0m3.184s
   sys 0m3.836s
3. Enter into /tmp/ folder and execute time git clone https://github.com/thiagoalvescosta/jose123.git
   Result:
   real 0m2.848s
   user 0m2.348s
   sys 0m0.912s

The difference is very big. I don't know why. I also tried with --writeback and was the same

I executed: juicefs bench /home/ubuntu/juicefs -p 4 and got:

Write big blocks count: 4096 / 4096 [==============================================================] done Read big blocks count: 4096 / 4096 [==============================================================] done Write small blocks count: 400 / 400 [==============================================================] done Read small blocks count: 400 / 400 [==============================================================] done Stat small files count: 400 / 400 [==============================================================] done Benchmark finished! BlockSize: 1 MiB, BigFileSize: 1024 MiB, SmallFileSize: 128 KiB, SmallFileCount: 100, NumThreads: 4 Time used: 37.5 s, CPU: 93.9%, Memory: 532.3 MiB +------------------+------------------+---------------+ | ITEM | VALUE | COST | +------------------+------------------+---------------+ | Write big file | 355.08 MiB/s | 11.54 s/file | | Read big file | 243.04 MiB/s | 16.85 s/file | | Write small file | 140.7 files/s | 28.43 ms/file | | Read small file | 594.6 files/s | 6.73 ms/file | | Stat file | 1646.3 files/s | 2.43 ms/file | | FUSE operation | 75643 operations | 1.96 ms/op | | Update meta | 1277 operations | 7.64 ms/op | | Put object | 1424 operations | 152.35 ms/op | | Get object | 0 operations | 0.00 ms/op | | Delete object | 0 operations | 0.00 ms/op | | Write into cache | 1424 operations | 218.87 ms/op | | Read from cache | 1430 operations | 290.81 ms/op | +------------------+------------------+---------------+

Environment:

• JuiceFS version: 1.0.4+2023-04-06.f1c475d
• Cloud provider or hardware configuration running JuiceFS: AWS EC2 Instance m5a.xlarge
• OS (e.g cat /etc/os-release):

NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

• Kernel: Linux ip-10-20-19-33 4.4.0-131-generic invalidate read buffer after write #157-Ubuntu SMP Thu Jul 12 15:51:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Object storage: memoryDB Redis 6.2 db.t4g.medium same subnet of EC2 instance
• Metadata engine info: Redis 6.2

[SandyXSD]

Basically JuiceFS (and most of distributed systems) is not suitable for workloads like git as it has too many "small" operations. However, if you'd like to take a try, there are two suggestions that may make it smoother:

1. For git clone, clone the repo locally first and then mv it into a JuiceFS mount point
2. For daily git use, aggressively cache the metadata by increasing attr-cache, entry-cache and dir-entry-cache in mount command (note this may lead to inconsistent view on multiple hosts, see JuiceFS metadata cache)

[thiagoalvescosta]

Thanks for the answer. For the use I need, I can't clone and then move. And the options did not help. It's weird that after the clone everything works really fast. I currently use s3backer and it appears to perform the same as a local disk both in the clone and in the rest of the use. What makes juicefs slow in clone?

[solracsf]

Did you try juicefs mount -d -o writeback_cache --writeback --upload-delay=30 if this helps?

[thiagoalvescosta]

I tried and same result!

[davies]

git issue many operations on metadata, which require fast meta engine. The MemoryDB is VERY slow to write, because it persistents log to a distributed storage for every transaction.

So, it's expected to be VERY slow for git when MemoryDB is used as meta engine.

[thiagoalvescosta]

What you recommend? Use a sql database?

[solracsf]

Redis should give you the best performance here. If you don't need access from multiple hosts, also consider BadgerDB.

[bck01215]

Having a similar issue. I'm using juicefs as a CSI. I have minio running with virtualbox and consul currently. The benchmarks are not great. Any idea where to troubleshoot slowness? metadata is in redis

[xylcbd]

与ubuntu系统自带的ext4文件系统相比，这个juicefs的小文件随机读写速度慢太多了。1万个小文件（大约120MB），读取需要7.6s（ext4只需要1.6秒），写入需要7分钟（ext4只需要30秒）。实验过程是：

# 安装
curl -sSL https://d.juicefs.com/install | sh -

# 创建文件系统
juicefs format sqlite3://myjfs.db myjfs

# 挂载文件系统
juicefs mount sqlite3://myjfs.db ~/tmp/myjfs

# 写入数据（120MB，共10709个文件），耗时7分钟14秒
time cp -r ~/samples ~/tmp/myjfs/

# 读取数据（普通磁盘），耗时1.8秒
time zip -qr ~/xxx.zip samples/

# 读取数据（juicefs），耗时7.6秒
time zip -qr ~/xxx.zip samples/

[flying-x]

I am facing the same issue and I started a discussion on an idea here: #4853

However, I think this problem might be solvable if there is a distributed in-memory KV store with local caching. If each juicefs mounts a redis endpoint like redis://localhost:6379/ the latency to talk to the local metadata server will be very low.

Now, the local KV store must be caching the reads and propagating the writes to the entire cluster. It seems on the surface that this thing can do it? https://github.com/distributedio/titan

Basically, 2 questions:

1. is there a distributed kv store with caching that people have used successfully in production here?
2. if so, does it support auto-scaling, i.e. I have add / remove more nodes to the metadata cluster easily as I am adding more compute node & juicefs client.

Any pointers?

[min-xu-et]

I tried compiling linux kernel on 4 machine in parallel, I think I got something bad:

rm: cannot remove 'test3/src/linux/Documentation/devicetree/bindings/hwmon': Operation not permitted

But I don't see any errors in dmesg or juicefs stdout/stderr. Is this a corrupted FS? I am already using your patched version. But do I need to (1) enable some flags or (2) reformat the using the patched version?

[williamstein]

Many thanks for looking at my patches.

Use juicefs fsck and juicefs gc to check for corruption. The sort of corruption I dealt with comes from multiple nodes assigning the SAME inodes to files... which later leads to a situation that I think gets detected by fsck and/or gc. It is I think normal behavior for juicefs fsck to output some "inodes with missing chunks", which is concerning at first, but it often just data that hasn't been garbage collected yet -- do juicefs gc --delete then try again.

If you do see actual corruption as detected by juicefs fsck, then that's very serious, in my opinion.

Is there any flags I needed to set when I mount juicefs? I don't want to accidentally not enable your code path and didn't test it properly.

Did you use the client-id flag option I mentioned? That is critical, and if you don't use it, then things can break:

  --client-id value       numerical id with 0 <= client-id < 1024 that you should explicitly assign to this client.  These client-id's should be DISTINCT between all current mounted clients!  The client id is used to ensure that there is no overlap between clients when assigning inode and slice ids.  This is critical to support asynchronous replication using Redis. (default: 0)

(I increased the default batch-size to 4096, which I think cuts down the probability of failure from upstream.)

If you want to very easily see corruption, I also added a different flag to mount:

--batch-size value      the batch size used for inode and slice ids. Set to something small to greatly increase the chances of filesystem corruption if you write form multiple clients and do not properly set client-id. (default: 4096)

You can do juicefs mount --batch-size=10 ... and do NOT give the client-id option, and then it should be easy to cause corruption by creating lots of files at the same time on different clients.

(If any JuiceFS devs are reading this, please be forgiving of my relative ignorance about the internals of JuiceFS, and thanks for any feedback. I really, really love what you have built!)

[williamstein]

Also relevant to this thread -- the lead KeyDB developer made some very relevant comments today about the future roadmap of KeyDB here: Snapchat/KeyDB#798
In particular, he talks about what Snapchat actually seriously uses in KeyDB, and why, and also how they prioritize what will come in the future. What's relevant to JuiceFS:

• they are putting major work into FLASH support, which makes it so less used data is stored on disk, which could help enormously for using JuiceFS with 10 million+ files. He sounds confident that this will get open sourced in keydb, but only when it's robustly working or them in production.
• they care a lot about vertically scaling. I don't know if this matters at all for JuiceFS?
• they do not care about multimaster replication. This is worrisome for this thread. Fortunately, the actual multimaster implementation is evidently a fairly minimal change, so this might not be too bad.

[flying-x]

Great news! After setting client-id, it seems the errors has gone away! The kernel compiling are going well on all 4 nodes simultaneously. After everything is done from the test, I will do a fsck to verify!

I wish there is a way to automatically set a client-id on mount. Otherwise, I need to manually assign and set the ID whenever I add a new machine/VM.

The keydb pointer is really interesting. It would be very sad if multi-master isn't supported by them. I don't know any other good solution as a multiple master metadata server for juicefs. Even the keydb multi-master is a bit too "eventual consistent" for me. I tried deleting a key from a master and it keeps on getting re-inserted by the replication. I don't know enough the internal workings of the multi-master consistency semantics, but it feels that we are a bit fragile here. But hey, it is probably good enough for a lot of use cases already!

Another thing could potentially be problematic is that with local metadata server (keydb process) sharing the same cpu/memory/network resources as the datapath (juicefs process and s3 data transfers), it can have bad resource competition and priority inversion. Maybe that's why some folks are not in favor of this kind of setup? I don't really know.

(ps. sorry for switching the accounts above. I forgot to switch from company github account to the personal one. lol)

[williamstein]

I wish there is a way to automatically set a client-id on mount. Otherwise, I need to manually assign and set the ID whenever I add a new machine/VM.

I wasn't able to solve that problem in a way I could 100% trust. For CoCalc.com I have a central server that allocates wireguard VPN credentials (the keydb's all communicate over a wireguard vpn), etc., to orchestrate all clients, so I have a single place where I can safely assign a global id, and it's all automated. This problem of client-id allocation can likely be solved using redis locking or pub/sub or something similar, but I didn't need a solution for my use case, so I didn't solve it.

It would be very sad if multi-master isn't supported by them.

I don't know any other good solution as a multiple master metadata server for juicefs.

I think this is the sort of thing that is typically only available in the complementary non-open source products. E.g., I think redis enterprise has multimaster "While native Redis doesn't support multi-master replication out-of-the-box, there are third-party solutions that have been developed to provide this capability. One popular solution is Redis Enterprise, offered by Redis Labs, which includes conflict-free replicated data types (CRDTs) that allow active-active geo-distributed deployments." (see https://www.dragonflydb.io/faq/redis-replication-multi-master). Also, maybe the JuiceFS commercial metadata server also does something similar.

It would be very sad if multi-master isn't supported by them.

My only solace is this section https://docs.keydb.dev/docs/active-rep#technical-implementation -- if you click through the link and look at the commit that implements most of multimaster, it's just a few lines of code.

Even the keydb multi-master is a bit too "eventual consistent" for me. I tried deleting a key from a master and it keeps on getting re-inserted by the replication.

I am not seeing this behavior. Maybe there is something wrong with how you are configuring multimaster. The following screenshot is a session where I create and delete a key on each side and it works. This is on two separate VM's entirely:

Regarding resources, it probably depends on the use case. Thanks for the clarification regarding your accounts. Finally, if you (or anybody else interested in this) wants to poke around any of this cocalc implementation of JuiceFS+KeyDB yourself, make an account at https://cocalc.com, click the help button and make a support request. I'll receive it and can give you some free trial credit. We also added this to our docs at https://doc.cocalc.com/cloud_file_system.html