Abysmal performance and OOM issues running a large mv task with reflink support enabled?

[Soukyuu]

Hello,

I'm quite new to ZFS (coming from btrfs) and am currently attempting to mv my data from a pool's root into a dataset:

mv -v /mnt/vault/myfolder/* /mnt/vault/dataset/

The process starts fine, though a bit slow for my taste as it should be using reflinks, not copying (bcloneused rising), then slows down more and more until it stalls the whole PC and I get messages about processes getting killed due to going OOM.

In htop I can see most of my CPU usage coming from the kernel, the mv process being stuck in disk sleep most of the time and weirdly, even starting something like a browser gets stuck in disk sleep even thought it's on a completely different disk (SSD) and even controller!

Memory usage in green (=used) also seems to be steadily climbing, with cache (orange) declining.

What I have tried so far (one fix active at a time) was:

• limiting ARC size -> does not seem to help in the long run, but gives me a few hours of copying
• setting zfs_arc_shrinker_limit=0 -> helps a bit, but on large mv operations it still runs OOM
• disabling MGLRU -> does not help, OOM under 10 minutes
• setting MGLRU's min_ttl_ms=0 does not really help, OOM under 20 minutes

The last three fix attempts come from reading #10255

At this point, having heard all about ZFS stability and ease of use, I'm most certainly doing something wrong. It does not really help that nothing zfs related seems to be popping up as using memory in any monitoring tools I tried so far.

• arch linux with current lts kernel
• openZFS 2.2.6
• 8GB of RAM

Any ideas?

[amotin]

I suspect it might be related to block cloning, used by mv between dataset. While in theory after source file is deleted, respective records in BRT table should be deleted also, reducing memory consumption back. In perfect case if it happen within the same transaction group it should not even reach the disk. But I guess that in case of a large files window between cloning and deleting can be big enough to require on-disk BRT writes, that may cause unaccounted additional memory overhead that might be impossible to evict on kernel request, that might trigger the OOMs you see. Last week I've made couple PR to reduce the memory overhead a bit, but that is a drop in the bucket. It needs a deeper look.

PS: What recordsize do you use and what are the sizes of files you move?

[Soukyuu]

Hmm I see. I am using whatever record size is the default on 2.2.6, I'm not sure where to check. I am copying several years worth of digital photos, so tens of thousands of ~20Mb files, mostly. The files are sorted into year/month_day/ folders, so not one huge set of files mv'ed at once.

There are also some 1GB+ files, but only a few. I'm already running OOM on the photos though, which is rather disturbing, hopefully those did not lead to any sort of silent corruption...

It seems that there is some sort of disk queue, as even after the process has finished, the HDDs are very busy for up to 10 minutes. Maybe the relatively small write of incrementing the ref count compared to actually writing the whole file puts more load on the HDDs?

edit: the only parameter I changed from default when creating the pool was ashift=12, the pool itself is a raidz with 3x1TB HDDs

[amotin]

It seems that there is some sort of disk queue, as even after the process has finished, the HDDs are very busy for up to 10 minutes.

I think we may underestimate the amount of required modifications when spreading the operations between transaction group. As result, when we finally get to the committing the transaction group, we get overwhelmed by the amount of BRT work that has to be done at once. 10 minutes after the move completion I would not even imagine, since transaction groups are expected to complete within seconds. What makes me wonder is whether you already have significant BRT table before even starting the data move. It would be good to see zdb -TT POOL output in a steady state. Or may be zpool get all | grep bclone.

[Soukyuu]

It would be good to see zdb -TT POOL output in a steady state.

BRT: used 256G; saved 256G; ratio 2.00x
BRT: vdev 0: refcnt 2.02M; used 256G; saved 256G

Or may be zpool get all | grep bclone

vault  bcloneused                     256G                           -
vault  bclonesaved                    256G                           -
vault  bcloneratio                    2.00x                          -

Ah yeah, what has led me to this situation was: initially, I created a pool without any datasets and copied two user folder into pool's root. I later on realized I should have created a dataset per user, so I went ahead and created two datasets, then attempted to mv my stuff into them. So initially, there was no BRT, from what I understand, it grew as mv was doing its cp --reflink=auto src dst && rm src thing.

[amotin]

If you don't have the mv running now, then you have 256GB of cloned data on the pool besides it. It does not sound like much to cause problems, but your system is not big either. I guess having that "static" part of the BRT table can make the move operation more expensive than it would be otherwise.

[Soukyuu]

Yes, it wasn't running. I have started removing the old folders using rm -rf folder/* and it's taking forever as well, running into disk sleep. I'm assuming this is because it's now cleaning up the cloned data.

At least to me, it looks like I shouldn't be using the reflink feature for now. It might save some space, but surely not my nerves 😄

I'll keep an eye on future releases and any performance improvements in regards to BRT/reflinks for now, thank you for taking your time to look into it!