Should we have a warning for primarycache=metadata, secondarycache=all?

[kftsehk]

TL;DR

When setting primarycache and secondarycache, unclear doc, no warning, misleading analogue. Shall this most likely misconfiguration

primarycache=metadata
secondarycache=all

deserves a warning or so?

The Problem

The candidate entry to populate L2ARC is from evicted entries of ARC, but not requested entries of ARC.
This is very different from how L1-L3 CPU cache works by pulling requested entries through high level cache into low level cache, although the name is somewhat suggestive of their comparability.

The configuration, suggesting that you might be caching metadata in ARC, but metadata + data in L2ARC

primarycache=metadata
secondarycache=all

is practically equal to

primarycache=metadata
secondarycache=metadata

because what pops up in ARC is what possibly cached L2ARC.

Quoting from some docs

I just googled zfs primarycache secondarycache and look for some trustworthy sources

https://docs.oracle.com/cd/E19253-01/819-5461/ghbxt/index.html

Setting cache properties on a ZFS file system – Two new ZFS file system properties enable you to control what is cached in the primary cache (ARC) and the secondary cache (L2ARC). The cache properties are set as follows:

primarycache – Controls what is cached in the ARC.

secondarycache – Controls what is cached in the L2ARC.

Possible values for both properties – all, none, and metadata. If set to all, both user data and metadata are cached. If set to none, neither user data nor metadata is cached. If set to metadata, only metadata is cached. The default is all.

You can set these properties on an existing file system or when a file system is created. For example:

zfs set primarycache=metadata tank/datab
zfs create -o primarycache=metadata tank/newdatab

No mention of eviction or what can get into L2ARC

https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Workload%20Tuning.html

Data is evicted from the first list while an effort is made to keep data in the second list. In this way, ARC is able to outperform LRU by providing a superior hit rate.

In addition, a dedicated cache device (typically a SSD) can be added to the pool, with zpool add POOLNAME cache DEVICENAME. The cache device is managed by the L2ARC, which scans entries that are next to be evicted and writes them to the cache device. The data stored in ARC and L2ARC can be controlled via the primarycache and secondarycache zfs properties respectively, which can be set on both zvols and datasets. Possible settings are all, none and metadata.

Mentioned eviction but unclear if data cached in ARC == data ever existed transiently in ARC, if any, in order to copy the value

https://youtu.be/ptY6-K78McY?t=1687

paraphrasing: ARC can stores data, or a pointer to L2ARC but not the actual data, so the data can be retrieved as an L2ARC hit
... much later in the talk, ZIO copy the requested block to DMU's dbuf then uio_t's buffer (?)
... compressed ARC ... responsibility of decompression goes up to the ARC, and decompressed data only momentarily exist in DMU

I don't think I understand everything here tbh, but seem to explains:

We have an ARC with only metadata entries, so we cannot find a pointer in the ARC that points to a piece of user data in the L2ARC. The title of the talk is "Lecture on OpenZFS read and write code paths", which in my opinion is too much for the purpose of setting this one flag.

Experiment

Conclusion: primarycache=metadata, secondarycache=all is equal to secondarycache=metadata

# primarycache=metadata, secondarycache=all
  time  read  miss%  dmis  dh%  pmis  ph%  mread  mm%  size     c  el2el  el2mfu  el2mru  el2inel  l2read  l2bytes  l2miss%  l2size  grow  need  free  avail  waste
16:57:12   413     49   203   50     0    0    210    0  109M  2.0G      0       0       0        0     203        0      100     16K     1     0  3.1G   1.1G    29K
16:57:13   644     33   192   69    22    0    430    0  152M  2.0G      0       0       0        0     214        0      100     16K     1     0  3.1G   1.1G    29K
16:57:14  1.3K     33     1   99   442    0    723    0  602M  2.0G      0       0       0        0     443        0      100     16K     1     0  2.6G   596M    29K
16:57:15  1.1K     30     0  100   327    0    562    0  931M  2.0G      0       0       0        0     327        0      100    7.0M     1     0  2.3G   285M    29K
16:57:16  1.6K     27     0  100   433    0    819    0  1.3G  2.0G      0       0       0        0     433        0      100     33M     0     0  1.9G  -145M    29K
16:57:17  1.1K     28     0  100   320    0    587    0  1.6G  2.0G      0       0       0        0     320        0      100     33M     0     0  1.6G  -426M    29K
16:57:18  1.5K     25     0  100   395    0    787    0  2.0G  2.0G      0       0       0        0     395        0      100     33M     0     0  1.2G  -785M    29K
$ date; zfs set primarycache=all brick; date
Fri Oct 13 04:57:11 PM HKT 2023
Fri Oct 13 04:57:19 PM HKT 2023
16:57:19   960     21     0  100   205    0    568    0  2.0G  2.0G   174M    174M       0      46M     205        0      100     33M     0     0  1.3G  -751M   7.0K
16:57:20  1.8K     25     0  100   468    0    929    0  2.1G  2.0G   429M    429M       0        0     468        0      100     33M     0  2.0M  1.3G  -750M   7.0K
16:57:21  1.0K     26     1   99   267    0    524    0  2.3G  2.0G    61M     61M       0     1.0M     268        0      100     33M     0     0  1.1G  -954M   7.0K
16:57:22  1.9K     25     1   99   484    0    974    0  2.0G  2.0G   704M    568M    136M     656K     485        0      100     33M     0     0  1.3G  -698M   7.0K
16:57:23   399     26     0  100   105    0    207    0  2.1G  2.0G    32M     23M    9.0M        0     105        0      100     33M     0     0  1.3G  -753M   7.0K

On background was ls * | xargs -P 8 -I {} bash -c "cat {} > /dev/null" where ls * has 1TB of data with blocksize 1M.

Confirmed that only after setting primarycache=all, that we have evicted entries eligible to store in L2ARC.

Whether it opted to store the block or not in such flush is another issue not discussed here, and indeed it populated to 16GB+ after typing this text

[VMatrix1900]

This is helpful!

[Rivers47]

Is there a way to change this eviction behavior?

also does this also mean

primarycache=none
secondarycache=all

would be equivalent to no cache at all?

[amotin]

Is there a way to change this eviction behavior?

No, not in the current design and for a reason. L2ARC is filled not with blocks that are evicted from ARC, but with blocks that "soon" will be evicted from ARC. It is done to not leave the system in low-memory condition if one happen during L2ARC writing. If you disable ARC, then "soon" never happens. Plus without ARC you can't know what blocks are more import to write into L2ARC.

would be equivalent to no cache at all?

Right. But I would not recommend primarycache=none. At least leave primarycache=metadata to avoid massively repeating reads.

[kftse-ust-hk]

imo it deserves a warning, as I misconfigured this again few weeks ago, this thread sent a notification to me and now I know I am misconfigured again.

It is a laptop dev machine with only 64GB memory, so one might instinctively set primarycache=metadata secondarycache=all. And, it doesn't work that way.

[kftse-ust-hk]

@Rivers47 If only the data buffer dbuf is considered cacheable in L2ARC. It is more a redesign or feature than a quick fix.

But I do see a scenario that L2ARC can be important.

Consider a ZFS system with some high-compression dataset. It is very advantageous for certain scenarios:

• Deep Learning, storing raw/plain-text readable format and get automatic compression by e.g. zstd-6 in parallel by ZFS.
  We get rid of (de)compression from the whole workflow, for example, no more decompressing .jsonl.gz that is 50GB per file.
  The L2ARC would then function as cache for the uncompressed data (require disable compressed LRC) for repeated reads
• Warm archive/highly dedupable & compressible data
  Consider those JIT languages with user-space package manager/virtual environment, e.g. javascript - node_modules, python - .venv, simple dedup, compress would make storage very compact and workflow much more efficient with L2ARC. For the fact that you might need 1 hour to download a Nvidia Toolkit from some mirror.

In fact for 2nd point I am looking to overlay / cow the cache on existing filesystem.