L2ARC only using ~200GB of a 1TB device

[FallingSnow]

I've noticed a strange relationship between the size of ARC and L2ARC. I had a 4GB ARC cache and L2ARC would only fill to 100GB over about a day of usage. It did this fairly quickly and just remained there, not increasing. So I increased my ARC to 8GB and instantly saw L2ARC size climbing, but it seems to have leveled off to ~200GB, twice what it was with a 4GB ARC. My ARC cache hit rate is between 50-90% depending on what I'm doing at the moment.

Now my understanding is that with my record size (128K) the L2ARC indexes (which take up space in ARC) should only be about 550MB of ARC using the formula 1TB / 128kB × 70 byte = 550MB. So there should be plenty of space for L2ARC to continue to grow.

Is there some kind of formula that limits the size of the L2ARC based on the size of ARC? Why isn't my L2ARC growing?

[IvanVolosyuk]

Actually records stored in L2ARC can be smaller than 128kB, thus ratio can be smaller. You can inspect ram overhead looking at arcstats. I think this should give you good estimate of ram usage for l2arc headers:

cat /proc/spl/kstat/zfs/arcstats | grep l2_hdr_size

You can adjust the ARC fraction used for L2 headers if you know what you are doing, see zfs_arc_meta_limit_percent

echo $percent >> /sys/module/zfs/parameters/zfs_arc_meta_limit_percent

[FallingSnow]

I found the command arc_summary which returns the information about the L2ARC. It does seem the l2arc headers are using much more than I originally thought. However it still doesn't hit 75%, currently it's only hitting about 30%.

$ cat /sys/module/zfs/parameters/zfs_arc_meta_limit_percent
75

$ cat /proc/spl/kstat/zfs/arcstats | grep l2_hdr_size

l2_hdr_size                     4    2487744480

$ arc_summary
ARC size (current):                                    99.7 %    8.0 GiB
        Target size (adaptive):                       100.0 %    8.0 GiB
        Min size (hard limit):                         12.1 %  990.7 MiB
        Max size (high water):                            8:1    8.0 GiB
        Most Frequently Used (MFU) cache size:        < 0.1 %  272.0 KiB
        Most Recently Used (MRU) cache size:          100.0 %    5.1 GiB
        Metadata cache size (hard limit):              75.0 %    6.0 GiB
        Metadata cache size (current):                 45.4 %    2.7 GiB
        Dnode cache size (hard limit):                 10.0 %  614.4 MiB
        Dnode cache size (current):                     1.2 %    7.2 MiB
...
L2ARC size (adaptive):                                         205.9 GiB
        Compressed:                                    97.4 %  200.5 GiB
        Header size:                                    1.1 %    2.3 GiB
        MFU allocated size:                            18.8 %   37.6 GiB
        MRU allocated size:                            81.0 %  162.3 GiB
        Prefetch allocated size:                        0.3 %  579.8 MiB
        Data (buffer content) allocated size:          98.8 %  198.0 GiB
        Metadata (buffer content) allocated size:       1.2 %    2.5 GiB