zpool-trim(8): cannot trim '⋯': device is in use as a cache #14231

grahamperrin · 2022-11-27T11:07:54Z

grahamperrin
Nov 27, 2022

From the autotrim section of zpoolprops.7 — OpenZFS documentation:

… TRIM on L2ARC devices is enabled by setting l2arc_trim_ahead > 0. …

root@mowa219-gjp4-8570p-freebsd:~ # zpool offline Transcend gpt/cache-transcend
root@mowa219-gjp4-8570p-freebsd:~ # zpool trim Transcend gpt/cache-transcend
cannot trim 'gpt/cache-transcend': device is in use as a cache
root@mowa219-gjp4-8570p-freebsd:~ # zpool online Transcend gpt/cache-transcend
root@mowa219-gjp4-8570p-freebsd:~ # sysctl vfs.zfs.l2arc.trim_ahead
vfs.zfs.l2arc.trim_ahead: 0
root@mowa219-gjp4-8570p-freebsd:~ #

Is the cannot trim response (to the zpool-trim(8) command) due to autotrim off for the pool?

The response surprised me, only because I expected a different message, to indicate that the device does not support TRIM; it's a very simple USB flash drive (Kingston DataTraveler 3.0).

Answered by IvanVolosyuk

Nov 27, 2022

You can try: echo 1 >/sys/module/zfs/parameters/l2arc_trim_ahead 'zpool trim' purpose is to trim empty space in the pool. It doesn't look like it was designed to work on L2ARC as it is just a ring buffer with cache. Trim ahead module parameter for L2ARC is different. AFAIK it just instructs ZFS to keep a bit of trimmed space in front of new L2ARC allocations.

View full answer

IvanVolosyuk · 2022-11-27T11:47:10Z

IvanVolosyuk
Nov 27, 2022

You can try: echo 1 >/sys/module/zfs/parameters/l2arc_trim_ahead 'zpool trim' purpose is to trim empty space in the pool. It doesn't look like it was designed to work on L2ARC as it is just a ring buffer with cache. Trim ahead module parameter for L2ARC is different. AFAIK it just instructs ZFS to keep a bit of trimmed space in front of new L2ARC allocations.

…

On Sun, Nov 27, 2022 at 10:08 PM Graham Perrin ***@***.***> wrote: From the *autotrim* section of zpoolprops.7 — OpenZFS documentation <https://openzfs.github.io/openzfs-docs/man/7/zpoolprops.7.html>: … TRIM on L2ARC devices is enabled by setting *l2arc_trim_ahead > 0*. … From helloSystem/ISO#412 (comment) <helloSystem/ISO#412 (comment)>: ***@***.***:~ # zpool offline Transcend gpt/cache-transcend ***@***.***:~ # zpool trim Transcend gpt/cache-transcend cannot trim 'gpt/cache-transcend': device is in use as a cache ***@***.***:~ # zpool online Transcend gpt/cache-transcend ***@***.***:~ # sysctl vfs.zfs.l2arc.trim_ahead vfs.zfs.l2arc.trim_ahead: 0 ***@***.***:~ # Is the *cannot trim* response (to the zpool-trim(8) command) due to autotrim off for the pool? The response surprised me, only because I expected a different message, to indicate that the *device* does not support TRIM; it's a very simple USB flash drive (Kingston DataTraveler 3.0). — Reply to this email directly, view it on GitHub <#14231>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABXQ6HKVWFK4FX5PISD5EATWKM6JLANCNFSM6AAAAAASMO2FCQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

5 replies

grahamperrin Nov 27, 2022
Author

echo 1 >/sys/module/zfs/parameters/l2arc_trim_ahead

Not applicable on FreeBSD, but thanks.

… doesn't look like it was designed to work on L2ARC …

If so, then I think that the related sentence (highlighted below) should become a separate paragraph:

𠄶– with some explanation, i.e. words to this effect:

AFAIK it just instructs ZFS to keep a bit of trimmed space in front of new L2ARC allocations.

rincebrain Nov 27, 2022
Collaborator

zpool trim on an offline device I don't expect to work in general, I think, since it just trims the unallocated space, not blanks the device like blkdiscard might.

What happens if you try it with the device online and that tunable set?

grahamperrin Nov 28, 2022
Author

… What happens if you try it with the device online and that tunable set?

Short story: no different after changing vfs.zfs.l2arc.trim_ahead from its default 0, to 1:

cannot trim 'gpt/cache-transcend': device is in use as a cache

Short story question: is the manual page true? 0 (zero) to enable?

Long story: it's remarkable that with both imports below, five seconds after the import, allocation is zero.

I'm not accustomed to seeing 0 in this context, for now I'm inclined to treat the peculiarity as orthogonal to the original question. (Generally: I'd like to test with a different pool, one that's disposable, with different physical media.)

root@mowa219-gjp4-8570p-freebsd:~ # zpool status -x
all pools are healthy
root@mowa219-gjp4-8570p-freebsd:~ # zpool trim Transcend
cannot trim: no devices in pool support trim operations
root@mowa219-gjp4-8570p-freebsd:~ # zpool trim Transcend gpt/Transcend
cannot trim 'gpt/Transcend': trim operations are not supported by this device
root@mowa219-gjp4-8570p-freebsd:~ # zpool trim Transcend gpt/cache-transcend
cannot trim 'gpt/cache-transcend': device is in use as a cache
root@mowa219-gjp4-8570p-freebsd:~ # sysctl vfs.zfs.l2arc.trim_ahead=1
vfs.zfs.l2arc.trim_ahead: 0 -> 1
root@mowa219-gjp4-8570p-freebsd:~ # zpool trim Transcend gpt/cache-transcend
cannot trim 'gpt/cache-transcend': device is in use as a cache
root@mowa219-gjp4-8570p-freebsd:~ # zpool offline Transcend gpt/cache-transcend
root@mowa219-gjp4-8570p-freebsd:~ # zpool trim Transcend gpt/cache-transcend
cannot trim 'gpt/cache-transcend': device is in use as a cache
root@mowa219-gjp4-8570p-freebsd:~ # zpool online Transcend gpt/cache-transcend
root@mowa219-gjp4-8570p-freebsd:~ # sysctl vfs.zfs.l2arc.trim_ahead=0
vfs.zfs.l2arc.trim_ahead: 1 -> 0
root@mowa219-gjp4-8570p-freebsd:~ # zpool status -x
all pools are healthy
root@mowa219-gjp4-8570p-freebsd:~ # zpool export Transcend
root@mowa219-gjp4-8570p-freebsd:~ # zpool import Transcend ; zpool status Transcend && zfs load-key Transcend/VirtualBox && zfs mount Transcend/VirtualBox ; mount | grep Transcend ; sleep 5 ; zpool iostat -v
  pool: Transcend
 state: ONLINE
  scan: scrub repaired 0B in 03:28:14 with 0 errors on Sat Nov 26 11:18:41 2022
remove: Removal of vdev 1 copied 704K in 0h0m, completed on Wed May  4 02:25:25 2022
        312 memory used for removed device mappings
config:

        NAME                   STATE     READ WRITE CKSUM
        Transcend              ONLINE       0     0     0
          gpt/Transcend        ONLINE       0     0     0
        cache
          gpt/cache-transcend  ONLINE       0     0     0

errors: No known data errors
Transcend on /media/t500 (zfs, local, nfsv4acls)
Transcend/VirtualBox on /media/t500/VirtualBox (zfs, local, nfsv4acls)
                         capacity     operations     bandwidth 
pool                   alloc   free   read  write   read  write
---------------------  -----  -----  -----  -----  -----  -----
Transcend               334G   130G     20     23   705K   346K
  gpt/Transcend         334G   130G     20     23   704K   346K
  indirect-1               -      -      0      0      0      0
cache                      -      -      -      -      -      -
  gpt/cache-transcend      0  14.4G      1      0  64.6K  3.80K
---------------------  -----  -----  -----  -----  -----  -----
august                  705G   207G     32     28   614K   566K
  ada0p3.eli            705G   207G     32     28   614K   566K
cache                      -      -      -      -      -      -
  gpt/cache-august     10.6G  18.2G     32      0   720K  38.1K
  gpt/cache2-august    13.8G   637M     33      0   776K  37.7K
---------------------  -----  -----  -----  -----  -----  -----
root@mowa219-gjp4-8570p-freebsd:~ # date 
Mon Nov 28 07:22:31 GMT 2022
root@mowa219-gjp4-8570p-freebsd:~ # zpool iostat -v Transcend
                         capacity     operations     bandwidth 
pool                   alloc   free   read  write   read  write
---------------------  -----  -----  -----  -----  -----  -----
Transcend               334G   130G      3      3   117K  57.4K
  gpt/Transcend         334G   130G      3      3   117K  57.4K
  indirect-1               -      -      0      0      0      0
cache                      -      -      -      -      -      -
  gpt/cache-transcend  1.01M  14.4G      0      0  8.64K  19.5K
---------------------  -----  -----  -----  -----  -----  -----
root@mowa219-gjp4-8570p-freebsd:~ # sysctl vfs.zfs.l2arc.trim_ahead=1
vfs.zfs.l2arc.trim_ahead: 0 -> 1
root@mowa219-gjp4-8570p-freebsd:~ # zpool trim Transcend gpt/cache-transcend
cannot trim 'gpt/cache-transcend': device is in use as a cache
root@mowa219-gjp4-8570p-freebsd:~ # sysctl vfs.zfs.l2arc.trim_ahead=0
vfs.zfs.l2arc.trim_ahead: 1 -> 0
root@mowa219-gjp4-8570p-freebsd:~ # date ; zpool status -x
Mon Nov 28 07:31:58 GMT 2022
all pools are healthy
root@mowa219-gjp4-8570p-freebsd:~ # zpool export Transcend
root@mowa219-gjp4-8570p-freebsd:~ # zpool import Transcend ; zpool status Transcend && zfs load-key Transcend/VirtualBox && zfs mount Transcend/VirtualBox ; mount | grep Transcend ; sleep 5 ; zpool iostat -v
  pool: Transcend
 state: ONLINE
  scan: scrub repaired 0B in 03:28:14 with 0 errors on Sat Nov 26 11:18:41 2022
remove: Removal of vdev 1 copied 704K in 0h0m, completed on Wed May  4 02:25:25 2022
        312 memory used for removed device mappings
config:

        NAME                   STATE     READ WRITE CKSUM
        Transcend              ONLINE       0     0     0
          gpt/Transcend        ONLINE       0     0     0
        cache
          gpt/cache-transcend  ONLINE       0     0     0

errors: No known data errors
Transcend on /media/t500 (zfs, local, nfsv4acls)
Transcend/VirtualBox on /media/t500/VirtualBox (zfs, local, nfsv4acls)
                         capacity     operations     bandwidth 
pool                   alloc   free   read  write   read  write
---------------------  -----  -----  -----  -----  -----  -----
Transcend               334G   130G     21     25   743K   340K
  gpt/Transcend         334G   130G     21     25   743K   340K
  indirect-1               -      -      0      0      0      0
cache                      -      -      -      -      -      -
  gpt/cache-transcend      0  14.4G      1      0  63.8K  3.75K
---------------------  -----  -----  -----  -----  -----  -----
august                  705G   207G     31     28   602K   564K
  ada0p3.eli            705G   207G     31     28   602K   564K
cache                      -      -      -      -      -      -
  gpt/cache-august     10.6G  18.2G     31      0   707K  37.5K
  gpt/cache2-august    13.8G   642M     32      0   761K  37.2K
---------------------  -----  -----  -----  -----  -----  -----
root@mowa219-gjp4-8570p-freebsd:~ # date ; zpool iostat -v Transcend
Mon Nov 28 07:32:47 GMT 2022
                         capacity     operations     bandwidth 
pool                   alloc   free   read  write   read  write
---------------------  -----  -----  -----  -----  -----  -----
Transcend               334G   130G      6      7   212K  97.0K
  gpt/Transcend         334G   130G      6      7   212K  97.0K
  indirect-1               -      -      0      0      0      0
cache                      -      -      -      -      -      -
  gpt/cache-transcend   164K  14.4G      0      0  15.1K  6.23K
---------------------  -----  -----  -----  -----  -----  -----
root@mowa219-gjp4-8570p-freebsd:~ # date ; zpool iostat -v Transcend
Mon Nov 28 07:39:30 GMT 2022
                         capacity     operations     bandwidth 
pool                   alloc   free   read  write   read  write
---------------------  -----  -----  -----  -----  -----  -----
Transcend               334G   130G      0      0  16.3K  7.46K
  gpt/Transcend         334G   130G      0      0  16.3K  7.46K
  indirect-1               -      -      0      0      0      0
cache                      -      -      -      -      -      -
  gpt/cache-transcend   684K  14.4G      0      0  1.09K  1.68K
---------------------  -----  -----  -----  -----  -----  -----
root@mowa219-gjp4-8570p-freebsd:~ # zpool export Transcend
root@mowa219-gjp4-8570p-freebsd:~ #

rincebrain Nov 28, 2022
Collaborator

zfs/module/zfs/arc.c

Lines 897 to 912 in b0657a5

    
           /* 
        
            * L2ARC TRIM 
        
            * l2arc_trim_ahead : A ZFS module parameter that controls how much ahead of 
        
            * 		the current write size (l2arc_write_max) we should TRIM if we 
        
            * 		have filled the device. It is defined as a percentage of the 
        
            * 		write size. If set to 100 we trim twice the space required to 
        
            * 		accommodate upcoming writes. A minimum of 64MB will be trimmed. 
        
            * 		It also enables TRIM of the whole L2ARC device upon creation or 
        
            * 		addition to an existing pool or if the header of the device is 
        
            * 		invalid upon importing a pool or onlining a cache device. The 
        
            * 		default is 0, which disables TRIM on L2ARC altogether as it can 
        
            * 		put significant stress on the underlying storage devices. This 
        
            * 		will vary depending of how well the specific device handles 
        
            * 		these commands. 
        
            */ 
        
           static uint64_t l2arc_trim_ahead = 0;

Also, the man page is telling you to "set l2arc_trim_ahead to a value greater than zero", not "set it to 0".

grahamperrin Nov 29, 2022
Author

greater than

D'oh! Thanks.

grahamperrin · 2022-11-29T22:01:52Z

grahamperrin
Nov 29, 2022
Author

#14231 (reply in thread)

… (Generally: I'd like to test with a different pool, one that's disposable, with different physical media.) …

A Verbatim device that certainly does not support trim operations:

root@mowa219-gjp4-8570p-freebsd:~ # zpool trim chunky da4p1
cannot trim 'da4p1': trim operations are not supported by this device
root@mowa219-gjp4-8570p-freebsd:~ # umount /chunky
root@mowa219-gjp4-8570p-freebsd:~ # zpool destroy chunky
root@mowa219-gjp4-8570p-freebsd:~ # zpool labelclear /dev/da4p1
root@mowa219-gjp4-8570p-freebsd:~ # geom disk list da4
Geom name: da4
Providers:
1. Name: da4
   Mediasize: 8011120640 (7.5G)
   Sectorsize: 512
   Mode: r0w0e0
   descr: Verbatim STORE N GO
   ident: 07072C6CB6D7A632
   rotationrate: unknown
   fwsectors: 63
   fwheads: 255

root@mowa219-gjp4-8570p-freebsd:~ #

I removed the Kingston cache device from the pool named Transcend, then reused the Verbatim device as the cache.

Result:

root@mowa219-gjp4-8570p-freebsd:~ # zpool trim Transcend gpt/cache-transcend
cannot trim 'gpt/cache-transcend': device is in use as a cache
root@mowa219-gjp4-8570p-freebsd:~ #

#14231 (comment)

… I expected a different message, to indicate that the device does not support TRIM; …

So, my expectation was wrong. The lack of support at device level is irrelevant.

I'll mark #14231 (comment) as the answer (thanks, @IvanVolosyuk); it's only slightly wrong in that part of what was suggested is for FreeBSD. This, essentially, seems to be the answer:

… It doesn't look like it was designed to work on L2ARC …

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

zpool-trim(8): cannot trim '⋯': device is in use as a cache #14231

{{title}}

Replies: 2 comments 5 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

zpool-trim(8): cannot trim '⋯': device is in use as a cache #14231

grahamperrin Nov 27, 2022

Replies: 2 comments · 5 replies

IvanVolosyuk Nov 27, 2022

grahamperrin Nov 27, 2022 Author

rincebrain Nov 27, 2022 Collaborator

grahamperrin Nov 28, 2022 Author

rincebrain Nov 28, 2022 Collaborator

grahamperrin Nov 29, 2022 Author

grahamperrin Nov 29, 2022 Author

grahamperrin
Nov 27, 2022

Replies: 2 comments 5 replies

IvanVolosyuk
Nov 27, 2022

grahamperrin Nov 27, 2022
Author

rincebrain Nov 27, 2022
Collaborator

grahamperrin Nov 28, 2022
Author

rincebrain Nov 28, 2022
Collaborator

grahamperrin Nov 29, 2022
Author

grahamperrin
Nov 29, 2022
Author