zfs-import-cache fails because zfs modules aren't loaded #9144

becky-ligon · 2019-08-08T21:55:14Z

becky-ligon
Aug 8, 2019

System information

Type	Version/Name
Distribution Name	Oracle Linux Server
Distribution Version	Release 8.0
Linux Kernel	4.18.0-80.4.2.el8_0.x86_64
Architecture	x86_64 using AMD EPYC processor
ZFS Version	0.8.1-1
SPL Version	0.8.1-1

Describe the problem you're observing

I want to say upfront that there is a similar issue but the workarounds didn't work for me. So, I'm opening this issue again.

My issue is that zfs-import-cache fails and zfs-mount subsequently fails. I believe zfs-import-cache is failing because the zfs module is not loaded. My zpools are defined using the multipath names dm-uuid-mpath- found in /dev/disk/by-id directory. Perhaps, zfs-import-cache is failing because multipathd hasn't finished initializing?

After the system comes up, zfs IS loaded. I can successfully run "systemctl restart zfs-import-cache", then "systemctl restart zfs-mount" and all is well. Why is it that the zfs module is actually loaded after the system completes booting but not by the time zfs-import-cache needs it?

Describe how to reproduce the problem

Simply boot the system

Include any warning/errors/backtraces from the system logs

Aug 08 16:42:08 burstbuff04.palmetto.clemson.edu systemd[1]: Found device /dev/disk/by-uuid/78a50021-a7e1-420a-b290-7fd5a2e21db9.
Aug 08 16:42:08 burstbuff04.palmetto.clemson.edu systemd[1]: Stopped Timer to wait for more drives before activating degraded array md125..
Aug 08 16:42:08 burstbuff04.palmetto.clemson.edu systemd[1]: Found device /dev/disk/by-uuid/2F5D-E051.
Aug 08 16:42:08 burstbuff04.palmetto.clemson.edu systemd[1]: Stopped Timer to wait for more drives before activating degraded array md124..
Aug 08 16:42:09 burstbuff04.palmetto.clemson.edu systemd-udevd[2119]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 08 16:42:16 burstbuff04.palmetto.clemson.edu systemd[1]: Started udev Coldplug all Devices.
Aug 08 16:42:16 burstbuff04.palmetto.clemson.edu systemd[1]: Starting udev Wait for Complete Device Initialization...
Aug 08 16:42:16 burstbuff04.palmetto.clemson.edu systemd[1]: Started udev Wait for Complete Device Initialization.
Aug 08 16:42:16 burstbuff04.palmetto.clemson.edu systemd[1]: Starting Device-Mapper Multipath Device Controller...
Aug 08 16:42:16 burstbuff04.palmetto.clemson.edu systemd[1]: Starting Import ZFS pools by cache file...
Aug 08 16:42:16 burstbuff04.palmetto.clemson.edu zpool[4764]: The ZFS modules are not loaded.
Aug 08 16:42:16 burstbuff04.palmetto.clemson.edu zpool[4764]: Try running '/sbin/modprobe zfs' as root to load them.
Aug 08 16:42:16 burstbuff04.palmetto.clemson.edu kernel: device-mapper: uevent: version 1.0.3
Aug 08 16:42:16 burstbuff04.palmetto.clemson.edu kernel: device-mapper: ioctl: 4.39.0-ioctl (2018-04-03) initialised: [email protected]
Aug 08 16:42:16 burstbuff04.palmetto.clemson.edu systemd[1]: zfs-import-cache.service: Main process exited, code=exited, status=1/FAILURE
Aug 08 16:42:16 burstbuff04.palmetto.clemson.edu systemd[1]: zfs-import-cache.service: Failed with result 'exit-code'.
Aug 08 16:42:16 burstbuff04.palmetto.clemson.edu systemd[1]: Failed to start Import ZFS pools by cache file.
Aug 08 16:42:16 burstbuff04.palmetto.clemson.edu systemd[1]: Reached target ZFS pool import target.
Aug 08 16:42:16 burstbuff04.palmetto.clemson.edu systemd[1]: Starting Mount ZFS filesystems...
Aug 08 16:42:16 burstbuff04.palmetto.clemson.edu zfs[4769]: The ZFS modules are not loaded.
Aug 08 16:42:16 burstbuff04.palmetto.clemson.edu zfs[4769]: Try running '/sbin/modprobe zfs' as root to load them.
Aug 08 16:42:16 burstbuff04.palmetto.clemson.edu systemd[1]: zfs-mount.service: Main process exited, code=exited, status=1/FAILURE
Aug 08 16:42:16 burstbuff04.palmetto.clemson.edu systemd[1]: zfs-mount.service: Failed with result 'exit-code'.
Aug 08 16:42:16 burstbuff04.palmetto.clemson.edu systemd[1]: Failed to start Mount ZFS filesystems.
Aug 08 16:42:16 burstbuff04.palmetto.clemson.edu multipathd[4777]: --------start up--------
Aug 08 16:42:16 burstbuff04.palmetto.clemson.edu multipathd[4777]: read /etc/multipath.conf
Aug 08 16:42:16 burstbuff04.palmetto.clemson.edu multipathd[4777]: path checkers start up
l

johnnyjacq16 · 2019-08-08T23:58:58Z

johnnyjacq16
Aug 8, 2019

To fix zfs-import-cache service not starting run the below commands:
zpool set cachefile=/etc/zfs/zpool.cache <pool>
systemctl restart zfs-import-cache.service

To ensure the zfs module is installed for the running kernel run the below command:
modinfo zfs

To ensure the zfs module is running, run the below command:
lsmod | grep zfs

0 replies

becky-ligon · 2019-08-09T14:18:44Z

becky-ligon
Aug 9, 2019
Author

Thanks, Johnny, for your message but these things don't work for me. However, I retried the steps and got the same results, zfs modules are not loaded. I believe there is a race condition where udev and multipath are still processing when zfs-import-cache fires. Is there a way to tell systemd to load the zfs modules before udev or multipath?

Becky

0 replies

MasterCATZ · 2019-12-11T13:27:18Z

MasterCATZ
Dec 11, 2019

issue for me is multipath is loading just after zfs
/etc/multipath.conf contains all the drives zfs needs

Dec 11 18:30:26 aio systemd[1]: Started Mount ZFS filesystems.
Dec 11 18:30:26 aio systemd[1]: Starting Mount ZFS filesystems...
Dec 11 18:30:26 aio systemd[1]: Reached target ZFS pool import target.
Dec 11 18:30:26 aio systemd[1]: Failed to start Import ZFS pools by cache file.
Dec 11 18:30:26 aio systemd[1]: zfs-import-cache.service: Failed with result 'exit-code'.
Dec 11 18:30:26 aio systemd[1]: zfs-import-cache.service: Main process exited, code=exited, status=1/FAILURE
Dec 11 18:30:26 aio zpool[3793]: a backup source.
Dec 11 18:30:26 aio zpool[3793]: Destroy and re-create the pool from
Dec 11 18:30:26 aio zpool[3793]: cannot import 'ZFS3WAY': no such pool or dataset
Dec 11 18:30:26 aio kernel: ZFS: Loaded module v0.8.2-1, ZFS pool version 5000, ZFS filesystem version 5
Dec 11 18:30:26 aio systemd[1]: Starting Import ZFS pools by cache file...
Dec 11 18:30:26 aio systemd[1]: Started Install ZFS kernel module.
Dec 11 18:30:25 aio multipathd[3778]: path checkers start up
Dec 11 18:30:25 aio multipathd[3778]: read /etc/multipath.conf
Dec 11 18:30:25 aio multipathd[3778]: --------start up--------

0 replies

becky-ligon · 2019-12-11T14:48:13Z

becky-ligon
Dec 11, 2019
Author

The given instructions fix the problem temporarily. Upon reboot, the same issue occurs again. Becky

…

On Thu, Aug 8, 2019 at 7:59 PM john ***@***.***> wrote: To fix zfs-import-cache service not starting run the below commands: zpool set cachefile=/etc/zfs/zpool.cache <pool> systemctl restart zfs-import-cache.service To ensure the zfs module is installed for the running kernel run the below command: modinfo zfs To ensure the zfs module is running, run the below command: lsmod | grep zfs — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/zfsonlinux/zfs/issues/9144?email_source=notifications&email_token=AM3LWOBYJXXL52H324NCMTDQDSXNZA5CNFSM4IKOZBW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD35G3SI#issuecomment-519728585>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AM3LWODL6CH2AUJXF4ONXPLQDSXNZANCNFSM4IKOZBWQ> .

-- Becky Ligon System Architect, Advanced Computing Infrastructure Clemson University Clemson, SC

0 replies

rafaelalvessa · 2020-01-02T21:44:35Z

rafaelalvessa
Jan 2, 2020

I was having a similar issue, where the ZFS modules weren't being loaded on boot. I had to edit /etc/modules (or /etc/modules-load.d/modules.conf) to include zfs:

# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
zfs

0 replies

behlendorf · 2020-01-02T21:57:06Z

behlendorf
Jan 2, 2020
Maintainer

Why is it that the zfs module is actually loaded after the system completes booting

The zfs.ko modules are automatically loaded by the udev rule 90-zfs.rules if blkid detects any block device on the system containing a zfs label. Similarly, they will be loaded on demand when running either the zfs or zpool command.

It sounds like it may be the case that the relevant zfs services are running before multipathd has made the devices available.

0 replies

becky-ligon · 2020-01-02T22:05:01Z

becky-ligon
Jan 2, 2020
Author

Brian: You are exactly right, and I haven't been able to determine how to wait until multipathd is done before ZFS kicks in. It's a bit of a race condition. Any ideas? Becky

…

On Thu, Jan 2, 2020 at 4:57 PM Brian Behlendorf ***@***.***> wrote: Why is it that the zfs module is actually loaded after the system completes booting The zfs.ko modules are automatically loaded by the udev rule 90-zfs.rules if blkid detects any block device on the system containing a zfs label. Similarly, they will be loaded on demand when running either the zfs or zpool command. It sounds like it may be the case that the relevant zfs services are running before multipathd has made the devices available. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/zfsonlinux/zfs/issues/9144?email_source=notifications&email_token=AM3LWOCHA247EXQIMLCU7ITQ3ZPLHA5CNFSM4IKOZBW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEH7R2PY#issuecomment-570367295>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM3LWOCG4ETAAFHWDLX4FTTQ3ZPLHANCNFSM4IKOZBWQ> .

-- Becky Ligon System Architect, Advanced Computing Infrastructure Clemson University Clemson, SC

0 replies

behlendorf · 2020-01-02T22:33:54Z

behlendorf
Jan 2, 2020
Maintainer

Sorry, I'm not sure of the right way to express that with systemd.

0 replies

broth-itk · 2020-03-26T07:11:48Z

broth-itk
Mar 26, 2020

I fixed the issue by modifying /lib/systemd/system/zfs-import-cache.service by adding After=multipathd.service in the [Unit] section.

When using multipath identifiers, importing the pool before multipath even starts is not a good idea.

Who is distributing the zfs-import-cache.service file?

0 replies

becky-ligon · 2020-03-26T15:50:07Z

becky-ligon
Mar 26, 2020
Author

Adding After=multipathd.service did not work for me. It seems that multipath continues to run after systemd is notified that it is done. In my case, I have 5 JBOD's on one system, so it takes multipath a bit of time to complete. Becky Ligon

…

On Thu, Mar 26, 2020 at 3:12 AM broth-itk ***@***.***> wrote: I fixed the issue by modifying /lib/systemd/system/zfs-import-cache.service by adding After=multipathd.service in the [Unit] section. When using multipath identifiers, importing the pool before multipath even starts is not a good idea. Who is distributing the zfs-import-cache.service file? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/openzfs/zfs/issues/9144#issuecomment-604267714>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM3LWOF75I3HOQEMHTOSQ5TRJL54DANCNFSM4IKOZBWQ> .

-- Becky Ligon System Architect, Advanced Computing Infrastructure Clemson University Clemson, SC

0 replies

gdevenyi · 2020-10-29T13:59:57Z

gdevenyi
Oct 29, 2020

It seems After=multipathd.service is already present in the 0.8.5 version of the systemd files. Like @becky-ligon this doesn't solve my problem, as I have a 90-disk JBOD that has not settled its multipath resolution before the systemd zfs tries to import the pool

0 replies

talbottmd · 2021-01-19T23:33:31Z

talbottmd
Jan 19, 2021

I also just ran into this exact issue while running CentOS 8 and ZoL 0.8.5 while using multiple JBODs and many disks. I wish I could find a better solution (probably exists somewhere..), but I created this one and it does make sure multipathd has "finished" before zfs attempts to import a pool.

I created this simple little script as /etc/multipath_check.sh

#!/bin/bash
PATH=/usr/bin:$PATH
for i in $(seq 1 180); do
    journalctl -x -u multipathd | grep finished >/dev/null && exit 0 || sleep 1
done
exit 1

and then add this ExecStartPre=/bin/bash /etc/multipath_check.sh to the zfs-import-cache.service file under the [Service] section. It's a short enough script that it could probably be wrapped into the single line on the service file, but, this way is easier to illustrate and test before implementing.

That'll check the multipathd journal entries every second for up to 180 seconds and once it receives the "finished" entry that shows up with journalctl -x, it'll signal the zfs import service it's ok to startup.

Hopefully this will help someone in the future or give a ZoL developer an idea to integrate some mechanism that could do this sort of thing in a future release.

I'm not sure if this is specifically a RedHat thing of if it also applies to other EL distros. To check if it'll work on your distro, see if you get the following output in the end of the journalctl entries:

journalctl -x -u multipathd
...
Jan 19 15:06:40 hostname multipathd[14036]: mpathbc: load table [0 15628053168 multipath 0 0 1 1 round-robin 0 2 1 67:240 1 71:176 1]
Jan 19 15:06:40 hostname systemd[1]: Started Device-Mapper Multipath Device Controller.
-- Subject: Unit multipathd.service has finished start-up
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
-- 
-- Unit multipathd.service has finished starting up.
-- 
-- The start-up result is done.

0 replies

becky-ligon · 2021-01-20T15:48:43Z

becky-ligon
Jan 20, 2021
Author

Thanks for this tip! I have the same issue on CentOS 8. Becky

…

On Tue, Jan 19, 2021 at 6:34 PM talbottmd ***@***.***> wrote: I also just ran into this exact issue while running CentOS 8 and ZoL 0.8.5 while using multiple JBODs and many disks. I wish I could find a better solution (probably exists somewhere..), but I created this one and it does make sure multipathd has "finished" before zfs attempts to import a pool. I created this simple little script as /etc/multipath_check.sh #!/bin/bash PATH=/usr/bin:$PATHfor i in $(seq 1 180); do journalctl -x -u multipathd | grep finished >/dev/null && exit 0 || sleep 1doneexit 1 and then add this ExecStartPre=/bin/bash /etc/multipath_check.sh to the zfs-import-cache.service file under the [Service] section. It's a short enough script that it could probably be wrapped into the single line on the service file, but, this way is easier to illustrate and test before implementing. That'll check the multipathd journal entries every second for up to 180 seconds and once it receives the "finished" entry that shows up with journalctl -x, it'll signal the zfs import service it's ok to startup. Hopefully this will help someone in the future or give a ZoL developer an idea to integrate some mechanism that could do this sort of thing in a future release. I'm not sure if this is specifically a RedHat thing of if it also applies to other EL distros. To check if it'll work on your distro, see if you get the following output in the end of the journalctl entries: journalctl -x -u multipathd ... Jan 19 15:06:40 hostname multipathd[14036]: mpathbc: load table [0 15628053168 multipath 0 0 1 1 round-robin 0 2 1 67:240 1 71:176 1] Jan 19 15:06:40 hostname systemd[1]: Started Device-Mapper Multipath Device Controller. -- Subject: Unit multipathd.service has finished start-up -- Defined-By: systemd -- Support: https://access.redhat.com/support -- -- Unit multipathd.service has finished starting up. -- -- The start-up result is done. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#9144 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM3LWOGBZSK2Z7OSU6UO4KDS2YJHFANCNFSM4WJXBM7A> .

-- Becky Ligon System Architect, Advanced Computing Infrastructure Clemson University Clemson, SC

0 replies

braddeicide · 2021-02-19T07:48:44Z

braddeicide
Feb 19, 2021

I'm getting this also on CentOS 8.2, zfs-0.8.5.

I added -x to the bash script and can see it takes 4 seconds before multipathd is ready.

I also had to remove ConditionPathIsDirectory=/sys/modules/zfs reverting f1de160, the grep works.
I can't tell when /sys/modules/zfs is created but the first file is created 4 seconds after zfs-import-cache.service starts (yes same time multipathd reports ready). Unless the conditional is removed the service is skipped due to conditional failed.

So by default it conditional fails, remove the condition and it fails due to zfs module not being loaded, but with the sleep waiting for multipathd the kernel module also loads. This doesn't feel solid, feels like I'm winning a race condition.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

zfs-import-cache fails because zfs modules aren't loaded #9144

{{title}}

Replies: 14 comments

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

zfs-import-cache fails because zfs modules aren't loaded #9144

System information

Describe the problem you're observing

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

Replies: 14 comments

becky-ligon Aug 9, 2019 Author

becky-ligon Dec 11, 2019 Author

behlendorf Jan 2, 2020 Maintainer

becky-ligon Jan 2, 2020 Author

behlendorf Jan 2, 2020 Maintainer

becky-ligon Mar 26, 2020 Author

becky-ligon Jan 20, 2021 Author

becky-ligon
Aug 9, 2019
Author

becky-ligon
Dec 11, 2019
Author

behlendorf
Jan 2, 2020
Maintainer

becky-ligon
Jan 2, 2020
Author

behlendorf
Jan 2, 2020
Maintainer

becky-ligon
Mar 26, 2020
Author

becky-ligon
Jan 20, 2021
Author