Cannot remove log device from encrypted pool #8748
-
System information
Describe the problem you're observingI initially created my pool on macOS using O3X 1.8.2. The system was an older Mac mini with a 500 GB SSD, on which I reserved two partitions for a log and a cache device. The mini started to have issues running the pool, possibly due to being rather underpowered (4G mem, 1.4GHz i5), so I decided to move it to a more beefy MacBook Pro. I exported on the mini, and imported it on the Pro, where of course the log and cache partitions were not available. I've now moved the pool to Ubuntu Server as macOS just has so many issues running it stably and performant (which may not be due to ZFS itself, but possibly the USB subsystem, as this is a USB-connected pool. When I tried to remove the orphaned log device, I got: Thing is, the encrypted datasets were mounted and happy (actually just the one, "titanic" itself (I haven't yet created any other filesystems)). Later, I was at the point where I was ready to add back a new log and cache device on the SSD on the Ubuntu Server rig. So I thought, given that I couldn't remove the missing log device, I'd try replacing it first. So, I created a partition on my Air's SSD, and did: This was a bit weird I thought (adding a log device to an ashift=12 pool seriously adds it as ashift=9 by default? Isn't that configuration pool-specific? I wasn't even aware that you could mix-and-match?), but adding The next thing happening was the pool now going into a resilver (why resilver to replace a log device?) at crazy speeds (looks related to this bug): EDIT NOTE: There's a couple of lines missing here at the end. I copied this from an O3X forum post I made before this one, and I have apparently made a bad copy. The actual output did include the original log device's GUID, which was This was yesterday, and it never really finished or actually resilvered anything. Today, I wanted to replace the device that is listing 17.1K checksum errors above, but after doing About the actual issue! :D I had to export the pool, remove the partition from the SSD for the new log device that was being replaced in, and re-import it to get it to not continue trying to "resilver it into the pool." Again, it didn't seem to actually be doing anything, but it was stuck in the resilvering loop. The current status now is this: This resilvering is for the device I'm actually currently replacing, so just disregard that. It's running along nicely, and it's not the fact that the pool is resilvering right now that is causing me to not be able to remove the log device, because I've tried this for a long time now, with the pool being in a normal state. My attempts at removing the log device now: Again, the pool is mounted, and writeable. I read a similar post online where the issue had to do with the log device being a mirror. This is not my case. The log device, when it was there, was just Describe how to reproduce the problemNot to sure. I don't have any more drives available to do something like this. But I hope the above description helps. I initially created the pool (on macOS with O3X 1.8.2) with Include any warning/errors/backtraces from the system logsI'm not sure exactly what to provide. Please let me know if I can provide anything. Thank you! :D |
Beta Was this translation helpful? Give feedback.
Replies: 20 comments
-
|
@DanielSmedegaardBuus thanks for the clear description of the issue Regarding |
Beta Was this translation helpful? Give feedback.
-
|
Thank you, @behlendorf, for picking up my report :) Good to know about the ashift caveat! I'll make sure to specify at least ashift=12 when adding new log and cache devices. About volumes, then no. These are my current resources: Could the snapshots perhaps play in here? I'll destroy them and try again. (BTW: /scratch is a single-disk zpool for temp stuffs) |
Beta Was this translation helpful? Give feedback.
-
|
Okay, so I destroyed all snapshots and tried again, but still no donut: Here's a zdb output in case it reveals anything: BTW: Those dashes in the device names are from me redacting serial numbers. Not sure if that's just paranoid, it just seems like most people do that, so I'm following suit :) |
Beta Was this translation helpful? Give feedback.
-
|
I have hit this also. Tried to remove a SLOG from a pool, and was told to supply the key for every encrypted dataset in order to replay the log. If this happens often, it basically breaks the promise of sending encrypted datasets to untrusted machines, and not having to supply the key to do basic pool maintenance. Hopefully it can only happen if the dataset was actually mounted prior to a shutdown? It's quite possible there was a slightly unclean shutdown in my case, as due to another outstanding issue, all pools need to be manually imported on startup with the latest stable Debian and 0.8.0 compiled from source. I'm sorry I don't remember the issue number, but it was discussed at length, there was a workaround involving creation of a symlink, but I seem to have lost it recently in going to 0.8.0. |
Beta Was this translation helpful? Give feedback.
-
|
@DanielSmedegaardBuus Hmm. This is interesting. Can you post a couple bits of debugging output? In particular, I would be interested in the output of: and Hopefully that should give us enough info to figure out what's going on here. |
Beta Was this translation helpful? Give feedback.
-
|
As i stated in #9576 i have the same problem on unencrypted pool:
|
Beta Was this translation helpful? Give feedback.
-
|
So it looks like the code that's causingg your issue is actually code thats been around for a long time. The commandline utility, however, is misinterpreting that EBUSY error code and turning it into a message about encryption. We should fix the error message to be more clear, but the actual ffending line of code is: @behlendorf Who would know about this bit of code? Does he just need to do a |
Beta Was this translation helpful? Give feedback.
-
|
@tcaputi So the line says that ZIL_REPLAY_NEEDED... What happens if i don't replay the log? Eg. i've lost the harddrive with the log device. All datasets are mounted and under use. Does that mean they are ok? At least i an way that data errors will not cause further data corruption? Like there might have been file corrupted or missing, but i will not lose data anymore when using this filesystem, storing and modifying data on it. Or do i? How can i remove the device that i cannot replay because i've lost it? |
Beta Was this translation helpful? Give feedback.
-
|
There is a module option which allows us to discard the zfs intent log instead of replaying it. You could try setting That said, I completely agree that really there should be a |
Beta Was this translation helpful? Give feedback.
-
So can i be sure that discarding ZIL will not cause any further data corruption? What about replaying the ZIL on pool/datasets that have been mounted and used since the pool was degraded. Is that safe? |
Beta Was this translation helpful? Give feedback.
-
|
The ZIL for each filesystem only contains data that was written in the last few seconds before the system crashed or the filesystem was unmounted. You should not have any corruption from removing a ZIL, although you may lose that last little bit of data. Replaying the ZIL should not cause any further corruption either, although to be honest I'm not 100% sure what will happen in that case, but I'm about 99% sure the ZIL will just be discarded. |
Beta Was this translation helpful? Give feedback.
-
I think it's important to define "any corruption" when used in a sentence like this (since naive users may read this, and get the wrong idea). For example, the application that wrote the data that's contained in the ZIL, may detect "corruption"; but ZFS itself will not detect any "corruption". |
Beta Was this translation helpful? Give feedback.
-
|
I've tried doing |
Beta Was this translation helpful? Give feedback.
-
|
So i rebooted with there's still obviously some problem. |
Beta Was this translation helpful? Give feedback.
-
|
With OpenZFS 2.0 we've fixed a few issues in the area and updated the documentation to mention that the encryption keys need to loaded in order to remove a log device from the pool. |
Beta Was this translation helpful? Give feedback.
-
|
I faced this problem as well, but I hasn't even encrypted my drive at all, so I'm trying to look for similar answers here. In case anyone wondered here is my zpool status and remove command results: |
Beta Was this translation helpful? Give feedback.
-
|
oh and i see that when i started doing write operation to rpool, the entire rpool frozen and made no progress. it was later I spent an entire evening transferring my ZFS root from rpool to rpool2 (remember reading is fine but writing is not) and start it there just right now I see that zed is stucking in a "zombie" like situation. zed occupied an entire core timeslice (as zed should indeed be single-threaded) and I tried to kill it via any means. Not even SIGKILL did anything. All the ZFS related operations stuck (including zpool, zfs) and the kernel also prints I/O hung message in dmesg: While zed is technically a mutant "zombie", can it really be considered like that? Since it is a core part of ZFS to be the event processing daemon, so killing it might mean certain death and thus there must be some kind of kernel level magic to prevent this process to be killed in the first place...thus: As you see it timed out in killing it, though I can understand why. Now back to the discussion. I suspect the reason my RAIDZ1 pool isn't progressing is due the faulty SLOG device, since when I tried to remove it, it tells me something I unexpected, that is, it claims my pool had encrypted dataset it in, while in reality it doesn't. So, this SLOG may had some degree of corruption to the point that it is considered a valid SLOG device but the content which the SLOG should replay itself after a catastrophic event such as a forced, unclean reset, isn't. It tried to replay the "encrypted section" of the data but well, it refuses, and thus the dataset is running in circle trying to replay it since it is important for SLOG to replay sequentially for consistency. I'll have to confess that I did a series of reboot before because I was trying to debug a weird issue where my SCSI drives under SAS controller, keep failing and I had lost one member of the pool not long ago, so there are 3 remaining and it looked quite dangerous to me so I saw this as a sign of something sinister going to happen. I quickly reused my another RAID1/mirror pool (rpool2) and created a big enough ZVOL inside rpool2 to replace it as a member for rpool. These two drives were originally my 12TB NAS drive that also happened to run ZFS so its a smooth criminal to get into the game. As a summary, what I said is I re-imported an existing ZFS pool and made a ZVOL inside the imported pool as the replacing RAID member for another dying pool. But the problem is I know ZFS nesting is a bad thing, as I remember someone from ServeTheHome said this would cause ZFS (txg) deadlock and I highly suspect I'm witnessing one such case. Terribly fascinating at the cost my data at neck point. I would never do this again but just hope that in a short-term after I "salvage" all my data (well at this point 3 member pool out of a RAIDZ1 is technically functional thanks to XOR/ECC magic but this SHTF anyway) it will be fine. It was never fine at any point. |
Beta Was this translation helpful? Give feedback.
-
|
I am seeing the same problem. Ubuntu 20.04 I've never had an encrypted dataset on any pool. This pool did hold some device volumes, but I destroyed them all. pool: p5 root@shed9:/services# zpool iostat -v p5 p5 6.51T 7.99T 94 556 732K 11.5M root@shed9:/services# zpool remove p5 nvme0n1p2 Although, according to the title of this thread, it is about encryption, most of this thread is about an error with a misleading message. With due respect to behlendorf, I don't believe it has been answered. I need to remove the device holding the log, but am prevented from doing so. |
Beta Was this translation helpful? Give feedback.
-
|
same for me, realised just today. pool never had encryption, ashift=12, for all devices. zfs is version (debian / buster-backports) the dbgmsg with full flags and after remove op is full of: all with the same timestamp from begin to end. pool otherwise claims to be HEALTHY, log device itself too. |
Beta Was this translation helpful? Give feedback.
-
|
I have the same issue. Tried the zil_replay_disable trick, also upgraded from 0.8.3 (the stock one on Ubuntu 20.04) to 2.1.0 (from Jonathon F), but always got the same message "Mount encrypted datasets to replay logs." The problem doesn't seem related to the "UNAVAIL" status of the log device. |
Beta Was this translation helpful? Give feedback.
With OpenZFS 2.0 we've fixed a few issues in the area and updated the documentation to mention that the encryption keys need to loaded in order to remove a log device from the pool.