Replies: 1 comment 16 replies
-
|
To be clear, the N_crypt drives are on top of LUKS, or ? The CKSUM errors being reported for all of a raidz is suspicious, to me, especially since all the errors in that zed excerpt you showed are in the same location across multiple disks - I could be mistaken (I don't run raidz in my home env, so I have much less recent experience with it), but I think I expect that to happen when 1) the data in a stripe fails checksum, 2) none of the parity reconstruction results in something that matches checksum, and 3) none of the disks spat up an IO error to show who to blame. Problems with this theory include:
Could you share the hardware+software configs both before and after your migration (motherboard/CPU/HBA/HBA firmware rev + distro/distro release/kernel ver should be sufficient), the models and firmware revs of the hard drives used in each of the two pools you referenced, the output of Presuming you have a copy elsewhere (and zdb doesn't error on the checksum issue, though it'd kind of moot the feature I'm thinking of if it did), would you be willing to share the ondisk representation of one of the affected files when it's intact on the same filesystem and when it's not? (I'm specifically thinking of requesting some output from |
Beta Was this translation helpful? Give feedback.
-
|
Most spinning hard drives don't do anything we'd think of as TRIM at all, some SMR drives support it because they rather care about the difference between "don't care about this any more" and "write new data here", but they will not, so far as I know, do anything like that without you telling them. Whatever's happening seems persistent, since zdb still sees the wrong bits after scrub does. (All this is prefaced with an "AIUI" - some may be incomplete or outright wrong if I absorbed incorrect information.) It affects all disks because ZFS writes stuff across all disks (when it's large enough), the checksums are across all the data in each record (which, again, spans the disks). If you get an IO error when you're reading a record, great, you know who's wrong, and can recover accordingly. If you don't, and none of your reconstruction attempts get you the right checksum, you don't know who was wrong, so you report errors for all the disks. That doesn't mean all the disks necessarily did something wrong, just that it couldn't recover and didn't know who to blame. The checksums returning bad means the data read from some of the disks is bad now (most likely). If it's not a transient error from your RAM or CPU or hard drive electronics being dodgy or something, then ZFS doesn't have to have done any writing when it encountered the checksum error for it to stay that way. (ZFS does log when it hits checksum errors, but not over existing data - ZFS effectively never overwrites things in-place.) ZFS, during scrubs, will attempt to repair things just like it would if you encountered them in normal operation - if it ultimately Scrubs require the pool be imported read/write, and will always, as far as I know, write a couple of things to disk. The only way this could result in your data getting turned from correct to mangled on-disk, I think, is if something is flipping bits, flips bits to make something look mangled, ZFS thinks it's computed a corrected version that passes checksum, and the corrected version's bits get flipped too before it goes to disk. But at that point, I'd be expecting you to see many, many more errors, at least corrected, and not to necessarily get errors if you examine the same data twice. You seem convinced scrubs are mangling your data. Is there some reason you haven't shared for that? Because I haven't seen any reason to believe scrubbing is causing you problems, versus just finding problems that something else caused. Even more generally, I've not seen any particular indication that this is a problem in ZFS, versus something that just might have happened to your data silently without it. It certainly could be, I'm not claiming it's not, just that there's nothing that obviously points to it being one, to me, so far. It's curious that, unless I'm overlooking something, you're not having a bunch of corrected errors, just the uncorrected ones. I'll keep thinking about it. Maybe something will jump out. |
Beta Was this translation helpful? Give feedback.
-
|
Well you are absolutely right that something else could be causing these errors to occur. And believe me I understand that zfs might not be the problem here. It might be some package/program that I just don't know which writes random data to raw disks or causes data mangling. Those programs would need root access, as far as I understand, to gain access to the raw device which is in use by LUKS and ZFS. Then again why only tiny portions of disks, why does it cause the kind of damage so that with dual parity ZFS scrub cannot repair it. Why are there no other (noticeable) errors on the system other than the once in a couple of months mangling of data which gets noticed by doing a scrub but only causes unrecoverable errors and no correctable ones. Those are the questions that I am asking myself or have asked myself to figure out what is going on and I don't know how to check for such things as they are kind of low level (raw device access monitoring). A hardware problem is possible but after exchanging hardware and getting a separate machine seems just so unlikely. And again I would expect the error to occur at the same time on both machines, otherwise where would the correlation in hardware be? They run their scrub at the same time because of debians scrub scheduling. From the software side, I tried to separate these to machines that have the same problem as much as possible by installing the second one from scratch and only use it for a samba share and a couple of virtual machines which aren't stored on the ZFS array. Since the second system has basically just been on and only gotten some use in the last 3 to 4 months for backups from the other ZFS host and the software on it is so far still kind of minimal, it again throws questions as to what would cause two separate systems to have the same thing running which causes those errors. I hope I am making some kind of sense in trying to explain my reasoning as to why I suspect that maybe ZFS might be involved in some way. If you have more questions about the hardware or software that might point you or me in some kind of direction, ask away. |
Beta Was this translation helpful? Give feedback.
-
|
Sure, it certainly could be, I'm just saying I've seen nothing obviously pointing to something in ZFS (or, for that matter, anything else...), to me. Perhaps someone else will come along and go "oh I see, it's [something]". |
Beta Was this translation helpful? Give feedback.
-
|
Hello uk3gaus, with great interest I've read this whole thread about your issue here. I'm not very deep into the code of ZFS but I've been using and working with system with ZFS since 2007. Just a quick question here, as I've not seen any indication of an action like this performed yet:
The background of this question is simple: Is the error persistent on all disks and in the on-disk-representation of all of them - or is the error introduced temporarily during the scrub while the data is inflight? Another thought came to my mind with regards to what rincebrain mentioned regarding the steps during scrub (if too many or all devices return mangled data (or a checksum calculation error happens) zfs won't be able to distinguish good from bad data and will have to report all disks as having the respective error) - taking this into account if you have a random error in the checksum calculation you will see a random error in the data.
I see this topic is more than 20 days old, but I do expect it to be still open. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, regarding your first point: when the checksum errors first occured at the beginning of 2020 I did exactly that. I cleared the pool errors and reran the scrub. The same errors popped up again. My only option was to restore with additional par2 data or in later instances by restoring from backup. After repairing the file (when restoring from non ZFS based storage or with par2) I had to purge all snapshots containing the file for the scrub to come back clean. I cannot realistically do more frequent scrubs at least not at the rate you are suggesting. Those are 12 disk wide raidz2 arrays consisting of 10/12 TB disks so a scrub takes between 24 - 40 hours depending on the workload while the scrub is running. And to be honest since the likelyhood of more frequent scrubs mangling more data I really don't want to do that. I already disabled automatic scrubs since so far the problem only occured during scrubs since taking care of restoring single files is a pain and I will most likey lose snapshots again. Just to reiterate: I have the same symptom on two hardware independent systems of which one system was completely rebuilt component by component after the error first occured: new mainboard, cpu, ecc memory, hba (+added fan) and power supply. Two two disk ZFS mirror arrays running in one server also scrubbed at the same time as the big array in the same server. And while the big array had errors on all disks, the two mirrors were completely clean. |
Beta Was this translation helpful? Give feedback.
-
System information
Describe the problem you're observing
I am running a 12 disk raidz2 and I am getting checksum errors on all disks during scrubbing. This happens during my monthly scrubs but it doesn't throw an error at every scrub. Sometimes all goes well for a couple of months and then it happens again. A hardware malfunction/defect can almost be excluded since after first getting this error in early 2020 I have upgraded my systems HBA, power supply, motherboard, CPU, RAM -> switched to ECC. So with basically everything but the disks replaced it still happens. Sadly now on a second system that I was planning to use as a backup in such a case the same thing started to happen on my most recent scrub. I am also using a UPS since July 2020.
The pools on both machines were at above 90% when this happens.
I should mention that I have never had a cksum or any error during normal operation of both pools, so I can only assume this is some edge case. Has anyone else experienced something like this or has any suggestions on what to do about it?
Describe how to reproduce the problem
I have no idea how to reproduce it since it appears completely randomly but always during scrub.
Include any warning/errors/backtraces from the system logs
Output from zpool status -v
zfs list
Output from syslog since dmesg reports no errors whatsoever:
Beta Was this translation helpful? Give feedback.
All reactions