Getting error ZFS-8000-8A with zpool status, tried scrubbing which results in a crash everytime, what do I do?

[aafrasht]

System information

Distribution Name | Ubuntu
Distribution Version | 22.04.3 LTS
Kernel Version | Linux 5.15.0-78-generic x86_64
Architecture | AMD64
OpenZFS Version | zfs-2.1.5-1ubuntu622.04.1
zfs-kmod-2.1.5-1ubuntu622.04.1
Hardware name |
Corsair Vengeance LPX 16GB (2x8GB) DDR4 DRAM 3200MHz C16
Gigabyte B450 I AORUS PRO WIFI
AMD Ryzen 3 3200G 4-Core
3x --> WD120EMAZ-11BLFA0 (12 TB Western Digital drives)

Describe the problem you're observing

Recently I ran into problems with my pool and I have no idea how to fix it.
I don't care per say about the data I may have lost, I'm just trying to bypass (or fix?) the sections
that are in error to get my pool working again. I tried scrubbing it but resulted in a very strange error
that required me to reboot the machine. See below for long error. See below for other relevant technical info.

I have a 250 gig backup file on my pool that when I try to copy to my main computer (windows 10) I get a file I/O error. That being said, I can play other movies off my drive so that works....again, I think scrubbing will fix it but a very strange error below pops up around the 12% mark during the scrub.

Note that this happened shortly after I upgraded my M/B bios to the latest version and Ubuntu from 20.04 to 22.04. While this happened I was changing around a lot of files and compressing some stuff and doing general file clean up (read, deleting old worthless crap).

If anyone needs other logs, debugging stuff or general info, just ask and I'll try to provide it. Please let me know how to get the logs (commands or files) as I'm not a Ubuntu expert. No GUI either, I just putty into my machine.

Include any warning/errors/backtraces from the system logs

zpool Info:
pool: datapool
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
scan: scrub canceled on Wed Aug 9 00:23:42 2023
config:

        NAME        STATE     READ WRITE CKSUM
        datapool    ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            sdd     ONLINE       0     0   140
            sda     ONLINE       0     0   140
            sdb     ONLINE       0     0   140

 errors: Permanent errors have been detected in the following files:

        datapool:<0xb37>
        datapool:<0x8080>

Zpool Config:

NAME      PROPERTY                       VALUE                          SOURCE
datapool  size                           32.5T                          -
datapool  capacity                       58%                            -
datapool  altroot                        -                              default
datapool  health                         ONLINE                         -
datapool  guid                           11586357166166851864           -
datapool  version                        -                              default
datapool  bootfs                         -                              default
datapool  delegation                     on                             default
datapool  autoreplace                    off                            default
datapool  cachefile                      -                              default
datapool  failmode                       wait                           default
datapool  listsnapshots                  off                            default
datapool  autoexpand                     off                            default
datapool  dedupratio                     1.00x                          -
datapool  free                           13.5T                          -
datapool  allocated                      19.0T                          -
datapool  readonly                       off                            -
datapool  ashift                         0                              default
datapool  comment                        -                              default
datapool  expandsize                     -                              -
datapool  freeing                        0                              -
datapool  fragmentation                  2%                             -
datapool  leaked                         0                              -
datapool  multihost                      off                            default
datapool  checkpoint                     -                              -
datapool  load_guid                      10375314273603775311           -
datapool  autotrim                       off                            default
datapool  compatibility                  off                            default
datapool  feature@async_destroy          enabled                        local
datapool  feature@empty_bpobj            enabled                        local
datapool  feature@lz4_compress           active                         local
datapool  feature@multi_vdev_crash_dump  enabled                        local
datapool  feature@spacemap_histogram     active                         local
datapool  feature@enabled_txg            active                         local
datapool  feature@hole_birth             active                         local
datapool  feature@extensible_dataset     active                         local
datapool  feature@embedded_data          active                         local
datapool  feature@bookmarks              enabled                        local
datapool  feature@filesystem_limits      enabled                        local
datapool  feature@large_blocks           enabled                        local
datapool  feature@large_dnode            enabled                        local
datapool  feature@sha512                 enabled                        local
datapool  feature@skein                  enabled                        local
datapool  feature@edonr                  enabled                        local
datapool  feature@userobj_accounting     active                         local
datapool  feature@encryption             disabled                       local
datapool  feature@project_quota          disabled                       local
datapool  feature@device_removal         disabled                       local
datapool  feature@obsolete_counts        disabled                       local
datapool  feature@zpool_checkpoint       disabled                       local
datapool  feature@spacemap_v2            disabled                       local
datapool  feature@allocation_classes     disabled                       local
datapool  feature@resilver_defer         disabled                       local
datapool  feature@bookmark_v2            disabled                       local
datapool  feature@redaction_bookmarks    disabled                       local
datapool  feature@redacted_datasets      disabled                       local
datapool  feature@bookmark_written       disabled                       local
datapool  feature@log_spacemap           disabled                       local
datapool  feature@livelist               disabled                       local
datapool  feature@device_rebuild         disabled                       local
datapool  feature@zstd_compress          disabled                       local
datapool  feature@draid                  disabled                       local

Error DURING Scrub (had to reboot):

Aug  8 21:16:11 SomeAdmin-FileServer kernel: [14710.405063] general protection fault, probably for non-canonical address 0x10000000000000: 0000 [#1] SMP NOPTI
 Aug  8 21:16:11 SomeAdmin-FileServer kernel: [14710.405105] CPU: 1 PID: 788723 Comm: dsl_scan_iss Tainted: P           O      5.15.0-78-generic #85-Ubuntu
 Aug  8 21:16:11 SomeAdmin-FileServer kernel: [14710.405137] Hardware name: Gigabyte Technology Co., Ltd. B450 I AORUS PRO WIFI/B450 I AORUS PRO WIFI-CF, BIOS F64a 02/09/2023
 Aug  8 21:16:11 SomeAdmin-FileServer kernel: [14710.405171] RIP: 0010:avl_walk+0x58/0x140 [zavl]
 Aug  8 21:16:11 SomeAdmin-FileServer kernel: [14710.405191] Code: 48 83 fa 01 0f 87 ce 00 00 00 4d 8b 34 d6 4d 85 f6 74 4c 48 83 fa 01 0f 87 a7 00 00 00 48 63 db 48 83 fb 01 0f 87 ca 00 00 00 <49> 8b 04 de 48 85 c0 75 05 eb 55 48 89 d0 48 83 fb 01 77 51 48 8b
 Aug  8 21:16:11 SomeAdmin-FileServer kernel: [14710.405246] RSP: 0018:ffffb55591237cf0 EFLAGS: 00010297
 Aug  8 21:16:11 SomeAdmin-FileServer kernel: [14710.405265] RAX: 00000600f96a0000 RBX: 0000000000000000 RCX: 000000000000000c
 Aug  8 21:16:11 SomeAdmin-FileServer kernel: [14710.405287] RDX: 0000000000000001 RSI: ffff8c2bfa1765d8 RDI: ffff8c2d32823060
 Aug  8 21:16:11 SomeAdmin-FileServer kernel: [14710.405310] RBP: ffffb55591237d18 R08: ffffb55591237db8 R09: ffff8c2d32823060
 Aug  8 21:16:11 SomeAdmin-FileServer kernel: [14710.405332] R10: ffff8c2bfa176eb8 R11: ffff8c2bfa176638 R12: ffff8c2bfa176638
 Aug  8 21:16:11 SomeAdmin-FileServer kernel: [14710.405354] R13: 0000000000000060 R14: 0010000000000000 R15: ffff8c2bfa1765d8
 Aug  8 21:16:11 SomeAdmin-FileServer kernel: [14710.405377] FS:  0000000000000000(0000) GS:ffff8c2eb0a40000(0000) knlGS:0000000000000000
 Aug  8 21:16:11 SomeAdmin-FileServer kernel: [14710.405403] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 Aug  8 21:16:11 SomeAdmin-FileServer kernel: [14710.405421] CR2: 000000c00c88b000 CR3: 000000010f7a6000 CR4: 00000000003506e0
 Aug  8 21:16:11 SomeAdmin-FileServer kernel: [14710.405444] Call Trace:
 Aug  8 21:16:11 SomeAdmin-FileServer kernel: [14710.405454]  <TASK>
 Aug  8 21:16:11 SomeAdmin-FileServer kernel: [14710.405464]  scan_io_queue_gather+0xe2/0x270 [zfs]
 Aug  8 21:16:11 SomeAdmin-FileServer kernel: [14710.405581]  scan_io_queues_run_one+0x137/0x290 [zfs]
 Aug  8 21:16:11 SomeAdmin-FileServer kernel: [14710.405695]  taskq_thread+0x221/0x400 [spl]
 Aug  8 21:16:11 SomeAdmin-FileServer kernel: [14710.405717]  ? wake_up_q+0x90/0x90
 Aug  8 21:16:11 SomeAdmin-FileServer kernel: [14710.405734]  ? param_set_taskq_kick+0xf0/0xf0 [spl]
 Aug  8 21:16:11 SomeAdmin-FileServer kernel: [14710.405757]  kthread+0x12a/0x150
 Aug  8 21:16:11 SomeAdmin-FileServer kernel: [14710.405771]  ? set_kthread_struct+0x50/0x50
 Aug  8 21:16:11 SomeAdmin-FileServer kernel: [14710.405786]  ret_from_fork+0x22/0x30
 Aug  8 21:16:11 SomeAdmin-FileServer kernel: [14710.405802]  </TASK>
 Aug  8 21:16:11 SomeAdmin-FileServer kernel: [14710.405810] Modules linked in: tls rfcomm cmac algif_hash algif_skcipher af_alg bnep binfmt_misc nls_iso8859_1 zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) intel_rapl_msr icp(PO) btusb zcommon(PO) btrtl btbcm znvpair(PO) btintel spl(O) joydev input_leds bluetooth ecdh_generic ecc intel_rapl_common edac_mce_amd snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio iwlmvm kvm snd_hda_codec_hdmi mac80211 snd_hda_intel snd_intel_dspcfg rapl snd_intel_sdw_acpi libarc4 snd_hda_codec snd_hda_core snd_hwdep snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi wmi_bmof gigabyte_wmi snd_seq snd_seq_device iwlwifi snd_timer snd k10temp cfg80211 soundcore ccp mac_hid sch_fq_codel it87 hwmon_vid msr parport_pc ppdev lp parport ramoops pstore_blk reed_solomon pstore_zone efi_pstore ip_tables x_tables autofs4 dm_crypt uas usb_storage amdgpu hid_generic iommu_v2 gpu_sched drm_ttm_helper usbhid ttm hid drm_kms_helper syscopyarea crct10dif_pclmul crc32_pclmul sysfillrect sysimgblt
 Aug  8 21:16:11 SomeAdmin-FileServer kernel: [14710.405871]  ghash_clmulni_intel fb_sys_fops aesni_intel cec rc_core crypto_simd igb cryptd drm i2c_piix4 dca i2c_algo_bit xhci_pci ahci xhci_pci_renesas libahci gpio_amdpt wmi video gpio_generic
 Aug  8 21:16:11 SomeAdmin-FileServer kernel: [14710.406185] ---[ end trace 2b794df975b88f99 ]---
 Aug  8 21:16:12 SomeAdmin-FileServer kernel: [14710.546338] RIP: 0010:avl_walk+0x58/0x140 [zavl]
 Aug  8 21:16:12 SomeAdmin-FileServer kernel: [14710.546361] Code: 48 83 fa 01 0f 87 ce 00 00 00 4d 8b 34 d6 4d 85 f6 74 4c 48 83 fa 01 0f 87 a7 00 00 00 48 63 db 48 83 fb 01 0f 87 ca 00 00 00 <49> 8b 04 de 48 85 c0 75 05 eb 55 48 89 d0 48 83 fb 01 77 51 48 8b
 Aug  8 21:16:12 SomeAdmin-FileServer kernel: [14710.546404] RSP: 0018:ffffb55591237cf0 EFLAGS: 00010297
 Aug  8 21:16:12 SomeAdmin-FileServer kernel: [14710.546418] RAX: 00000600f96a0000 RBX: 0000000000000000 RCX: 000000000000000c
 Aug  8 21:16:12 SomeAdmin-FileServer kernel: [14710.546435] RDX: 0000000000000001 RSI: ffff8c2bfa1765d8 RDI: ffff8c2d32823060
 Aug  8 21:16:12 SomeAdmin-FileServer kernel: [14710.546452] RBP: ffffb55591237d18 R08: ffffb55591237db8 R09: ffff8c2d32823060
 Aug  8 21:16:12 SomeAdmin-FileServer kernel: [14710.546469] R10: ffff8c2bfa176eb8 R11: ffff8c2bfa176638 R12: ffff8c2bfa176638
 Aug  8 21:16:12 SomeAdmin-FileServer kernel: [14710.546485] R13: 0000000000000060 R14: 0010000000000000 R15: ffff8c2bfa1765d8
 Aug  8 21:16:12 SomeAdmin-FileServer kernel: [14710.546503] FS:  0000000000000000(0000) GS:ffff8c2eb0a40000(0000) knlGS:0000000000000000
 Aug  8 21:16:12 SomeAdmin-FileServer kernel: [14710.546522] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 Aug  8 21:16:12 SomeAdmin-FileServer kernel: [14710.546536] CR2: 000000c00c88b000 CR3: 000000010f7a6000 CR4: 00000000003506e0

SMART OUTPUT (FOR EACH DRIVE)
NOTE Yes I know the /dev/sdd has 1 error under "UDMA_CRC_ERROR_COUNT", that has been there for over two years. Never had a problem. I assume
that is just one sector of the drive that is screwed and it knows to bypass it.

NOTE 2 I just ran smartctl -t extended on these drives, so these results are in the last 48 hours.

Drive /dev/sda:

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Ultrastar He10/12
Device Model:     WDC WD120EMAZ-11BLFA0
Serial Number:    8CK9WLJF
LU WWN Device Id: 5 000cca 26feec573
Firmware Version: 81.00A81
User Capacity:    12,000,138,625,024 bytes [12.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Aug  9 23:19:30 2023 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (   87) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (1311) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0004   126   126   054    Old_age   Offline      -       116
  3 Spin_Up_Time            0x0007   183   183   024    Pre-fail  Always       -       310 (Average 408)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       121
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000a   100   100   067    Old_age   Always       -       0
  8 Seek_Time_Performance   0x0004   136   136   020    Old_age   Offline      -       16
  9 Power_On_Hours          0x0012   096   096   000    Old_age   Always       -       30070
 10 Spin_Retry_Count        0x0012   100   100   060    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       121
 22 Helium_Level            0x0023   100   100   025    Pre-fail  Always       -       100
192 Power-Off_Retract_Count 0x0032   099   099   000    Old_age   Always       -       1746
193 Load_Cycle_Count        0x0012   099   099   000    Old_age   Always       -       1746
194 Temperature_Celsius     0x0002   162   162   000    Old_age   Always       -       40 (Min/Max 18/54)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     30070         -
# 2  Extended offline    Aborted by host               90%     30042         -
# 3  Short offline       Completed without error       00%     30042         -
# 4  Short offline       Completed without error       00%     30042         -
# 5  Short offline       Aborted by host               10%      1110         -
# 6  Short offline       Aborted by host               10%      1110         -
# 7  Short offline       Aborted by host               10%      1110         -
# 8  Short captive       Interrupted (host reset)      50%      1109         -
# 9  Short offline       Completed without error       00%         0         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Drive /dev/sdb:

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Ultrastar He10/12
Device Model:     WDC WD120EMAZ-11BLFA0
Serial Number:    8CK8Z51F
LU WWN Device Id: 5 000cca 26fee5ab5
Firmware Version: 81.00A81
User Capacity:    12,000,138,625,024 bytes [12.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Aug  9 23:19:50 2023 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (   87) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (1274) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0004   127   127   054    Old_age   Offline      -       112
  3 Spin_Up_Time            0x0007   174   174   024    Pre-fail  Always       -       344 (Average 414)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       119
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000a   100   100   067    Old_age   Always       -       0
  8 Seek_Time_Performance   0x0004   140   140   020    Old_age   Offline      -       15
  9 Power_On_Hours          0x0012   096   096   000    Old_age   Always       -       30070
 10 Spin_Retry_Count        0x0012   100   100   060    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       119
 22 Helium_Level            0x0023   100   100   025    Pre-fail  Always       -       100
192 Power-Off_Retract_Count 0x0032   099   099   000    Old_age   Always       -       1721
193 Load_Cycle_Count        0x0012   099   099   000    Old_age   Always       -       1721
194 Temperature_Celsius     0x0002   162   162   000    Old_age   Always       -       40 (Min/Max 19/50)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     30070         -
# 2  Extended offline    Aborted by host               90%     30042         -
# 3  Short offline       Completed without error       00%     30042         -
# 4  Short offline       Interrupted (host reset)      10%      1746         -
# 5  Short offline       Aborted by host               10%      1113         -
# 6  Short offline       Aborted by host               10%      1112         -
# 7  Extended captive    Interrupted (host reset)      90%      1109         -
# 8  Short captive       Interrupted (host reset)      50%      1109         -
# 9  Short offline       Completed without error       00%      1109         -
#10  Short offline       Completed without error       00%         0         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Drive /dev/sdd:

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Ultrastar He10/12
Device Model:     WDC WD120EMAZ-11BLFA0
Serial Number:    8CJWZH8E
LU WWN Device Id: 5 000cca 26fe8e77e
Firmware Version: 81.00A81
User Capacity:    12,000,138,625,024 bytes [12.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Aug  9 23:20:07 2023 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (   87) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (1290) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0004   127   127   054    Old_age   Offline      -       112
  3 Spin_Up_Time            0x0007   182   182   024    Pre-fail  Always       -       314 (Average 410)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       121
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000a   100   100   067    Old_age   Always       -       0
  8 Seek_Time_Performance   0x0004   128   128   020    Old_age   Offline      -       18
  9 Power_On_Hours          0x0012   096   096   000    Old_age   Always       -       30070
 10 Spin_Retry_Count        0x0012   100   100   060    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       121
 22 Helium_Level            0x0023   100   100   025    Pre-fail  Always       -       100
192 Power-Off_Retract_Count 0x0032   099   099   000    Old_age   Always       -       2214
193 Load_Cycle_Count        0x0012   099   099   000    Old_age   Always       -       2214
194 Temperature_Celsius     0x0002   175   175   000    Old_age   Always       -       37 (Min/Max 19/54)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       1

SMART Error Log Version: 1
ATA Error Count: 1
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 1 occurred at disk power-on lifetime: 1930 hours (80 days + 10 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 43 00 00 00 00 00  Error: ICRC, ABRT at LBA = 0x00000000 = 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 80 40 88 3d 71 40 08      02:03:58.662  READ FPDMA QUEUED
  60 00 88 20 90 5a 40 08      02:03:58.654  READ FPDMA QUEUED
  60 30 48 08 3e 71 40 08      02:03:58.645  READ FPDMA QUEUED
  60 80 38 20 8b 5a 40 08      02:03:58.640  READ FPDMA QUEUED
  60 00 c0 10 60 31 40 08      02:03:58.639  READ FPDMA QUEUED

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     30070         -
# 2  Extended offline    Aborted by host               90%     30042         -
# 3  Short offline       Completed without error       00%     30042         -
# 4  Short offline       Completed without error       00%      1114         -
# 5  Short offline       Completed without error       00%      1110         -
# 6  Short offline       Completed without error       00%      1044         -
# 7  Short offline       Completed without error       00%         0         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

[rincebrain]

Try running a newer version than what Ubuntu ships (e.g. 2.1.12 is current, 2.1.5 is what you're on) or report the bug to Ubuntu.

[aafrasht]

I have resolved the issue. When I updated my bios, I set my RAM frequency to 3200 MHZ, which was causing my system to be unstable. I have set my RAM freq back to 2933 MHZ, and my scrub is now complete. Note that it still shows "There are errors" so I'll do another scrub to see if that gets rid of those errors.

Odd though...my RAM is rated for 3200mhz. I'll have to look into that later.

btw, is there a guide on how to upgrade my existing openzfs to the latest openzfs? I'm worried that I may mess up my pool and lose all my data.

[rincebrain]

Ubuntu used to have a reliable third-party packager in jonathonf, but sadly he passed away.

I'm not aware of anyone providing consistent third-party built Ubuntu PPA packages atm, but I suspect someone will pop up if they haven't already.

As of OpenZFS 2.2 (which is still in RC, so I'm not suggesting you run it anywhere you're not just testing with data you don't care about), there will be a build target in the makefile for native Debian packaging like the Debian/Ubuntu packages, which will make it significantly easier, compared to the alien-generated debs you get right now, which are very differently laid out from how Ubuntu/Debian package things.

[aafrasht]

Ok, if there is a good step by step guide on how to upgrade I can do that. In the meantime my pool is working fine again.

btw, is there a strong reason to upgrade from 2.1.12 to 2.2 (once it comes out)? Critical bug or a highly desirable feature? My fileserver is there for simple needs (files, plex server, syncing), it doesn't do anything special or advanced. The data on it is important though and I use it for my main storage of my personal stuff.

[rincebrain]

I generally advise upgrading to point releases eventually and minor releases when the older minor's point releases stop being maintained and you continue to want support if something breaks, or if there's a killer feature (like cp --reflink or BLAKE3 support or zstd early abort or w/e) for your use case.

But people's reasons for upgrading may differ.