Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Linux 6.5.5 deadlock on lvrename? #8575

Closed
marmarek opened this issue Oct 7, 2023 · 5 comments
Closed

Linux 6.5.5 deadlock on lvrename? #8575

marmarek opened this issue Oct 7, 2023 · 5 comments
Labels
affects-4.2 This issue affects Qubes OS 4.2. C: kernel diagnosed Technical diagnosis has been performed (see issue comments). P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. r4.1-dom0-stable r4.2-host-stable
Milestone
Release 4.2

Comments

@marmarek
Copy link
Member

marmarek commented Oct 7, 2023

How to file a helpful issue

Qubes OS release

R4.2

Brief summary

A test hanged on VM shutdown, specifically lvrename step: https://openqa.qubes-os.org/tests/83530

I'm creating the issue to write down collected logs, but I'll investigate only when it will happen for the second time.

Steps to reproduce

Run integration tests and hope to trigger the issue. I'm not sure if it's hardware dependent, or just hard to reproduce

Expected behavior

No hang

Actual behavior

Shutting down a VM hangs.

Process list:

        root        7835  0.0  0.7 251848 31104 pts/8    D<L+ 21:30   0:00 lvm lvrename -- qubes_dom0/vm-test-inst-vm-private qubes_dom0/vm-test-inst-vm-private-1696714259-back
        root        7839  0.0  0.2 240888  9600 pts/8    S+   21:30   0:00 lvm lvremove --force -- qubes_dom0/vm-test-inst-vm-volatile
        root        7841  0.0  0.2 240888  9472 pts/8    S+   21:30   0:00 lvm lvremove --force -- qubes_dom0/vm-test-inst-vm-root-snap
        root        7846  0.0  0.2 240916  9592 pts/8    S+   21:31   0:00 lvm lvs --noheadings -o vg_name,pool_lv,name,lv_size,data_percent,lv_attr,origin,lv_metadata_size,metadata_percent --units b --reportformat=json
Task backtraces (via sysrq) 
[ 3485.834038] task:lvm             state:D stack:0     pid:7835  ppid:5665   flags:0x00004006
[ 3485.834043] Call Trace:
[ 3485.834046]  <TASK>
[ 3485.834048]  __schedule+0x23d/0x670
[ 3485.834052]  schedule+0x5e/0xd0
[ 3485.834057]  io_schedule+0x46/0x70
[ 3485.834061]  dm_wait_for_bios_completion+0xfc/0x110
[ 3485.834067]  ? __pfx_autoremove_wake_function+0x10/0x10
[ 3485.834072]  __dm_suspend+0x7e/0x1b0
[ 3485.834077]  dm_internal_suspend_noflush+0x5c/0x80
[ 3485.834083]  pool_presuspend+0xcc/0x130 [dm_thin_pool]
[ 3485.834092]  dm_table_presuspend_targets+0x3f/0x60
[ 3485.834098]  __dm_suspend+0x41/0x1b0
[ 3485.834103]  dm_suspend+0xc0/0xe0
[ 3485.834107]  dev_suspend+0xa5/0xd0
[ 3485.834111]  ctl_ioctl+0x26e/0x350
[ 3485.834115]  ? __pfx_dev_suspend+0x10/0x10
[ 3485.834120]  dm_ctl_ioctl+0xe/0x20
[ 3485.834124]  __x64_sys_ioctl+0x94/0xd0
[ 3485.834129]  do_syscall_64+0x5c/0x90
[ 3485.834133]  ? do_syscall_64+0x6b/0x90
[ 3485.834137]  ? do_syscall_64+0x6b/0x90
[ 3485.834141]  ? xen_pv_evtchn_do_upcall+0x54/0xb0
[ 3485.834146]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[ 3485.834151] RIP: 0033:0x7f1cb77cfe0f
[ 3485.834154] RSP: 002b:00007fff870f2560 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[ 3485.834159] RAX: ffffffffffffffda RBX: 00005b8d13c16580 RCX: 00007f1cb77cfe0f
[ 3485.834164] RDX: 00005b8d144a2180 RSI: 00000000c138fd06 RDI: 0000000000000003
[ 3485.834169] RBP: 00005b8d144a2180 R08: 00005b8d132b1190 R09: 00007fff870f2420
[ 3485.834174] R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000000c
[ 3485.834179] R13: 00005b8d132aacf0 R14: 00005b8d1324414d R15: 00005b8d144a21b0
[ 3485.834185]  </TASK>
[ 3485.834187] task:lvm             state:S stack:0     pid:7839  ppid:5665   flags:0x00000002
[ 3485.834193] Call Trace:
[ 3485.834196]  <TASK>
[ 3485.834198]  __schedule+0x23d/0x670
[ 3485.834202]  schedule+0x5e/0xd0
[ 3485.834207]  locks_lock_inode_wait+0xb7/0x1d0
[ 3485.834214]  ? __pfx_autoremove_wake_function+0x10/0x10
[ 3485.834218]  __do_sys_flock+0x1a4/0x1f0
[ 3485.834224]  do_syscall_64+0x5c/0x90
[ 3485.834231]  ? syscall_exit_to_user_mode+0x2b/0x40
[ 3485.834235]  ? do_syscall_64+0x6b/0x90
[ 3485.834239]  ? do_syscall_64+0x6b/0x90
[ 3485.834243]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[ 3485.834248] RIP: 0033:0x7b558416087b
[ 3485.834251] RSP: 002b:00007ffdb5220108 EFLAGS: 00000202 ORIG_RAX: 0000000000000049
[ 3485.834256] RAX: ffffffffffffffda RBX: 00007ffdb5220380 RCX: 00007b558416087b
[ 3485.834261] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000006
[ 3485.834266] RBP: 00007ffdb5220320 R08: 00005ee1a9d908c0 R09: 0000000000000042
[ 3485.834271] R10: 0000000000000008 R11: 0000000000000202 R12: 00005ee1ab110020
[ 3485.834276] R13: 0000000000000002 R14: 00007ffdb5220380 R15: 0000000000000000
[ 3485.834282]  </TASK>
[ 3485.834285] task:lvm             state:S stack:0     pid:7841  ppid:5665   flags:0x00000002
[ 3485.834291] Call Trace:
[ 3485.834293]  <TASK>
[ 3485.834296]  __schedule+0x23d/0x670
[ 3485.834300]  schedule+0x5e/0xd0
[ 3485.834304]  locks_lock_inode_wait+0xb7/0x1d0
[ 3485.834309]  ? __pfx_autoremove_wake_function+0x10/0x10
[ 3485.834314]  __do_sys_flock+0x1a4/0x1f0
[ 3485.834319]  do_syscall_64+0x5c/0x90
[ 3485.834324]  ? exc_page_fault+0x7c/0x170
[ 3485.834327]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[ 3485.834331] RIP: 0033:0x782067da787b
[ 3485.834334] RSP: 002b:00007fff11cdab08 EFLAGS: 00000202 ORIG_RAX: 0000000000000049
[ 3485.834340] RAX: ffffffffffffffda RBX: 00007fff11cdad80 RCX: 0000782067da787b
[ 3485.834345] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000005
[ 3485.834350] RBP: 00007fff11cdad20 R08: 0000557e4813c8c0 R09: 0000000000000042
[ 3485.834355] R10: 0000000000000008 R11: 0000000000000202 R12: 00007fff11cdace4
[ 3485.834360] R13: 0000000000000002 R14: 00007fff11cdaca0 R15: 0000000000000000
[ 3485.834365]  </TASK>
[ 3485.834368] task:lvm             state:S stack:0     pid:7846  ppid:5665   flags:0x00000002
[ 3485.834374] Call Trace:
[ 3485.834376]  <TASK>
[ 3485.834378]  __schedule+0x23d/0x670
[ 3485.834383]  schedule+0x5e/0xd0
[ 3485.834387]  locks_lock_inode_wait+0xb7/0x1d0
[ 3485.834392]  ? __pfx_autoremove_wake_function+0x10/0x10
[ 3485.834397]  __do_sys_flock+0x1a4/0x1f0
[ 3485.834402]  do_syscall_64+0x5c/0x90
[ 3485.834406]  ? syscall_exit_work+0x103/0x130
[ 3485.834410]  ? syscall_exit_to_user_mode+0x2b/0x40
[ 3485.834414]  ? do_syscall_64+0x6b/0x90
[ 3485.834418]  ? __x64_sys_rt_sigaction+0xbd/0x110
[ 3485.834423]  ? syscall_exit_work+0x103/0x130
[ 3485.834427]  ? syscall_exit_to_user_mode+0x2b/0x40
[ 3485.834431]  ? do_syscall_64+0x6b/0x90
[ 3485.834435]  ? do_syscall_64+0x6b/0x90
[ 3485.834439]  ? do_syscall_64+0x6b/0x90
[ 3485.834444]  ? exc_page_fault+0x7c/0x170
[ 3485.834447]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[ 3485.834451] RIP: 0033:0x7e46fda1987b
[ 3485.834454] RSP: 002b:00007ffdc0e266f8 EFLAGS: 00000206 ORIG_RAX: 0000000000000049
[ 3485.834460] RAX: ffffffffffffffda RBX: 00007ffdc0e26970 RCX: 00007e46fda1987b
[ 3485.834465] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000006
[ 3485.834470] RBP: 00007ffdc0e26910 R08: 00005ee4e9a708c0 R09: 0000000000000042
[ 3485.834475] R10: 0000000000000008 R11: 0000000000000206 R12: 00007ffdc0e268d4
[ 3485.834480] R13: 0000000000000002 R14: 00007ffdc0e26890 R15: 0000000000000000
[ 3485.834485]  </TASK>
...
[ 3485.846084] Showing busy workqueues and worker pools:
[ 3485.846090] workqueue writeback: flags=0x4a
[ 3485.846094]   pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3
[ 3485.846096]     in-flight: 8033:wb_workfn
[ 3485.846121] pool 8: cpus=0-3 flags=0x4 nice=0 hung=0s workers=5 idle: 8178 7976 12 1959
Blocked tasks 
[ 4246.558313] sysrq: Show Blocked State
[ 4246.558388] task:journal-offline state:D stack:0     pid:8098  ppid:1      flags:0x00000002
[ 4246.558407] Call Trace:
[ 4246.558414]  <TASK>
[ 4246.558422]  __schedule+0x23d/0x670
[ 4246.558440]  schedule+0x5e/0xd0
[ 4246.558450]  io_schedule+0x46/0x70
[ 4246.558461]  folio_wait_bit_common+0x13d/0x350
[ 4246.558475]  ? __pfx_wake_page_function+0x10/0x10
[ 4246.558488]  folio_wait_writeback+0x2c/0x90
[ 4246.558498]  mpage_prepare_extent_to_map+0x15c/0x4d0
[ 4246.558512]  ext4_do_writepages+0x25f/0x770
[ 4246.558523]  ext4_writepages+0xad/0x180
[ 4246.558533]  do_writepages+0xcf/0x1e0
[ 4246.558543]  ? __seccomp_filter+0x32a/0x4f0
[ 4246.558554]  filemap_fdatawrite_wbc+0x63/0x90
[ 4246.558567]  __filemap_fdatawrite_range+0x5c/0x80
[ 4246.558578]  file_write_and_wait_range+0x4a/0xb0
[ 4246.558588]  ext4_sync_file+0x88/0x380
[ 4246.558598]  __x64_sys_fsync+0x3b/0x70
[ 4246.558609]  do_syscall_64+0x5c/0x90
[ 4246.558621]  ? exit_to_user_mode_prepare+0xb2/0xd0
[ 4246.558632]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[ 4246.558644] RIP: 0033:0x7710cf124d0a
[ 4246.558654] RSP: 002b:00007710ccdfda40 EFLAGS: 00000293 ORIG_RAX: 000000000000004a
[ 4246.558668] RAX: ffffffffffffffda RBX: 000064bb92f67e60 RCX: 00007710cf124d0a
[ 4246.558679] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000028
[ 4246.558691] RBP: 000064bb92f72670 R08: 0000000000000000 R09: 00007710ccdfe6c0
[ 4246.558702] R10: 00007710cf0adfee R11: 0000000000000293 R12: 000064bb92505940
[ 4246.558713] R13: 0000000000000002 R14: 00007ffc05649500 R15: 00007710cc5fe000
[ 4246.558728]  </TASK>
[ 4246.558836] task:lvm             state:D stack:0     pid:7835  ppid:5665   flags:0x00004006
[ 4246.558852] Call Trace:
[ 4246.558857]  <TASK>
[ 4246.558863]  __schedule+0x23d/0x670
[ 4246.558874]  schedule+0x5e/0xd0
[ 4246.558884]  io_schedule+0x46/0x70
[ 4246.558894]  dm_wait_for_bios_completion+0xfc/0x110
[ 4246.558909]  ? __pfx_autoremove_wake_function+0x10/0x10
[ 4246.558922]  __dm_suspend+0x7e/0x1b0
[ 4246.558932]  dm_internal_suspend_noflush+0x5c/0x80
[ 4246.558946]  pool_presuspend+0xcc/0x130 [dm_thin_pool]
[ 4246.558968]  dm_table_presuspend_targets+0x3f/0x60
[ 4246.558980]  __dm_suspend+0x41/0x1b0
[ 4246.558991]  dm_suspend+0xc0/0xe0
[ 4246.559001]  dev_suspend+0xa5/0xd0
[ 4246.559011]  ctl_ioctl+0x26e/0x350
[ 4246.559020]  ? __pfx_dev_suspend+0x10/0x10
[ 4246.559032]  dm_ctl_ioctl+0xe/0x20
[ 4246.559041]  __x64_sys_ioctl+0x94/0xd0
[ 4246.559052]  do_syscall_64+0x5c/0x90
[ 4246.559062]  ? do_syscall_64+0x6b/0x90
[ 4246.559072]  ? do_syscall_64+0x6b/0x90
[ 4246.559081]  ? xen_pv_evtchn_do_upcall+0x54/0xb0
[ 4246.559093]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[ 4246.559104] RIP: 0033:0x7f1cb77cfe0f
[ 4246.559112] RSP: 002b:00007fff870f2560 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[ 4246.559141] RAX: ffffffffffffffda RBX: 00005b8d13c16580 RCX: 00007f1cb77cfe0f
[ 4246.559152] RDX: 00005b8d144a2180 RSI: 00000000c138fd06 RDI: 0000000000000003
[ 4246.559164] RBP: 00005b8d144a2180 R08: 00005b8d132b1190 R09: 00007fff870f2420
[ 4246.559175] R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000000c
[ 4246.559186] R13: 00005b8d132aacf0 R14: 00005b8d1324414d R15: 00005b8d144a21b0
[ 4246.559199]  </TASK>
[ 4246.559207] task:kworker/u8:3    state:D stack:0     pid:8033  ppid:2      flags:0x00004000
[ 4246.559222] Workqueue: writeback wb_workfn (flush-253:4)
[ 4246.559238] Call Trace:
[ 4246.559244]  <TASK>
[ 4246.559249]  __schedule+0x23d/0x670
[ 4246.559260]  schedule+0x5e/0xd0
[ 4246.559270]  io_schedule+0x46/0x70
[ 4246.559280]  folio_wait_bit_common+0x13d/0x350
[ 4246.559290]  ? __pfx_wake_page_function+0x10/0x10
[ 4246.559302]  mpage_prepare_extent_to_map+0x309/0x4d0
[ 4246.559314]  ext4_do_writepages+0x25f/0x770
[ 4246.559324]  ext4_writepages+0xad/0x180
[ 4246.559334]  do_writepages+0xcf/0x1e0
[ 4246.559344]  ? find_busiest_group+0x42/0x1a0
[ 4246.559354]  __writeback_single_inode+0x3d/0x280
[ 4246.559368]  writeback_sb_inodes+0x1ed/0x4a0
[ 4246.559381]  __writeback_inodes_wb+0x4c/0xf0
[ 4246.559393]  wb_writeback+0x298/0x310
[ 4246.559403]  wb_do_writeback+0x230/0x2b0
[ 4246.559414]  wb_workfn+0x5f/0x260
[ 4246.559424]  ? _raw_spin_unlock+0xe/0x30
[ 4246.559434]  ? finish_task_switch.isra.0+0x95/0x2b0
[ 4246.559447]  ? __schedule+0x245/0x670
[ 4246.559457]  process_one_work+0x1df/0x3e0
[ 4246.559466]  worker_thread+0x51/0x390
[ 4246.559475]  ? __pfx_worker_thread+0x10/0x10
[ 4246.559484]  kthread+0xe5/0x120
[ 4246.559495]  ? __pfx_kthread+0x10/0x10
[ 4246.559504]  ret_from_fork+0x31/0x50
[ 4246.559514]  ? __pfx_kthread+0x10/0x10
[ 4246.559523]  ret_from_fork_asm+0x1b/0x30
[ 4246.559536]  </TASK>
@marmarek marmarek added T: bug C: kernel P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. affects-4.2 This issue affects Qubes OS 4.2. labels Oct 7, 2023
@rapenne-s rapenne-s mentioned this issue Oct 9, 2023
Closed
@marmarek
Copy link
Member Author

marmarek commented Oct 13, 2023 •
edited
Loading

Similar thing happens on 6.5.6 too. Whole disk I/O got frozen, no lvrename involved this time.

sysrq: Show Blocked State 
[  522.190523] sysrq: Show Blocked State
[  522.190621] task:dmcrypt_write/2 state:D stack:0     pid:1245  ppid:2      flags:0x00004000
[  522.190649] Call Trace:
[  522.190660]  <TASK>
[  522.190681]  __schedule+0x23d/0x670
[  522.190710]  ? __pfx_wbt_inflight_cb+0x10/0x10
[  522.190735]  ? __pfx_wbt_cleanup_cb+0x10/0x10
[  522.190757]  ? __pfx_dmcrypt_write+0x10/0x10 [dm_crypt]
[  522.190786]  schedule+0x5e/0xd0
[  522.190802]  io_schedule+0x46/0x70
[  522.190818]  rq_qos_wait+0xc0/0x140
[  522.190835]  ? __pfx_rq_qos_wake_function+0x10/0x10
[  522.190850]  ? __pfx_wbt_inflight_cb+0x10/0x10
[  522.190867]  wbt_wait+0xa6/0x110
[  522.190882]  __rq_qos_throttle+0x24/0x40
[  522.190895]  blk_mq_get_new_requests+0xc9/0x180
[  522.190912]  blk_mq_submit_bio+0x332/0x550
[  522.190927]  __submit_bio+0x8b/0x170
[  522.190941]  submit_bio_noacct_nocheck+0x120/0x1f0
[  522.190955]  ? submit_bio_noacct+0x49/0x560
[  522.190967]  dmcrypt_write+0x11a/0x140 [dm_crypt]
[  522.191059]  kthread+0xe5/0x120
[  522.191075]  ? __pfx_kthread+0x10/0x10
[  522.191095]  ret_from_fork+0x31/0x50
[  522.191108]  ? __pfx_kthread+0x10/0x10
[  522.191121]  ret_from_fork_asm+0x1b/0x30
[  522.191141]  </TASK>
[  522.191151] task:kworker/u12:5   state:D stack:0     pid:1324  ppid:2      flags:0x00004000
[  522.191172] Workqueue: writeback wb_workfn (flush-253:4)
[  522.191194] Call Trace:
[  522.191201]  <TASK>
[  522.191208]  __schedule+0x23d/0x670
[  522.191224]  schedule+0x5e/0xd0
[  522.191238]  schedule_preempt_disabled+0x15/0x30
[  522.191255]  rwsem_down_read_slowpath+0x29e/0x4f0
[  522.191272]  down_read+0x48/0xa0
[  522.191286]  dm_thin_find_block+0x32/0x80 [dm_thin_pool]
[  522.191319]  thin_bio_map.isra.0+0x15d/0x380 [dm_thin_pool]
[  522.191348]  __map_bio+0x44/0x1b0
[  522.191363]  dm_split_and_process_bio+0x1fa/0x510
[  522.191378]  dm_submit_bio+0x4e/0x100
[  522.191389]  __submit_bio+0xb0/0x170
[  522.191402]  __submit_bio_noacct+0x90/0x220
[  522.191415]  __block_write_full_folio+0x1e1/0x400
[  522.191431]  ? __pfx_end_buffer_async_write+0x10/0x10
[  522.191445]  ? __pfx_blkdev_get_block+0x10/0x10
[  522.191462]  writepage_cb+0x1a/0x70
[  522.191476]  write_cache_pages+0x143/0x3a0
[  522.191489]  ? __pfx_writepage_cb+0x10/0x10
[  522.191508]  do_writepages+0x164/0x1e0
[  522.191524]  __writeback_single_inode+0x3d/0x280
[  522.191542]  writeback_sb_inodes+0x1ed/0x4a0
[  522.191560]  __writeback_inodes_wb+0x4c/0xf0
[  522.191577]  wb_writeback+0x298/0x310
[  522.191592]  wb_do_writeback+0x1f8/0x2b0
[  522.191608]  wb_workfn+0x5f/0x260
[  522.191623]  ? _raw_spin_unlock+0xe/0x30
[  522.191634]  ? finish_task_switch.isra.0+0x95/0x2b0
[  522.191651]  ? __schedule+0x245/0x670
[  522.191666]  process_one_work+0x1df/0x3e0
[  522.191679]  worker_thread+0x51/0x390
[  522.191691]  ? __pfx_worker_thread+0x10/0x10
[  522.191704]  kthread+0xe5/0x120
[  522.191716]  ? __pfx_kthread+0x10/0x10
[  522.191729]  ret_from_fork+0x31/0x50
[  522.191740]  ? __pfx_kthread+0x10/0x10
[  522.191752]  ret_from_fork_asm+0x1b/0x30
[  522.191769]  </TASK>
[  522.191778] task:jbd2/dm-4-8     state:D stack:0     pid:1359  ppid:2      flags:0x00004000
[  522.191795] Call Trace:
[  522.191802]  <TASK>
[  522.191809]  __schedule+0x23d/0x670
[  522.191825]  schedule+0x5e/0xd0
[  522.191839]  schedule_preempt_disabled+0x15/0x30
[  522.191854]  rwsem_down_read_slowpath+0x29e/0x4f0
[  522.191869]  down_read+0x48/0xa0
[  522.191880]  dm_thin_find_block+0x32/0x80 [dm_thin_pool]
[  522.191905]  thin_bio_map.isra.0+0x15d/0x380 [dm_thin_pool]
[  522.191931]  __map_bio+0x44/0x1b0
[  522.191941]  dm_split_and_process_bio+0x1fa/0x510
[  522.191954]  dm_submit_bio+0x4e/0x100
[  522.191965]  __submit_bio+0xb0/0x170
[  522.191976]  __submit_bio_noacct+0x90/0x220
[  522.191987]  jbd2_journal_commit_transaction+0x63d/0x1a10
[  522.192030]  ? finish_task_switch.isra.0+0x95/0x2b0
[  522.192046]  kjournald2+0xad/0x280
[  522.192060]  ? __pfx_autoremove_wake_function+0x10/0x10
[  522.192076]  ? __pfx_kjournald2+0x10/0x10
[  522.192088]  kthread+0xe5/0x120
[  522.192099]  ? __pfx_kthread+0x10/0x10
[  522.192111]  ret_from_fork+0x31/0x50
[  522.192120]  ? __pfx_kthread+0x10/0x10
[  522.192131]  ret_from_fork_asm+0x1b/0x30
[  522.192146]  </TASK>
[  522.192153] task:systemd-journal state:D stack:0     pid:1445  ppid:1      flags:0x00000002
[  522.192169] Call Trace:
[  522.192176]  <TASK>
[  522.192183]  __schedule+0x23d/0x670
[  522.192197]  schedule+0x5e/0xd0
[  522.192210]  schedule_preempt_disabled+0x15/0x30
[  522.192225]  rwsem_down_read_slowpath+0x29e/0x4f0
[  522.192239]  down_read+0x48/0xa0
[  522.192249]  dm_thin_find_block+0x32/0x80 [dm_thin_pool]
[  522.192274]  thin_bio_map.isra.0+0x15d/0x380 [dm_thin_pool]
[  522.192299]  __map_bio+0x44/0x1b0
[  522.192310]  dm_split_and_process_bio+0x1fa/0x510
[  522.192322]  dm_submit_bio+0x4e/0x100
[  522.192332]  __submit_bio+0xb0/0x170
[  522.192343]  __submit_bio_noacct+0x90/0x220
[  522.192354]  ext4_read_bh+0x46/0x90
[  522.192367]  ext4_bread+0x52/0x70
[  522.192378]  __ext4_read_dirblock+0x52/0x340
[  522.192393]  htree_dirblock_to_tree+0xac/0x370
[  522.192409]  ext4_htree_fill_tree+0x113/0x3d0
[  522.192425]  ext4_dx_readdir+0x108/0x370
[  522.192437]  ext4_readdir+0x5b2/0x6f0
[  522.192448]  iterate_dir+0x90/0x170
[  522.192463]  __x64_sys_getdents64+0x88/0x130
[  522.192477]  ? __pfx_filldir64+0x10/0x10
[  522.192490]  ? syscall_trace_enter.constprop.0+0x124/0x1a0
[  522.192504]  do_syscall_64+0x5c/0x90
[  522.192519]  ? syscall_exit_work+0x103/0x130
[  522.192531]  ? syscall_exit_to_user_mode+0x2b/0x40
[  522.192544]  ? do_syscall_64+0x6b/0x90
[  522.192557]  ? syscall_exit_work+0x103/0x130
[  522.192569]  ? syscall_exit_to_user_mode+0x2b/0x40
[  522.192580]  ? do_syscall_64+0x6b/0x90
[  522.192593]  ? syscall_exit_work+0x103/0x130
[  522.192604]  ? syscall_exit_to_user_mode+0x2b/0x40
[  522.192615]  ? do_syscall_64+0x6b/0x90
[  522.192628]  ? exit_to_user_mode_prepare+0xbc/0xd0
[  522.192640]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[  522.192656] RIP: 0033:0x7fb522cf5bc7
[  522.192668] RSP: 002b:00007ffe8c54c5b8 EFLAGS: 00000293 ORIG_RAX: 00000000000000d9
[  522.192686] RAX: ffffffffffffffda RBX: 0000564cdf0a27a0 RCX: 00007fb522cf5bc7
[  522.192700] RDX: 0000000000008000 RSI: 0000564cdf0a27a0 RDI: 0000000000000025
[  522.192714] RBP: 0000564cdf0a2774 R08: 0000000000008030 R09: 0000000000000000
[  522.192727] R10: 0000000000000000 R11: 0000000000000293 R12: fffffffffffffdb8
[  522.192741] R13: 0000000000000000 R14: 0000564cdf0a2770 R15: 00007ffe8c54d068
[  522.192757]  </TASK>
[  522.192764] task:journal-offline state:D stack:0     pid:4053  ppid:1      flags:0x00000002
[  522.192778] Call Trace:
[  522.192787]  <TASK>
[  522.192794]  __schedule+0x23d/0x670
[  522.192807]  schedule+0x5e/0xd0
[  522.192818]  schedule_timeout+0x151/0x160
[  522.192833]  io_schedule_timeout+0x50/0x80
[  522.192845]  wait_for_completion_io+0x8a/0x160
[  522.192856]  submit_bio_wait+0x61/0x90
[  522.192869]  blkdev_issue_flush+0x4d/0x80
[  522.192882]  ? __pfx_submit_bio_wait_endio+0x10/0x10
[  522.192896]  ext4_sync_file+0x2b8/0x380
[  522.192908]  __x64_sys_fsync+0x3b/0x70
[  522.192919]  do_syscall_64+0x5c/0x90
[  522.192931]  ? syscall_exit_work+0x103/0x130
[  522.192942]  ? syscall_exit_to_user_mode+0x2b/0x40
[  522.192953]  ? do_syscall_64+0x6b/0x90
[  522.192965]  ? syscall_exit_work+0x103/0x130
[  522.192975]  ? syscall_exit_to_user_mode+0x2b/0x40
[  522.192986]  ? do_syscall_64+0x6b/0x90
[  522.193011]  ? switch_fpu_return+0x50/0xe0
[  522.193023]  ? exit_to_user_mode_prepare+0xb2/0xd0
[  522.193034]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[  522.193046] RIP: 0033:0x7fb522d24d0a
[  522.193054] RSP: 002b:00007fb5211fea40 EFLAGS: 00000293 ORIG_RAX: 000000000000004a
[  522.193068] RAX: ffffffffffffffda RBX: 0000564cdf05ee70 RCX: 00007fb522d24d0a
[  522.193079] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000017
[  522.193091] RBP: 0000564cdf053d80 R08: 0000000000000000 R09: 00007fb5211ff6c0
[  522.193104] R10: 00007fb522cadfee R11: 0000000000000293 R12: 0000564cde251940
[  522.193115] R13: 0000000000000002 R14: 00007ffe8c54cac0 R15: 00007fb5209ff000
[  522.193129]  </TASK>
[  522.193134] task:journal-offline state:D stack:0     pid:4054  ppid:1      flags:0x00000002
[  522.193147] Call Trace:
[  522.193153]  <TASK>
[  522.193159]  __schedule+0x23d/0x670
[  522.193171]  schedule+0x5e/0xd0
[  522.193183]  schedule_timeout+0x151/0x160
[  522.193195]  io_schedule_timeout+0x50/0x80
[  522.193206]  wait_for_completion_io+0x8a/0x160
[  522.193217]  submit_bio_wait+0x61/0x90
[  522.193229]  blkdev_issue_flush+0x4d/0x80
[  522.193241]  ? __pfx_submit_bio_wait_endio+0x10/0x10
[  522.193255]  ext4_sync_file+0x2b8/0x380
[  522.193265]  __x64_sys_fsync+0x3b/0x70
[  522.193275]  do_syscall_64+0x5c/0x90
[  522.193289]  ? syscall_exit_work+0x103/0x130
[  522.193299]  ? syscall_exit_to_user_mode+0x2b/0x40
[  522.193312]  ? do_syscall_64+0x6b/0x90
[  522.193324]  ? exit_to_user_mode_prepare+0xb2/0xd0
[  522.193335]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[  522.193403] RIP: 0033:0x7fb522d24d0a
[  522.193411] RSP: 002b:00007fb522310a40 EFLAGS: 00000293 ORIG_RAX: 000000000000004a
[  522.193424] RAX: ffffffffffffffda RBX: 0000564cdf06f450 RCX: 00007fb522d24d0a
[  522.193436] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000000000002a
[  522.193447] RBP: 0000564cdf052c60 R08: 0000000000000000 R09: 00007fb5223116c0
[  522.193459] R10: 00007fb522cadfee R11: 0000000000000293 R12: 0000564cde251940
[  522.193471] R13: 0000000000000002 R14: 00007ffe8c54cac0 R15: 00007fb521b11000
[  522.193484]  </TASK>
[  522.193491] task:dmeventd        state:D stack:0     pid:2049  ppid:1      flags:0x00000002
[  522.193505] Call Trace:
[  522.193510]  <TASK>
[  522.193516]  __schedule+0x23d/0x670
[  522.193529]  schedule+0x5e/0xd0
[  522.193541]  schedule_preempt_disabled+0x15/0x30
[  522.193554]  rwsem_down_read_slowpath+0x29e/0x4f0
[  522.193567]  down_read+0x48/0xa0
[  522.193577]  dm_pool_get_metadata_transaction_id+0x22/0x60 [dm_thin_pool]
[  522.193602]  pool_status+0x122/0x600 [dm_thin_pool]
[  522.193626]  retrieve_status+0x15d/0x210
[  522.193638]  table_status+0x9b/0x140
[  522.193650]  ctl_ioctl+0x26e/0x350
[  522.193660]  ? __pfx_table_status+0x10/0x10
[  522.193675]  dm_ctl_ioctl+0xe/0x20
[  522.193685]  __x64_sys_ioctl+0x94/0xd0
[  522.193697]  do_syscall_64+0x5c/0x90
[  522.193709]  ? do_syscall_64+0x6b/0x90
[  522.193720]  ? __rseq_handle_notify_resume+0x36/0x60
[  522.193733]  ? exit_to_user_mode_loop+0x42/0x150
[  522.193746]  ? exit_to_user_mode_prepare+0xbc/0xd0
[  522.193756]  ? syscall_exit_to_user_mode+0x2b/0x40
[  522.193766]  ? do_syscall_64+0x6b/0x90
[  522.193776]  ? syscall_exit_to_user_mode+0x2b/0x40
[  522.193786]  ? do_syscall_64+0x6b/0x90
[  522.193796]  ? do_syscall_64+0x6b/0x90
[  522.193806]  ? do_syscall_64+0x6b/0x90
[  522.193817]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[  522.193828] RIP: 0033:0x72fbc9b09e0f
[  522.193835] RSP: 002b:000072fbc91229a0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[  522.193846] RAX: ffffffffffffffda RBX: 000072fbc4a4e4a0 RCX: 000072fbc9b09e0f
[  522.193859] RDX: 000072fbc4067c30 RSI: 00000000c138fd0c RDI: 0000000000000007
[  522.193870] RBP: 000000000000001e R08: 0000000000000004 R09: 000072fbc9c51858
[  522.193881] R10: 000072fbc9c4f576 R11: 0000000000000246 R12: 000072fbc9c4f576
[  522.193892] R13: 000072fbc9c4f576 R14: 000072fbc4067c60 R15: 000072fbc9c4f576
[  522.193904]  </TASK>
[  522.193916] task:alsactl         state:D stack:0     pid:2658  ppid:1      flags:0x00000002
[  522.193928] Call Trace:
[  522.193933]  <TASK>
[  522.193939]  __schedule+0x23d/0x670
[  522.193951]  schedule+0x5e/0xd0
[  522.193961]  schedule_preempt_disabled+0x15/0x30
[  522.193974]  rwsem_down_read_slowpath+0x29e/0x4f0
[  522.193986]  down_read+0x48/0xa0
[  522.194006]  dm_thin_find_block+0x32/0x80 [dm_thin_pool]
[  522.194028]  thin_bio_map.isra.0+0x15d/0x380 [dm_thin_pool]
[  522.194049]  __map_bio+0x44/0x1b0
[  522.194058]  dm_split_and_process_bio+0x1fa/0x510
[  522.194069]  dm_submit_bio+0x4e/0x100
[  522.194078]  __submit_bio+0xb0/0x170
[  522.194087]  __submit_bio_noacct+0x90/0x220
[  522.194097]  ext4_mpage_readpages+0x2ba/0x7b0
[  522.194110]  read_pages+0x85/0x260
[  522.194121]  page_cache_ra_unbounded+0x12d/0x180
[  522.194135]  filemap_get_pages+0xfc/0x350
[  522.194146]  filemap_read+0xd9/0x350
[  522.194155]  ? generic_fillattr+0x49/0x100
[  522.194168]  vfs_read+0x1fe/0x350
[  522.194181]  ksys_read+0x6f/0xf0
[  522.194191]  do_syscall_64+0x5c/0x90
[  522.194202]  ? do_syscall_64+0x6b/0x90
[  522.194212]  ? do_syscall_64+0x6b/0x90
[  522.194223]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[  522.194234] RIP: 0033:0x7612dae96281
[  522.194242] RSP: 002b:00007ffd04324f88 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[  522.194253] RAX: ffffffffffffffda RBX: 000061446b3502c0 RCX: 00007612dae96281
[  522.194264] RDX: 0000000000001000 RSI: 000061446b35d190 RDI: 0000000000000006
[  522.194275] RBP: 00007612daf6b600 R08: 0000000000001000 R09: 0000000000000000
[  522.194286] R10: 0000000000000000 R11: 0000000000000246 R12: 000061446b351430
[  522.194297] R13: 0000000000000d68 R14: 00007612daf6aa00 R15: 0000000000000d68
[  522.194309]  </TASK>
[  522.194318] task:chronyd         state:D stack:0     pid:4065  ppid:1      flags:0x00000002
[  522.194331] Call Trace:
[  522.194335]  <TASK>
[  522.194341]  __schedule+0x23d/0x670
[  522.194352]  schedule+0x5e/0xd0
[  522.194363]  schedule_preempt_disabled+0x15/0x30
[  522.194375]  rwsem_down_read_slowpath+0x29e/0x4f0
[  522.194388]  down_read+0x48/0xa0
[  522.194397]  dm_thin_find_block+0x32/0x80 [dm_thin_pool]
[  522.194417]  thin_bio_map.isra.0+0x15d/0x380 [dm_thin_pool]
[  522.194438]  __map_bio+0x44/0x1b0
[  522.194447]  dm_split_and_process_bio+0x1fa/0x510
[  522.194458]  dm_submit_bio+0x4e/0x100
[  522.194466]  __submit_bio+0xb0/0x170
[  522.194475]  __submit_bio_noacct+0x90/0x220
[  522.194485]  ext4_mpage_readpages+0x2ba/0x7b0
[  522.194497]  read_pages+0x85/0x260
[  522.194508]  page_cache_ra_unbounded+0x12d/0x180
[  522.194522]  filemap_get_pages+0xfc/0x350
[  522.194532]  filemap_read+0xd9/0x350
[  522.194541]  ? dput+0x3a/0x310
[  522.194552]  vfs_read+0x1fe/0x350
[  522.194561]  ksys_read+0x6f/0xf0
[  522.194570]  do_syscall_64+0x5c/0x90
[  522.194580]  ? switch_fpu_return+0x50/0xe0
[  522.194591]  ? exit_to_user_mode_prepare+0xb2/0xd0
[  522.194601]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[  522.194612] RIP: 0033:0x7545397231b8
[  522.194619] RSP: 002b:000075453922b238 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[  522.194631] RAX: ffffffffffffffda RBX: 0000754534000ba0 RCX: 00007545397231b8
[  522.194642] RDX: 0000000000001000 RSI: 0000754534003660 RDI: 0000000000000009
[  522.194652] RBP: 00007545397f3600 R08: 0000000000000001 R09: 0000000000000000
[  522.194663] R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000000a
[  522.194673] R13: 0000000000000d68 R14: 00007545397f2a00 R15: 0000000000000d68
[  522.194686]  </TASK>
[  522.194691] task:libvirtd        state:D stack:0     pid:2708  ppid:1      flags:0x00000002
[  522.194704] Call Trace:
[  522.194709]  <TASK>
[  522.194714]  __schedule+0x23d/0x670
[  522.194727]  schedule+0x5e/0xd0
[  522.194737]  schedule_preempt_disabled+0x15/0x30
[  522.194749]  rwsem_down_read_slowpath+0x29e/0x4f0
[  522.194760]  down_read+0x48/0xa0
[  522.194768]  dm_thin_find_block+0x32/0x80 [dm_thin_pool]
[  522.194787]  thin_bio_map.isra.0+0x15d/0x380 [dm_thin_pool]
[  522.194806]  __map_bio+0x44/0x1b0
[  522.194814]  dm_split_and_process_bio+0x1fa/0x510
[  522.194824]  dm_submit_bio+0x4e/0x100
[  522.194832]  __submit_bio+0xb0/0x170
[  522.194840]  __submit_bio_noacct+0x90/0x220
[  522.194849]  ext4_mpage_readpages+0x2ba/0x7b0
[  522.194861]  read_pages+0x85/0x260
[  522.194871]  page_cache_ra_unbounded+0x12d/0x180
[  522.194884]  filemap_get_pages+0xfc/0x350
[  522.194894]  filemap_read+0xd9/0x350
[  522.194902]  ? dput+0x3a/0x310
[  522.194912]  vfs_read+0x1fe/0x350
[  522.194921]  ksys_read+0x6f/0xf0
[  522.194929]  do_syscall_64+0x5c/0x90
[  522.194939]  ? count_memcg_events.constprop.0+0x24/0x30
[  522.194951]  ? handle_mm_fault+0x9e/0x350
[  522.194959]  ? do_user_addr_fault+0x179/0x640
[  522.194970]  ? exc_page_fault+0x7c/0x170
[  522.194978]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[  522.194989] RIP: 0033:0x70c45f5331b8
[  522.195005] RSP: 002b:00007ffcb9677588 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[  522.195017] RAX: ffffffffffffffda RBX: 00006526d8896000 RCX: 000070c45f5331b8
[  522.195027] RDX: 0000000000001000 RSI: 00006526d88a3710 RDI: 0000000000000016
[  522.195037] RBP: 000070c45f603600 R08: 0000000000000001 R09: 0000000000000000
[  522.195047] R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000000a
[  522.195057] R13: 0000000000000d68 R14: 000070c45f602a00 R15: 0000000000000d68
[  522.195069]  </TASK>
[  522.195080] task:kworker/u12:7   state:D stack:0     pid:2731  ppid:2      flags:0x00004000
[  522.195093] Workqueue: dm-thin do_worker [dm_thin_pool]
[  522.195113] Call Trace:
[  522.195118]  <TASK>
[  522.195123]  __schedule+0x23d/0x670
[  522.195134]  schedule+0x5e/0xd0
[  522.195144]  io_schedule+0x46/0x70
[  522.195154]  bit_wait_io+0x11/0x70
[  522.195162]  __wait_on_bit+0x4c/0x150
[  522.195169]  ? __pfx_bit_wait_io+0x10/0x10
[  522.195177]  out_of_line_wait_on_bit+0x95/0xc0
[  522.195187]  ? __pfx_wake_bit_function+0x10/0x10
[  522.195199]  dm_bufio_write_dirty_buffers+0x21f/0x2a0
[  522.195212]  ? metadata_ll_commit+0xbb/0xe0 [dm_persistent_data]
[  522.195244]  __commit_transaction+0x18f/0x280 [dm_thin_pool]
[  522.195265]  ? __slab_free+0xf1/0x320
[  522.195276]  dm_pool_commit_metadata+0x30/0x60 [dm_thin_pool]
[  522.195298]  commit+0x31/0x120 [dm_thin_pool]
[  522.195317]  process_deferred_bios+0x2b2/0x2e0 [dm_thin_pool]
[  522.195337]  do_worker+0xfb/0x170 [dm_thin_pool]
[  522.195356]  process_one_work+0x1df/0x3e0
[  522.195365]  worker_thread+0x51/0x390
[  522.195373]  ? __pfx_worker_thread+0x10/0x10
[  522.195382]  kthread+0xe5/0x120
[  522.195391]  ? __pfx_kthread+0x10/0x10
[  522.195400]  ret_from_fork+0x31/0x50
[  522.195407]  ? __pfx_kthread+0x10/0x10
[  522.195416]  ret_from_fork_asm+0x1b/0x30
[  522.195428]  </TASK>
[  522.195478] task:rpm2archive     state:D stack:0     pid:4040  ppid:4039   flags:0x00000002
[  522.195490] Call Trace:
[  522.195495]  <TASK>
[  522.195500]  __schedule+0x23d/0x670
[  522.195511]  schedule+0x5e/0xd0
[  522.195523]  schedule_preempt_disabled+0x15/0x30
[  522.195534]  rwsem_down_read_slowpath+0x29e/0x4f0
[  522.195546]  down_read+0x48/0xa0
[  522.195554]  dm_thin_find_block+0x32/0x80 [dm_thin_pool]
[  522.195573]  thin_bio_map.isra.0+0x15d/0x380 [dm_thin_pool]
[  522.195593]  __map_bio+0x44/0x1b0
[  522.195601]  dm_split_and_process_bio+0x1fa/0x510
[  522.195611]  dm_submit_bio+0x4e/0x100
[  522.195619]  __submit_bio+0xb0/0x170
[  522.195627]  __submit_bio_noacct+0x90/0x220
[  522.195636]  ext4_mpage_readpages+0x2ba/0x7b0
[  522.195648]  read_pages+0x85/0x260
[  522.195658]  page_cache_ra_unbounded+0x12d/0x180
[  522.195670]  filemap_get_pages+0x2cf/0x350
[  522.195680]  filemap_read+0xd9/0x350
[  522.195688]  ? perf_event_task_tick+0x56/0xa0
[  522.195700]  vfs_read+0x1fe/0x350
[  522.195709]  ksys_read+0x6f/0xf0
[  522.195716]  do_syscall_64+0x5c/0x90
[  522.195727]  ? handle_percpu_irq+0x3b/0x60
[  522.195737]  ? handle_irq_desc+0x3b/0x50
[  522.195746]  ? __evtchn_fifo_handle_events+0x1b4/0x1e0
[  522.195760]  ? xen_sched_clock+0x15/0x30
[  522.195770]  ? sched_clock+0x10/0x30
[  522.195780]  ? sched_clock_cpu+0xf/0x190
[  522.195790]  ? irqtime_account_irq+0x40/0xc0
[  522.195800]  ? __irq_exit_rcu+0x4b/0xc0
[  522.195810]  ? xen_pv_evtchn_do_upcall+0x54/0xb0
[  522.195823]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[  522.195833] RIP: 0033:0x750fb0b38281
[  522.195839] RSP: 002b:00007ffc465a6e38 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[  522.195849] RAX: ffffffffffffffda RBX: 000060cea227b5d0 RCX: 0000750fb0b38281
[  522.195858] RDX: 000000000001f000 RSI: 000060cea22be0f7 RDI: 0000000000000003
[  522.195868] RBP: 0000750fb0c0d600 R08: 0000000000000000 R09: 0000000000000172
[  522.195877] R10: 00007ffc465a6ee0 R11: 0000000000000246 R12: 000060cea22be0f7
[  522.195887] R13: 000000000001f8cc R14: 0000000000000d68 R15: 0000750fb0c0ca00
[  522.195898]  </TASK>
[  522.195904] task:kworker/u12:9   state:D stack:0     pid:4045  ppid:2      flags:0x00004000
[  522.195915] Workqueue: writeback wb_workfn (flush-253:4)
[  522.195928] Call Trace:
[  522.195932]  <TASK>
[  522.195937]  __schedule+0x23d/0x670
[  522.195947]  schedule+0x5e/0xd0
[  522.195956]  schedule_preempt_disabled+0x15/0x30
[  522.195967]  rwsem_down_read_slowpath+0x29e/0x4f0
[  522.195977]  down_read+0x48/0xa0
[  522.195985]  dm_thin_find_block+0x32/0x80 [dm_thin_pool]
[  522.196013]  thin_bio_map.isra.0+0x15d/0x380 [dm_thin_pool]
[  522.196032]  __map_bio+0x44/0x1b0
[  522.196039]  dm_split_and_process_bio+0x1fa/0x510
[  522.196048]  dm_submit_bio+0x4e/0x100
[  522.196057]  __submit_bio+0xb0/0x170
[  522.196066]  __submit_bio_noacct+0x90/0x220
[  522.196074]  ext4_io_submit+0x24/0x40
[  522.196081]  ext4_do_writepages+0x3af/0x770
[  522.196091]  ext4_writepages+0xad/0x180
[  522.196101]  do_writepages+0xcf/0x1e0
[  522.196110]  __writeback_single_inode+0x3d/0x280
[  522.196121]  writeback_sb_inodes+0x1ed/0x4a0
[  522.196133]  __writeback_inodes_wb+0x4c/0xf0
[  522.196144]  wb_writeback+0x298/0x310
[  522.196154]  wb_do_writeback+0x230/0x2b0
[  522.196164]  wb_workfn+0x5f/0x260
[  522.196173]  ? _raw_spin_unlock+0xe/0x30
[  522.196180]  ? finish_task_switch.isra.0+0x95/0x2b0
[  522.196191]  ? __schedule+0x245/0x670
[  522.196200]  process_one_work+0x1df/0x3e0
[  522.196208]  worker_thread+0x51/0x390
[  522.196216]  ? __pfx_worker_thread+0x10/0x10
[  522.196224]  kthread+0xe5/0x120
[  522.196232]  ? __pfx_kthread+0x10/0x10
[  522.196241]  ret_from_fork+0x31/0x50
[  522.196247]  ? __pfx_kthread+0x10/0x10
[  522.196255]  ret_from_fork_asm+0x1b/0x30
[  522.196266]  </TASK>
[  522.196271] task:(tmpfiles)      state:D stack:0     pid:4059  ppid:3425   flags:0x00000002
[  522.196283] Call Trace:
[  522.196287]  <TASK>
[  522.196292]  __schedule+0x23d/0x670
[  522.196302]  schedule+0x5e/0xd0
[  522.196311]  schedule_preempt_disabled+0x15/0x30
[  522.196322]  rwsem_down_read_slowpath+0x29e/0x4f0
[  522.196332]  down_read+0x48/0xa0
[  522.196340]  dm_thin_find_block+0x32/0x80 [dm_thin_pool]
[  522.196357]  thin_bio_map.isra.0+0x15d/0x380 [dm_thin_pool]
[  522.196375]  __map_bio+0x44/0x1b0
[  522.196383]  dm_split_and_process_bio+0x1fa/0x510
[  522.196392]  dm_submit_bio+0x4e/0x100
[  522.196400]  __submit_bio+0xb0/0x170
[  522.196407]  __submit_bio_noacct+0x90/0x220
[  522.196415]  ext4_mpage_readpages+0x2ba/0x7b0
[  522.196426]  read_pages+0x85/0x260
[  522.196436]  page_cache_ra_unbounded+0x12d/0x180
[  522.196447]  filemap_get_pages+0xfc/0x350
[  522.196456]  filemap_read+0xd9/0x350
[  522.196465]  __kernel_read+0x134/0x2e0
[  522.196474]  search_binary_handler+0x71/0x260
[  522.196485]  exec_binprm+0x51/0x1a0
[  522.196495]  bprm_execve.part.0+0x180/0x220
[  522.196505]  do_execveat_common.isra.0+0x1ad/0x250
[  522.196516]  __x64_sys_execve+0x36/0x40
[  522.196524]  do_syscall_64+0x5c/0x90
[  522.196534]  ? __count_memcg_events+0x4d/0x90
[  522.196543]  ? count_memcg_events.constprop.0+0x24/0x30
[  522.196553]  ? handle_mm_fault+0x9e/0x350
[  522.196561]  ? do_user_addr_fault+0x225/0x640
[  522.196569]  ? syscall_exit_work+0x103/0x130
[  522.196578]  ? exc_page_fault+0x7c/0x170
[  522.196586]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[  522.196595] RIP: 0033:0x7f2bd24fa4db
[  522.196602] RSP: 002b:00007fff1c6a1328 EFLAGS: 00000206 ORIG_RAX: 000000000000003b
[  522.196613] RAX: ffffffffffffffda RBX: 00005d221a754700 RCX: 00007f2bd24fa4db
[  522.196623] RDX: 00005d221a7a3610 RSI: 00005d221a754700 RDI: 00005d221a8baad0
[  522.196632] RBP: 00007fff1c6a15f0 R08: 0000000000000007 R09: 0000000000000030
[  522.196641] R10: 0000000000000011 R11: 0000000000000206 R12: 00005d221a791ad0
[  522.196651] R13: 00000000ffffffff R14: 0000000000000000 R15: 00005d221a792038
[  522.196662]  </TASK>
[  522.196667] task:lvm             state:D stack:0     pid:4062  ppid:2673   flags:0x00000002
[  522.196678] Call Trace:
[  522.196682]  <TASK>
[  522.196687]  __schedule+0x23d/0x670
[  522.196697]  schedule+0x5e/0xd0
[  522.196706]  schedule_preempt_disabled+0x15/0x30
[  522.196717]  rwsem_down_read_slowpath+0x29e/0x4f0
[  522.196727]  down_read+0x48/0xa0
[  522.196735]  dm_pool_get_metadata_transaction_id+0x22/0x60 [dm_thin_pool]
[  522.196755]  pool_status+0x122/0x600 [dm_thin_pool]
[  522.196773]  retrieve_status+0x15d/0x210
[  522.196783]  table_status+0x9b/0x140
[  522.196792]  ctl_ioctl+0x26e/0x350
[  522.196800]  ? __pfx_table_status+0x10/0x10
[  522.196813]  dm_ctl_ioctl+0xe/0x20
[  522.196821]  __x64_sys_ioctl+0x94/0xd0
[  522.196829]  do_syscall_64+0x5c/0x90
[  522.196838]  ? syscall_exit_work+0x103/0x130
[  522.196846]  ? syscall_exit_to_user_mode+0x2b/0x40
[  522.196855]  ? do_syscall_64+0x6b/0x90
[  522.196863]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[  522.196873] RIP: 0033:0x7eea6b7c4e0f
[  522.196878] RSP: 002b:00007ffe60c65e60 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[  522.196888] RAX: ffffffffffffffda RBX: 00005644067b2c20 RCX: 00007eea6b7c4e0f
[  522.196896] RDX: 0000564406ffd0e0 RSI: 00000000c138fd0c RDI: 0000000000000004
[  522.196905] RBP: 0000564406ffd0e0 R08: 0000564404e71190 R09: 00007ffe60c65d20
[  522.196914] R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000001e
[  522.196923] R13: 0000564404e6acf0 R14: 0000564404e6acf0 R15: 0000564406ffd110
[  522.196936]  </TASK>

@andrewdavidwong andrewdavidwong added the needs diagnosis Requires technical diagnosis from developer. Replace with "diagnosed" or remove if otherwise closed. label Oct 13, 2023
@marmarek marmarek mentioned this issue Oct 17, 2023
Closed
@marmarek
Copy link
Member Author

At one time I've got the issue without lvm among blocked processes

Blocked processes 
[  299.388109] sysrq: Show Blocked State
[  299.391268] task:rpm2archive     state:D stack:0     pid:3050  ppid:3049   flags:0x00000002
[  299.392837] Call Trace:
[  299.393430]  <TASK>
[  299.393880]  __schedule+0x526/0x18b0
[  299.394338]  ? folio_wait_bit_common+0xd1/0x360
[  299.394793]  schedule+0x5e/0xd0
[  299.395247]  io_schedule+0x46/0x70
[  299.395693]  folio_wait_bit_common+0x13d/0x360
[  299.396174]  ? __pfx_wake_page_function+0x10/0x10
[  299.396717]  filemap_get_pages+0x60b/0x640
[  299.397247]  ? lock_release+0xd1/0x280
[  299.397686]  filemap_read+0xd9/0x350
[  299.398127]  vfs_read+0x1f3/0x320
[  299.398547]  ksys_read+0x75/0x100
[  299.399007]  do_syscall_64+0x5c/0x90
[  299.399504]  ? irqtime_account_irq+0x40/0xc0
[  299.399996]  ? __irq_exit_rcu+0x59/0x100
[  299.400507]  ? irq_exit_rcu+0xe/0x20
[  299.400998]  ? lockdep_hardirqs_on+0x51/0x100
[  299.401511]  entry_SYSCALL_64_after_hwframe+0x72/0xdc
[  299.402011] RIP: 0033:0x7b8f18bf9281
[  299.402521] RSP: 002b:00007ffebf0070d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[  299.403031] RAX: ffffffffffffffda RBX: 000059320f2ca5d0 RCX: 00007b8f18bf9281
[  299.403561] RDX: 000000000001f000 RSI: 000059320f30d35f RDI: 0000000000000003
[  299.404102] RBP: 00007b8f18cce600 R08: 000059320f30c9c0 R09: 00000000000027a9
[  299.404575] R10: 00007ffebf007180 R11: 0000000000000246 R12: 000059320f30d35f
[  299.405006] R13: 000000000001f664 R14: 0000000000000d68 R15: 00007b8f18ccda00
[  299.405459]  </TASK>

@marmarek
Copy link
Member Author

Upstream report: https://lore.kernel.org/all/ZTNH0qtmint%2FzLJZ@mail-itl/

marmarek added a commit to QubesOS/qubes-linux-kernel that referenced this issue Oct 21, 2023
@marmarek
Revert commit causing storage freeze
83ff48a 
Temporary workaround until real fix is found.

QubesOS/qubes-issues#8575
@andrewdavidwong andrewdavidwong added the waiting for upstream This issue is waiting for something from an upstream project to arrive in Qubes. Remove when closed. label Oct 21, 2023
This was referenced Oct 22, 2023
Closed
Closed
marmarek added a commit to marmarek/qubes-linux-kernel that referenced this issue Oct 28, 2023
@marmarek
Replace revert with upstream patch regarding dm-crypt bug
c689047 
Fixes QubesOS/qubes-issues#8575
This was referenced Oct 28, 2023
Merged
Closed
@andrewdavidwong andrewdavidwong added diagnosed Technical diagnosis has been performed (see issue comments). and removed needs diagnosis Requires technical diagnosis from developer. Replace with "diagnosed" or remove if otherwise closed. waiting for upstream This issue is waiting for something from an upstream project to arrive in Qubes. Remove when closed. labels Nov 16, 2023
@andrewdavidwong andrewdavidwong added this to the Release 4.2 milestone Nov 16, 2023
@qubesos-bot qubesos-bot mentioned this issue Nov 28, 2023
Closed
@qubesos-bot
Copy link

Automated announcement from builder-github

The component linux-kernel-latest (including package kernel-latest-6.6.2-1.qubes.fc32) has been pushed to the r4.1 testing repository for dom0.
To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

@qubesos-bot qubesos-bot mentioned this issue Nov 28, 2023
Closed
@qubesos-bot
Copy link

Automated announcement from builder-github

The component linux-kernel-latest (including package kernel-latest-6.6.2-1.qubes.fc32) has been pushed to the r4.1 stable repository for dom0.
To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-4.2 This issue affects Qubes OS 4.2. C: kernel diagnosed Technical diagnosis has been performed (see issue comments). P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. r4.1-dom0-stable r4.2-host-stable
Projects
None yet
Milestone
Release 4.2
Development

No branches or pull requests
3 participants
@marmarek @andrewdavidwong @qubesos-bot