Fix deadlock on block cache. #6412

jimmygchen · 2024-09-18T22:43:37Z

Issue Addressed

I ran into another deadlock scenario again today during testing.

On thread A, read lock (1) for block_cache is acquired here:

lighthouse/beacon_node/beacon_chain/src/eth1_chain.rs

Line 478 in 99e53b8

let blocks = self.core.blocks().read();

lighthouse/beacon_node/eth1/src/service.rs

Lines 477 to 479 in 9b3b730

    
           pub fn blocks(&self) -> &RwLock<BlockCache> { 
        
               &self.inner.block_cache 
        
           }

On thread B, a write lock (2) for block_cache is acquired in prune_blocks, so it waits for (1) to release the lock

lighthouse/beacon_node/eth1/src/inner.rs

Line 63 in c824142

self.block_cache.write().truncate(block_cache_truncation);

On thread A, a read lock (3) for block_cache is acquired again, now this is waiting on (2), but (1) may not have release the lock before this, so we're in a deadlock.

lighthouse/beacon_node/beacon_chain/src/eth1_chain.rs

Line 516 in 99e53b8

"earliest_block_timestamp" => self.core.earliest_block_timestamp(),

lighthouse/beacon_node/eth1/src/service.rs

Lines 499 to 501 in 9b3b730

    
           pub fn earliest_block_timestamp(&self) -> Option<u64> { 
        
               self.inner.block_cache.read().earliest_block_timestamp() 
        
           }

Strangely it's on code that hasn't been touched for ages, maybe it's more easily reproducible when resources are very constrained (I'm running a kurtosis devnet).

Proposed Changes

Drop the block_cache at (1) after using, and before it gets acquired again.

jimmygchen · 2024-09-19T00:23:57Z

Thread dump after node is stuck for future reference:

Thread 1:
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000055e8287f3cb2 in std::sys::sync::condvar::futex::Condvar::wait ()
#2  0x000055e8286c9169 in _ZN5tokio7runtime4park5Inner4park17hb2690810ffa414cdE.llvm.16672836764291932862 ()
#3  0x000055e826c23397 in tokio::runtime::park::CachedParkThread::block_on ()
#4  0x000055e825e53b80 in tokio::runtime::runtime::Runtime::block_on ()
#5  0x000055e825bf350b in environment::Environment<E>::block_until_shutdown_requested ()
#6  0x000055e826e4663c in lighthouse::main::{{closure}} ()
#7  0x000055e826e431e6 in lighthouse::main ()
#8  0x000055e825ec0373 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#9  0x000055e826e77f79 in _ZN3std2rt10lang_start28_$u7b$$u7b$closure$u7d$$u7d$17h12e93a8472bd2878E.llvm.2173934129280139846 ()
#10 0x000055e8287ddacd in std::rt::lang_start_internal ()
#11 0x000055e826e474a5 in main ()
[Switching to thread 2 (Thread 0x7fbc6ffff640 (LWP 7))]
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7fbc70616630)
    at ./nptl/futex-internal.c:57
57	./nptl/futex-internal.c: No such file or directory.

Thread 2:
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7fbc70616630)
    at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7fbc70616630)
    at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7fbc70616630, expected=expected@entry=0,
    clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007fbc708dda41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7fbc70616678, cond=0x7fbc70616608)
    at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7fbc70616608, mutex=0x7fbc70616678) at ./nptl/pthread_cond_wait.c:627
#5  0x000055e82735a250 in background_thread_entry ()
#6  0x00007fbc708deac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#7  0x00007fbc7096fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
[Switching to thread 3 (Thread 0x7fbc6f07e640 (LWP 8))]
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
38	../sysdeps/unix/sysv/linux/x86_64/syscall.S: No such file or directory.

Thread 3:
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000055e8287de293 in std::thread::park ()
#2  0x000055e826563386 in crossbeam_channel::context::Context::wait_until ()
#3  0x000055e826565a30 in crossbeam_channel::flavors::array::Channel<T>::recv::{{closure}} ()
#4  0x000055e82656547a in crossbeam_channel::flavors::array::Channel<T>::recv ()
#5  0x000055e826ace3e4 in crossbeam_channel::channel::Receiver<T>::recv ()
#6  0x000055e82687c4fb in std::panicking::try ()
#7  0x000055e825ec03a9 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#8  0x000055e825ec8622 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#9  0x000055e8287f1d4b in std::sys::pal::unix::thread::Thread::new::thread_start ()
#10 0x00007fbc708deac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#11 0x00007fbc7096fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
[Switching to thread 4 (Thread 0x7fbc6e4ff640 (LWP 9))]
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7fbc70616704)
    at ./nptl/futex-internal.c:57
57	./nptl/futex-internal.c: No such file or directory.

Thread 4:
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7fbc70616704)
    at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7fbc70616704) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7fbc70616704, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007fbc708dda41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7fbc70616748, cond=0x7fbc706166d8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7fbc706166d8, mutex=0x7fbc70616748) at ./nptl/pthread_cond_wait.c:627
#5  0x000055e827359ac9 in background_thread_entry ()
#6  0x00007fbc708deac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#7  0x00007fbc7096fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
[Switching to thread 5 (Thread 0x7fbc6dcfe640 (LWP 10))]
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
38	../sysdeps/unix/sysv/linux/x86_64/syscall.S: No such file or directory.

Thread 5:
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000055e8287de293 in std::thread::park ()
#2  0x000055e827796346 in crossbeam_channel::context::Context::wait_until ()
#3  0x000055e82779890f in crossbeam_channel::flavors::array::Channel<T>::recv::{{closure}} ()
#4  0x000055e827798326 in crossbeam_channel::flavors::array::Channel<T>::recv ()
#5  0x000055e8277a1564 in crossbeam_channel::channel::Receiver<T>::recv ()
#6  0x000055e8277b1bab in std::panicking::try ()
#7  0x000055e8277a2449 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#8  0x000055e82778ed2b in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#9  0x000055e8287f1d4b in std::sys::pal::unix::thread::Thread::new::thread_start ()
#10 0x00007fbc708deac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#11 0x00007fbc7096fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
[Switching to thread 6 (Thread 0x7fbc6d5ff640 (LWP 11))]
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7fbc706167d4) at ./nptl/futex-internal.c:57
57	./nptl/futex-internal.c: No such file or directory.

Thread 6:
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7fbc706167d4) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7fbc706167d4) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7fbc706167d4, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007fbc708dda41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7fbc70616818, cond=0x7fbc706167a8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7fbc706167a8, mutex=0x7fbc70616818) at ./nptl/pthread_cond_wait.c:627
#5  0x000055e827359ac9 in background_thread_entry ()
#6  0x00007fbc708deac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#7  0x00007fbc7096fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
[Switching to thread 7 (Thread 0x7fbc6cdfe640 (LWP 12))]
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
38	../sysdeps/unix/sysv/linux/x86_64/syscall.S: No such file or directory.

Thread 7:
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000055e82850aa2d in parking_lot::raw_rwlock::RawRwLock::wait_for_readers ()
#2  0x000055e825bb8129 in parking_lot::raw_rwlock::RawRwLock::lock_exclusive_slow ()
#3  0x000055e82746d5c1 in eth1::inner::Inner::prune_blocks ()
#4  0x000055e827495b87 in <futures_util::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll ()
#5  0x000055e82748312b in <futures_util::future::poll_fn::PollFn<F> as core::future::future::Future>::poll ()
#6  0x000055e8274a22b5 in eth1::service::Service::do_update::{{closure}} ()
#7  0x000055e82749deb1 in futures_util::future::future::FutureExt::poll_unpin ()
#8  0x000055e8274954c6 in <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll ()
#9  0x000055e8274a96b0 in _ZN13task_executor12TaskExecutor12spawn_handle28_$u7b$$u7b$closure$u7d$$u7d$17h777d20d0bb75b0b0E.llvm.17418404132664207095 ()
#10 0x000055e8274a9f01 in std::panicking::try ()
#11 0x000055e82748f598 in tokio::runtime::task::harness::Harness<T,S>::poll ()
#12 0x000055e8286cc47b in tokio::runtime::scheduler::multi_thread::worker::Context::run_task ()
#13 0x000055e8286cb32b in tokio::runtime::scheduler::multi_thread::worker::Context::run ()
#14 0x000055e8286da787 in tokio::runtime::context::scoped::Scoped<T>::set ()
#15 0x000055e8286e033d in tokio::runtime::context::runtime::enter_runtime ()
#16 0x000055e8286cac46 in tokio::runtime::scheduler::multi_thread::worker::run ()
#17 0x000055e8286d3507 in <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll ()
#18 0x000055e8286ce5d3 in tokio::runtime::task::core::Core<T,S>::poll ()
#19 0x000055e8286d2852 in tokio::runtime::task::harness::Harness<T,S>::poll ()
#20 0x000055e8286eb45d in tokio::runtime::blocking::pool::Inner::run ()
#21 0x000055e8286ec36f in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#22 0x000055e8286ecd72 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#23 0x000055e8287f1d4b in std::sys::pal::unix::thread::Thread::new::thread_start ()
#24 0x00007fbc708deac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#25 0x00007fbc7096fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
[Switching to thread 8 (Thread 0x7fbc6c5fd640 (LWP 13))]
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
38	in ../sysdeps/unix/sysv/linux/x86_64/syscall.S

Thread 8:
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000055e8287de293 in std::thread::park ()
#2  0x000055e82776bc26 in crossbeam_channel::context::Context::wait_until ()
#3  0x000055e82776c900 in crossbeam_channel::flavors::array::Channel<T>::recv::{{closure}} ()
#4  0x000055e82776c2f4 in crossbeam_channel::flavors::array::Channel<T>::recv ()
#5  0x000055e82775e1dd in crossbeam_channel::channel::Receiver<T>::recv ()
#6  0x000055e82777ab83 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#7  0x000055e82777c72b in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#8  0x000055e8287f1d4b in std::sys::pal::unix::thread::Thread::new::thread_start ()
#9  0x00007fbc708deac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007fbc7096fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
[Switching to thread 9 (Thread 0x7fbc5f5ff640 (LWP 14))]
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7fbc706168a0) at ./nptl/futex-internal.c:57
57	./nptl/futex-internal.c: No such file or directory.

Thread 9:
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7fbc706168a0) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7fbc706168a0) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7fbc706168a0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007fbc708dda41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7fbc706168e8, cond=0x7fbc70616878) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7fbc70616878, mutex=0x7fbc706168e8) at ./nptl/pthread_cond_wait.c:627
#5  0x000055e827359ac9 in background_thread_entry ()
#6  0x00007fbc708deac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#7  0x00007fbc7096fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
[Switching to thread 10 (Thread 0x7fbc5edfe640 (LWP 15))]
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
38	../sysdeps/unix/sysv/linux/x86_64/syscall.S: No such file or directory.

Thread 10:
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000055e8287de293 in std::thread::park ()
#2  0x000055e82776bc26 in crossbeam_channel::context::Context::wait_until ()
#3  0x000055e82776c900 in crossbeam_channel::flavors::array::Channel<T>::recv::{{closure}} ()
#4  0x000055e82776c2f4 in crossbeam_channel::flavors::array::Channel<T>::recv ()
#5  0x000055e82775e1dd in crossbeam_channel::channel::Receiver<T>::recv ()
#6  0x000055e82777ab83 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#7  0x000055e82777c72b in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#8  0x000055e8287f1d4b in std::sys::pal::unix::thread::Thread::new::thread_start ()
#9  0x00007fbc708deac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007fbc7096fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
[Switching to thread 11 (Thread 0x7fbc5cfff640 (LWP 16))]
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
38	in ../sysdeps/unix/sysv/linux/x86_64/syscall.S

Thread 11:
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000055e8287f3cb2 in std::sys::sync::condvar::futex::Condvar::wait ()
#2  0x000055e825bbdc39 in rayon_core::sleep::Sleep::sleep ()
#3  0x000055e825bbd601 in rayon_core::registry::WorkerThread::wait_until_cold ()
#4  0x000055e82852baf2 in rayon_core::registry::ThreadBuilder::run ()
#5  0x000055e82852f7ea in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#6  0x000055e828530b62 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#7  0x000055e8287f1d4b in std::sys::pal::unix::thread::Thread::new::thread_start ()
#8  0x00007fbc708deac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#9  0x00007fbc7096fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
[Switching to thread 12 (Thread 0x7fbc450ff640 (LWP 17))]
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
38	in ../sysdeps/unix/sysv/linux/x86_64/syscall.S

Thread 12:
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000055e8287de293 in std::thread::park ()
#2  0x000055e82657a022 in std::sync::mpmc::list::Channel<T>::recv::{{closure}} ()
#3  0x000055e826579e2c in std::sync::mpmc::list::Channel<T>::recv ()
#4  0x000055e825ebf6ee in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#5  0x000055e825ec882e in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#6  0x000055e8287f1d4b in std::sys::pal::unix::thread::Thread::new::thread_start ()
#7  0x00007fbc708deac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8  0x00007fbc7096fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
[Switching to thread 13 (Thread 0x7fbc44afd640 (LWP 19))]
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
38	in ../sysdeps/unix/sysv/linux/x86_64/syscall.S

Thread 13:
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000055e8287de293 in std::thread::park ()
#2  0x000055e827d0e636 in futures_timer::native::global::run ()
#3  0x000055e827d0e872 in _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h02f1516c3a4f636dE.llvm.14523895506203702223 ()
#4  0x000055e827d0fa57 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#5  0x000055e8287f1d4b in std::sys::pal::unix::thread::Thread::new::thread_start ()
#6  0x00007fbc708deac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#7  0x00007fbc7096fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
[Switching to thread 14 (Thread 0x7fbc371ff640 (LWP 21))]
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
38	in ../sysdeps/unix/sysv/linux/x86_64/syscall.S

Thread 14:
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000055e8287de293 in std::thread::park ()
#2  0x000055e828589f22 in std::sync::mpmc::list::Channel<T>::recv::{{closure}} ()
#3  0x000055e8285899a9 in std::sync::mpmc::list::Channel<T>::recv ()
#4  0x000055e82858ce4f in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#5  0x000055e82858dec3 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#6  0x000055e8287f1d4b in std::sys::pal::unix::thread::Thread::new::thread_start ()
#7  0x00007fbc708deac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8  0x00007fbc7096fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
[Switching to thread 15 (Thread 0x7fbc369ff640 (LWP 22))]
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
38	in ../sysdeps/unix/sysv/linux/x86_64/syscall.S

Thread 15:
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000055e825bb8a0d in parking_lot::raw_rwlock::RawRwLock::lock_shared_slow ()
#2  0x000055e827467709 in eth1::service::Service::earliest_block_timestamp ()
#3  0x000055e826c96574 in beacon_chain::eth1_chain::Eth1Chain<T,E>::eth1_data_for_block_production ()
#4  0x000055e826910636 in <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll ()
#5  0x000055e826809b11 in tokio::runtime::task::core::Core<T,S>::poll ()
#6  0x000055e826094d5a in tokio::runtime::task::harness::Harness<T,S>::poll ()
#7  0x000055e8286eb45d in tokio::runtime::blocking::pool::Inner::run ()
#8  0x000055e8286ec36f in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#9  0x000055e8286ecd72 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#10 0x000055e8287f1d4b in std::sys::pal::unix::thread::Thread::new::thread_start ()
#11 0x00007fbc708deac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#12 0x00007fbc7096fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
[Switching to thread 16 (Thread 0x7fbc1d429640 (LWP 26))]
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55e82a58b93c <leveldb::Env::Default()::env_container+92>) at ./nptl/futex-internal.c:57
57	./nptl/futex-internal.c: No such file or directory.

Thread 16:
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55e82a58b93c <leveldb::Env::Default()::env_container+92>) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x55e82a58b93c <leveldb::Env::Default()::env_container+92>) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55e82a58b93c <leveldb::Env::Default()::env_container+92>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007fbc708dda41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55e82a58b8e8 <leveldb::Env::Default()::env_container+8>, cond=0x55e82a58b910 <leveldb::Env::Default()::env_container+48>) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x55e82a58b910 <leveldb::Env::Default()::env_container+48>, mutex=0x55e82a58b8e8 <leveldb::Env::Default()::env_container+8>) at ./nptl/pthread_cond_wait.c:627
#5  0x00007fbc70c32747 in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x000055e827d41338 in leveldb::(anonymous namespace)::PosixEnv::BackgroundThreadEntryPoint(leveldb::(anonymous namespace)::PosixEnv*) ()
#7  0x00007fbc70c62253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007fbc708deac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#9  0x00007fbc7096fa04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

michaelsproul

LGTM, nice find.

I would love to get the deadlock detector running again

michaelsproul · 2024-09-19T02:00:59Z

Looks like lockbud has had some recent updates and might be usable again:

Get lockbud running again #6413

michaelsproul · 2024-09-19T02:01:20Z

@mergify queue

mergify · 2024-09-19T02:01:45Z

queue

🛑 The pull request has been removed from the queue `default`

The merge conditions cannot be satisfied due to failing checks.

You can take a look at Queue: Embarked in merge queue check runs for more details.

In case of a failure due to a flaky test, you should first retrigger the CI.
Then, re-embark the pull request into the merge queue by posting the comment
@mergifyio refresh on the pull request.

michaelsproul · 2024-09-19T02:51:23Z

CI is failing because the Windows runner was updated to 1.81 and we need:

Fix test failure on Rust v1.81 #6407

jimmygchen · 2024-09-19T03:55:43Z

@mergify requeue

mergify · 2024-09-19T03:55:51Z

requeue

✅ This pull request will be re-embarked automatically

The followup queue command will be automatically executed to re-embark the pull request

mergify · 2024-09-19T03:55:57Z

queue

✅ The pull request has been merged automatically

The pull request has been merged automatically at 46e0d66

Fix deadlock on block cache.

51fd721

jimmygchen added the ready-for-review The code is ready for review label Sep 18, 2024

michaelsproul added bug Something isn't working v6.0.0 New major release for hierarchical state diffs database labels Sep 19, 2024

michaelsproul approved these changes Sep 19, 2024

View reviewed changes

michaelsproul mentioned this pull request Sep 19, 2024

Get lockbud running again #6413

Closed

michaelsproul added ready-for-merge This PR is ready to merge. and removed ready-for-review The code is ready for review labels Sep 19, 2024

mergify bot added a commit that referenced this pull request Sep 19, 2024

Merge of #6412

a682d44

This was referenced Sep 19, 2024

merge queue: embarking unstable (8a085fc) and [#6412 + #6402] together #6415

Closed

Correct ENR decoding on extension trait #6402

Merged

mergify bot added a commit that referenced this pull request Sep 19, 2024

Merge of #6412

90f8032

mergify bot mentioned this pull request Sep 19, 2024

merge queue: embarking unstable (8a085fc) and #6412 together #6416

Closed

5 tasks

mergify bot merged commit 46e0d66 into sigp:unstable Sep 19, 2024
27 of 28 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix deadlock on block cache. #6412

Fix deadlock on block cache. #6412

jimmygchen commented Sep 18, 2024

jimmygchen commented Sep 19, 2024

michaelsproul left a comment

michaelsproul commented Sep 19, 2024

michaelsproul commented Sep 19, 2024

mergify bot commented Sep 19, 2024 •

edited

Loading

michaelsproul commented Sep 19, 2024

jimmygchen commented Sep 19, 2024

mergify bot commented Sep 19, 2024

mergify bot commented Sep 19, 2024 •

edited

Loading

	pub fn blocks(&self) -> &RwLock<BlockCache> {
	&self.inner.block_cache
	}

	pub fn earliest_block_timestamp(&self) -> Option<u64> {
	self.inner.block_cache.read().earliest_block_timestamp()
	}

Fix deadlock on block cache. #6412

Fix deadlock on block cache. #6412

Conversation

jimmygchen commented Sep 18, 2024

Issue Addressed

Proposed Changes

jimmygchen commented Sep 19, 2024

michaelsproul left a comment

Choose a reason for hiding this comment

michaelsproul commented Sep 19, 2024

michaelsproul commented Sep 19, 2024

mergify bot commented Sep 19, 2024 • edited Loading

🛑 The pull request has been removed from the queue default

michaelsproul commented Sep 19, 2024

jimmygchen commented Sep 19, 2024

mergify bot commented Sep 19, 2024

✅ This pull request will be re-embarked automatically

mergify bot commented Sep 19, 2024 • edited Loading

✅ The pull request has been merged automatically

mergify bot commented Sep 19, 2024 •

edited

Loading

🛑 The pull request has been removed from the queue `default`

mergify bot commented Sep 19, 2024 •

edited

Loading