Parachain import fails with Storage root must match that calculated

[5tan-enj]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Description of bug

After updating to polkadot-v1.11.0, unable to use previously synced DB, block import fails with Storage root must match that calculated

Example

2024-09-25 16:31:31 [Parachain] 💔 Error importing block 0x40c56441e955b11304864483ca8e5e91a76512dab29785c819837eca0ae36470: consensus error: Import failed: Error at calling runtime api: Execution failed: Runtime panicked: Storage root must match that calculated.
Hash: given=331bab4406ad2abc186318c9f9ef49df465193f7169ef2f4f95f2070b30e2c28, expected=c04a4c1d318293cce519a81cf2156d881a812a33a7999a26fcae75fe1d9cf92d

====================

Version: 1.1.2-b58c3c2cd1a

   0: backtrace::capture::Backtrace::create
   1: backtrace::capture::Backtrace::new
   2: sp_panic_handler::set::{{closure}}
   3: std::panicking::rust_panic_with_hook
   4: std::panicking::begin_panic_handler::{{closure}}
   5: std::sys_common::backtrace::__rust_end_short_backtrace
   6: _rust_begin_unwind
   7: core::panicking::panic_fmt
   8: frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::final_checks
   9: tracing::span::Span::in_scope
  10: frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::execute_block
  11: canary_matrix_runtime::api::dispatch
  12: environmental::using
  13: sc_executor::executor::WasmExecutor<H>::with_instance::{{closure}}
  14: sc_executor::wasm_runtime::RuntimeCache::with_instance
  15: <sc_executor::executor::NativeElseWasmExecutor<D> as sp_core::traits::CodeExecutor>::call
  16: sp_state_machine::execution::StateMachine<B,H,Exec>::execute
  17: <sc_service::client::call_executor::LocalCallExecutor<Block,B,E> as sc_client_api::call_executor::CallExecutor<Block>>::contextual_call
  18: <sc_service::client::client::Client<B,E,Block,RA> as sp_api::CallApiAt<Block>>::call_api_at
  19: <canary_matrix_runtime::RuntimeApiImpl<__SrApiBlock__,RuntimeApiImplCall> as sp_api::Core<__SrApiBlock__>>::__runtime_api_internal_call_api_at
  20: sp_api::Core::execute_block
  21: <&sc_service::client::client::Client<B,E,Block,RA> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
  22: <alloc::sync::Arc<T> as sc_consensus::block_import::BlockImport<B>>::import_block::{{closure}}
  23: <cumulus_client_consensus_common::ParachainBlockImport<Block,BI,BE> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
  24: <alloc::boxed::Box<dyn sc_consensus::block_import::BlockImport<B>+Error = sp_consensus::error::Error+core::marker::Send+core::marker::Sync> as sc_consensus::block_import::BlockImport<B>>::import_block::{{closure}}
  25: futures_util::future::future::FutureExt::poll_unpin
  26: sc_consensus::import_queue::basic_queue::BlockImportWorker<B>::new::{{closure}}
  27: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
  28: <sc_service::task_manager::prometheus_future::PrometheusFuture<T> as core::future::future::Future>::poll
  29: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
  30: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
  31: tokio::runtime::task::core::Core<T,S>::poll
  32: tokio::runtime::task::harness::Harness<T,S>::poll
  33: tokio::runtime::blocking::pool::Inner::run
  34: std::sys_common::backtrace::__rust_begin_short_backtrace
  35: core::ops::function::FnOnce::call_once{{vtable.shim}}
  36: std::sys::pal::unix::thread::Thread::new::thread_start
  37: __pthread_joiner_wake


Thread 'tokio-runtime-worker' panicked at 'Storage root must match that calculated.', /Users/ubuntu/.cargo/git/checkouts/polkadot-sdk-a7b255f686157f80/314adba/substrate/frame/executive/src/lib.rs:834

This is a bug. Please report it at:

	https://github.com/substrate-developer-hub/substrate-parachain-template/issues/new

Pruning the DB and syncing from scratch works without issues, the issue only happens when trying to use an already synced DB. I am aware of the changes for storage reclaim and have already implemented the steps here : https://github.com/paritytech/polkadot-sdk/blob/master/docs/sdk/src/guides/enable_pov_reclaim.rs

Steps to reproduce

No response

[bkchr]

From which version are you upgraded? Did you bumped the spec_version?

[5tan-enj]

from polkadot-v1.9.0, this is a client upgrade btw, not a runtime upgrade (in preparation for a runtime upgrade), to be clear

client built with v1.9.0 -> synced DB
replaced with client built with v1.11.0 -> synced DB