Running init_send_tx for the second time is causing tokio-runtime-worker to panic and Owner API disconnects with no response

[marekyggdrasil]

Describe the bug

I am using Owner API. I perform a transaction by opening wallet with open_wallet and init tx using init_send_tx. I noticed that after success if I do it again the remote just disconnects but only for init_send_tx. Other methods appear to work fine. I examined wallet logs and I found this

20220109 04:13:02.634 ERROR grin_util::logger - 
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: Any { .. }': impls/src/client_utils/client.rs:324   0: grin_util::logger::send_panic_to_log::{{closure}}
   1: std::panicking::rust_panic_with_hook
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:628:17
   2: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:521:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys_common/backtrace.rs:139:18
   4: rust_begin_unwind
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:517:5
   5: core::panicking::panic_fmt
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/panicking.rs:100:14
   6: core::result::unwrap_failed
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/result.rs:1616:5
   7: grin_wallet_impls::client_utils::client::Client::send_request
   8: grin_wallet_impls::client_utils::client::Client::post
   9: <grin_wallet_impls::node_clients::http::HTTPNodeClient as grin_wallet_libwallet::types::NodeClient>::get_chain_tip
  10: grin_wallet_libwallet::internal::tx::new_tx_slate
  11: grin_wallet_libwallet::api_impl::owner::init_send_tx
  12: grin_wallet_api::owner::Owner<L,C,K>::init_send_tx
  13: <grin_wallet_api::owner::Owner<L,C,K> as grin_wallet_api::owner_rpc::OwnerRpc>::init_send_tx
  14: <dyn grin_wallet_api::owner_rpc::OwnerRpc as easy_jsonrpc_mw::Handler>::handle
  15: easy_jsonrpc_mw::handle_call
  16: easy_jsonrpc_mw::handle_parsed_request
  17: easy_jsonrpc_mw::Handler::handle_request
  18: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  19: <hyper::proto::h1::dispatch::Server<S,hyper::body::body::Body> as hyper::proto::h1::dispatch::Dispatch>::poll_msg
  20: hyper::proto::h1::dispatch::Dispatcher<D,Bs,I,T>::poll_catch
  21: <hyper::server::conn::upgrades::UpgradeableConnection<I,S,E> as core::future::future::Future>::poll
  22: <hyper::server::conn::spawn_all::NewSvcTask<I,N,S,E,W> as core::future::future::Future>::poll
  23: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
  24: tokio::runtime::task::raw::poll
  25: std::thread::local::LocalKey<T>::with
  26: tokio::runtime::thread_pool::worker::Context::run_task
  27: tokio::runtime::thread_pool::worker::Context::run
  28: tokio::macros::scoped_tls::ScopedKey<T>::set
  29: tokio::runtime::thread_pool::worker::run
  30: tokio::runtime::task::core::Core<T,S>::poll
  31: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
  32: tokio::runtime::task::harness::Harness<T,S>::poll
  33: tokio::runtime::blocking::pool::Inner::run
  34: tokio::runtime::context::enter
  35: std::sys_common::backtrace::__rust_begin_short_backtrace
  36: core::ops::function::FnOnce::call_once{{vtable.shim}}
  37: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/alloc/src/boxed.rs:1691:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/alloc/src/boxed.rs:1691:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys/unix/thread.rs:106:17
  38: start_thread
  39: clone

If I restart the Owner API and do the init_send_tx it is fine.

To Reproduce
Steps to reproduce the behavior:

1. Make sure your wallet has unlocked inputs, otherwise you will encounter Indicate in the WalletError when inputs are locked instead of just throwing InvalidKeychainMask #638 also have another wallet ready to receive.
2. Start the Owner API listener
3. Init secure API connection
4. open_wallet, otherwise you will get invalid keychain mask
5. init_send_tx that will give you a slate.
6. Receive the above slate using another wallet. This will give you another slate.
7. Use slate from (6) to finalize_tx which also gives a slate
8. Provide slate from (7) to post_tx
9. Wait for the transaction to get processed and inputs get unlocked.
10. Repeat step (5), your API request will not receive any response because remote will disconnect. The wallet logs will contain the tokio runtime panic error I stated above. Other API methods seem to be working fine. I tried start_updater and it is unaffected.
11. Stop Owner API and start again.
12. Repeat steps (3), (4) and (5) this time it will work properly.

Expected behavior

I would expect the init_send_tx to work without restarting Owner API and I would expect not to have tokio runtime panic in the logs.

Screenshots
N/A

Desktop (please complete the following information):
OS:

$ uname -s -r -v -m -p -i -o
Linux 4.19.0-18-amd64 #1 SMP Debian 4.19.208-1 (2021-09-29) x86_64 unknown unknown GNU/Linux

Wallet version

$ grin-wallet -V
grin-wallet 5.1.0-alpha.1

Build at commit

$ git rev-parse HEAD
34d23eb17ada3ffc09d76b1fa66ba9778abc29ea

Additional context

“Anakin Skywalker was weak. I destroyed him.” – Darth Vader