Crash: with_new_exprs for Distinct does not support sort expressions

[oherrala]

What type of bug is this?

Crash

What subsystems are affected?

Query Engine

Minimal reproduce step

Table:

CREATE TABLE IF NOT EXISTS "user" (
  "from" STRING NULL,
  "short_name" STRING NULL,
  "long_name" STRING NULL,
  "is_licensed" BOOLEAN NULL,
  "hw_model" STRING NULL,
  "role" STRING NULL,
  "ts" TIMESTAMP(9) NOT NULL,
  TIME INDEX ("ts"),
  PRIMARY KEY ("from")
)

ENGINE=mito
WITH(
  merge_mode = 'last_non_null'
)

Query:

SELECT DISTINCT ON (from) * FROM user ORDER BY from, ts DESC;

What did you expect to see?

I did not expect crash, but query results or error from query.

What did you see instead?

Stacktrace and console output:

Well, this is embarrassing.

GreptimeDB had a problem and crashed. To help us diagnose the problem you can send us a crash report.

We have generated a report file at "/tmp/report-395d04f7-5663-4a45-817c-9646ede049f8.toml". Submit an issue or email with the subject of "GreptimeDB Crash Report" and include the report as an attachment.

• Homepage: https://github.com/GreptimeTeam/greptimedb/discussions

We take privacy seriously, and do not perform any automated error collection. In order to improve the software, we rely on people to submit reports.

Thank you kindly!

What operating system did you use?

Docker image greptime/greptimedb:v0.13.1

What version of GreptimeDB did you use?

0.13.1

Relevant log output and stack trace

name = "GreptimeDB"
operating_system = "Ubuntu 22.4.0 [64-bit]"
crate_version = "0.13.0"
explanation = """
Panic occurred in file '/root/.cargo/git/checkouts/datafusion-11a8b534adb6bd68-shallow/2464703/datafusion/expr/src/logical_plan/plan.rs' at line 1035
"""
cause = "with_new_exprs for Distinct does not support sort expressions"
method = "Panic"
backtrace = """

   0: 0xaaaab0fe5374 - core::panicking::panic_fmt::h29b5025e6a1f313d
   1: 0xaaaab381c82c - datafusion_expr::logical_plan::plan::LogicalPlan::with_new_exprs::h7a930eee688f6b4c
   2: 0xaaaab8220ca8 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &mut F>::call_once::h971ba9cc1d2d6c76
   3: 0xaaaab800a61c - datafusion_common::tree_node::TreeNode::transform_up::transform_up_impl::{{closure}}::h2c5c46949fe3143c
   4: 0xaaaab80082b4 - datafusion_common::tree_node::TreeNode::transform_up::transform_up_impl::hdebc0d2df4b19a42
   5: 0xaaaab826ddac - <query::optimizer::type_conversion::TypeConversionRule as query::optimizer::ExtensionAnalyzerRule>::analyze::h38010704f7176c6e
   6: 0xaaaab827283c - query::query_engine::state::QueryEngineState::optimize_by_extension_rules::h2a4167ee29d043d4
   7: 0xaaaab835326c - query::planner::DfLogicalPlanner::plan_sql::{{closure}}::{{closure}}::h9638ee8d84df333b
   8: 0xaaaab8356eec - <query::planner::DfLogicalPlanner as query::planner::LogicalPlanner>::plan::{{closure}}::{{closure}}::hf5a9f3a468c74fb1
   9: 0xaaaab8356954 - <query::planner::DfLogicalPlanner as query::planner::LogicalPlanner>::plan::{{closure}}::h8f9e4646bd494740
  10: 0xaaaab576eeb0 - operator::statement::StatementExecutor::plan::{{closure}}::{{closure}}::hfebe2335fc05c6a8
  11: 0xaaaab576ebd8 - operator::statement::StatementExecutor::plan::{{closure}}::h39938ee8c5553bc3
  12: 0xaaaab57a517c - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::hb965e8411d0a0168
  13: 0xaaaab569b8f0 - <frontend::instance::Instance as servers::query_handler::sql::SqlQueryHandler>::do_query::{{closure}}::h42ae5cf0dfdbfec6
  14: 0xaaaab6048b4c - <servers::query_handler::sql::ServerSqlQueryHandlerAdapter<E> as servers::query_handler::sql::SqlQueryHandler>::do_query::{{closure}}::h4c814e0d1cf6f9c7
  15: 0xaaaab8bf4fe4 - servers::http::handler::sql::{{closure}}::{{closure}}::ha91283a49b861f8f
  16: 0xaaaab8eca748 - <F as axum::handler::Handler<(M,T1,T2,T3,T4),S>>::call::{{closure}}::h72802fd387b1b53a
  17: 0xaaaab8a7efe0 - <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::hfa390c7c4678c22c
  18: 0xaaaab8a7e204 - <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::h4fcdc56121b744c8
  19: 0xaaaab89ab4b0 - <tower::util::oneshot::Oneshot<S,Req> as core::future::future::Future>::poll::h573dfb5301f77fde
  20: 0xaaaab8acf260 - <axum::routing::route::RouteFuture<E> as core::future::future::Future>::poll::hda717c4d540cbbbe
  21: 0xaaaab8a7e0e8 - <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::h49ff5940f135a30a
  22: 0xaaaab8a7e4f8 - <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::h703f7d9e2d5e5a2d
  23: 0xaaaab8a7e73c - <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::h929cd4709e533bce
  24: 0xaaaab89ab4b0 - <tower::util::oneshot::Oneshot<S,Req> as core::future::future::Future>::poll::h573dfb5301f77fde
  25: 0xaaaab8acf260 - <axum::routing::route::RouteFuture<E> as core::future::future::Future>::poll::hda717c4d540cbbbe
  26: 0xaaaab8a7e0e8 - <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::h49ff5940f135a30a
  27: 0xaaaab8e8b8a8 - axum::middleware::from_fn::Next::run::{{closure}}::hf45747571f34573c
  28: 0xaaaab8e76e28 - <axum::middleware::from_fn::FromFn<F,S,I,(T1,)> as tower_service::Service<http::request::Request<axum_core::body::Body>>>::call::{{closure}}::h6aa10800ee908348
  29: 0xaaaab8a7e398 - <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::h5eb73ce198d82fcc
  30: 0xaaaab8a7da34 - <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::h1b12ac231974bf0c
  31: 0xaaaab8a7dd10 - <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::h289900ccfba1dea1
  32: 0xaaaab89ab4b0 - <tower::util::oneshot::Oneshot<S,Req> as core::future::future::Future>::poll::h573dfb5301f77fde
  33: 0xaaaab8acf260 - <axum::routing::route::RouteFuture<E> as core::future::future::Future>::poll::hda717c4d540cbbbe
  34: 0xaaaab8a7e0e8 - <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::h49ff5940f135a30a
  35: 0xaaaab8e8b8a8 - axum::middleware::from_fn::Next::run::{{closure}}::hf45747571f34573c
  36: 0xaaaab8e76604 - <axum::middleware::from_fn::FromFn<F,S,I,(T1,)> as tower_service::Service<http::request::Request<axum_core::body::Body>>>::call::{{closure}}::h33f9ce5ac8847412
  37: 0xaaaab8a7e398 - <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::h5eb73ce198d82fcc
  38: 0xaaaab8e8b8a8 - axum::middleware::from_fn::Next::run::{{closure}}::hf45747571f34573c
  39: 0xaaaab8e78238 - <axum::middleware::from_fn::FromFn<F,S,I,(T1,T2)> as tower_service::Service<http::request::Request<axum_core::body::Body>>>::call::{{closure}}::h8fb9b3509375c894
  40: 0xaaaab89c5c24 - <servers::http::timeout::ResponseFuture<F> as core::future::future::Future>::poll::h812fdfd6f15c1ba9
  41: 0xaaaab8e11460 - <tower_http::cors::ResponseFuture<F> as core::future::future::Future>::poll::h4d257f8199576db7
  42: 0xaaaab8b6b04c - <tower_http::trace::future::ResponseFuture<Fut,C,OnResponseT,OnBodyChunkT,OnEosT,OnFailureT> as core::future::future::Future>::poll::he4e6882a331bedfa
  43: 0xaaaab8a7ec14 - <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::hda549dc6cb7bdec2
  44: 0xaaaab8a7dfc0 - <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::h344ccaeb77b121a2
  45: 0xaaaab8a7de9c - <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::h2b6daaa0ec561a51
  46: 0xaaaab89ab4b0 - <tower::util::oneshot::Oneshot<S,Req> as core::future::future::Future>::poll::h573dfb5301f77fde
  47: 0xaaaab8acf260 - <axum::routing::route::RouteFuture<E> as core::future::future::Future>::poll::hda717c4d540cbbbe
  48: 0xaaaab8a7d600 - <hyper_util::service::oneshot::Oneshot<S,Req> as core::future::future::Future>::poll::ha540fd4a87c0a203
  49: 0xaaaab8c4cf8c - hyper::proto::h1::dispatch::Dispatcher<D,Bs,I,T>::poll_catch::ha07b8bd1cd6ad0f3
  50: 0xaaaab8d22c04 - <core::pin::Pin<P> as core::future::future::Future>::poll::h739b5b92cd724e93
  51: 0xaaaab8ed3ff8 - <core::future::poll_fn::PollFn<F> as core::future::future::Future>::poll::h87fe16053ddf2a87
  52: 0xaaaab8c06b08 - <axum::serve::WithGracefulShutdown<L,M,S,F> as core::future::into_future::IntoFuture>::into_future::{{closure}}::{{closure}}::h163b9bcbf7d7bc8f
  53: 0xaaaab8c54a4c - tokio::runtime::task::core::Core<T,S>::poll::h9bfa0abf3b055b31
  54: 0xaaaab8a18a84 - tokio::runtime::task::harness::Harness<T,S>::poll::ha1cd9b750d842eb5
  55: 0xaaaab9d8aeac - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::hf6e524d42db63f9e
  56: 0xaaaab9d8a0e4 - tokio::runtime::scheduler::multi_thread::worker::Context::run::h1217f9f474716a81
  57: 0xaaaab9d7c4a4 - tokio::runtime::context::runtime::enter_runtime::h256a4e4c32931b41
  58: 0xaaaab9d89f1c - tokio::runtime::scheduler::multi_thread::worker::run::he244a7e319b06408
  59: 0xaaaab9d94fd8 - tokio::runtime::task::core::Core<T,S>::poll::h23d4482b28529f00
  60: 0xaaaab9d6fa3c - tokio::runtime::task::harness::Harness<T,S>::poll::h2e8f61dac7e093bc
  61: 0xaaaab9d80de4 - tokio::runtime::blocking::pool::Inner::run::h714bef9c00645016
  62: 0xaaaab9d97450 - std::sys::backtrace::__rust_begin_short_backtrace::he272859db0fdda4f
  63: 0xaaaab9d98a68 - core::ops::function::FnOnce::call_once{{vtable.shim}}::hd183e867811de9db
  64: 0xaaaab98340cc - std::sys::pal::unix::thread::Thread::new::thread_start::hdfd7c4e42ef24c96
  65: 0xffff9ccad5c8 - <unresolved>
  66: 0xffff9cd15edc - <unresolved>
  67:        0x0 - <unresolved>"""

[evenyag]

Looks like an issue related to upstream: apache/datafusion#12177

We may need to handle Distinct specially like Join......

[evenyag]

The expressions() method of DistinctOn returns the sort_expr.
https://github.com/apache/datafusion/blob/46.0.0/datafusion/expr/src/logical_plan/tree_node.rs#L451-L456

But with_new_exprs() doesn't support this.
https://github.com/apache/datafusion/blob/46.0.0/datafusion/expr/src/logical_plan/plan.rs#L1035-L1058

I created a quick fix for this in #5823

[killme2008]

Fixed in v0.13.2