Is it possible to return a LogicalPlan back to a query string?

[seve-martinez]

I'm working on a project that wants to make use of the excellent work within DataFusion and its subcrates. My org is using snowflake under the hood, but we want to leverage DataFusion's Analyzer, Optimizer, LogicalPlan, and Parser to essentially parse user's SQL, implement our own cost functions for optimizations, and then send the improved query to snowflake.

My understanding of the standard "flow" in DataFusion is: Query String >> AST >> LogicalPlan >> PhysicalPlan. I've not seen a way to render a query string from the Plans in the later steps.

Is this possible?

[alamb]

My understanding of the standard "flow" in DataFusion is: Query String >> AST >> LogicalPlan >> PhysicalPlan. I've not seen a way to render a query string from the Plans in the later steps.

This is not possible as of today as I understand it.

This is a neat use case. I think daskql does something like this (uses just the analyzer/parser/etc) -- maybe @jdye64 or @andygrove have something more they can share

[Jefffrey]

Also, here's an issue with the same question and some prior discussion: #8661

[andygrove]

@seve-martinez You may get some inspiration from https://github.com/andygrove/sqlfuzz/blob/main/src/sql_writer.rs

It isn't working directly from the DataFusion logical plan, but from a simplified version.

[devinjdangelo]

You may be interested in the datafusion-federation effort started by @backkem which aims to extend datafusion for use-cases like this, including where there may be multiple execution engines participating in a single query.

See some code here for transforming a logical plan into an AST (which can be rendered into a sql string).
https://github.com/backkem/datafusion-federation/blob/742718e6587e3bdb0dfd3c6cf7256ab5a67d26ae/sources/sql/src/producer.rs#L26

[backkem]

Thanks for the tag @devinjdangelo. datafusion-federation does indeed have a first version of this. Not all statements are supported yet though. Feel free to log tickets and I can take a look at filling in those gaps.

If there is more interest around this effort I'm happy to move it to a (new or existing) org.

[alamb]

If you ever want a repo in https://github.com/datafusion-contrib I would be happy to make one for you @backkem

[backkem]

Yea, I'm happy to move the project there. I'm not sure if there is a decent workflow to move the existing repo. Alternatively, you can create a repo with the same/similar name.

[alamb]

I made https://github.com/datafusion-contrib/datafusion-federation and invited you as an admin @backkem

[seve-martinez]

Thank you all! This is very helpful.