Datafusion-cli: Redesign the datafusion-cli execution and print, make it totally streaming printing without memory overhead

[zhuqi-lucas]

Which issue does this PR close?

• Closes Further improve datafusion-cli memory usage if we setting huge number for maxrow size. #14810

Rationale for this change

Redesign the datafusion-cli execution and print, make it totally streaming printing without memory overhead.
Almost no extra memory usage for datafusion-cli now after this PR.

What changes are included in this PR?

Redesign the datafusion-cli execution and print, make it totally streaming printing without memory overhead.
Almost no extra memory usage for datafusion-cli now after this PR.

1. Change all datafusion-cli printing to streaming
2. Support preview the column width format printing, default is 1000 lines
3. Also support csv/json, etc format, max rows support which not supporting before
4. Totally removing the datafusion-cli memory overhead, the only occupied memory is the current batch for batchs iterator.

Are these changes tested?

Unit test added.

Also mannul tested the datafusion-cli.

Are there any user-facing changes?

Almost no extra memory usage for datafusion-cli now!

[alamb]

Thank you @zhuqi-lucas -- this is really cool!

I have a suggestion that I think would make the code better, but we could do it as a follow on PR or never in my opinion.

In addition to the very nice tests you have in this PR I tested it manaully (notes below) and it worked great

Testing without limit:

cargo run -p datafusion-cli -- --maxrows inf -c "select * from 'benchmarks/data/hits.parquet'"

And output was produced immediately and streamed out

I also tested with

cargo run -p datafusion-cli --  -c "select * from 'benchmarks/data/hits.parquet'"

Which printed out the first 40 rows and then continued running but did not print anything, as expected

[alamb]

not related to this PR, but I don't understand this comment about Memory safety

[zhuqi-lucas]

It's also confusing to me, maybe we can remove the message. It should be memory safe, we only keep one batch size.

[alamb]

most of the newly added functions don't use fields on PrintFormat

I think we could probably avoid the

    #[allow(clippy::too_many_arguments)]

By encapsulating all the mut variables here in a new struct

Maybe something like

struct OutputStreamState {
        preview_batches: Vec<RecordBatch>,
        preview_row_count: usize,
        preview_limit: usize,
        precomputed_widths: Option<Vec<usize>>,
        header_printed: bool,
        writer: &mut dyn std::io::Write,
}

impl OutputStreamState {
  fn process_batch(&mut self, batch: RecordBatch) { ... }
...
  fn compute_column_widths(&self, 
    batches: &Vec<RecordBatch>,
    schema: SchemaRef,
  ) {...}

}

[zhuqi-lucas]

Thanks @alamb for review, this is a very good suggestion!

[zhuqi-lucas]

Created a follow-up ticket:
#14886

[alamb]

👍

[zhuqi-lucas]

Thank you @zhuqi-lucas -- this is really cool!

I have a suggestion that I think would make the code better, but we could do it as a follow on PR or never in my opinion.

In addition to the very nice tests you have in this PR I tested it manaully (notes below) and it worked great

Testing without limit:

cargo run -p datafusion-cli -- --maxrows inf -c "select * from 'benchmarks/data/hits.parquet'"

And output was produced immediately and streamed out

I also tested with

cargo run -p datafusion-cli --  -c "select * from 'benchmarks/data/hits.parquet'"

Which printed out the first 40 rows and then continued running but did not print anything, as expected

Thank you @alamb for review, i created a follow-up to do the code clean:
#14886

And we can do some code clean work including:

1. Add streaming state struct and clean up print batch logic
2. Remove older logics about max-rows
3. Add more docs and remove confusing docs
4. Etc

[alamb]

Thanks again @zhuqi-lucas

[alamb]

FYI we found one issue with this code here (fixed in #14921):

• New Datafusion-cli streaming printing way should handle corner case for only one small batch which lines are less than max_rows #14920
• Datafusion-cli: regression after commit 3d64de4, all results are not displayed #14926

[alamb]

I believe I also found another related issue with explain plans

• datafusion-cli regression: explain plan output looks bad (error rendering multi-lines) #14947