Add roadmap to readme #1616
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -141,6 +141,60 @@ datafusion = "6.0.0" | |
|
|
||
| DataFusion also includes a simple command-line interactive SQL utility. See the [CLI reference](https://arrow.apache.org/datafusion/cli/index.html) for more information. | ||
|
|
||
| # Roadmap | ||
|
|
||
| A quarterly roadmap will be published to give the DataFusion community visibility into the priorities of the projects contributors. This roadmap is not binding. | ||
|
|
||
| ## 2022 Q1 | ||
|
|
||
| ### DataFusion Core | ||
|
|
||
| - Publish official Arrow2 branch | ||
| - Implementation of memory manager (i.e. to enable spilling to disk as needed) | ||
|
|
||
| ### Benchmarking | ||
|
|
||
| - Inclusion in Db-Benchmark with all quries covered | ||
| - All TPCH queries covered | ||
|
Comment on lines
+157
to
+158
There was a problem hiding this comment. To my knowledge, no. I thought they were two independent benchmarks that we wanted to cover. However, I don't have much experience on the TPCH side / i've only been working on the db-benchmark solution. I don't see TPCH mentioned on db-benchmark. Would you be able to expand on how you think they are duplicates? There was a problem hiding this comment. I'll check it after getting up There was a problem hiding this comment. Sorry, my mistake. I believe the Db-Benchmark you mentioned is There was a problem hiding this comment. The db-benchmark im referring to is getting datafusion included here https://h2oai.github.io/db-benchmark/ I've opened a PR there to get it added. |
||
|
|
||
| ### Performance Improvements | ||
|
|
||
| - Predicate evaluation | ||
| - Improve multi-column comparisons (that can't be vectorized at the moment) | ||
| - Null constant support | ||
|
|
||
| ### New Features | ||
|
|
||
| - Read JSON as table | ||
| - Simplify DDL with Datafusion-Cli | ||
| - Add Decimal128 data type and the attendant features such as Arrow Kernel and UDF support | ||
| - Add new experimental e-graph based optimizer | ||
|
|
||
| ### Ballista | ||
|
|
||
| - Begin work on design documents and plan / priorities for development | ||
|
|
||
| ### Extensions ([datafusion-contrib](https://github.com/datafusion-contrib])) | ||
|
|
||
| - Stable S3 support | ||
| - Begin design discussions and prototyping of a stream provider | ||
|
|
||
| ## Beyond 2022 Q1 | ||
|
|
||
| There is no clear timeline for the below, but community members have expressed interest in working on these topics. | ||
|
|
||
| ### DataFusion Core | ||
|
|
||
| - Custom SQL support | ||
| - Split DataFusion into multiple crates | ||
| - Push based query execution and code generation | ||
|
|
||
| ### Ballista | ||
|
|
||
| - Evolve architecture so that it can be deployed in a multi-tenant cloud native environment | ||
| - Ensure Ballista is scalable, elastic, and stable for production usage | ||
| - Develop distributed ML capabilities | ||
|
|
||
| # Status | ||
|
|
||
| ## General | ||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @alamb! I'm curious what "arrow2" means here. Is it related to https://github.com/jorgecarleitao/arrow2?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes that's what it refers to and it has been completed. However it's not completely up to date with master.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, that is exactly what arrow2 means
There is a branch https://github.com/apache/arrow-datafusion/tree/arrow2 and a discussion ticket #1532 that has more information if you are interested
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you!