-
Notifications
You must be signed in to change notification settings - Fork 51
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Connection for Dremio Lakehouse #427
Comments
Thanks for this! We'll have a quick look into this today! |
Here's a first draft #446, it doesn't work and still needs filling in quite a bit but I think it should give you the general structure. There are a few things to add:
I mostly did it out of curiosity for:
|
Love the idea of internal format of Arrow - it looks very sweet! |
I'll pull the branch and try to get some feedback by Wed |
There's a first draft with this it in being pushed at the moment, it works with username/password/nossl
Outlines the variables you need: DREMIO_HOST, DREMIO_PORT, DREMIO_USE_SSL, DREMIO_USER, DREMIO_PASSWORD and they can be stored in
Is what I got working locally for this "setup" With the config dremio:
dremio_space: test
dremio_space_folder: test |
Please Describe The Problem To Be Solved
The Problem: Other than Snowflake, there is a lack of connectors to other lakehouse solutions. While a Databricks connector would be nice for many corporate production runs, in the interest of open source, a Dremio connector might be more appreciated by the community. This request is to build a Dremio connector for Quary.
Optional: Suggest A Solution
Looking into the code architecture, it seems that the bulk of connectors are maintained within
rust/quary-databases/src/databases_<flavor>.rs
andrust/core/src/database_<flavor>.rs
. Inspection shows a common class interface already designed across both. For Dremio, there are a number of protocols available including REST, JDBC, & ODBC. However, with a RUST build, it may be advantageous to use the ARROW Flight protocol as Dremio highly support it - can lead to 20X speed-up over JDBC. *In fact, could even extend this issue to a generic "Arrow Flight Connector" type.A possible plan includes:
rust/quary-databases/src/databases_dremio.rs
.*_dremio.rs
files).Happy to help on this to build out my Rust expertise...
The text was updated successfully, but these errors were encountered: