Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Polars SQL datasets #853

Open
AntonNikishin opened this issue Sep 27, 2024 · 4 comments
Open

Polars SQL datasets #853

AntonNikishin opened this issue Sep 27, 2024 · 4 comments
Labels
Community Issue/PR opened by the open-source community

Comments

@AntonNikishin
Copy link

Description

It would be great to have Polars implementations of SQLQueryDataset and SQLTableDataset, similar to the Pandas versions: pandas.SQLTableDataset and pandas.SQLQueryDataset.

Context

Sometimes users would like to read / write polars DataFrames directly from SQL databases.

Possible Implementation

The datasets will have similar implementation to Pandas versions, but will use polars built-in functions read_database and write_database.

P.S. I'm happy to work on that ☺️

@noklam
Copy link
Contributor

noklam commented Sep 27, 2024

Would the ibis dataset already support polar as a backend?

@deepyaman
Copy link
Member

Would the ibis dataset already support polar as a backend?

It does, but:

  1. I'm guessing the read_database would need to be implemented in Ibis.
  2. If a user just wants to use Polars syntax in their nodes, I guess it's a fair ask.

It's a separate question whether Polars is the best way to manipulate data in a database (definite downside is pulling it into memit. For manipulation, rather than pushing down compute), but a user may still want to do it.

@deepyaman
Copy link
Member

I would recommend to create polars.DatabaseDataset instead of mirroring the pandas datasets, because:

  1. Polars provides symmetrical read and write methods.
  2. SQL is less explicit, because Polars SQL is also a thing.

@noklam
Copy link
Contributor

noklam commented Nov 11, 2024

Make sense, @AntonNikishin is this something you would like to work on?

@noklam noklam added the Community Issue/PR opened by the open-source community label Nov 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Community Issue/PR opened by the open-source community
Projects
Status: No status
Development

No branches or pull requests

3 participants