Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(datasets): Added the Experimental PolarsDatabaseDataset #990

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

MinuraPunchihewa
Copy link
Contributor

Description

This PR adds the PolarsDatabaseDataset to support interactions with databases using Polars.

Fixes #853

Development notes

I have extended the SQLQueryDataset to implement this dataset.

These changes have been tested,

1. Manually, by running the code locally to load and save tensors from and to Safetensors files.
2. Via the existing and newly added unit tests.

Checklist

  • Opened this PR as a 'Draft Pull Request' if it is work-in-progress
  • Updated the documentation to reflect the code changes
  • Added a description of this change in the relevant RELEASE.md file
  • Added tests to cover my changes
  • Received approvals from at least half of the TSC (required for adding a new, non-experimental dataset)

@MinuraPunchihewa
Copy link
Contributor Author

Hey @noklam, @deepyaman,
I was able to come up with this implementation for the PolarsDatabaseDataset by extending SQLQueryDataset and it seems to work quite well (at least load() does).

Should we implement save() as well? This would require a table name to be provided as parameter.

Or do you have different thoughts on how this dataset ought to be implemented?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Polars SQL datasets
1 participant