Skip to content

Add support for running pandas queries with cudf.pandas enabled #148

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

vyasr
Copy link
Contributor

@vyasr vyasr commented Apr 23, 2025

This PR makes it possible to run the pandas queries with GPU acceleration analogous to the support for the Polars GPU engine. To support this, cudf is added to the requirements list (which means we should also be able to run the Polars GPU engine benchmarks with the virtual environment now).

@ritchie46
Copy link
Member

This needs a rebase.

@vyasr vyasr force-pushed the feat/pandas_queries branch from 9deb6fa to d8baf65 Compare April 28, 2025 19:33
@vyasr vyasr force-pushed the feat/pandas_queries branch from d8baf65 to 306e6fb Compare April 28, 2025 19:52
@vyasr
Copy link
Contributor Author

vyasr commented Apr 28, 2025

Done. However, the last release of cudf has an upper bound on the supported version that bumps us back to 1.25 here. I don't know if that is compatible with the polars cloud bits that you recently added. If you prefer, I can revert the changes adding cudf to the environment and we can continue relying on the *-no-env variants of the Makefile targets for now for the GPU benchmarks, and revisit adding cudf to the environment at a later date.

Note that when we first added GPU benchmarks to this repo cudf was not yet available on PyPI, only NVIDIA's pip index, so there was an even stronger reason not to add it to the environment here. Now that we can get cudf from PyPI it is feasible to do this, with the main issue being if the upper bounds that we impose for stability reasons are prohibitive for your use cases in this repo. Ideally we'd be able to relax that bound eventually, but I don't think we're quite comfortable enough to do that yet.

@vyasr
Copy link
Contributor Author

vyasr commented May 24, 2025

The 25.06 release of cudf will support Polars 1.28, so perhaps the best option here is to wait for that release so that we don't have to change the supported Polars version here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants