Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add GPU support to the Polars LazyFrame profiler #17723

Draft
wants to merge 11 commits into
base: branch-25.04
Choose a base branch
from

Conversation

Matt711
Copy link
Contributor

@Matt711 Matt711 commented Jan 13, 2025

Description

Closes #16224. Depends on pola-rs/polars#20693

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@Matt711 Matt711 added feature request New feature or request non-breaking Non-breaking change labels Jan 13, 2025
Copy link

copy-pr-bot bot commented Jan 13, 2025

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions bot added Python Affects Python cuDF API. cudf.polars Issues specific to cudf.polars labels Jan 13, 2025
@Matt711
Copy link
Contributor Author

Matt711 commented Jan 13, 2025

This is what the profile would looks like.
TODO

  • report the time in us
  • time each node separately
In [1]: import polars as pl

In [2]: lf = pl.LazyFrame(
   ...:     {
   ...:         "a": ["a", "b", "a", "b", "b", "c"],
   ...:         "b": [1, 2, 3, 4, 5, 6],
   ...:         "c": [6, 5, 4, 3, 2, 1],
   ...:     }
   ...: )
   ...: lf.group_by("a", maintain_order=True).agg(pl.all().sum()).sort(
   ...:     "a"
   ...: ).profile(engine="gpu")
Out[2]: 
(shape: (3, 3)
 ┌─────┬─────┬─────┐
 │ abc   │
 │ --------- │
 │ stri64i64 │
 ╞═════╪═════╪═════╡
 │ a410  │
 │ b1110  │
 │ c61   │
 └─────┴─────┴─────┘,
 shape: (2, 3)
 ┌──────────────┬───────────────┬───────────────┐
 │ nodestartend           │
 │ ---------           │
 │ stru64u64           │
 ╞══════════════╪═══════════════╪═══════════════╡
 │ optimization05700362213737 │
 │ GPU callback57003622137375700362795311 │
 └──────────────┴───────────────┴───────────────┘)

Copy link

copy-pr-bot bot commented Jan 23, 2025

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@Matt711 Matt711 changed the base branch from branch-25.02 to branch-25.04 February 10, 2025 13:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cudf.polars Issues specific to cudf.polars feature request New feature or request non-breaking Non-breaking change Python Affects Python cuDF API.
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

[FEA] Support GPU execution engine in LazyFrame profiler
1 participant