Skip to content

Commit

Permalink
Add a modified user-guide section back
Browse files Browse the repository at this point in the history
  • Loading branch information
Edwin Vehmaanperä committed Sep 24, 2024
1 parent ff415b1 commit 319b22a
Show file tree
Hide file tree
Showing 2 changed files with 28 additions and 0 deletions.
9 changes: 9 additions & 0 deletions docs/source/src/python/user-guide/lazy/execution.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,4 +28,13 @@
.collect(streaming=True)
)
# --8<-- [end:stream]
# --8<-- [start:partial]
q9 = (
pl.scan_csv(f"docs/assets/data/reddit.csv")
.head(10)
.with_columns(pl.col("name").str.to_uppercase())
.filter(pl.col("comment_karma") > 0)
.collect()
)
# --8<-- [end:partial]
"""
19 changes: 19 additions & 0 deletions docs/source/user-guide/lazy/execution.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,3 +48,22 @@ If your data requires more memory than you have available Polars may be able to
{{code_block('user-guide/lazy/execution','stream',['scan_csv','collect'])}}

We look at [streaming in more detail here](streaming.md).

### Execution on a partial dataset

While you're writing, optimizing or checking your query on a large dataset, querying all available data may lead to a slow development process.

You can instead limit the number of scanned partitions or use .head early in the query when testing. Keep in mind that aggregations and filters may behave unpredictably on subsets of data.

{{code_block('user-guide/lazy/execution','partial',['scan_csv','collect','head'])}}

```text
shape: (1, 6)
┌─────┬─────────────────────────┬─────────────┬────────────┬───────────────┬────────────┐
│ id ┆ name ┆ created_utc ┆ updated_on ┆ comment_karma ┆ link_karma │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
╞═════╪═════════════════════════╪═════════════╪════════════╪═══════════════╪════════════╡
│ 6 ┆ TAOJIANLONG_JASONBROKEN ┆ 1397113510 ┆ 1536527864 ┆ 4 ┆ 0 │
└─────┴─────────────────────────┴─────────────┴────────────┴───────────────┴────────────┘
```

0 comments on commit 319b22a

Please sign in to comment.