-
-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(python): Issue correct PolarsInefficientMapWarning
for lshift/rshift operations
#12385
fix(python): Issue correct PolarsInefficientMapWarning
for lshift/rshift operations
#12385
Conversation
map_elements
replacement for lshift/rshift operators
map_elements
replacement for lshift/rshift operatorsInefficientMapWarning
for lshift/rshift operators
InefficientMapWarning
for lshift/rshift operatorsPolarsInefficientMapWarning
for lshift/rshift operators
PolarsInefficientMapWarning
for lshift/rshift operatorsPolarsInefficientMapWarning
for lshift/rshift operations
Perhaps #12090 (comment) could be suggested as an alternative.
|
Consider alternative proposed by cmdlineuser
nice! I'll give this a go |
PolarsInefficientMapWarning
for lshift/rshift operationsPolarsInefficientMapWarning
for lshift/rshift operations
That seems to work wonderfully, thanks! Demo with the originally reported issue: In [3]:
...: df = pl.DataFrame({'ipvers': [4, 4], 'prefix': ['172.16.2.0/24', '172.16.2.101/32'], 'prefixlen': [24, 32]})
...: (
...: df
...: .filter(pl.col('ipvers') == 4)
...: .with_columns(
...: netmask = pl.col('prefixlen')
...: .map_elements(lambda x: (0xffffffff >> (32 - x)) &
...: 0xffffffff)
...: )
...: )
<ipython-input-3-cf7f706f16ef>:7: PolarsInefficientMapWarning:
Expr.map_elements is significantly slower than the native expressions API.
Only use if you absolutely CANNOT implement your logic otherwise.
In this case, you can replace your `map_elements` with the following:
- pl.col("prefixlen").map_elements(lambda x: ...)
+ (4294967295 / (2**(32 - pl.col("prefixlen")))).cast(pl.Int64) & 4294967295
.map_elements(lambda x: (0xffffffff >> (32 - x)) &
Out[3]:
shape: (2, 4)
┌────────┬─────────────────┬───────────┬────────────┐
│ ipvers ┆ prefix ┆ prefixlen ┆ netmask │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ i64 ┆ i64 │
╞════════╪═════════════════╪═══════════╪════════════╡
│ 4 ┆ 172.16.2.0/24 ┆ 24 ┆ 16777215 │
│ 4 ┆ 172.16.2.101/32 ┆ 32 ┆ 4294967295 │
└────────┴─────────────────┴───────────┴────────────┘
In [4]:
...: df = pl.DataFrame({'ipvers': [4, 4], 'prefix': ['172.16.2.0/24', '172.16.2.101/32'], 'prefixlen': [24, 32]})
...: (
...: df
...: .filter(pl.col('ipvers') == 4)
...: .with_columns(
...: netmask = (4294967295 / (2**(32 - pl.col("prefixlen")))).cast(pl.Int64) & 4294967295
...: )
...: )
Out[4]:
shape: (2, 4)
┌────────┬─────────────────┬───────────┬────────────┐
│ ipvers ┆ prefix ┆ prefixlen ┆ netmask │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ i64 ┆ i64 │
╞════════╪═════════════════╪═══════════╪════════════╡
│ 4 ┆ 172.16.2.0/24 ┆ 24 ┆ 16777215 │
│ 4 ┆ 172.16.2.101/32 ┆ 32 ┆ 4294967295 │
└────────┴─────────────────┴───────────┴────────────┘ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great, thanks Marco!
closes #12384