feat: add all, any and null_count Spark Expressions
#1724
Conversation
|
Hey @EdAbati , Took an initial swing at implementing the |
narwhals/_spark_like/expr.py
Outdated
|
|
||
| return self._from_call(_null_count, "null_count", returns_scalar=True) | ||
|
|
||
| def replace_strict( |
There was a problem hiding this comment.
I am tempted to say that this should not be implemented for now and just raise a NotImplementedError. (as we do in Dask)
We would need to be able to access the dataframe (and collect the results) to get the distinct values of the column.
@FBruzzesi and @MarcoGorelli any thoughts?
There was a problem hiding this comment.
I am tempted to say that this should not be implemented for now and just raise a NotImplementedError. (as we do in Dask)
Sure we can evaluate if and how to support replace_strict later on. Super excited to ship the rest for now 🙌🏼
Co-authored-by: Edoardo Abati <[email protected]>
Co-authored-by: Edoardo Abati <[email protected]>
|
Thanks everyone - committed the changes above. Lmk if anything else needs to be changed |
|
awesome, thanks! just looks like there's some merge conflicts |
|
Might've committed some git crimes resolving that merge conflict :') apologies in advance |
| result = df.select(nw.all().any()) | ||
| result = df.select(nw.col("a", "b", "c").any()) |
There was a problem hiding this comment.
@MarcoGorelli pyspark fails to maintain the original name with nw.all().<expression_method>. @EdAbati maybe you know why and where this is happening already.
Here it would result in:
FAILED tests/expr_and_series/any_all_test.py::test_any_all[pyspark] - AssertionError: Expected column name a at index 0, found bool_or(a)
There was a problem hiding this comment.
Let's ship this🚀 ! Thanks @lucas-nelson-uiuc and apologies for such a long back and forth on this PR
FBruzzesi
changed the title
all, any and null_count Spark Expressions
What type of PR is this? (check all applicable)
Related issues
Checklist
If you have comments or can explain your changes, please do so below
Added the following methods to
SparkLikeExprandSparkLikeNamespace:anyallnull_countany_horizontalCopied respective tests over - couldn't run them without Java on my machine but running them locally on their respective test datasets worked for me.
Let me know if anything needs to be updated!