Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement any and all for pyarrow numpy strings #54591

Merged
merged 32 commits into from
Aug 28, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
b24afc9
Start new string array
phofl Aug 13, 2023
b306c6f
Add missing methods
phofl Aug 13, 2023
2dbcfb0
Implement Arrow String Array that is compatible with NumPy semantics
phofl Aug 13, 2023
d9e61e5
Move methods
phofl Aug 13, 2023
3188c25
Refactor
phofl Aug 13, 2023
df231f0
Merge remote-tracking branch 'upstream/main' into string_array_numpy_…
phofl Aug 14, 2023
cd19bfb
Refactor
phofl Aug 14, 2023
c73c6b0
Remove
phofl Aug 14, 2023
6b26309
Fix
phofl Aug 14, 2023
da6d67c
Update
phofl Aug 14, 2023
d862eca
Na return value
phofl Aug 16, 2023
4be0ee8
Merge remote-tracking branch 'upstream/main' into string_array_numpy_…
phofl Aug 16, 2023
6cf2639
Fix
phofl Aug 16, 2023
333dcae
Merge remote-tracking branch 'origin/string_array_numpy_semantics' in…
phofl Aug 16, 2023
48bd626
Update
phofl Aug 16, 2023
4f9387a
Implement any and all for pyarrow numpy strings
phofl Aug 16, 2023
606cd71
Fix typing
phofl Aug 21, 2023
a44b042
Merge remote-tracking branch 'upstream/main' into string_array_numpy_…
phofl Aug 21, 2023
6414501
Update
phofl Aug 21, 2023
27b5057
Merge remote-tracking branch 'upstream/main' into any_all
phofl Aug 21, 2023
3fec6d3
Merge remote-tracking branch 'upstream/main' into string_array_numpy_…
phofl Aug 21, 2023
68acc32
Fix
phofl Aug 21, 2023
fbab6fb
Fix
phofl Aug 21, 2023
dd0f9a8
Merge branch 'string_array_numpy_semantics_na_val' into any_all
phofl Aug 21, 2023
68e5f8f
Fix
phofl Aug 21, 2023
0322006
Move test
phofl Aug 21, 2023
8bb52f4
Skip test when no pa
phofl Aug 21, 2023
cc8e6f7
Fix typing
phofl Aug 22, 2023
a3ef88d
Merge remote-tracking branch 'upstream/main' into any_all
phofl Aug 23, 2023
33355a7
Fix tests
phofl Aug 23, 2023
f337cd9
Merge remote-tracking branch 'upstream/main' into any_all
phofl Aug 26, 2023
1facb79
move + rename test
jorisvandenbossche Aug 28, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions pandas/core/arrays/string_arrow.py
Original file line number Diff line number Diff line change
Expand Up @@ -554,3 +554,16 @@ def value_counts(self, dropna: bool = True):
return Series(
result._values.to_numpy(), index=result.index, name=result.name, copy=False
)

def _reduce(
self, name: str, *, skipna: bool = True, keepdims: bool = False, **kwargs
):
if name in ["any", "all"]:
arr = pc.and_kleene(
pc.invert(pc.is_null(self._pa_array)), pc.not_equal(self._pa_array, "")
)
return ArrowExtensionArray(arr)._reduce(
name, skipna=skipna, keepdims=keepdims, **kwargs
)
else:
return super()._reduce(name, skipna=skipna, keepdims=keepdims, **kwargs)
6 changes: 5 additions & 1 deletion pandas/tests/extension/test_string.py
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,11 @@ def test_fillna_no_op_returns_copy(self, data):

class TestReduce(base.BaseReduceTests):
def _supports_reduction(self, ser: pd.Series, op_name: str) -> bool:
return op_name in ["min", "max"]
return (
op_name in ["min", "max"]
or ser.dtype.storage == "pyarrow_numpy" # type: ignore[union-attr]
and op_name in ("any", "all")
)


class TestMethods(base.BaseMethodsTests):
Expand Down
19 changes: 19 additions & 0 deletions pandas/tests/reductions/test_reductions.py
Original file line number Diff line number Diff line change
Expand Up @@ -1078,6 +1078,25 @@ def test_any_all_datetimelike(self):
assert df.any().all()
assert not df.all().any()

def test_any_all_pyarrow_string(self):
# GH#54591
pytest.importorskip("pyarrow")
ser = Series(["", "a"], dtype="string[pyarrow_numpy]")
assert ser.any()
assert not ser.all()

ser = Series([None, "a"], dtype="string[pyarrow_numpy]")
assert ser.any()
assert not ser.all()

ser = Series([None, ""], dtype="string[pyarrow_numpy]")
assert not ser.any()
assert not ser.all()

ser = Series(["a", "b"], dtype="string[pyarrow_numpy]")
assert ser.any()
assert ser.all()

def test_timedelta64_analytics(self):
# index min/max
dti = date_range("2012-1-1", periods=3, freq="D")
Expand Down