Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Expand distributed indexing, match numpy indexing scheme #938

Draft
wants to merge 150 commits into
base: main
Choose a base branch
from
Draft

Expand distributed indexing, match numpy indexing scheme #938

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
150 commits
Select commit Hold shift + click to select a range
445fc94
Broken. __getitem__ refactoring in prep for distributed/non-ordered i…
ClaudiaComito Feb 17, 2022
6641d1e
Preprocess key, workaround torch_proxy for advanced indexing, simplif…
Feb 22, 2022
cd78ecb
put advanced index shape in the dimensions name to get the correct po…
Feb 22, 2022
7d97ea2
first changes to setitem
Feb 22, 2022
0c37abf
Expand `__process_key()` to address advanced indexing.
ClaudiaComito Feb 24, 2022
b1508b9
Address boolean indexing
ClaudiaComito Feb 26, 2022
ae5af94
separate advanced indexing on dim 0 from adv ind across dimensions
ClaudiaComito Feb 28, 2022
ace900a
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito Mar 11, 2022
0a8cb35
Replace `sanitize_in` with `try:...except:` construct
ClaudiaComito Mar 11, 2022
6c7c10a
`nonzero()`: do not assume input DNDarray is load-balanced
ClaudiaComito Mar 11, 2022
fb3524b
Memory management
ClaudiaComito Mar 14, 2022
eb297fb
fix #925: ht.nonzero() returns tuple of 1-D arrays instead of n-D arr…
Mystic-Slice Apr 8, 2022
8485a31
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito May 24, 2022
a52e518
calculate output_shape, split axis bookkeeping for advanced indexing
ClaudiaComito Jun 7, 2022
5995639
`__process_key()` to return expanded array, expanded key, output gsha…
ClaudiaComito Jul 12, 2022
3830e62
in , copy before manipulations
ClaudiaComito Aug 25, 2022
82b2508
nonzero() to return tuple of 1D arrays, stable distributed results
ClaudiaComito Aug 27, 2022
aafaf99
update __process_key(), get rid of recursive calls, __getitem__ broken
ClaudiaComito Aug 27, 2022
b746872
deal with scalar key, local and distributed cases
ClaudiaComito Aug 30, 2022
00fe538
test getitem separately, follow numpy Indexing on ndarray examples
ClaudiaComito Aug 30, 2022
4360bd1
test for 0-dim DNDarray key
ClaudiaComito Aug 30, 2022
231c1de
Expand __process_key() to deal with distributed boolean mask
ClaudiaComito Aug 31, 2022
f19f902
Expand test_getitem for distributed single-element indexing, non-dist…
ClaudiaComito Aug 31, 2022
7ed435f
Add check for matching boolean index / indexed array shapes
ClaudiaComito Aug 31, 2022
0da7f56
Only sort result if input.split != 0
ClaudiaComito Sep 3, 2022
e55c7f9
BROKEN: distributed boolean indexing to return stable result for all …
ClaudiaComito Sep 3, 2022
75d9314
Add tests for distributed boolean indexing
ClaudiaComito Sep 3, 2022
15a8a28
BROKEN: Fixed key redistribution for input.split != 0.
ClaudiaComito Sep 4, 2022
8db0511
Expanded boolean indexing tests
ClaudiaComito Sep 4, 2022
291329e
Set up communication matrix for boolean indexing along non-zero split
ClaudiaComito Sep 8, 2022
6d986dd
Implement getitem for non-ordered key along split axis
ClaudiaComito Nov 4, 2022
f46ae67
Fix edge-case contiguity mismatch for Allgatherv
ClaudiaComito Dec 12, 2022
4da69fd
merge branch release/1.2.x
ClaudiaComito Dec 12, 2022
27ea911
Update ubuntu
ClaudiaComito Dec 12, 2022
d0fb6c8
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 12, 2022
0e704d4
switch back to ubuntu 20.04
ClaudiaComito Dec 12, 2022
f5d7850
pull
ClaudiaComito Dec 12, 2022
acfe9bd
Upgrade CI to ubuntu 22.04 and cuda 11.7.1
ClaudiaComito Dec 12, 2022
0fd3d87
avoid unnecessary gathering of test DNDarrays
ClaudiaComito Dec 20, 2022
3c4c07c
early out for resplit of non-distributed DNDarrays
ClaudiaComito Dec 20, 2022
989e0f4
match split of comparison array to expected output
ClaudiaComito Dec 20, 2022
6d66fad
avoid MPI calls in non-distributed cases
ClaudiaComito Dec 20, 2022
a37b4d3
avoid MPI calls in non-distributed resplit
ClaudiaComito Dec 20, 2022
8eebe10
set default to None
ClaudiaComito Dec 20, 2022
22c5c68
remove print statement
ClaudiaComito Dec 20, 2022
c692bff
upgrade torch version
ClaudiaComito Dec 20, 2022
df6a4e5
copy to cpu before comparing
ClaudiaComito Dec 20, 2022
af0e721
use ht.allclose instead of np.allclose
ClaudiaComito Dec 23, 2022
bac6d4e
cast different dtype operands to promoted dtype within torch call
ClaudiaComito Dec 23, 2022
c0c6362
compare local tensors to corresponding slice of expected_array only
ClaudiaComito Dec 23, 2022
587bc05
expand tests
ClaudiaComito Dec 23, 2022
24239a1
remove redundant code
ClaudiaComito Dec 23, 2022
cd65b37
Implement slicing with negative step
ClaudiaComito Dec 26, 2022
86e8801
test slicing with negative step
ClaudiaComito Dec 26, 2022
6779010
merge branch bugs/#1057-Allgatherv-contiguity-mismatch
ClaudiaComito Dec 26, 2022
3b1f46d
Fix single-element indexing within mixed-type key
ClaudiaComito Dec 27, 2022
1a4bf97
Non-ordered indexing, split != 0
ClaudiaComito Dec 27, 2022
9e42156
generalize negative step slicing to all splits, loss of dims
ClaudiaComito Dec 28, 2022
1a310a9
loop over active ranks only when key in descending order
ClaudiaComito Dec 28, 2022
c2ba0d9
replace list-on-list mapping with argsort mapping for non-ordered key
ClaudiaComito Dec 29, 2022
f6bb5c3
replace list-on-list mapping with argsort mapping for boolean indexing
ClaudiaComito Dec 30, 2022
cad9975
fix advanced indexing via list, remove last key-mapping bottleneck fo…
ClaudiaComito Dec 31, 2022
83e6950
fix local slices, expand tests
ClaudiaComito Jan 2, 2023
28ab925
fix and test dimensional indexing
ClaudiaComito Jan 2, 2023
bc226fc
Fix same-dim advanced indexing, expand tests
ClaudiaComito Jan 5, 2023
c48c66e
[skip ci] implement single-element indexing along split axis w/ Itera…
ClaudiaComito Jan 25, 2023
18329a1
[skip ci] generalize advanced indexing incl. distributed DNDarray key
ClaudiaComito Jan 26, 2023
f024ebb
[skip ci] Expand tests combined advanced / basic indexing
ClaudiaComito Jan 29, 2023
6ae2788
[skip ci] fix advanced dimensional indexing on non-distributed array
ClaudiaComito Feb 5, 2023
178d7f8
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito Jul 26, 2023
09e586c
fix distr advanced indexing with broadcasted shape
ClaudiaComito Jul 27, 2023
c56ebf4
transpose without copying
ClaudiaComito Jul 29, 2023
86f704a
[skip ci] document __process_key(), clean up code
ClaudiaComito Aug 1, 2023
68ead71
[skip ci] docs edits
ClaudiaComito Aug 1, 2023
252995c
fix Ellipsis dimensions
ClaudiaComito Aug 4, 2023
c2a7e20
fix shape and split bookkeeping within advanced indexing
ClaudiaComito Aug 4, 2023
235a7b8
test adv indexing on non consecutive dims
ClaudiaComito Aug 4, 2023
4e936e8
abstract scalar key checks for both getitem and setitem
ClaudiaComito Aug 7, 2023
8a74cd9
setitem scalar key
ClaudiaComito Aug 8, 2023
8cf3ff1
DRAFT - abstraction common utilities for getitem and setitem
ClaudiaComito Aug 9, 2023
b45578a
handle all single-element indexing along split axis in same block
ClaudiaComito Aug 9, 2023
cec4bb9
resolve send/recv dimensions mismatch in a few edge cases
ClaudiaComito Aug 10, 2023
cc49a49
transpose self back to original shape after indexing
ClaudiaComito Aug 12, 2023
fe26ae8
add setitem tests
ClaudiaComito Aug 30, 2023
611b46d
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito Aug 30, 2023
affdb60
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito Nov 27, 2023
7e5be66
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito Dec 6, 2023
6d2e369
do not index input unnecessarily for sanitation
ClaudiaComito Dec 7, 2023
f528356
test named split dimension for torch_proxy
ClaudiaComito Dec 7, 2023
01a1140
value broadcasting abstraction
ClaudiaComito Dec 8, 2023
f8264a9
introduce distr sanitation for value when key is ordered
ClaudiaComito Dec 13, 2023
b1cd02f
keep track of original key
ClaudiaComito Dec 13, 2023
31bdb34
fix value broadcasting for advanced setitem
ClaudiaComito Dec 14, 2023
c4d6749
match broadcasting to numpy
ClaudiaComito Dec 16, 2023
5782d6e
finalize broadcast_value and fix test
ClaudiaComito Dec 20, 2023
2174e84
assignment to negative slice along split axis
ClaudiaComito Dec 20, 2023
782bde2
getitem: index underlying tensor with processed key in non-distr case
ClaudiaComito Jan 8, 2024
084371d
setitem: test neg step slice along non-zero split axis
ClaudiaComito Jan 8, 2024
b1aa7aa
allow for nominal value/self split mismatch
ClaudiaComito Jan 8, 2024
1c2b71e
expand test negative step along split axis
ClaudiaComito Jan 8, 2024
7201a89
allow value.ndim > indexed_dims if extra dims are singletons
ClaudiaComito Jan 12, 2024
dfc7266
BROKEN: expand negative step tests
ClaudiaComito Jan 12, 2024
8bbe242
squeeze out singleton dimensions when broadcasting value
ClaudiaComito Jan 15, 2024
00a17e6
fix negative step slicing on 1 process
ClaudiaComito Jan 15, 2024
bdd2dd8
setitem w. dimensional indexing, add tests
ClaudiaComito Jan 16, 2024
1fbd4d6
setitem w. advanced indexing on first dim
ClaudiaComito Jan 17, 2024
95d3c92
setitem: test boolean indexing, local and split=0
ClaudiaComito Jan 17, 2024
f335aa8
fix output shape for boolean indexing w. split>0
ClaudiaComito Jan 18, 2024
d520ddf
setitem with non-ordered, mask-like key and non-distr value
ClaudiaComito Jan 18, 2024
d754a9c
allow for partial boolean indexing on first key.ndim dims of array
ClaudiaComito Jan 19, 2024
5e69fe6
remove unnecessary check
ClaudiaComito Jan 19, 2024
8d9849e
add tests for partial boolean indexing
ClaudiaComito Jan 19, 2024
66ae371
set w. single-tensor key and non-distr value
ClaudiaComito Jan 22, 2024
980e8f0
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito Feb 5, 2024
ae4d423
non-ordered, non-mask-like key and local value
ClaudiaComito Feb 5, 2024
b695e5a
broken: set up comm map for full distributed setitem
ClaudiaComito Feb 7, 2024
e6c1e10
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Feb 7, 2024
d42f1cb
implement setitem w. distributed non-ordered key
ClaudiaComito Feb 8, 2024
7868fa0
[skip ci] broken: add tests for distr value non-ordered key
ClaudiaComito Feb 8, 2024
f8055ff
Merge branch '914_adv-indexing-outshape-outsplit' of github.com:helmh…
ClaudiaComito Feb 8, 2024
2944903
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Feb 8, 2024
1bffd26
__process_key(): refactor adv indexing tensor extraction
ClaudiaComito Feb 12, 2024
83842ec
working: setitem w. mask-like adv indexing, non-ordered split key
ClaudiaComito Feb 12, 2024
366aaf9
adapt tests
ClaudiaComito Feb 12, 2024
bbe0a7b
refactor __process_key(): address boolean ind within adv ind
ClaudiaComito Feb 12, 2024
24c6cd1
Merge branch '914_adv-indexing-outshape-outsplit' of github.com:helmh…
ClaudiaComito Feb 12, 2024
1c47b42
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Feb 12, 2024
4ee9b96
getitem: address mask-like key
ClaudiaComito Feb 15, 2024
3c88f8b
Merge branch '914_adv-indexing-outshape-outsplit' of github.com:helmh…
ClaudiaComito Feb 16, 2024
54db23d
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito Feb 16, 2024
15cce44
define nonzero_size in non-distr case
ClaudiaComito Feb 18, 2024
09fb199
handle split_bookkeeping when key is mask-like
ClaudiaComito Feb 18, 2024
9c8d051
fix key type mismatch in advanced indexing
ClaudiaComito Feb 18, 2024
41fba0a
getitem: address n-D key along split axis, free memory
ClaudiaComito Feb 18, 2024
e4a90de
balance indexed array before eq()
ClaudiaComito Feb 18, 2024
c8967e7
remove print statements
ClaudiaComito Feb 18, 2024
2d443d8
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito Feb 18, 2024
95eaaeb
test adv ind on non-consecutive dims
ClaudiaComito Feb 18, 2024
835a13f
remove print statement
ClaudiaComito Feb 20, 2024
216a1a0
setitem: mixed indexing w. shape broadcasting
ClaudiaComito Feb 20, 2024
b62bad2
expand tests for mixed indexing w. broadcasting
ClaudiaComito Feb 20, 2024
7ea2abe
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito Feb 20, 2024
435ff0c
reinstate tests for specific bugs
ClaudiaComito Feb 20, 2024
30efe59
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito Mar 8, 2024
e6679a0
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito Apr 10, 2024
ad96822
prep send_buffer - expand value dimension if necessary
ClaudiaComito Apr 10, 2024
c9d44ae
fix send_indices dims when key is not mask-like
ClaudiaComito Apr 10, 2024
cc70400
test split mismatch on comm.size > 1
ClaudiaComito Apr 11, 2024
b78de30
broadcasting assignment along split axis
ClaudiaComito Apr 11, 2024
a8f2d57
expand tests
ClaudiaComito Apr 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading