Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] expose python & C API for prefiltered brute force #174

Merged
merged 25 commits into from
Jul 29, 2024

Conversation

rhdong
Copy link
Member

@rhdong rhdong commented Jun 5, 2024

No description provided.

@rhdong rhdong requested review from benfred and cjnolet June 5, 2024 00:05
@rhdong rhdong requested review from a team as code owners June 5, 2024 00:05
@rhdong rhdong added feature request New feature or request non-breaking Introduces a non-breaking change C and removed cpp CMake labels Jun 5, 2024
Copy link
Member

@benfred benfred left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks great! thanks for getting this in.

some minor comments / feedback below:

python/cuvs/cuvs/neighbors/brute_force/brute_force.pyx Outdated Show resolved Hide resolved
python/cuvs/cuvs/neighbors/brute_force/brute_force.pxd Outdated Show resolved Hide resolved
python/cuvs/cuvs/neighbors/prefilters/prefilters.pyx Outdated Show resolved Hide resolved
Copy link

copy-pr-bot bot commented Jun 6, 2024

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@rhdong rhdong requested a review from benfred June 6, 2024 01:49
@rhdong
Copy link
Member Author

rhdong commented Jun 6, 2024

/ok to test

@benfred benfred requested a review from a team as a code owner June 6, 2024 19:18
@rhdong rhdong requested a review from benfred June 20, 2024 05:37
Copy link
Member

@benfred benfred left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm! thanks for this,

@benfred
Copy link
Member

benfred commented Jun 25, 2024

/ok to test

@rhdong
Copy link
Member Author

rhdong commented Jun 26, 2024

/ok to test

@rhdong
Copy link
Member Author

rhdong commented Jun 26, 2024

/ok to test

@rhdong rhdong requested a review from benfred June 26, 2024 22:32
Copy link
Member

@cjnolet cjnolet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this API is coming along great! Moslty minor things at this point, but I think this should be ready very soon.

cpp/include/cuvs/neighbors/prefilters.h Outdated Show resolved Hide resolved
*/
enum cuvsPrefilterType {
/* No filter */
NO_FILTER,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will need a way to specify a custom filter function (maybe to accept a function pointer?) also. While bfknn currently only supports bitmap / bitset, we also need to expose the filtering APIs for the other ANN indexes (most ideally through the same API).

I'm okay pushing this to a future change, since it's going to be needed ASAP so that we can expose the remaining filtering functions for the other algorithm. Can you create an issue for this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I will!

cpp/test/neighbors/brute_force_c.cu Outdated Show resolved Hide resolved
python/cuvs/cuvs/neighbors/prefilters/__init__.py Outdated Show resolved Hide resolved
python/cuvs/cuvs/neighbors/prefilters/prefilters.pxd Outdated Show resolved Hide resolved
python/cuvs/cuvs/neighbors/prefilters/prefilters.pyx Outdated Show resolved Hide resolved
@rhdong
Copy link
Member Author

rhdong commented Jul 2, 2024

/ok to test

@rhdong rhdong requested a review from cjnolet July 2, 2024 02:27
@@ -90,48 +302,167 @@ void recall_eval(T* query_data,
min_recall));
};

template <typename T, typename IdxT, typename bitmap_t = uint32_t>
void recall_eval_with_filter(T* query_data,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please make sure these are testing all possible combinations of the filter struct/type? Eg NULL, NO_FILTER, BITMAP, and BITSET?

We should set a good precedent for testing these early, so that all follow-on filtering implementations can do the same thing.

Copy link
Member Author

@rhdong rhdong Jul 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that makes sense. I just checked the code, the NO_FILTER, BITMAP have been covered(pls refer to https://github.com/rapidsai/cuvs/pull/174/files#diff-1b162130023c13d62c06d91f3aab663d943091dddd8a38813c27f65e7d6c61dfR93), and the BITSET is reserved for other algorithms to compatible which is not be implemented in brute-force. Would you like me to implement BITSET in brute-force and test it?

@rhdong rhdong requested a review from cjnolet July 4, 2024 21:36
Copy link
Member

@benfred benfred left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@benfred
Copy link
Member

benfred commented Jul 17, 2024

/ok to test

@benfred
Copy link
Member

benfred commented Jul 22, 2024

/ok to test

@cjnolet
Copy link
Member

cjnolet commented Jul 29, 2024

/merge

@rapids-bot rapids-bot bot merged commit 2517826 into rapidsai:branch-24.08 Jul 29, 2024
59 checks passed
divyegala pushed a commit to divyegala/cuvs that referenced this pull request Jul 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C CMake cpp feature request New feature or request non-breaking Introduces a non-breaking change Python
Projects
Development

Successfully merging this pull request may close these issues.

3 participants