Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

release 24.04 [skip ci] #654

Merged
merged 47 commits into from
May 10, 2024
Merged

release 24.04 [skip ci] #654

merged 47 commits into from
May 10, 2024

Conversation

YanxuanLiu
Copy link
Collaborator

Merge to main for 24.04 release

Release notes as follows:

  • Feature standardization in logistic regression for sparse vectors.
  • GPU accelerated Density Based Spatial Clustering for Applications with Noise (DBSCAN) algorithm with example notebook.
  • GPU accelerated IVF-Flat Approximate Nearest Neighbor algorithm with example notebook
  • Stage level scheduling support for Yarn and K8s.
  • Update of RAPIDS dependencies to 24.04.

NOTE: this PR must be merged as create a merge commit

nvauto and others added 30 commits January 24, 2024 18:08
@nvauto
Merge pull request #555 from NVIDIA/branch-24.02
f6470e2 
[auto-merge] branch-24.02 to branch-24.04 [skip ci] [bot]
@nvauto
Merge pull request #557 from NVIDIA/branch-24.02
2f588af 
[auto-merge] branch-24.02 to branch-24.04 [skip ci] [bot]
@nvauto
Merge pull request #559 from NVIDIA/branch-24.02
a360488 
[auto-merge] branch-24.02 to branch-24.04 [skip ci] [bot]
@nvauto
Merge pull request #561 from NVIDIA/branch-24.02
ee68cb9 
[auto-merge] branch-24.02 to branch-24.04 [skip ci] [bot]
@nvauto
Merge pull request #564 from NVIDIA/branch-24.02
fe68115 
[auto-merge] branch-24.02 to branch-24.04 [skip ci] [bot]
@nvauto
Merge pull request #567 from NVIDIA/branch-24.02
37adfdd 
[auto-merge] branch-24.02 to branch-24.04 [skip ci] [bot]
@nvauto
Merge pull request #571 from NVIDIA/branch-24.02
1eb3466 
[auto-merge] branch-24.02 to branch-24.04 [skip ci] [bot]
@nvauto
Merge pull request #572 from NVIDIA/branch-24.02
2f11551 
[auto-merge] branch-24.02 to branch-24.04 [skip ci] [bot]
@nvauto
Merge pull request #575 from NVIDIA/branch-24.02
f1a0bf9 
[auto-merge] branch-24.02 to branch-24.04 [skip ci] [bot]
@nvauto
Merge pull request #577 from NVIDIA/branch-24.02
8630114 
[auto-merge] branch-24.02 to branch-24.04 [skip ci] [bot]
@nvauto
Merge pull request #580 from NVIDIA/branch-24.02
ed73617 
[auto-merge] branch-24.02 to branch-24.04 [skip ci] [bot]
@nvauto
Merge pull request #583 from NVIDIA/branch-24.02
3b1fdaa 
[auto-merge] branch-24.02 to branch-24.04 [skip ci] [bot]
@nvauto
Merge pull request #584 from NVIDIA/branch-24.02
3ebb856 
[auto-merge] branch-24.02 to branch-24.04 [skip ci] [bot]
@NvTimLiu
Move K8s cloud name into common lib for Jenkins CI (#582)
680e261 
Signed-off-by: Tim Liu <[email protected]>
@nvauto
Merge pull request #589 from NVIDIA/branch-24.02
1a1ad7b 
[auto-merge] branch-24.02 to branch-24.04 [skip ci] [bot]
@nvauto
Merge pull request #590 from NVIDIA/branch-24.02
5be6546 
[auto-merge] branch-24.02 to branch-24.04 [skip ci] [bot]
@nvauto
Merge pull request #592 from NVIDIA/branch-24.02
219cddf 
[auto-merge] branch-24.02 to branch-24.04 [skip ci] [bot]
@YanxuanLiu
add cloud guardword (#594)
b6556a2 
Signed-off-by: YanxuanLiu <[email protected]>
@NvTimLiu
Auto merge PRs to branch-24.06 from branch-24.04 [skip ci] (#595)
6989591 
Signed-off-by: Tim Liu <[email protected]>
@Er1cCheng
DBSCAN Algorithm for Spark RAPIDS ML (#598)
ca2975e 
* DBSCAN basis

* Precomputed distance testcase, usecase comments

Signed-off-by: nvssh nssswitch user account <[email protected]>

* Code structure cleaning and update for ColID extraction

* testfile fix

* Remove Precomputed mode support

* idCol fix

* Syntax fix for CI python version

* ColID fix, code cleaning for sparse vector input

* Avoid core indices calc kernel by default

---------

Signed-off-by: nvssh nssswitch user account <[email protected]>
Co-authored-by: nvssh nssswitch user account <[email protected]>
@Er1cCheng
kNN update for colID extraction (#601)
ca730ec 
* kNN update for colID extraction

* comment fix

Signed-off-by: nvssh nssswitch user account <[email protected]>

---------

Signed-off-by: nvssh nssswitch user account <[email protected]>
Co-authored-by: nvssh nssswitch user account <[email protected]>
@YanxuanLiu
update actions version (#597)
0d7743c 
Signed-off-by: YanxuanLiu <[email protected]>
@YanxuanLiu
revert action version (#609)
dafe207 
Signed-off-by: YanxuanLiu <[email protected]>
@Er1cCheng
knn overwrite for colID extraction (#612)
d17ad0c 
Signed-off-by: Hongzhe Cheng <[email protected]>
@Er1cCheng
Sparse Datagen Fix (#607)
41f6c2b 
* Datagen Fix

* Auth fix

Signed-off-by: nvssh nssswitch user account <[email protected]>

* Auth fix

Signed-off-by: Hongzhe Cheng <[email protected]>

---------

Signed-off-by: nvssh nssswitch user account <[email protected]>
Signed-off-by: Hongzhe Cheng <[email protected]>
Co-authored-by: nvssh nssswitch user account <[email protected]>
@Er1cCheng
DBSCAN code move, notebook, benchmark (#608)
d3be33c 
* DBSCAN notebook, benchmark script

* Move DBSCAN to clustering

Signed-off-by: Hongzhe Cheng <[email protected]>

* File check in

* benchmark script support

* Benchmark parameter fix

* Transform time fix

* DBSCAN data broadcast fix, comment fix

* separate change for DBSCAN source code fix

* cmdline switch for benchmark score compute

* style fix

---------

Signed-off-by: Hongzhe Cheng <[email protected]>
@NvTimLiu
Remove build link from the pre-merge-CI workflow (#605)
6408c52 
We need to remove all the CI job's link from the pre-merge-CI workflow for security concern.

Signed-off-by: Tim Liu <[email protected]>
@Er1cCheng
DBSCAN broadcast fix (#617)
6e1c0d1 
* DBSCAN broadcast fix

Signed-off-by: Hongzhe Cheng <[email protected]>

* Comment delete

---------

Signed-off-by: Hongzhe Cheng <[email protected]>
@eordentlich
update to rapids 24.04; temp. patch dbscan test till new algorithm pa…
3d50d0e 
…rameter is supported (#619)

Signed-off-by: Erik Ordentlich <[email protected]>
@Er1cCheng
Algorithm parameter support for DBSCAN (#622)
7202ff4 
* Add in algorithm parameter support for DBSCAN

Signed-off-by: Hongzhe Cheng <[email protected]>

* comment fix

---------

Signed-off-by: Hongzhe Cheng <[email protected]>
wbo4958 and others added 17 commits April 17, 2024 07:54
@wbo4958
[data-gen] add example in the help (#621)
793851a 
Signed-off-by: Bobby Wang <[email protected]>
@eordentlich
work around to support pyspark 3.3 with rapids 24.04 (#623)
d487be3 
Signed-off-by: Erik Ordentlich <[email protected]>
@Er1cCheng
Databricks notebook readme fix (#627)
9e95ef8 
Signed-off-by: Hongzhe Cheng <[email protected]>
@lijinf2
Support standardization for sparse matrix in logistic regression (#629)
49f5069 
* support standardization for sparse vectors per cuml 24.04

* revise test cases to test sparse standardization

* revise

* revise docstring regarding sparse standardization

---------

Signed-off-by: Jinfeng <[email protected]>
@eordentlich
add dbscan to api docs (#600)
4d58143 
* add dbscan to api docs

Signed-off-by: Erik Ordentlich <[email protected]>

* make consistent with source changes

Signed-off-by: Erik Ordentlich <[email protected]>

---------

Signed-off-by: Erik Ordentlich <[email protected]>
@eordentlich
keep eval related computations and data on gpu (#587)
6d7ff10 
* keep eval related computations and data on gpu

Signed-off-by: Erik Ordentlich <[email protected]>

* extend to regression and rf

Signed-off-by: Erik Ordentlich <[email protected]>

* clean up

Signed-off-by: Erik Ordentlich <[email protected]>

* fix types

Signed-off-by: Erik Ordentlich <[email protected]>

---------

Signed-off-by: Erik Ordentlich <[email protected]>
@Er1cCheng
DBSCAN support for 64bit indextype (#634)
8432305 
Signed-off-by: Hongzhe Cheng <[email protected]>
@Er1cCheng
Log print fix for logistic regression (#636)
96de8d2 
Signed-off-by: Hongzhe Cheng <[email protected]>
@Er1cCheng
DBSCAN notebook Twitter exmaple (#638)
9f9b4e6 
* Twitter DBSCAN exmaple

Signed-off-by: Hongzhe Cheng <[email protected]>

* Parquet Save

---------

Signed-off-by: Hongzhe Cheng <[email protected]>
@wbo4958
Support stage-level scheduling for Yarn and K8s (#640)
97826c3 
Signed-off-by: Bobby Wang <[email protected]>
@lijinf2
Add Approximate Nearest Neighbors algorithm (i.e. IVF_FLAT) from Cuml (…
c12d1b4 
…#630)

* Get toy example working

* square ivfflat dists, add join API, add test with/without setting idCol

* test key APIs, and add parametrize

* fix a bug relates to returned id

* move dictionary typeconverter to a class

* revised per comments that can be easily addressed

* remove brute option from approximatenearestneighbors

* add example and docstring to Estimator class and tested the examples in pyspark shell

* add docstring to kneighbors and approxSimilarityJoin

* reuse code: exact knn working

* get ann working, runslow tested

* test getter setter

* support metric argument and all values except cosine and correlation

* fix mypy error

---------

Signed-off-by: Jinfeng <[email protected]>
@eordentlich
add approx knn to api docs [skip ci] (#644)
b244341 
* minor doc updates

Signed-off-by: Erik Ordentlich <[email protected]>

* fix bullets in doc string

Signed-off-by: Erik Ordentlich <[email protected]>

---------

Signed-off-by: Erik Ordentlich <[email protected]>
@lijinf2
Fix NearestNeighbors _ensureIdCol to check whether id_col in df.colum…
1788e9d 
…ns instead of relying on isSet(idCol) (#642)

* fix ensureIdCol to avoid using isSet(idCol)

* simply the logic of ensureIdCol

* try set idCol to None

---------

Signed-off-by: Jinfeng <[email protected]>
@YanxuanLiu
update blossom-ci auth list (#647)
9fff486 
Signed-off-by: YanxuanLiu <[email protected]>
@lijinf2
add notebook for ann and remove exactNearestNeighborsJoin API from AN…
c59795a 
…N class (#650)

Signed-off-by: Jinfeng <[email protected]>
@eordentlich
remove unsupported save,load,read,write from api docs for knn estimat… (
0198d3f 
#646)

* remove unsupported save,load,read,write from api docs for knn estimator, model classes

Signed-off-by: Erik Ordentlich <[email protected]>

* fix class names in error messages

Signed-off-by: Erik Ordentlich <[email protected]>

* typo

Signed-off-by: Erik Ordentlich <[email protected]>

---------

Signed-off-by: Erik Ordentlich <[email protected]>
@GaryShen2008
Add one more guardword for uploaded log (#586)
ac4785c 
Signed-off-by: Gary Shen <[email protected]>
@YanxuanLiu YanxuanLiu requested a review from eordentlich May 10, 2024 01:42
@YanxuanLiu YanxuanLiu self-assigned this May 10, 2024
@YanxuanLiu
Copy link
Collaborator Author

build

eordentlich
eordentlich approved these changes May 10, 2024
View reviewed changes
Copy link
Collaborator

@eordentlich eordentlich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@YanxuanLiu YanxuanLiu merged commit df01b39 into main May 10, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@eordentlich eordentlich eordentlich approved these changes

Assignees

@YanxuanLiu YanxuanLiu

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@YanxuanLiu @eordentlich @nvauto @NvTimLiu @Er1cCheng @wbo4958 @lijinf2 @GaryShen2008