Batch-parallel K-means and K-medians #1288

mrfh92 · 2023-11-30T16:14:40Z

Due Diligence

General:
- base branch must be main for new features, latest release branch (e.g. release/1.3.x) for bug fixes
- title of the PR is suitable to appear in the Release Notes
Implementation:
- unit tests: all split configurations tested (here: only split=0 allowed)
- unit tests: multiple dtypes tested
- documentation updated where needed

Description

This PR implements a variant of K-Means that can take advantage of Heat data distribution structure. The algorithm is described in the paper https://doi.org/10.1016/j.cie.2020.107023. In essence, the idea is to perform K-means on each chunk of data separately in parallel as a first step. In the second step, the obtained cluster centers are "merged": the set of all process-local cluster centers is collected, and then another application of K-means is used to determine the final "global" cluster centers als cluster centers of the set of "local" cluster centers.
Of course, this idea generalizes to K-Medians in a straightforward way. To improve scalability of this approach even to very high number of processes, the "merging" can also be done in hierarchical manor.

Caveat: This does not necessarily yield the same results as the classical K-Means.

Issue/s resolved: #1287

Changes proposed:

New classes heat.cluster.BatchParallelKMeans and heat.cluster.BatchParallelKMedians

Type of change

new feature

Memory requirements

TBD

Performance

Example of ~130GB on up to 8 GPU-nodes of the Terrabyte-Cluster. Left plot shows standard K-means (4 clusters) and right plot shows BatchParallelKmeans (4 and 40 clusters)

Does this change modify the behaviour of other functions? If so, which?

no

ghost · 2023-11-30T16:15:24Z

👇 Click on the image for a new way to code review

Legend

…s due to instabilities

github-actions · 2023-12-01T09:51:56Z

Thank you for the PR!

codecov · 2023-12-01T10:18:46Z

Codecov Report

Attention: Patch coverage is 99.54955% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 92.06%. Comparing base (850dcfb) to head (6e9e9ec).

Files	Patch %	Lines
heat/cluster/_kcluster.py	94.73%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1288      +/-   ##
==========================================
+ Coverage   91.92%   92.06%   +0.14%     
==========================================
  Files          79       80       +1     
  Lines       11721    11941     +220     
==========================================
+ Hits        10774    10993     +219     
- Misses        947      948       +1

Flag	Coverage Δ
unit	`92.06% <99.54%> (+0.14%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2023-12-01T14:42:15Z

Thank you for the PR!

github-actions · 2023-12-01T16:10:00Z

Thank you for the PR!

github-actions · 2023-12-04T12:25:02Z

Thank you for the PR!

… for storing the value of the K-Clustering functional after predict

github-actions · 2023-12-04T13:06:47Z

Thank you for the PR!

github-actions · 2023-12-04T13:10:45Z

Thank you for the PR!

github-actions · 2023-12-04T13:21:19Z

Thank you for the PR!

… is now also possible in a hierarchical manor that scales better to high numbers of processes

github-actions · 2023-12-07T15:49:24Z

Thank you for the PR!

ClaudiaComito

@mrfh92 this is great.

Would it be possible or advisable to make batch-parallel clustering available from within a general ht.cluster.KMeans or ht.cluster.KMedians call? I'm thinking something like:

kmeans = ht.cluster.KMeans(n_clusters=k, parallel="batch")

where parallel can be "batch" or "standard".

github-actions · 2024-03-01T11:45:14Z

Thank you for the PR!

github-actions · 2024-03-11T10:13:20Z

Thank you for the PR!

…edians

github-actions · 2024-03-11T10:19:09Z

Thank you for the PR!

…edians

github-actions · 2024-03-25T09:17:25Z

Thank you for the PR!

…edians

github-actions · 2024-04-08T08:35:49Z

Thank you for the PR!

…edians

github-actions · 2024-04-11T07:59:58Z

Thank you for the PR!

ClaudiaComito

Looks great to me @mrfh92 , I only have a few edits to the documentation. Otherwise we can merge.

heat/cluster/batchparallelclustering.py

Co-authored-by: Claudia Comito <[email protected]>

github-actions · 2024-04-11T08:54:53Z

Thank you for the PR!

github-actions · 2024-04-11T08:56:24Z

Thank you for the PR!

github-actions · 2024-04-11T08:56:32Z

Thank you for the PR!

…edians

mrfh92 · 2024-04-11T09:20:12Z

@ClaudiaComito I have committed all suggested changes.

github-actions · 2024-04-11T09:24:22Z

Thank you for the PR!

ClaudiaComito

👏 @mrfh92 !

added batch-parallel k-means and k-medians

fa6f778

Hoppe added 3 commits November 30, 2023 17:15

just removed debugging prints

dac4622

...

c884329

some bug fixes, removed option init="random" for BatchParallelKMedian…

b3410e3

…s due to instabilities

mrfh92 self-assigned this Dec 1, 2023

mrfh92 added cluster enhancement New feature or request labels Dec 1, 2023

mrfh92 added the high-level functions High-level machine-learning algorithms label Dec 1, 2023

further bug fixes, removed option init="random" completely

48bb7a0

found out that unit tests must start with "test..."...

92b204e

added tests for some nonconvered lines

0c0317c

Hoppe added 2 commits December 4, 2023 14:02

added attribute "functional_value_" to BatchParallelKClustering class…

f804c84

… for storing the value of the K-Clustering functional after predict

added one missing test...

786bb3d

small bug fix

f35420b

ClaudiaComito added this to the 1.4.0 milestone Dec 6, 2023

the "merging" process of the "local" cluster centers to "global" ones…

8e8390b

… is now also possible in a hierarchical manor that scales better to high numbers of processes

mrfh92 marked this pull request as ready for review December 8, 2023 08:25

mrfh92 requested review from mtar and ClaudiaComito December 8, 2023 08:54

ClaudiaComito requested changes Dec 11, 2023

View reviewed changes

mrfh92 added the PR talk label Mar 4, 2024

added docs for batchparallel initialization in KMeans and KMedians

4ccb8a3

Merge branch 'main' into features/1287-Batch-parallel_K-means_and_K-m…

575ad4f

…edians

mrfh92 removed the PR talk label Mar 25, 2024

Merge branch 'main' into features/1287-Batch-parallel_K-means_and_K-m…

7ba1152

…edians

mrfh92 added the Review me label Apr 8, 2024

Merge branch 'main' into features/1287-Batch-parallel_K-means_and_K-m…

8933eb3

…edians

Merge branch 'main' into features/1287-Batch-parallel_K-means_and_K-m…

e3ed04f

…edians

ClaudiaComito requested changes Apr 11, 2024

View reviewed changes

heat/cluster/batchparallelclustering.py Outdated Show resolved Hide resolved

heat/cluster/batchparallelclustering.py Outdated Show resolved Hide resolved

heat/cluster/batchparallelclustering.py Outdated Show resolved Hide resolved

mrfh92 and others added 3 commits April 11, 2024 10:50

Update heat/cluster/batchparallelclustering.py

f427ca4

Co-authored-by: Claudia Comito <[email protected]>

Update heat/cluster/batchparallelclustering.py

e7e0f80

Co-authored-by: Claudia Comito <[email protected]>

Update heat/cluster/batchparallelclustering.py

1c642c3

Co-authored-by: Claudia Comito <[email protected]>

Merge branch 'main' into features/1287-Batch-parallel_K-means_and_K-m…

6e9e9ec

…edians

ClaudiaComito approved these changes Apr 12, 2024

View reviewed changes

ClaudiaComito added merge queue and removed Review me labels Apr 12, 2024

mrfh92 merged commit ef44340 into main Apr 12, 2024
56 checks passed

mrfh92 deleted the features/1287-Batch-parallel_K-means_and_K-medians branch April 12, 2024 12:34

mrfh92 mentioned this pull request Apr 12, 2024

Batch-parallel K-means and K-medians #1287

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Batch-parallel K-means and K-medians #1288

Batch-parallel K-means and K-medians #1288

mrfh92 commented Nov 30, 2023 •

edited

Loading

ghost commented Nov 30, 2023 •

edited by ghost

Loading

Legend

github-actions bot commented Dec 1, 2023

codecov bot commented Dec 1, 2023 •

edited

Loading

github-actions bot commented Dec 1, 2023

github-actions bot commented Dec 1, 2023

github-actions bot commented Dec 4, 2023

github-actions bot commented Dec 4, 2023

github-actions bot commented Dec 4, 2023

github-actions bot commented Dec 4, 2023

github-actions bot commented Dec 7, 2023

ClaudiaComito left a comment •

edited

Loading

github-actions bot commented Mar 1, 2024

github-actions bot commented Mar 11, 2024

github-actions bot commented Mar 11, 2024

github-actions bot commented Mar 25, 2024

github-actions bot commented Apr 8, 2024

github-actions bot commented Apr 11, 2024

ClaudiaComito left a comment

github-actions bot commented Apr 11, 2024

github-actions bot commented Apr 11, 2024

github-actions bot commented Apr 11, 2024

mrfh92 commented Apr 11, 2024

github-actions bot commented Apr 11, 2024

ClaudiaComito left a comment

Batch-parallel K-means and K-medians #1288

Batch-parallel K-means and K-medians #1288

Conversation

mrfh92 commented Nov 30, 2023 • edited Loading

Due Diligence

Description

Changes proposed:

Type of change

Memory requirements

Performance

Does this change modify the behaviour of other functions? If so, which?

ghost commented Nov 30, 2023 • edited by ghost Loading

Legend

github-actions bot commented Dec 1, 2023

codecov bot commented Dec 1, 2023 • edited Loading

Codecov Report

github-actions bot commented Dec 1, 2023

github-actions bot commented Dec 1, 2023

github-actions bot commented Dec 4, 2023

github-actions bot commented Dec 4, 2023

github-actions bot commented Dec 4, 2023

github-actions bot commented Dec 4, 2023

github-actions bot commented Dec 7, 2023

ClaudiaComito left a comment • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Mar 1, 2024

github-actions bot commented Mar 11, 2024

github-actions bot commented Mar 11, 2024

github-actions bot commented Mar 25, 2024

github-actions bot commented Apr 8, 2024

github-actions bot commented Apr 11, 2024

ClaudiaComito left a comment

Choose a reason for hiding this comment

github-actions bot commented Apr 11, 2024

github-actions bot commented Apr 11, 2024

github-actions bot commented Apr 11, 2024

mrfh92 commented Apr 11, 2024

github-actions bot commented Apr 11, 2024

ClaudiaComito left a comment

Choose a reason for hiding this comment

mrfh92 commented Nov 30, 2023 •

edited

Loading

ghost commented Nov 30, 2023 •

edited by ghost

Loading

codecov bot commented Dec 1, 2023 •

edited

Loading

ClaudiaComito left a comment •

edited

Loading