Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance/skmeans convergence #285

Merged
merged 2 commits into from
Oct 31, 2024

Conversation

weefuzzy
Copy link
Member

@weefuzzy weefuzzy commented Oct 30, 2024

Responding to queries from @tremblap about how well skmeans was behaving, this makes two changes

  1. Normalize the input data to the unit sphere. Otherwise taking the dot product of the centroids and data doesn't yield cosine similarities.
  2. When the embedding matrix is made sparse after assignment, the assigned cells should be 1 if the subsequent multiplication with the data is to yield usefully convergent means (well, sums in this case).

This PR to be followed closely with an update to the initialisation.


Note to future me:

There's maybe a more Eigenonic way to do (2) that can get us a sparse matrix to make updating the means more efficient. But for now I've just kept the changes as small as possible.

    // an expression that will return a matrix with each col filled with m's column-wise maxima 
    auto maxes = m.colwise().maxCoeff().replicate(m.rows(),1);
   // convert to 0s and 1s by testing for equality with the maxima 
    SparseMatrix<double> s =  (m.array() == maxes.array()).matrix().cast<double>().sparseView(); 

(With this, we could probably also get rid of a separate assignClusters step, as that would be encoded in the sparse matrix. However, now that I re-read it, I see that there's a possible ambiguous situation when a column has two exactly equal values that are maxima...)

before computing the means, otherwise convergence is hindered.
@weefuzzy weefuzzy requested a review from tremblap October 30, 2024 23:34
@weefuzzy weefuzzy self-assigned this Oct 30, 2024
@tremblap
Copy link
Member

it helps a little indeed. looking forward to get the next part

@weefuzzy weefuzzy merged commit c76186c into flucoma:main Oct 31, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants