Guided topic modelling changes the embeddings object in my environment

[zilch42]

Hi MaartenGr,

I have noticed something that I am having trouble understanding and wondering if you can help. When using guided topic modelling using seed_topic_list and pre-calculated embeddings, after fitting the model the embeddings object in my environment is actually changed, even though it's not reassigned in my code (as far as I can tell).

I am relatively new to Python (I come from R land), but how is this happening? I would have thought that when calling topics, probs = topic_model.fit_transform(docs, embeddings) the only objects that could change in my environment were topic_model, topics, and probs.

from bertopic import BERTopic
from sklearn.datasets import fetch_20newsgroups
from sentence_transformers import SentenceTransformer
import numpy as np

data = fetch_20newsgroups(subset='all',  remove=('headers', 'footers', 'quotes'))
docs = data['data']
seed_topics = np.unique(data['target_names']).tolist()
seed_topics = [t.split(".") for t in seed_topics]

embedding_model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = embedding_model.encode(docs, show_progress_bar=True)
embeddings[0,0:5]

>>>array([ 0.00207799,  0.02345042,  0.02480884, -0.01011017,  0.04626137], dtype=float32)

topic_model = BERTopic(embedding_model=embedding_model, 
                       seed_topic_list=seed_topics,
                       calculate_probabilities=False)
topics, probs = topic_model.fit_transform(docs, embeddings)
embeddings[0,0:5]

>>>array([-0.0108137 ,  0.01694239,  0.017736  , -0.02159058,  0.03590234], dtype=float32)

I'm guessing that this is happening because guided topic modelling actually nudges the embeddings in different directions based on the seeds and it is important to have those changed embeddings if later supplying them to another function like visualise_documents. Is that correct...? But I have had a look at the code and still don't understand exactly how my embeddings object is being changed.

The reason I discovered this was because I was lazily refitting the model without reinitialising BERTopic() to investigate the variability in UMAP, expecting it to just retrain the model from scratch, but I got very different document visualisations after the first and second fittings.

Initial fit:

2nd fit:

(note: this is not the newsgroups data)

The second one is actually very pretty so I am wondering what's going on here? Is each subsequent training causing the document embeddings to converge around the seeds, leading to less local variation which means UMAP has an easier time evenly distributing the topics among a smaller set of dimensions?

The other curious thing is that the second fitting take substantially longer to run than the first (2-10x longer with 40k docs).

[MaartenGr]

Nice catch! It indeed seems that the embeddings get overwritten in the calculation of the topics. In practice, the document embeddings get averaged with the guided topic word embeddings in order to nudge them towards the same topic. You can find more about this process here. It seems that there are some namespace issues that cause the embeddings to also chance outside of BERTopic, which is not intentional as those need to remain the same. I'll have to look into that to see where this is happening. My guess though, is that this is the culprit:

BERTopic/bertopic/_bertopic.py

Line 2967 in d665d3f

embeddings[indices] = np.average([embeddings[indices], seed_topic_embeddings[seed_topic]], weights=[3, 1])

[zilch42]

Thanks Maarten!