How to calculate perplexity value?

[powerhorse1986]

Hi Maarten,

How are you? So sorry to bother you again. Recently our group finished a project by using BERTopic and submitted a manuscript. One of the reviewers insisted that we must calculate the perplexity score. I totally have no idea how to get it done. Would you please give some hints? Thank you. And, happy new year in advance.

Li

[MaartenGr]

Does the reviewer also have a suggestion of how to calculate the perplexity? BERTopic is not a probabilistic model and as such does not (directly) calculate word probabilities / document-topic probabilities the same way that something like LDA does.

That said, you can approximate the word probabilities by using the c-TF-IDF matrix for each topic and push them between 0 and 1. Then, you can get the document-topic probabilities by setting calculate_probabilities=True.

So I think it would be possible to do so and get the perplexity, I'm just not sure how meaningful it would be.

[powerhorse1986]

Hi Maarten,

Thanks for your response. Unfortunately, the reviewer did not suggest us how to calculate the perplexity. We will try to calculate the coherence scores, which you showed how to get them in other thread. And we will quote your comment to convince the reviewer adopt the coherence score as alternative to the perplexity score.

Thank you again.
Li