[number of topics] Who is correct, my supervisor or myself

[SongLin99]

Dear Author,

Hello!

I am a master's student using BERTopic for bibliometric analysis, with the theme of understanding how AI can help the development of medicine.
I obtained abstracts of 10000 articles and analyzed them
I have obtained 20 topics and 4000 articles that belong to TOPIC-1
But my supervisor told me,

- I should only obtain 6-8 larger TOPICs, so that they can be better understood
- I cannot have 4000 articles that have not been classified

However,in my opinion

- I should obtain more small TOPICs (20+) and then summarize them into 6-7 large TOPICs through hierarchical modeling or manual analysis
- There is no problem with 4000 unclassified documents

Dear developers and researchers with bibliometric experience, could you please take a look and see who is more accurate？Thank you！

Example of BIG TOPIC ("Medical Image Processing")
Example of Small TOPIC("CNN and Brain Imaging", "CT and Lung Disease")

[MaartenGr]

Let me start by saying that I'm certainly not going to go between your supervisor and yourself. I don't think that's a particularly helpful way of approaching a problem like this. Rather, I believe it's best to highlight the different perspectives that you can take and evaluate them according to the standards/metrics/philosophy that fits best with the particular use case or application.

That said, there are two things mentioned.

The number and size of topics

There is something to be said for both views.

If you directly get 6-7 large topics, then they are typically modeled directly using the underlying clustering (which is not the case with the hierarchical approach). Moreover, from a psychological perspective, sometimes having fewer topics helps readers/users understand more easily what the corpus is about. Information overload is a real thing that shouldn't be underestimated.

Smaller topics, and then using hierarchical topic modeling, is a nice strategy for getting multi-level topics. It gives you a more thorough understand of high level topics by looking at the smaller topics they contain. Because the strategy for merging topics is just finding the best (even among all poor choices), you cannot be certain that 6-7 topics you obtain is the same quality as those you would get by directly modeling 6-7 topics. You can, however, more easily identify when the model should stop merging topics. This is an important feature as 6-7 topics might not be ideal. What about 15? Or 45? With hierarchical topic modeling, you can more easily identify this.

Outliers

Documents not classified isn't inherently good or bad and again depends on both the use case as well as your perspective. From a topic representation perspective (i.e., what are the names of the topics?), I see no problems when you ignore a part of the corpus since we are only interested in representing the topics and naming them.

For topic assignment (i.e., which topic belongs to which document), I can definitely imagine wanting to reduce the number of outliers, especially if those documents are quite important.

But here's the thing, when you run .fit_transform, you are primarily doing topic representation and actually not topic assignment. The main focus is on creating these nice topic representations and finding topics, topic assignment is more of a side-effect of this process.

As such, you can easily just reduce outliers after .fit_transform, which I believe is a valid technique of reducing outliers.

[SongLin99]

Thank you very much @MaartenGr , i will try both way to find a better solution.