Time series of topics for large texts and a response variable

[dragonattheend]

Thank you for making BERTopic available and really helpful website and code examples.

I have a time-series of fairly large texts, hundreds/thousands of words, I have four columns in my dataset: id, time, text and response.

I've been looking for a way to get topics for each text in id and time and then relate them (their probabilities and changes in these probabilities) to my response variable. I want to see whether certain topics and/or change in the intensity of their discussion over time has an association with my response variable (and its changes).

Since my texts are large, I split them into sentences. Half of them got assigned to topic -1 and I intend to remove them.

Is it okay to group and average the probabilities of the remaining topics? Will it be representative of the whole text?

Is it okay to take time differences between probabilities for each topic?

It is very possible that I'm missing something and there is a simpler approach to what I'm trying to do. Please suggest if so. Thanks again!

[MaartenGr]

Is it okay to group and average the probabilities of the remaining topics? Will it be representative of the whole text?

That generally should be okay if the sub-documents are of a relatively equal size.

Is it okay to take time differences between probabilities for each topic?

If I understand you correctly, then I think so yes since the dynamic topic modeling in BERTopic is doing something similar.

It is very possible that I'm missing something and there is a simpler approach to what I'm trying to do. Please suggest if so. Thanks again!

If you are looking for statistically comparing response variable then it might be worthwhile to check out this thread that demonstrates the use of covariate analysis in BERTopic.

[dragonattheend]

Thank you, this is very helpful.

Do you think it is okay to run separate models for each id and then try to overlap their results, like here?

[MaartenGr]

You could do that but since their representations are meant to be quite different at certain timestamps, I am not sure whether that would work. Definitely worth trying out though.