Merging Topic Models

[jwegner99]

Hello,

I am trying to topic model approximately ~4.5 million tweets.

I have been unable to successfully do so given the memory requirements in UMAP. I have tried the suggested solutions to memory problems (i.e., low_memory=True, setting min_df, and setting calculate probabilities to False). I also have tried online topic modelling, but the topics seem less sensible as compared to those in the traditional full model.

I am currently trying two new solutions, one is merging topic models and the other is to train on a sample and then transforming on the rest. My question is, with these methods, is there a way to then assign each tweet in an original df a topic. I know the index length remains the same with the traditional bertopic, but I wasn't sure that was the case in merged models.

Thank you for your help!

[MaartenGr]

Have you also tried using cuML? It is a GPU-enabled method for UMAP and HDBSCAN which are typically the main bottlenecks with respect to speed? You can find a nice guide here.

I am currently trying two new solutions, one is merging topic models and the other is to train on a sample and then transforming on the rest. My question is, with these methods, is there a way to then assign each tweet in an original df a topic. I know the index length remains the same with the traditional bertopic, but I wasn't sure that was the case in merged models.

When you merge models, it should retain the order of assignment according to the order of models that you put in the models parameter. I would generally advise merging models since that allows you to go over all of the data and create multiple decently-sized models to combine. I wouldn't advise merging model if you generate hundreds of tiny models since you would end up with a lot of micro-clusters.