soft clustering using gaussian mixtures

[Rosyparadise]

Hello and thank you for your hard work on bertopic. It really is amazing!

I was wondering if it's possible to use a gaussian mixtures model with soft clustering instead of HDBSCAN with the calculate_probabilities=True option. In my understanding, bertopic only supports hard clustering unless its the built-in soft clustering of HDBSCAN.

Is this true? If not, could you point me to the right direction? It would be of immense help.

[MaartenGr]

Thank you for the kind words!

I was wondering if it's possible to use a gaussian mixtures model with soft clustering instead of HDBSCAN with the calculate_probabilities=True option. In my understanding, bertopic only supports hard clustering unless its the built-in soft clustering of HDBSCAN.

If you do not use HDBSCAN, you can still extract the cosine similarity scores between the documents and topics using .transform if you set the clustering algorithm to a BaseCluster() after training or if you save and load the model using safetensors.

Having said that, it indeed does not support the predict_proba functionality of clustering algorithms in sklearn yet. You can still use it to create the clusters but for the probabilities, you will have to first reduce the dimensionality of your input data and then run topic_model.hdbscan_model.predict_proba on the reduced embeddings.