Coherence score of different model represeantion

[TalaN1993]

@MaartenGr I have a question and would like to know your opinion. In a BERTopic model, if we want to compare the coherence of different representations while accepting the other parts as default, is it logical for the coherence of a model like KeyBERT to be higher than that of an OpenAI model? Should we expect the coherence score of OpenAI to always be higher?

[MaartenGr]

A couple of things first. OpenAI is not the model itself, so it highly depends on which underlying LLM you are actually choosing. Second, if you are using an LLM, you are generally generating labels and not keywords. Calculating traditional coherence (e.g., "c_v") is therefore not possible since it requires keywords instead.

That said, if you are generating keywords with an LLM and/or you use a coherence metric that can compare keywords with labels, then you cannot be sure the LLM will always be higher. There are so many factors related to this, such as the data, language, size/quality of the LLM, coherence metric, etc. that you cannot expect these things beforehand.

[TalaN1993]

Yes, now I understand what you mean. So, if after receiving the output from GPT, I split it and remove the stop words again, I can convert the labels into keywords. Isn't that right?

[MaartenGr]

I really would advise against doing that. The labels that are extracted are not keywords (even if you split them into words) and not directly extracted from the corpus. Coherence measures the keywords against the corpus, so this really would not be helpful to you.

Another helpful resource is this paper. It explains how coherence works and allows you to get more intuition about why labels don't work (even if you convert them into separate words).

[MaartenGr]

You will also find more information about coherence here and my view on this metric.

[TalaN1993]

@MaartenGr Hello.I tried to obtain topic diversity and NPMI coherence for my model using OCTIS. However, the challenge I’m facing is related to calculating coherence. I managed to get the values for diversity, but my NPMI output is coming out as ERROR.

ValueError: unable to interpret topic as either a list of tokens or a list of ids

[MaartenGr]

@TalaN1993 That seems to be related to OCTIS, so I would advise opening up an issue over there as they know about OCTIS and its internals more than I do. Also, the link I shared previously about my view on the metric also contains some example code I believe.