Assign existing topics to new sentences based on embeddings

[9j7axvsLuF]

I can't figure out how to do the following. I have a corpus of interview transcripts. My pipeline involves segmenting this corpus into sentences (about 10,000 sentences, let's call this corpus A), pre-computing embeddings for these sentences, then fitting a topic model with the pre-computed embeddings. So far, so good – I get a nice topic model of corpus A.

The problem is that I have another corpus of passages that have been manually extracted from interview transcripts and assigned timestamps. I want to do dynamic topic modeling on that corpus, using the existing topic I fit for corpus A. After segmenting these timestamps passages into sentences, I get about 5,000 sentences (let's call this corpus B). The problem is that while conceptually, corpus B should be a strict subset of corpus A, in practice it isn't, both due to slight differences in how the passages were manually extracted from corpus A (such as correcting small transcription mistakes) and the differences in the output of the sentence segmentation algorithm (some sentences get segmented at different joints).

I pre-computed embeddings for corpus B using the same embedding model as for corpus A. What I would like to do is assign existing topics (from the topic model of corpus A) to sentences in corpus B, to do dynamic topic modeling with timestamps. But I'd like to use the pre-computed embeddings for corpus B to fit these existing topics.

Basically I want to avoid a situation in which the topic I see in my dynamic topic model of corpus B are different from the topic I had for my static topic model of corpus A, since conceptually B is a subset of A, and sentences should be distributed in the same topics. What would be the best way to achieve that?

[MaartenGr]

Hmmm, I'm not sure if this is possible in BERTopic at the moment since dynamic topic modeling was meant to be used on the data it was fitted on (so corpus A). Having said that, it might be possible if you do something like this:

# Fit model on corpus A
topic_model.fit(corpus_A)

# Assign topics for corpus B using the precomputed embeddings
topics_B = topic_model.transform(corpus_B, pre_computed_embeddings_B)

# Apply dynamic topic modeling
topics_over_time = topic_model.topics_over_time(
    docs=corpus_B, 
    timestamps=timestamps_B, 
    topics=topics_B
)

This might work but I haven't tried it myself.

[9j7axvsLuF]

Thanks! Indeed, this doesn't work, unfortunately. It returns the following:

ValueError                                Traceback (most recent call last)
Cell In[56], [line 3](vscode-notebook-cell:?execution_count=56&line=3)
      [1](vscode-notebook-cell:?execution_count=56&line=1) # topic_model.visualize_topics_over_time(topics_over_time)
      [2](vscode-notebook-cell:?execution_count=56&line=2) # Apply dynamic topic modeling
----> [3](vscode-notebook-cell:?execution_count=56&line=3) topics_over_time = topic_model.topics_over_time(
      [4](vscode-notebook-cell:?execution_count=56&line=4)     docs=timestamped_corpus['sentence'], 
      [5](vscode-notebook-cell:?execution_count=56&line=5)     timestamps=timestamped_corpus['time'], 
      [6](vscode-notebook-cell:?execution_count=56&line=6)     topics=timestamped_topics
      [7](vscode-notebook-cell:?execution_count=56&line=7) )

File c:\lib\site-packages\bertopic\_bertopic.py:766, in BERTopic.topics_over_time(self, docs, timestamps, topics, nr_bins, datetime_format, evolution_tuning, global_tuning)
    [764](file:///C:/lib/site-packages/bertopic/_bertopic.py:764) check_is_fitted(self)
    [765](file:///C:/lib/site-packages/bertopic/_bertopic.py:765) check_documents_type(docs)
--> [766](file:///C:/lib/site-packages/bertopic/_bertopic.py:766) selected_topics = topics if topics else self.topics_
    [767](file:///C:/lib/site-packages/bertopic/_bertopic.py:767) documents = pd.DataFrame({"Document": docs, "Topic": selected_topics, "Timestamps": timestamps})
    [768](file:///C:/lib/site-packages/bertopic/_bertopic.py:768) global_c_tf_idf = normalize(self.c_tf_idf_, axis=1, norm='l1', copy=False)

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

So instead, I've trying making a custom version of the topics_over_time function as follows:

def custom_topics_over_time(topic_model,
                            docs: List[str],
                            timestamps: Union[List[str], List[int]],
                            topics: List[int],
                            c_tf_idf: csr_matrix,
                            words: List[str],
                            nr_bins: int = None,
                            datetime_format: str = None,
                            evolution_tuning: bool = True,
                            global_tuning: bool = True) -> pd.DataFrame:
    """
    Custom topics over time function for corpus B.

    Arguments:
        topic_model: The trained BERTopic model.
        docs: The documents from corpus B.
        timestamps: The timestamp of each document in corpus B.
        topics: The topics assigned to each document in corpus B (i.e., timestamped_topics).
        c_tf_idf: The pre-computed c-TF-IDF matrix for corpus B.
        words: The words corresponding to the columns of the c_tf_idf matrix.
        nr_bins: The number of bins to create for the timestamps.
        datetime_format: The datetime format of the timestamps if they are strings.
        evolution_tuning: Whether to fine-tune each topic representation based on the previous timestamp.
        global_tuning: Whether to fine-tune each topic representation based on the global c-TF-IDF matrix.

    Returns:
        A DataFrame with the topic, words, frequency, and timestamp for each topic at each timestamp.
    """
    documents = pd.DataFrame({"Document": docs, "Topic": topics, "Timestamps": timestamps})
    global_c_tf_idf = normalize(topic_model.c_tf_idf_, axis=1, norm='l1', copy=False)

    all_topics = sorted(list(documents.Topic.unique()))
    all_topics_indices = {topic: index for index, topic in enumerate(all_topics)}

    if isinstance(timestamps[0], str):
        infer_datetime_format = True if not datetime_format else False
        documents["Timestamps"] = pd.to_datetime(documents["Timestamps"],
                                                 infer_datetime_format=infer_datetime_format,
                                                 format=datetime_format)

    if nr_bins:
        documents["Bins"] = pd.cut(documents.Timestamps, bins=nr_bins)
        documents["Timestamps"] = documents.apply(lambda row: row.Bins.left, 1)

    documents = documents.sort_values("Timestamps")
    timestamps = documents.Timestamps.unique()

    topics_over_time = []
    for index, timestamp in tqdm(enumerate(timestamps), disable=not topic_model.verbose):
        selection = documents.loc[documents.Timestamps == timestamp, :]
        documents_per_topic = selection.groupby(['Topic'], as_index=False).agg({'Document': ' '.join,
                                                                                "Timestamps": "count"})
        c_tf_idf_sel = c_tf_idf[documents_per_topic.Topic.values + topic_model._outliers]

        if global_tuning or evolution_tuning:
            c_tf_idf_sel = normalize(c_tf_idf_sel, axis=1, norm='l1', copy=False)

        if evolution_tuning and index != 0:
            current_topics = sorted(list(documents_per_topic.Topic.values))
            overlapping_topics = sorted(list(set(previous_topics).intersection(set(current_topics))))

            current_overlap_idx = [current_topics.index(topic) for topic in overlapping_topics]
            previous_overlap_idx = [previous_topics.index(topic) for topic in overlapping_topics]

            c_tf_idf_sel.tolil()[current_overlap_idx] = ((c_tf_idf_sel[current_overlap_idx] +
                                                          previous_c_tf_idf[previous_overlap_idx]) / 2.0).tolil()

        if global_tuning:
            selected_topics = [all_topics_indices[topic] for topic in documents_per_topic.Topic.values]
            c_tf_idf_sel = (global_c_tf_idf[selected_topics] + c_tf_idf_sel) / 2.0

        words_per_topic = topic_model._extract_words_per_topic(words, selection, c_tf_idf_sel, calculate_aspects=False)
        topic_frequency = pd.Series(documents_per_topic.Timestamps.values,
                                    index=documents_per_topic.Topic).to_dict()

        topics_at_timestamp = [(topic,
                                ", ".join([words[0] for words in values][:5]),
                                topic_frequency[topic],
                                timestamp) for topic, values in words_per_topic.items()]
        topics_over_time.extend(topics_at_timestamp)

        if evolution_tuning:
            previous_topics = sorted(list(documents_per_topic.Topic.values))
            previous_c_tf_idf = c_tf_idf_sel.copy()

    return pd.DataFrame(topics_over_time, columns=["Topic", "Words", "Frequency", "Timestamp"])


# Use the `transform` function to assign existing topics from `topic_model` to corpus B
timestamped_topics, timestamped_probs = topic_model.transform(timestamped_corpus['sentence'], timestamped_embeddings)

# Compute the c-TF-IDF matrix and get the words for corpus B
c_tf_idf, words = topic_model._c_tf_idf(pd.DataFrame({"Document": timestamped_corpus['sentence'], "Topic": timestamped_topics}))

# Run the custom topics over time function
topics_over_time = custom_topics_over_time(topic_model,
                                           timestamped_corpus['sentence'],
                                           timestamped_corpus['time'],
                                           timestamped_topics,
                                           c_tf_idf,
                                           words,
                                           nr_bins=50,
                                           evolution_tuning=True,
                                           global_tuning=False)

That seems to work, but I want to make sure I'm not doing anything wrong.

[MaartenGr]

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Ah, I made a small error. It should work if you do this instead:

# Fit model on corpus A
topic_model.fit(corpus_A)

# Assign topics for corpus B using the precomputed embeddings
topics_B, _ = topic_model.transform(corpus_B, pre_computed_embeddings_B)

# Apply dynamic topic modeling
topics_over_time = topic_model.topics_over_time(
    docs=corpus_B, 
    timestamps=timestamps_B, 
    topics=topics_B
)

[owwdesilva]

Hi @MaartenGr ,

Experiencing the same error. Even though i have tried your above updated code when i try to get the topics_over_time I'm getting same error as below.

ValueError                                Traceback (most recent call last)
Cell In[39], line 1
----> 1 topics_over_time = text_topic_model.topics_over_time(
      2     docs=transcript_data,
      3     timestamps=timestamps,
      4     topics=topics
      5 )

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/bertopic/_bertopic.py:766, in BERTopic.topics_over_time(self, docs, timestamps, topics, nr_bins, datetime_format, evolution_tuning, global_tuning)
    764 check_is_fitted(self)
    765 check_documents_type(docs)
--> 766 selected_topics = topics if topics else self.topics_
    767 documents = pd.DataFrame({"Document": docs, "Topic": selected_topics, "Timestamps": timestamps})
    768 global_c_tf_idf = normalize(self.c_tf_idf_, axis=1, norm='l1', copy=False)

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

[MaartenGr]

@owwdesilva I think if you convert topics_B to a list it should work.

[owwdesilva]

@MaartenGr Thank you for your reply.

It worked for me when i convert my topics as a list. Really appreciate your support.

Ps. I got another error as i have not set save_ctfidf as True while saving the model. Able to fix that too when enabling that.