Issue/ saving model save_ctfidf=False

[RomanBlaaas]

Could someone help me with the following problem ?

When I try to save my model I got the following error message:

Traceback (most recent call last):
  File "C:\Users\wkblar386\PycharmProjects\TextUnderstanding\main.py", line 71, in <module>
    main_train_model(corpus)
  File "C:\Users\wkblar386\PycharmProjects\TextUnderstanding\topicModel\train_model.py", line 153, in main_train_model
    pre_trained_model.save(path,
  File "C:\Users\wkblar386\PycharmProjects\TextUnderstanding\.venv\lib\site-packages\bertopic\_bertopic.py", line 3355, in save
    save_utils.save_ctfidf_config(model=self, path=save_directory / "ctfidf_config.json")
  File "C:\Users\wkblar386\PycharmProjects\TextUnderstanding\.venv\lib\site-packages\bertopic\_save_utils.py", line 372, in save_ctfidf_config
    json.dump(config, f, indent=2)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.3056.0_x64__qbz5n2kfra8p0\lib\json\__init__.py", line 179, in dump
    for chunk in iterable:
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.3056.0_x64__qbz5n2kfra8p0\lib\json\encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.3056.0_x64__qbz5n2kfra8p0\lib\json\encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.3056.0_x64__qbz5n2kfra8p0\lib\json\encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.3056.0_x64__qbz5n2kfra8p0\lib\json\encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.3056.0_x64__qbz5n2kfra8p0\lib\json\encoder.py", line 438, in _iterencode
    o = _default(o)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.3056.0_x64__qbz5n2kfra8p0\lib\json\encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type int32 is not JSON serializable

This is the BERTopic version I use:

(.venv) PS C:\Users\wkblar386\PycharmProjects\TextUnderstanding> pip show bertopic
Name: bertopic
Version: 0.16.4
Summary: BERTopic performs topic Modeling with state-of-the-art transformer models.
Home-page: https://github.com/MaartenGr/BERTopic
Author:
Author-email: "Maarten P. Grootendorst" maartengrootendorst@gmail.com
License: MIT License

And this my Code:

import os

from bertopic import BERTopic
from bertopic.representation import KeyBERTInspired

from sklearn.feature_extraction.text import TfidfVectorizer
from hdbscan import HDBSCAN
from umap import UMAP
from sentence_transformers import SentenceTransformer

random_state = 42


def train_topic_model(features):

    min_cluster_size = int(input("Please enter your chosen value for min_cluster_size. "
                                 "This is the minimum size of a cluster which get created: "))
    print()

    min_sample_size = int(input("Please enter your chosen value for min_sample_size. "
                                  "Setting this value significantly lower than min_cluster_size might help you to "
                                  "reduce the amount of noise you will get: "))
    print()

    n_components = int(input("Please enter your chosen value for dimension reduction: "))
    print()

    top_words = int(input("Please enter your chosen value for top words extraction: "))
    print()

    # paraphrase-multilingual-mpnet-base-v2/ paraphrase-multilingual-MiniLM-L12-v2/ all-distilroberta-v1
    embedding_model = SentenceTransformer('paraphrase-multilingual-mpnet-base-v2')
    embeddings = embedding_model.encode(features, show_progress_bar=True)

    umap_model = UMAP(
        n_neighbors=10,
        n_components=n_components,
        min_dist=0.01, metric='cosine',
        random_state=random_state,
        low_memory=True
    )

    hdbscan_model = HDBSCAN(
        min_cluster_size=min_cluster_size,
        min_samples=min_sample_size,
        metric='euclidean',
        cluster_selection_method='eom',
        prediction_data=True,
        gen_min_span_tree=True
    )

    custom_stopwords = ['hallo', 'person', 'gruß', 'danke', 'bitte', 'ticket', 'grüßen', 'http']

    vectorizer_model = TfidfVectorizer(
        stop_words=custom_stopwords,
        max_df=float(0.8),
        min_df=int(2),
        max_features=int(5000),
        use_idf=True,
        smooth_idf=True,
        norm='l2',
        ngram_range=(1, 1))

    keybert_model = KeyBERTInspired()
    representation_model = {"KeyBERT": keybert_model}

    topic_model = BERTopic(
        language="de",
        verbose=True,
        top_n_words=top_words,
        embedding_model=embedding_model,
        n_gram_range=(1, 1),
        hdbscan_model=hdbscan_model,
        umap_model=umap_model,
        calculate_probabilities=True,
        representation_model=representation_model,
        vectorizer_model=vectorizer_model,
    )

    topics, _ = topic_model.fit_transform(features,embeddings)

    topic_info = topic_model.get_topic_info()
    topic_numb = len(topic_info)

    print()
    print(f'BERTopic model Info: {topic_info}')
    print(f'Found topics: {topic_numb}')
    print()

    keybert_topic_labels = {topic: " ".join(list(zip(*values))[0][:1]) for topic, values in
                            topic_model.topic_aspects_["KeyBERT"].items()}

    topic_model.set_topic_labels(keybert_topic_labels)

    # Visualization of the Intertopic Distance Map
    fig_idm = topic_model.visualize_topics(custom_labels=True)
    print("Plot Inter Topic Distance Map")
    print("****************************")
    fig_idm.show()

    # Visualization of the barchart plot
    fig_barchart = topic_model.visualize_barchart(top_n_topics=topic_numb, n_words=10, custom_labels=True)
    print("Plot Barchart")
    print("****************************")

    fig_barchart.show()

    # Get reduced embeddings
    reduced_embeddings = UMAP(n_neighbors=15, n_components=2, min_dist=0.0, metric='cosine').fit_transform(embeddings)

    # Visualization of the documents and topic clustering
    fig_cluster = topic_model.visualize_documents(features,
                                                  reduced_embeddings=reduced_embeddings,
                                                  hide_annotations=True,
                                                  custom_labels=True)
    print("Plot Topic Cluster")
    print("****************************")
    fig_cluster.show()

    return topic_model


def main_train_model(dataframe):

    features = list(dataframe['DOC_TEXT_CLEANED_PP'])

    pre_trained_model = train_topic_model(features)

    while True:

        print()
        print("Please choose one of the following options:")
        print("Input 'y' : for saving the trained model and exit to the main menu")
        print("Input 'n' : for exit to the main menu without saving of the model")
        print()

        user_choice = input("Please enter your chosen option: ")

        if user_choice.lower() == "y":

            print()
            user_model_name = input("Please enter your chosen model name: ")

            path = os.path.join("C:\\Users\\wkblar386\\PycharmProjects\\TextUnderstanding\\topicModel\\Model",
                                f"TopicModel_{user_model_name.strip()}")

            pre_trained_model.save(path,
                                   serialization="pytorch",
                                   save_ctfidf=True,
                                   save_embedding_model=True)

            print("Topic model get saved. Please restart the program!")
            print()
            break

        elif user_choice.lower() == "n":
            print("Exit without saving ... ")
            break

        else:
            print("Invalid option. Please try again.")

[MaartenGr]

@RomanBlaaas I updated your post since the formatting of the code was a bit wonky. As a tip, you can use ``` to format your code but you will have to put it for the whole text and not just parts of it.

With respect to your issue, I'm not sure what is happening here. Could you try saving the model without the save_ctfidf parameter?

[RomanBlaaas]

@MaartenGr Thank you very much for your tip, I'm not that experienced with GitHub yet. Saving without the save_ctfidf parameter works. But when I load this saved model the Representative_Docs are NaN. Have I missed something? Is the issue because of the Numpy library version?

[MaartenGr]

The representative documents are not saved to enable federated learning (which is part of my work). If you want representative documents, you would have to recalculate them (as the topics change when combining the topics). I believe there are a couple of issues open that describe how you could do this.

[RomanBlaaas]

OK, all clearly understood. Thank you very much for your great support.

[RomanBlaaas]

This is the used Numpy version:

Name: numpy
Version: 1.26.4
Summary: Fundamental package for array computing in Python