BERTopic - Predicting topic assignments through cosine similarity of topic and document embeddings.

[ilanit1997]

Hi all,
I have been using BERTopic for a while now and this week I upgraded to latest BERTopic version (0.16.2).
Now for some reason I have the following message printed as part of my code:
70/74 [09:02<00:31, 7.76s/it]2024-06-19 13:08:49,777 - BERTopic - Predicting topic assignments through cosine similarity of topic and document embeddings.

The code:

loaded_model.calculate_probabilities = True
for i in tqdm(range(0, len(texts), batch_size)):
        batch_texts = texts[i:i + batch_size]
        batch_embeddings = embeddings[i:i + batch_size]
        batch_predictions, batch_topic_probabilities = loaded_model.transform(batch_texts, batch_embeddings)
        gc.collect()

        predictions.extend(batch_predictions)
        topic_probabilities.extend(batch_topic_probabilities)

The message appears to be a part of the tqdm generator, which I guess is coming from somewhere inside transform implementation.
I couldn't find the reference for this in the package.

In addition, I was under the impression that applying transform results in extracting the HDBSCAN probabilities and not the cosine similarity.

Would appreciate any assistance in the matter.

[MaartenGr]

That is a normal message indicating what is happening under the hood. When you run .transform it can extract probabilities in one of two ways:

• Through HDBSCAN
• Using cosine similarity between document and topic embeddings

The latter happens when you run .transform after you saved the model with safetensors or pytorch as the underlying UMAP and HDBSCAN models are not saved. If you want the former, then you will need to save the model using pickle instead.

[ilanit1997]

when trying to save using pickle I get a pickling error:
topic_model.save(os.path.join(model_folder, "topic_model_full.pkl"), save_embedding_model=False)
_pickle.PicklingError: Can't pickle <function Embedding.forward at 0x7f9b69584e50>: it's not the same object as torch.nn.modules.sparse.Embedding.forward
my definition of a module is as following:

embedding_model = SentenceTransformer(self.config.get("embedding_model"))
embedding_model.to(self.device)

ctfidf_model, hdbscan_model, representation_model, vectorizer_model, umap_model = self.create_base_models()
        print("finished init base models")
        if optimize_flag:
            reduced_embeddings = umap_model.fit_transform(output_dict.get("embeddings"))
            umap_model = Dimensionality(reduced_embeddings)
            output_dict["clusters"] = hdbscan_model.fit(reduced_embeddings).labels_
            output_dict["hdbscan_probs"] = hdbscan_model.probabilities_
            hdbscan_model = BaseCluster()

        topic_model = BERTopic(
            embedding_model=embedding_model,
            ctfidf_model=ctfidf_model,
            umap_model=umap_model,
            hdbscan_model=hdbscan_model,
            representation_model=representation_model,
            vectorizer_model=vectorizer_model,
            nr_topics=self.config.get("nr_topics"),
            n_gram_range=self.config.get("n_gram_range"),
            min_topic_size=self.config.get("min_topic_size"),
            top_n_words=self.config.get("top_n_words"),
            seed_topic_list=self.config.get("seed_topic_list"),
            calculate_probabilities=self.config.get("calculate_probabilities", False),
            verbose=True
        )
        return topic_model

The general models are per the best practices (e.g. ctfidf_model,...).

do you happen to have any idea why this would happen?

[MaartenGr]

when trying to save using pickle I get a pickling error:

Is that the full error log? If not, can you provide it? That would give me more insight on the origin of the issue.

Small notes:

embedding_model.to(self.device)

This is not necessary since this is done under the hood.

self.create_base_models()

Could you show what this function is doing?

[ilanit1997]

Sure this is the full log:

Traceback (most recent call last):
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/data/home/ilanit/Research/code_files/semantic_modeling/runners/topics_per_user_multiple_datasets.py", line 180, in <module>
    topic_model_obj.run_post_processing(topic_model, output_dict, to_save_folder, run,
  File "/data/home/ilanit/Research/code_files/semantic_modeling/analysis/timeseries_analysis/utils/semantic_modeling_utils.py", line 822, in run_post_processing
    self.save_topic_model(topic_model, to_save_folder, run)
  File "/data/home/ilanit/Research/code_files/semantic_modeling/analysis/timeseries_analysis/utils/semantic_modeling_utils.py", line 894, in save_topic_model
    topic_model.save(os.path.join(model_folder, "topic_model_full.pkl"), save_embedding_model=False)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/pyAudioAnalysis/../bertopic/_bertopic.py", line 3085, in save
    joblib.dump(self, file)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 555, in dump
    NumpyPickler(filename, protocol=protocol).dump(value)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 487, in dump
    self.save(obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 717, in save_reduce
    save(state)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 717, in save_reduce
    save(state)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 717, in save_reduce
    save(state)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 717, in save_reduce
    save(state)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 713, in save_reduce
    self._batch_setitems(dictitems)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 717, in save_reduce
    save(state)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 713, in save_reduce
    self._batch_setitems(dictitems)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 717, in save_reduce
    save(state)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 1075, in save_global
    raise PicklingError(
_pickle.PicklingError: Can't pickle <function Embedding.forward at 0x7f9b69584e50>: it's not the same object as torch.nn.modules.sparse.Embedding.forward
Traceback (most recent call last):
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/data/home/ilanit/Research/code_files/semantic_modeling/runners/topics_per_user_multiple_datasets.py", line 180, in <module>
    topic_model_obj.run_post_processing(topic_model, output_dict, to_save_folder, run,
  File "/data/home/ilanit/Research/code_files/semantic_modeling/analysis/timeseries_analysis/utils/semantic_modeling_utils.py", line 822, in run_post_processing
    self.save_topic_model(topic_model, to_save_folder, run)
  File "/data/home/ilanit/Research/code_files/semantic_modeling/analysis/timeseries_analysis/utils/semantic_modeling_utils.py", line 894, in save_topic_model
    topic_model.save(os.path.join(model_folder, "topic_model_full.pkl"), save_embedding_model=False)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/pyAudioAnalysis/../bertopic/_bertopic.py", line 3085, in save
    joblib.dump(self, file)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 555, in dump
    NumpyPickler(filename, protocol=protocol).dump(value)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 487, in dump
    self.save(obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 717, in save_reduce
    save(state)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 717, in save_reduce
    save(state)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 717, in save_reduce
    save(state)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 717, in save_reduce
    save(state)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 713, in save_reduce
    self._batch_setitems(dictitems)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 717, in save_reduce
    save(state)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 713, in save_reduce
    self._batch_setitems(dictitems)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 717, in save_reduce
    save(state)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/data/home/ilanit/anaconda3/envs/llms_env/lib/python3.9/pickle.py", line 1075, in save_global
    raise PicklingError(
_pickle.PicklingError: Can't pickle <function Embedding.forward at 0x7f9b69584e50>: it's not the same object as torch.nn.modules.sparse.Embedding.forward

The function is:

def create_base_models(self):
     ctfidf_model = self._create_ctfidf_model()
     hdbscan_model = self._create_hdbscan_model()
     representation_model = self._create_representation_model()
     vectorizer_model = self._crate_vectorizer_model()
     umap_model = self._create_umap_model()

     return ctfidf_model, hdbscan_model, representation_model, vectorizer_model, umap_model

where each one is as following:

def _crate_vectorizer_model(self):
     # Extract vocab to be used in BERTopic
     vectorizer_model = CountVectorizer(ngram_range=self.config.get("n_gram_range"),
                                        stop_words="english", min_df=self.config.get("min_df"))
     return vectorizer_model

 def _create_ctfidf_model(self):
     return ClassTfidfTransformer(reduce_frequent_words=self.config.get("reduce_frequent_words", True))

 def _create_umap_model(self):
     params = self.config.get("umap_params")
     if params:
         return UMAP(
             n_neighbors=params.get("n_neighbors"),
             n_components=params.get("n_components"),
             min_dist=params.get("min_dist"),
             metric=params.get("metric"),
             random_state=params.get("random_state"),
             low_memory=params.get("low_memory", False),
             verbose=True
         )
     return None

 def _create_hdbscan_model(self):
     params = self.config.get("hdbscan_params")
     if params:
         return HDBSCAN(
             min_cluster_size=params.get("min_cluster_size"),
             metric=params.get("metric"),
             cluster_selection_method=params.get("cluster_selection_method"),
             prediction_data=params.get("prediction_data"),
             min_samples=params.get("min_samples"),
             gen_min_span_tree=params.get("gen_min_span_tree")
         )
     return None


 def _create_representation_model(self):
     if self.config.get("representation_models"):
         keybert = KeyBERTInspired()
         llama2 = TextGeneration(self.get_llama_generator(), prompt=self.config.get("prompt"))
         models = {
             "KeyBERT": keybert,
             "Llama2": llama2,
         }
         if self.config.get("use_gpt_for_labeling", False):
             tokenizer = tiktoken.encoding_for_model("gpt-3.5-turbo")
             representation_model_gpt = AzureOpenAIRep(
                 self.client,
                 model=self.config.get("gpt_model"),
                 delay_in_seconds=2,
                 chat=True,
                 nr_docs=3,
                 doc_length=400,
                 tokenizer=tokenizer
             )
             models[self.config.get("gpt_model")] = representation_model_gpt
         return models
     return None

and AzureOpenAIRep is a representation model inherited from bertopic.representation import OpenAI to handle exceptions thrown by Azure (safety):

class AzureOpenAIRep(OpenAI):
    def extract_topics(self,
                       topic_model,
                       documents: pd.DataFrame,
                       c_tf_idf: csr_matrix,
                       topics: Mapping[str, List[Tuple[str, float]]]
                       ) -> Mapping[str, List[Tuple[str, float]]]:
        """ Extract topics

        Arguments:
            topic_model: A BERTopic model
            documents: All input documents
            c_tf_idf: The topic c-TF-IDF representation
            topics: The candidate topics as calculated with c-TF-IDF

        Returns:
            updated_topics: Updated topic representations
        """
        # Extract the top n representative documents per topic
        repr_docs_mappings, _, _, _ = topic_model._extract_representative_docs(c_tf_idf, documents, topics, 500, self.nr_docs, self.diversity)

        # Generate using OpenAI's Language Model
        updated_topics = {}
        for topic, docs in tqdm(repr_docs_mappings.items(), disable=not topic_model.verbose):
            truncated_docs = [truncate_document(topic_model, self.doc_length, self.tokenizer, doc) for doc in docs]
            keywords = list(zip(*topics[topic]))[0]
            topk_words = " ".join(keywords[:5])

            prompt = self._create_prompt(truncated_docs, topic, topics)
            self.prompts_.append(prompt)

            # Delay
            if self.delay_in_seconds:
                time.sleep(self.delay_in_seconds)

            try:

                if self.chat:
                    messages = [
                        {"role": "system", "content": "You are a helpful assistant."},
                        {"role": "user", "content": prompt}
                    ]
                    kwargs = {"model": self.model, "messages": messages, **self.generator_kwargs}

                    if self.exponential_backoff:
                        response = chat_completions_with_backoff(self.client, **kwargs)
                    else:
                        response = self.client.chat.completions.create(**kwargs)

                    # Check whether content was actually generated
                    # Adresses #1570 for potential issues with OpenAI's content filter
                    if hasattr(response.choices[0].message, "content"):
                        content = response.choices[0].message.content
                        if content:
                            label = content.strip().replace("topic: ", "")
                        else:
                            label = topk_words
                    else:
                        label = topk_words
                else:
                    if self.exponential_backoff:
                        response = completions_with_backoff(self.client, model=self.model, prompt=prompt, **self.generator_kwargs)
                    else:
                        response = self.client.completions.create(model=self.model, prompt=prompt, **self.generator_kwargs)
                    label = response.choices[0].text.strip()
            except Exception as e:
                print(f"Error in extract_topics: {e} - prompt {prompt}")
                label = topk_words

            updated_topics[topic] = [(label, 1)]

        return updated_topics

[MaartenGr]

Unfortunately, I am not sure why pickle would give an error here. The only related issues I could find are this and this.

In the meanwhile, you could use safetensors or pytorch instead (which is recommended over pickle).

[ilanit1997]

Thank you!

to clarify - If I save the model without pickle I wouldn't be able to use HDBSCAN for probability inference on new data.

[MaartenGr]

That's correct but it will still generate "probabilities" through the cosine similarity between document embeddings and topic embeddings. That will give you an indication of distributions/probabilities (especially if you apply softmax on top of it).